December 2018 - Zuul-discuss - lists.opendev.org

CFP Open Until January 23, Open Infrastructure Summit in Denver
by Claire Massey 22 Jan '19

22 Jan '19

Hi everyone, FYI - the CFP is now open for the first Open Infrastructure Summit <https://www.openstack.org/summit/denver-2019/> (formerly the OpenStack Summit) which will be held in Denver, Colorado April 29 - May 1, 2019. Wednesday, *January 23* is the deadline to Submit presentations <https://www.openstack.org/summit/denver-2019/call-for-presentations>. The Open Infrastructure Summit is organized by OSF and designed to be a place where open source infrastructure communities can come together and collaborate in the open. Zuul will have a large and prominent presence at the event so please submit talks to the CFP and plan to attend! SUBMIT YOUR PRESENTATION <https://www.openstack.org/summit/denver-2019/call-for-presentations> Important info: Based on previous Program Committee and attendee feedback, we have added / updated three Tracks: Security, Getting Started, and Open Development (previously Open Source Community). You can find the Track descriptions here <https://www.openstack.org/summit/denver-2019/summit-categories>. All of the OSF pilot projects —including Airship, Kata Containers, StarlingX and Zuul — will be front and center alongside other open source communities like Ansible, Cloud Foundry, Docker, Kubernetes, and many more. The Open Infrastructure Summit (formerly the OpenStack Summit), has evolved to recognize our diverse audience, and to signal to the market that the event is relevant for all IT infrastructure decision makers. If you’re interested in influencing the Summit content, apply to be a Programming Committee member <https://openstackfoundation.formstack.com/forms/openstackdenver2019_program…>*, where you can also find a full list of time requirements and expectations. Nominations will close on January 4, 2019. The content submission process for the Forum and Project Teams Gathering will be managed separately in the upcoming months. *OSF Staff will serve as the Programming Committee for the Getting Started Track. Denver Summit registration and sponsor sales are currently open. Learn more <https://www.openstack.org/summit/denver-2019> and email summit(a)openstack.org <mailto:summit@openstack.org> with any questions. Please email speakersupport(a)openstack.org <mailto:speakersupport@openstack.org> with any questions or feedback. Thanks, Claire

1 1

AW: AW: Different queues across branches?
by Mike Pagel 29 Dec '18

29 Dec '18

Happy holidays to everyone! I wrote: > In that case however, having multiple queue names is an > erroneous condition. Just wanted to wrap this up. I realize that raising an exception (or similar behavior) to complain about this condition will break existing installations, which is not acceptable. In particular not for this kind of thing. Nevertheless the discussion helped me to understand the behavior in my setup. Thanks a lot, Mike -----Ursprüngliche Nachricht----- Von: Mike Pagel [mailto:mike@mpagel.de] Gesendet: Donnerstag, 20. Dezember 2018 18:42 An: 'zuul-discuss(a)lists.zuul-ci.org' <zuul-discuss(a)lists.zuul-ci.org> Betreff: AW: AW: Different queues across branches? > Yes -- when creating shared change queues from projects, the first > appearance of a queue name is used, and this is true for any branch. > So, unlike jobs definitions and invocations, a project's membership in a shared queue is global and immutable. > It doesn't change based on branch, or the change under test. It is simply a characteristic of the project itself. I understand and I accept that this is the current state. In that case however, having multiple queue names is an erroneous condition. Depending on what "first" actually means I could imagine scenarios, where setting a different queue name in a branch can remove my project from the original queue used before on master and essentially break master's gating, because other (required) projects are still using the original queue name... Would you agree? Would it make sense to guard against this? Where would that kind of checking code go? What kind of exception should be thrown? Where would I find an example of such kind of configuration check code and associated tests? Mike

1 0

Different queues across branches?
by Mike Pagel 20 Dec '18

20 Dec '18

Hey zuulers, we recently had to create a maintenance branch across all repositories required to build the project. Since the two branches no longer affect each other in terms of a green build I intended to decouple the gating process of master and branch by changing the gate's queue name in the branch. I was expecting the branch gates to be executed in the new queue and the master gates being executed in the original one. However, it looks like all gates are still happening in the same original queue. Did I expect something wrong? In case it matters, we are using implied branch matching for our jobs. Thanks a lot, Mike

3 8

How to define zuul.executor.inventory_file on a localhost
by Akhil Jayakumar (akjayaku) 19 Dec '18

19 Dec '18

Hi All, Our openstack cinder ci jobs are failing due to below error. Do you know where we need to define zuul.executor variable “inventory_file”? Appreciate your help. 2018-12-18 06:48:58.993454 | TASK [log-inventory : Copy ansible inventory to logs dir] 2018-12-18 06:48:59.045029 | primary | ERROR 2018-12-18 06:48:59.045544 | primary | { 2018-12-18 06:48:59.045637 | primary | "msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'inventory_file'\n\nThe error appears to have been in '/tmp/03e62a72bdbc4480937f15e45000abb3/trusted/project_1/git.zuul-ci.org/zuul-jobs/roles/log-inventory/tasks/main.yaml': line 11, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n# https://docs.ansible.com/ansible/latest/porting_guides/porting_guide_2.4.ht… name: Copy ansible inventory to logs dir\n ^ here\n" 2018-12-18 06:48:59.045713 | primary | } 2018-12-18 06:48:59.048055 | 2018-12-18 06:48:59.048294 | PLAY RECAP 2018-12-18 06:48:59.048470 | primary | ok: 18 changed: 10 unreachable: 0 failed: 1 Thanks, Akhil.

3 3

Zuul project update 2018-12-17
by corvus＠inaugust.com 18 Dec '18

18 Dec '18

Hi, The past two weeks in the Zuul project have seen: David discovered an issue in node deletion that was leaving empty ZooKeeper nodes and added code to nodepool to clean those up automatically. He also fixed a race in the test_handler_poll_session_expired nodepool test. Tobias worked on spec for scale out scheduler (what we've been informally calling Zuul v4 -- where we likely connect all components via Zookeeper and remove our use of Gearman). He also fixed an issue with github status checks. Paul added nodepool.node_id support for job inventory files (so that performance problems in jobs can be traced back to the hypervisor on which they ran). And I've continued the relative-priority work (including a change to group projects in check as they are grouped in gate), as well as a revision to our proposed governance document. A number of us are likely to be less active over the next few weeks. I don't plan to send another status update email until January 7. Please help write the next update email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email -Jim

3 2

Creation of OSF Project Confirmation Guidelines
by Claire Massey 13 Dec '18

13 Dec '18

Hi everyone, As you know, Zuul is a pilot project supported by the OpenStack Foundation (OSF). As such, we wanted to make sure you’re aware of an effort recently kicked off a sub group of the OSF Board of Directors to draft guidelines[1] for confirming pilot projects as full, top-level, open infrastructure project of the Foundation. The current pilot projects that will be eligible for confirmation review in 2019-2020 are Airship, Kata Containers, StarlingX and Zuul. Currently the OpenStack project is the only confirmed project. The discussion about drafting these guidelines is just getting started, and *we invite you to participate in this effort* by reviewing the working draft in this etherpad and adding in your feedback and thoughts starting on line 139: https://etherpad.openstack.org/p/BrainstormingOSFProjectConfirmationGuideli… <https://etherpad.openstack.org/p/BrainstormingOSFProjectConfirmationGuideli…> There will most likely be an open call scheduled in early 2019 where additional feedback will also be welcomed. I will share that invitation with you once it’s schedule. The goal is to come up with a set of points that the Board can use when reviewing pilot projects for confirmation. This topic has previously been discussed at a very high-level at previous Board meetings, including the September 18th meeting[2] (starting around slide 38). 1. https://etherpad.openstack.org/p/ProjectConfirmationGuidelines <https://etherpad.openstack.org/p/ProjectConfirmationGuidelines> 2. https://docs.google.com/presentation/d/10UyCpxkjPqC3kT-dYRpBxzNT39i2OhlggJv… <https://docs.google.com/presentation/d/10UyCpxkjPqC3kT-dYRpBxzNT39i2OhlggJv…> Thanks, Claire

1 0

Projects config introspection
by Tristan Cacqueray 11 Dec '18

11 Dec '18

Hi all, I'd like to start the discussion around introspecting projects configs. The goal is to be able to efficiently answer the following questions: * Which projects use a specific job, * Which jobs use a specific nodeset, * Which nodesets or jobs use a specific label, and * Which jobs, labels or nodesets are not used. Putting aside the job variants model (which isn't well exposed in the current REST APIs), the easiest solution would be to add new endpoints, such as /api/job-per-project/{job_name}, or /api/label-per-project/{label_name}. But this feels sub-optimal and we need a new route for each type of query. Another solution would be to add a global toDict() method for the whole Tenant configuration object, which can be exposed at /api/config (or simply dumped using a zuul-scheduler command line argument). This would enable custom search to be done client-side. Another solution would be to implement the query using GraphQL custom resolver as started in https://review.openstack.org/574625 Regards, -Tristan

8 14

Adding an unconditional cleanup job phase
by Brendan Jackman 11 Dec '18

11 Dec '18

Hi all, Looking, for the moment, for lower-hanging fruit in my quest to reduce test resource pressure with Zuul. One unfortunate hack in my use-case is that we currently lock and release test resources which are shared with another system. This is handled by an ansible role that sends API calls to that other system. This means summarily killing a running job can prevent a resource from getting unlocked. To get around this I've had to disable pipeline.dequeue-on-new-patchset. I'd like to add stage where playbooks can be added that still gets run even if the job gets stopped. My guess is that this is also useful to other people who are using the static nodepool driver and would like to unconditionally clean up files left behind by jobs - does that sound correct? It generally seems that there would be value in having and unconditional cleanup stage, anyway. If you agree so far, can we talk semantics & implementation? I think it would be sensible that these playbooks are _never_ killed, but that obviously poses a risk that users can get stuck with an unkillable job running. I guess a simple solution is to have a distinct timeout from the job timeout, which covers just the running of these playbooks. We recommend setting this pretty short (default 30 secs, say), and strictly minimising the logic in these playbooks. If I'm still making sense then from what I've read so far, it seems all the complexity here can just go in AnsibleJob - the gearman communication would already support this. The client would just send the executor:stop TextJob as normal, and then the server can return its response asynchronously when the cleanup playbooks have finished? How does that sound? Cheers, Brendan

2 3

Roadmap for running jobs on OpenShift
by Tristan Cacqueray 10 Dec '18

10 Dec '18

Hello everyone, Please find below a list of topics I think we should address in order to land the OpenShift driver. Kube config integration in zuul-executor ---------------------------------------- As of now, the k8s and the openshift drivers need a setup function in the exector module in order to be usable: https://review.openstack.org/570668 The Openshift drivers --------------------- https://review.openstack.org/570667 and https://review.openstack.org/590335 are 2 variants to support different kinds of cluster accounts: self-provisioner and user. The code may be refactored in a single driver. Inventory update by job ----------------------- We should consider a pre-run phase to create new resources and update the job inventory using zuul_return to enable the test phase to uses "hosts: pod", as proposed in: https://review.openstack.org/590092 Roles and parent jobs --------------------- The containers use-cases need a lot of love to manage builds and to enable new workflows, for example, to promote an image from one cluster to a central registry and trigger updates in others clusters. Here is an initial build&deploy Openshift pod as well as a prepare-workspace implementation using oc rsync proposed here: https://review.openstack.org/570669 Even though this doesn't have to be part of Zuul, I think we should build new roles in zuul-jobs as a community. Define the security context --------------------------- It seems like opinions diverge on the security requirements for running untrusted jobs in pods. At the summit, I was told that we should keep on using the root user so that bindep and other service operations stay the same. I strongly disagree with this approach because containers that run as root have far more permissions than their workload requires and this could be abused by untrusted playbooks. Nodepool accesses to a k8s or Openshift cluster shall be restricted to non-root users to mitigate privilege escallation. Even though this can be discussed later, I think it's important to set the expectation immediately and keep users safe. How many users are interested in these topics and what are your review bandwidths? Can we get an estimated release date for OpenShift users? Regards, -Tristan

7 16

[Newsletter] What's Happening in the Open Infrastructure Community
by Claire Massey 05 Dec '18

05 Dec '18

Hi everyone, This week, the OSF team distributed the first Open Infrastructure Community Newsletter. Our goal with the bi-weekly newsletter is to provide a digest of the latest developments and activities across open infrastructure projects, events and users. This week, the newsletter highlighted the Berlin Summit as well as brief updates from Zuul and other OSF projects - OpenStack, Airship, Kata Containers and StarlingX. You can checkout the full newsletter on Superuser [1], and if you are interested in receiving the upcoming newsletters, you can subscribe here [2]. If you would like to contribute to a future newsletter or have feedback, please reach out to community(a)openstack.org <mailto:community@openstack.org>. Thanks, Claire [1] http://superuser.openstack.org/articles/inside-open-infrastructure-newslett… <http://superuser.openstack.org/articles/inside-open-infrastructure-newslett…> [2] https://www.openstack.org/community/email-signup <https://www.openstack.org/community/email-signup>

1 0