- Zuul-discuss - lists.opendev.org

Projects config introspection
by Tristan Cacqueray 12 Dec '18

12 Dec '18

Hi all, I'd like to start the discussion around introspecting projects configs. The goal is to be able to efficiently answer the following questions: * Which projects use a specific job, * Which jobs use a specific nodeset, * Which nodesets or jobs use a specific label, and * Which jobs, labels or nodesets are not used. Putting aside the job variants model (which isn't well exposed in the current REST APIs), the easiest solution would be to add new endpoints, such as /api/job-per-project/{job_name}, or /api/label-per-project/{label_name}. But this feels sub-optimal and we need a new route for each type of query. Another solution would be to add a global toDict() method for the whole Tenant configuration object, which can be exposed at /api/config (or simply dumped using a zuul-scheduler command line argument). This would enable custom search to be done client-side. Another solution would be to implement the query using GraphQL custom resolver as started in https://review.openstack.org/574625 Regards, -Tristan

8 14

Adding an unconditional cleanup job phase
by Brendan Jackman 11 Dec '18

11 Dec '18

Hi all, Looking, for the moment, for lower-hanging fruit in my quest to reduce test resource pressure with Zuul. One unfortunate hack in my use-case is that we currently lock and release test resources which are shared with another system. This is handled by an ansible role that sends API calls to that other system. This means summarily killing a running job can prevent a resource from getting unlocked. To get around this I've had to disable pipeline.dequeue-on-new-patchset. I'd like to add stage where playbooks can be added that still gets run even if the job gets stopped. My guess is that this is also useful to other people who are using the static nodepool driver and would like to unconditionally clean up files left behind by jobs - does that sound correct? It generally seems that there would be value in having and unconditional cleanup stage, anyway. If you agree so far, can we talk semantics & implementation? I think it would be sensible that these playbooks are _never_ killed, but that obviously poses a risk that users can get stuck with an unkillable job running. I guess a simple solution is to have a distinct timeout from the job timeout, which covers just the running of these playbooks. We recommend setting this pretty short (default 30 secs, say), and strictly minimising the logic in these playbooks. If I'm still making sense then from what I've read so far, it seems all the complexity here can just go in AnsibleJob - the gearman communication would already support this. The client would just send the executor:stop TextJob as normal, and then the server can return its response asynchronously when the cleanup playbooks have finished? How does that sound? Cheers, Brendan

2 3

Roadmap for running jobs on OpenShift
by Tristan Cacqueray 10 Dec '18

10 Dec '18

Hello everyone, Please find below a list of topics I think we should address in order to land the OpenShift driver. Kube config integration in zuul-executor ---------------------------------------- As of now, the k8s and the openshift drivers need a setup function in the exector module in order to be usable: https://review.openstack.org/570668 The Openshift drivers --------------------- https://review.openstack.org/570667 and https://review.openstack.org/590335 are 2 variants to support different kinds of cluster accounts: self-provisioner and user. The code may be refactored in a single driver. Inventory update by job ----------------------- We should consider a pre-run phase to create new resources and update the job inventory using zuul_return to enable the test phase to uses "hosts: pod", as proposed in: https://review.openstack.org/590092 Roles and parent jobs --------------------- The containers use-cases need a lot of love to manage builds and to enable new workflows, for example, to promote an image from one cluster to a central registry and trigger updates in others clusters. Here is an initial build&deploy Openshift pod as well as a prepare-workspace implementation using oc rsync proposed here: https://review.openstack.org/570669 Even though this doesn't have to be part of Zuul, I think we should build new roles in zuul-jobs as a community. Define the security context --------------------------- It seems like opinions diverge on the security requirements for running untrusted jobs in pods. At the summit, I was told that we should keep on using the root user so that bindep and other service operations stay the same. I strongly disagree with this approach because containers that run as root have far more permissions than their workload requires and this could be abused by untrusted playbooks. Nodepool accesses to a k8s or Openshift cluster shall be restricted to non-root users to mitigate privilege escallation. Even though this can be discussed later, I think it's important to set the expectation immediately and keep users safe. How many users are interested in these topics and what are your review bandwidths? Can we get an estimated release date for OpenShift users? Regards, -Tristan

7 16

[Newsletter] What's Happening in the Open Infrastructure Community
by Claire Massey 06 Dec '18

06 Dec '18

Hi everyone, This week, the OSF team distributed the first Open Infrastructure Community Newsletter. Our goal with the bi-weekly newsletter is to provide a digest of the latest developments and activities across open infrastructure projects, events and users. This week, the newsletter highlighted the Berlin Summit as well as brief updates from Zuul and other OSF projects - OpenStack, Airship, Kata Containers and StarlingX. You can checkout the full newsletter on Superuser [1], and if you are interested in receiving the upcoming newsletters, you can subscribe here [2]. If you would like to contribute to a future newsletter or have feedback, please reach out to community(a)openstack.org <mailto:community@openstack.org>. Thanks, Claire [1] http://superuser.openstack.org/articles/inside-open-infrastructure-newslett… <http://superuser.openstack.org/articles/inside-open-infrastructure-newslett…> [2] https://www.openstack.org/community/email-signup <https://www.openstack.org/community/email-signup>

1 0

Software Factory 3.2 released
by Tristan Cacqueray 05 Dec '18

05 Dec '18

We are pleased to announce the release of Software Factory version 3.2: https://www.softwarefactory-project.io/releases/3.2 This new release consolidates the 3.1 version with updated packages, minor fixes and some new features such as an improved update mechanism. Software Factory version 3.2 integrates Zuul and Nodepool version 3.3.1. Here is the list of features we picked in the Software Factory version of Zuul and Nodepool that are still under review upstream. We are confident these features will eventually land upstream, but they might evolve by then. They should therefore be considered a "tech preview", although we will do our best to integrate these in the least disruptive way possible: * Improved dashboard: https://review.openstack.org/#/q/topic:react-zuul * Openshift driver: https://review.openstack.org/#/q/topic:container-build-resources * RunC driver: https://review.openstack.org/535556 The Nodepool package also contains some backports to mitigate stability issues found in the (upstream) version 3.3.1: * Set pool for error'ed instances: https://review.openstack.org/621681 * Set type for error'ed instances: https://review.openstack.org/622101 * Log exceptions deleting ZK nodes: https://review.openstack.org/621301 * Fix race when deleting Node znodes: https://review.openstack.org/622403 * Count leaked nodes in unmanaged quota: https://review.openstack.org/621040 * Cleanup routine to delete empty nodes: https://review.openstack.org/622616 Best regards, The Software Factory Team

2 1

Zuul project update 2018-12-03
by corvus＠inaugust.com 04 Dec '18

04 Dec '18

Hi, It's been a while since I've sent one of these -- mostly due to all the conferences and holidays which seemed to all happen at once. There's a lot going on: I'm working on support for dynamically adjusting node request priorities. Because of the current heavy load on OpenStack's Zuul system, this should help us get results faster on changes to Zuul. I also have changes in progress to add support for storing information about artifacts (but not the artifacts themselves) in the SQL database. That will make it easier for jobs to store artifacts in arbitrary locations and have later jobs use those artifacts. In particular, this is in service of the idea of a "promotion" pipeline. Finally, I'll be proposing a formal governance structure for the project this week -- something I've been drafting for a while. Paul has been adding support for executor zones -- a long-standing feature request. This allows admins to locate executors near to the node resources they use. Shrews added support for generic node attributes via config file to nodepool (mainly supporting the executor zone feature). Monty is continuing work on refactoring of nodepool TaskManager across openstacksdk and keystoneauth libraries, and is also working with tristanC towards supporting both statsd and prometheus for metrics collection. There is a new project "promstats" as part of this effort. Tobias has been working on zk caching in nodepool, and per project and tenant usage statistics, and a spec for a scale-out scheduler. Tristan has proposed a roadmap for Openshift support, started discussion about config introspection and worked on a web ui to handle generic API errors. Clint is working on producing reliable unit test(s) for AWS/EC2 driver. Thanks everyone! -Jim

1 0

Road to fully scale out Zuul
by Tobias.Henkel＠bmw.de 03 Dec '18

03 Dec '18

Hi Zuulers, currently the zuul-scheduler is the only component of zuul that is not scale out ready and thus a single point of failure. We always had the goal to make the scheduler scale out ready at some point in time. In order to make this more concrete so we can start to work on this I tried to transform my thoughts on this into a spec: https://review.openstack.org/621479 I'd be interested in what you think about this. Cheers, Tobias

1 0

Testing multiple commits in one job with Gerrit
by Brendan Jackman 28 Nov '18

28 Nov '18

Hi all, I'm gradually setting up Zuul for my org, and basically I have just one impedance mismatch between Zuul's behaviour and my requirements: Due to pressure on our test hardware I don't think running full end-to-end tests on every individual commit will scale. What I'd ultimately like is a check/gate setup where jobs in the check pipeline runs on every single commit, but jobs in the gate pipeline only run once for an entire "stack" of commits, and then that entire stack can be merged at once. IIUC this is basically the behaviour you get already with GitHub, where the zuul notion of "change" maps to a whole Pull Request rather than an individual commit. I started looking into ways something similar can be achieved with Gerrit. After some comments from #zuul and starting at the code a bit this strategy is the one I'm considering: enable one Zuul change to encapsulate multiple Gerrit changes. (Gerrit and Zuul both use the word "change", which works because Gerrit changes and Zuul changes are currently 1:1. That wouldn't be the caes any more with my proposal so I'll always say "Gerrit change" or "Zuul change"). I haven't figured out exactly how the Gerrit changes should be grouped together, and how this should be configured in the Gerrit trigger, but I think the basic idea would be to inject extra events into the Gerrit event stream (somewhere around GerritEventConnector._handleEvent). One example might be that we detect when an entire stack of Gerrit changes has received some common set of approvals, we inject an event called (say) "zuul-change-stack-approved". The user can then have a pipeline.trigger.gerrit entry with this event, and that trigger would create a Zuul change encapsulating the entire stack of Gerrit changes. These wouldn't need to actually be treated in the code like events coming from Gerrit, but this basic idea would allow expressing this behaviour with the existing structure of the Gerrit trigger configuration. I think it would then be a matter of finding the places where the Scheduler and pipeline Managers interact with the Gerrit driver using references to Zuul changes, and ensuring that those parts of the Gerrit driver can handling being passed the new class of Zuul change. I think most of these would involve checking for if the Zuul change is of the new type and then iterating over the constituent Gerrit changes to performe some action against each (e.g. GerritConnection.canMerge, GerritRefFilter.matches, and, with a little more complication, GerritReporter.report). Does that seem like a reasonable approach to the problem? Cheers, Brendan

3 6

Multi-repo merge
by Mike Pagel 27 Nov '18

27 Nov '18

Zuul is great in gating projects consisting of many repositories. It even supports dependencies of changes within a single or across multiple repositories. Our project is making heavy use of this. Every now and then we are facing dependencies that unfortunately are not unidirectional. I know, we should have better thought about clean repo structures in the first place, but one not too artificial example is a situation where repositories are structured to express ownership, visibility and write access. In Github for instance, all this is configured at the repository level. If you like see the sample how such a situation could arise. --- Start of sample --- Assume company A is the owner of repository A, holding operating system and a top-level application framework (the "main" function in C). Further assume company B is the owner of repository B, implementing a module on top of the operating system, and the module is invoked by A's top level application. At the build level the dependencies are clear and well defined, but at the repo level they are bidirectional. Some changes in A and B will depend on each other. It turns out that a clean, backward compatible PR design, leading to unidirectional changes as required by zuul is almost always possible, but not always easy. E.g. think of a change where A replaces a BSP in their operating system, which requires adaptation in B. It might not be possible to temporarily have two BSPs in A, the the change to A will break the build. Nor can B simply switch its usage of APIs, as that, by itself would break the build as well. The only chance to get this merged would be to temporarily disable some checks, allow for the first merge, then reenable and merge the second change. Ugly. --- End of sample --- It would very much simplify our workflows if an "atomic" merge (from zuul's point of view) would be possible. I can think of some ways to sort of realize this with zuul, but none of them is really safe, even if zuul is the only party executing merges to protected branches. What would be your suggestion to accommodate such a requirement? Thanks a lot for any insight and ideas you might have. Best, Mike

6 12

Zuul CI and Gitlab
by Алексей Степаненко 26 Nov '18

26 Nov '18

Hi. At the summit in Berlin was interested in how the Zuul CI system. GitHub support has been announced, but we're using GitLab. Will there be added support for Networks in the future ? -- Aleksei Stepanenko, Selectel Team

1 0