June 2024 - Zuul-discuss - lists.opendev.org

Merge templates with jobs containing dependencies
by Guillaume Vincent 01 Jul '24

01 Jul '24

Hello Zuul memo list, This is the first time I'm posting here, I hope this is the right place to suggest an improvement to the tool. Or learn a new solution. I'm trying to improve the definition of my jobs and templates on Zuul. Imagine the following context with these 3 jobs: - build-el8-rpm-job: a job that builds a rpm for el8 - build-el9-rpm-job: a job that builds a rpm for el9 - rpm-publish-job: a job which publishes the rpms of the jobs it has as a dependency The definition of a Job on zuul could look like this ```yml - project: name: project_1 check: jobs: - build-el8-rpm-job - build-el9-rpm-job gate: jobs: - build-el8-rpm-job - build-el9-rpm-job post: jobs: - build-el8-rpm-job - build-el9-rpm-job - rpm-publish-job: dependencies: - build-el8-rpm-job - build-el9-rpm-job ``` On this job we produce the 2 rpms (el8 and el9) and we publish everything in a post job. But there are projects that don't need to be built for el9 for example. the job definition would look like this: ```yml - project: name: project_2 check: jobs: - build-el8-rpm-job gate: jobs: - build-el8-rpm-job post: jobs: - build-el8-rpm-job - rpm-publish-job: dependencies: - build-el8-rpm-job ``` As I have a lot of projects I will use templates. ## 1st solution: One template per environment TLDR: This doesn't work. Dependencies in templates are not merged. ```yml - project-template: name: build-and-publish-el8-template check: jobs: - build-el8-rpm-job gate: jobs: - build-el8-rpm-job post: jobs: - build-el8-rpm-job - rpm-publish-job: dependencies: - build-el8-rpm-job - project-template: name: build-and-publish-el9-template check: jobs: - build-el9-rpm-job gate: jobs: - build-el9-rpm-job post: jobs: - build-el9-rpm-job - rpm-publish-job: dependencies: - build-el9-rpm-job ``` If I use these 2 templates: ```yml - project: name: project_3 templates: - build-and-publish-el8-template - build-and-publish-el9-template ``` The resulting project definition is: ```yml - project: name: project_3 check: jobs: - build-el8-rpm-job - build-el9-rpm-job gate: jobs: - build-el8-rpm-job - build-el9-rpm-job post: jobs: - build-el8-rpm-job - build-el9-rpm-job - rpm-publish-job: dependencies: # missing build-el8-rpm-job here - build-el9-rpm-job ``` The el8 rpm is not published because the definition of `rpm-publish-job` of the last template overwrites that of the previous template. ## 2nd solution: The only solution I found, create templates by combination TLDR: The number of templates grows exponentially. The solution I use is to create one template per publish definition: ```yml - project-template: name: build-el8-template check: jobs: - build-el8-rpm-job gate: jobs: - build-el8-rpm-job - project-template: name: publish-el8-template post: jobs: - build-el8-rpm-job - rpm-publish-job: dependencies: - build-el8-rpm-job - project-template: name: build-el9-template check: jobs: - build-el9-rpm-job gate: jobs: - build-el9-rpm-job - project-template: name: publish-el9-template post: jobs: - build-el9-rpm-job - rpm-publish-job: dependencies: - build-el9-rpm-job - project-template: name: publish-el8-and-el9-template post: jobs: - build-el8-rpm-job - build-el9-rpm-job - rpm-publish-job: dependencies: - build-el8-rpm-job - build-el9-rpm-job ``` And now I can do ```yml - project: name: project_3 templates: - build-el8-template - build-el9-template - publish-el8-and-el9-template - project: name: project_4 templates: - build-el8-template - publish-el8-template - project: name: project_5 templates: - build-el9-template - publish-el9-template ``` But we realize that I will have to create too many templates. The number of dependencies grows exponentially. ## Improvement proposal A more elegant way to do this would be to merge the dependencies array: ```js merge_templates([ { post: { jobs: { "rpm-publish-job": { dependencies: ["build-el8-rpm-job"], }, }, }, }, { post: { jobs: { "rpm-publish-job": { dependencies: ["build-el9-rpm-job"], }, }, }, }, ]) == { post: { jobs: { "rpm-publish-job": { dependencies: ["build-el8-rpm-job", "build-el9-rpm-job"], }, }, }, }; ``` And duplications are removed: ```js merge_templates([ { post: { jobs: { "rpm-publish-job": { dependencies: ["build-el8-rpm-job", "build-el9-rpm-job"], }, }, }, }, { post: { jobs: { "rpm-publish-job": { dependencies: ["build-el9-rpm-job"], }, }, }, }, ]) == { post: { jobs: { "rpm-publish-job": { dependencies: ["build-el8-rpm-job", "build-el9-rpm-job"], }, }, }, }; ``` We will be able to use the first solution and compose our templates together. Maybe I'm using the tool in the wrong way. Maybe there is a way to merge the dependencies array? Thank you Guillaume

2 1

Superproject subscription to submodules updates managed by Zuul
by Fredrik Medley 01 Jul '24

01 Jul '24

Hi, I'm working on better support for submodules in Zuul. The basic idea is to support something like Gerrit's super project subscription, https://gerrit-review.googlesource.com/Documentation/user-submodules.html, but not just for Gerrit. 1. Let Zuul know about project subscriptions. 2. Start jobs for the super projects, now when Zuul knows about the update in the submodule. 3. Teach the Zuul mergers about bumping submodules. 4. Let Zuul create bump commits in the super projects when the submodules get submitted. This allows for cross server functionality and makes the feature function for any Git server. The plan is to implement things in the following way: 1. Extend the project configuration with "project-subscriptions": [ { "name": "(Canonical) name of the submodule project", "branch": "'.' or another branch name", "path": "Optional path in the repository to update", }, ... ] While doing this, also let Zuul read the .gitmodules file and generate such configuration automatically, according to speculative merging. 2. Extend QueueItem with extra BumpChanges for each super project. This can be done recursively. Zuul already today support dependency cycles, i.e. having multiple changes in a single QueueItem. Changes ahead can affect what super projects are subscribing, so the list of BumpChanges need to be reset when a QueueItem is dequeued and calculated every time it is enqueued. 3. Let the Zuul merger process the projects and branches in order. When a merge commit has been created for a submodule, the commit hash can be used in further merges of super projects, recursively. 4. To create a super project bump commit with the submitted submodule commits, Zuul need to know resulting commit hashes. One way is to submit the projects in order, but what happens if there is a problem with submitting some of them? To solve this, I would like to make the merger deterministic, i.e. let the commit date be the enqueue time of the QueueItem. When the gate has finished, push the tested commits to `refs/zuul/gate/<target-branch>`. When the commits have been pushed to all the projects, we know that uploading has succeeded and a second step of pushing to the actual target ref can be done "for free". Hopefully this will can be done much quicker and might also close existing changes or pull requests on the server. Questions: Shall reading .gitmodules be feature gated per tenant and overridden per super project? What is the best way to handle the submit process? If Zuul owns the merging, then the commit hash tested in gate will also end up in the repository, which is a great benefit. Maybe there is a possibility to discuss these things on the OpenInfra Day Sweden on Tuesday (7th May, 2024). Best regards, Fredrik

3 6

PTG October 2024 Team Signup
by Kendall Nelson 24 Jun '24

24 Jun '24

Hello Everyone, Hopefully you have seen by now, the next OpenInfra PTG[1] which will take place October 21-25, 2024 virtually! Registration is now open[2]! To sign up your team, you must complete the survey[3] by August 30th at 7:00 UTC. We NEED accurate contact information (email and IRC nickname ideally) for the moderator of your team’s sessions. This is because the survey information will be used to organize the schedule signups which will be done via the PTGBot. If you are not on IRC, please get setup[4] on the OFTC network and join #openinfra-events. You are also encouraged to familiarize yourself with the PTGBot documentation[5] as well. If you have any questions, please reach out to ptg(a)openinfra.dev! Information about signing up for time slots will be sent to moderators after the team signup deadline. Continue to visit openinfra.dev/ptg for updates. [1] PTG Site: https://openinfra.dev/ptg/ [2] PTG Registration: http://ptg.openinfra.dev/ [3] Team Survey: https://openinfrafoundation.formstack.com/forms/oct2024_ptg_survey [4] Setup IRC: https://docs.openstack.org/contributors/common/irc.html [5] PTGBot README: https://opendev.org/openstack/ptgbot/src/branch/master/README.rst -- Kendall Nelson *Senior Upstream Developer Advocate* The OpenInfra Foundation <https://openinfra.dev/>

1 0

Elasticsearch driver
by James E. Blair 19 Jun '24

19 Jun '24

Hi, In reviewing our current dependency caps, I noticed that we have pinned the python elasticsearch client as "elasticsearch<8.0.0". I looked into removing that cap, and found that we have no test coverage of the elasticsearch library itself. Typically, we try to have some kind of fake or mock that exercises at least some library code, if not all of it. But in this case, there is none. Version 8 of the library is known to break Zuul in the initializer for the library, but our tests won't catch that. If I were to propose to remove the cap, we won't have any feedback that it's broken until someone tries it in production. The compatibility page for the Python ES client has this to say: "Elasticsearch language clients are only backwards compatible with default distributions and without guarantees made." [1] That is a little confusing to me. But I read it as "You might be able to use the python elasticsearch client version 8 with an elasticsearch version 7 server, but we won't promise it will work. If you really want guarantees, you should install two versions simultaneously." I think it's worth asking the following questions: * Should we continue to have an ES driver in Zuul, or does the tight coupling of client library to server version make it more appropriate to offload this to an external service that moves MQTT messages into ES? * If we want to keep the ES driver, can we support multiple server versions with a single client library? * If we can not, should we switch to installing both elasticsearch7 and elasticsearch8 in Zuul and switch between the two at runtime? Is there a volunteer that will do this work? * If we can support both with version 8 of the elasticsearch library, is there a volunteer to do this work? * Can we update the testing to exercise more of the client library (via a mock or fake elasticsearch)? Is there a volunteer for this work? Thanks, Jim [1] https://elasticsearch-py.readthedocs.io/en/v8.14.0/#compatibility

2 2

Status page spec
by James E. Blair 17 Jun '24

17 Jun '24

Hi, Felix has posted a spec which aims to collect user stories related to the status page, for aid in review of his changes to the status page. The spec has seen some initial positive review already. Given that, I would like to merge it next Monday, June 24, if it looks like we have consensus then, or of course delay it if folks identify problems or need more time to review. It is located here: https://review.opendev.org/918505 Thanks, Jim

1 0

Spec for allowing untrusted projects to configure other projects
by James E. Blair 15 Jun '24

15 Jun '24

Hi, I have proposed a spec at https://review.opendev.org/922061 which describes a system where we can selectively allow certain untrusted-projects to configure the project-pipeline config for certain other untrusted-projects. This has some relevance to work involving superprojects, in that it will allow superprojects to configure the jobs that run on subprojects. But it is also useful outside that context, in that it will allow for centralized control of what jobs run on certain projects without all the permissions (and restrictions) that come with a config-project. Please take a look and provide any feedback. Thanks, Jim

1 0

New status page for the Zuul UI
by Felix.Edel＠bmw.de 06 Jun '24

06 Jun '24

Hi, I’ve been working on a stack of changes to rework the status view in the Zuul UI: https://review.opendev.org/c/zuul/zuul/+/916284 In our Zuul setup we have some tenants which are quite large and have a lot of pipelines and longer (gate) queues. For those tenants, the status page becomes more and more unusable as too much information is shown on this single page. This makes it hard for the “operators” of this tenant to get a clear picture of what’s going on and for the individual “developers” it’s hard to have a look at their individual changes (or queues). So, we have done surveys with some of our key users and tried to find out how the current status view could be improved and what they would expect from such a view (which information should be shown, which actions/options should be available). Based on this and our own experience with the Zuul UI we came up with some improvements which are implemented in the stack mentioned above. Basically, the idea behind those changes is to provide both a “operator-oriented” view and a “developer-oriented” view without mixing both into a single view (which is kind of what the current status view tries to do). Instead of showing all information on a single page, the new approach splits that information into different “scopes”: 1. Status / pipeline overview 2. Pipeline details 3. Individual change Based on those scopes, the pages that handle those focus only on the relevant information, but don’t show too many details about the other scopes. The stack currently covers 1. and 2., whereby 3. is already existing in form of the “single change panel/view”. I’m planning to do a rework of that as well later. Please note, that this is still an ongoing work and not yet finished, so there might be some flaws. If you are interested in this topic or have any questions, don’t hesitate to contact me, or reply to this email. I’d be happy to get some feedback/reviews on those changes. A preview of the new UI can be found here: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/… Br, Felix

2 2