- Zuul-discuss - lists.opendev.org

Zuul Stickers
by James Cole 08 May '18

08 May '18

Hey everyone! I just wanted to share a pic of the stickers we’re giving away at the Summit in Vancouver. A link is below in case you can’t see the attachment. https://www.dropbox.com/s/dwb8df3wuqey5i6/Zuul_Stickers.jpg?dl=0 Come grab one in Vancouver! James Cole Graphic Designer OpenStack Foundation

1 0

Polling users of Nodepool's static node driver
by David Shrewsbury 08 May '18

08 May '18

Hi, Is anyone currently using the static driver? And if so, are you making us of the multi-label feature it supports for the static nodes? I ask because I've been trying to change the behavior of the driver to pre-populate ZooKeeper with the static nodes (more in line with how the OpenStack driver works), rather than only having the ZK nodes appear when the node is actually in-use. Doing so fundamentally changes how the static driver works, and this deep dive of the code has revealed this multi-label usage which I think many of us (certainly me) have missed. Changing the driver to operate more like how we expect will mean changing nodepool to support multiple labels across *all* drivers. That's not a simple change, and affects other things, such as statsd reporting (which reports per-label stats). (And if you were wondering how the static driver was getting away with supporting multiple labels before, it was the fact that it *didn't* pre-register its nodes with ZK that allowed that.) One one hand, I can see how supporting multiple labels might be useful (e.g., a node labeled with both ubuntu-xenial and ubuntu-latest). But I don't see a great benefit to doing so. If no one is depending on this multi-label behavior, I'd like to remove it from the static driver. -Dave -- David Shrewsbury (Shrews)

4 6

Zuul project update 2018-05-07
by corvus＠inaugust.com 07 May '18

07 May '18

Hi, This past week has seen the following: Shrews began attempting to fix nodepool static driver to pre-register static nodes in ZooKeeper. This led to the discussion in [1] I pushed patches to add our new logo to the website and docs, as well as some other changes to the website suggested by the marketing team. The OpenDev conference and OpenStack summit are fast approaching. I have a lot to prepare before then -- both in terms of preparing for talks as well as polishing the website and documentation as much as possible, so I'm probably going to focus on those for the next two weeks rather than take up any significant new development work. Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email Thanks, Jim [1] http://lists.zuul-ci.org/pipermail/zuul-discuss/2018-May/000197.html

1 0

Re: [openstack-dev] Overriding project-templates in Zuul
by Joshua Hesketh 02 May '18

02 May '18

> > I think in actuality, both operations would end up as intersections: > > ================ ======== ======= ======= > Matcher Template Project Result > ================ ======== ======= ======= > files AB BC B > irrelevant-files AB BC B > ================ ======== ======= ======= > > So with the "combine" method, it's always possible to further restrict > where the job runs, but never to expand it. Ignoring the 'files' above, in the example of 'irrelevant-files' haven't you just combined the results to expand when it runs? ie, A and C are /not/ excluded and therefore the job will run when there are changes to A or C? I would expect the table to be something like: ================ ======== ======= ======= Matcher Template Project Result ================ ======== ======= ======= files AB BC B irrelevant-files AB BC ABC ================ ======== ======= ======= > > So a job with "files: tests/" and "irrelevant-files: docs/" would do > > whatever it is that happens when you specify both. > > In this case, I'm pretty sure that would mean it reduces to just "files: > tests/", but I've never claimed to understand irrelevant-files and I > won't start now. > Yes, I think you are right that this would reduce to that. However, what about the use case of: files: tests/* irrelevant-files: tests/docs/* I could see a use case where both of those would be helpful. Yes you could describe that as one regex but to the end user the above may be expected to work. Unless we make the two options mutually exclusive I feel like this is a feature we should support. (That said, it's likely a separate feature/functionality than what is being described now). Anyway, I feel like Proposal #2 is more how I would expect the system to behave. I can see an argument for combining the results (and feel like you could evaulate that at the end after combining the branch-matching variants) to give something like: ================ ======== ======= ======= Matcher Template Project Result ================ ======== ======= ======= files AB BC ABC irrelevant-files AB BC ABC ================ ======== ======= ======= However, that gives the user no way to remove a previously listed option. Thus overwriting may be the better solution (ie proposal #2 as written) unless we want to explore the option of allowing a syntax that says "extend" or "overwrite". Yours in hoping that made sense, Josh

2 1

Re: [openstack-dev] Overriding project-templates in Zuul
by corvus＠inaugust.com 01 May '18

01 May '18

Joshua Hesketh <joshua.hesketh(a)gmail.com> writes: > I might be misunderstanding at which point a job is chosen to be ran and > therefore when it's too late to dissuade it. However, if possible, would it > make more sense for the project-local copy of a job to overwrite the > supplied files and irrelevant-files? This would allow a project to run a > job when it otherwise doesn't match. Imagine that a project with branches has a job added via a template. project-config/zuul.yaml@master: - job: name: my-job vars: {jobvar: true} - project-template: name: myjobs check: jobs: - my-job: vars: {templatevar: true} project/zuul.yaml@master: - project: templates: - myjobs check: jobs: - my-job: vars: {projectvar: true} project/zuul.yaml@stable: - project: templates: - myjobs check: jobs: - my-job: vars: {projectvar: true} The resulting project config is: - project: jobs: - my-job (branches: master; project-local job) - my-job (branches: master; project-template job) - my-job (branches: stable; project-local job) - my-job (branches: stable; project-template job) When Zuul decides what to run, it goes through each of those in order, evaluates their matchers, and pulls in parents and their variants for each that matches. So a change on the master branch would collect the following variants to apply: my-job (branch: master; project-local job) my-job (job) base (job) my-job (branch: master; project-template job) my-job (job) base (job) It would then apply them in this order: base (job) my-job (job) my-job (branch: master; project-template job) my-job (branch: master; project-local job) To further restrict a project-local job with a "files:" matcher would cause the project-local job not to match, but the project-template job will still match, so the job gets run. That's the situation we have today, which is what I meant by "it's too late to dissuade it". Regarding the suggestion to overwrite it, we would need to decide which of the possible variants to overwrite. Keep in mind that there are 3 independent matchers operating on all the variants (branches, files, irrelevant-files). Does a project-local job with a "files:" matcher overwrite all of the variants? Just the ones which match the same branch? That would probably be the most reasonable thing to do. In my mind, that effectively splits the matchers into two categories: branch matchers, and file matchers. And they would behave differently. Zuul could collect the variants as above, considering only the branch matchers. It could then apply all of the variants in the normal manner, treating files and irrelevant-files as normal attributes which can be overwritten. Then, once it has composed the job to run based on all the matching variants, it would only *then* evaluate the files matchers. If they don't match, then it would not run the job after all. I think that's a very reasonable way to solve the issue as well, and I believe it would match people's expectations. Ultimately, the outcome will be very similar to the proposal I made except that rather than being combined, the matchers will be overwritten. That means that if you want to expand the set of irrelevant-files for a job, you would have to copy the set from the parent. There's one other aspect to consider -- it's possible to create a job like this: - job: name: doc-job - jobs: name: doc-job files: docs/index.rst vars: {rebuild_index: true} Which means: there's a normal docs job with no variables, but if docs/index.rst is changed, set the rebuild_index variable to true. Either approach (combine vs overwrite) eliminates the ability to do this within a project or project-template stanza. But the "combine" approach still lets us do this at the job level. We could still support this in the overwrite approach, however, I think it might be simpler to work with if we eliminated it as well and just always treated files and irrelevant-files matchers as overwriteable attributes. It would no longer be possible to implement the above example, but I'm not sure it's that useful anyway? > What happens when something is in both files and irrelevant-files? If the > project-template is trying to say A is in 'files', but the project-local > says A is in 'irrelevant-files', should that overwrite it? I think my statement and table below was erroneous: >> This effectively causes the "files" and "irrelevant-files" attributes on >> all of the project-local job definitions matching a given branch to be >> combined. The combination of multiple files matchers behaves as a >> union, and irrelevant-files matchers as an intersection. >> >> ================ ======== ======= ======= >> Matcher Template Project Result >> ================ ======== ======= ======= >> files AB BC ABC >> irrelevant-files AB BC B >> ================ ======== ======= ======= I think in actuality, both operations would end up as intersections: ================ ======== ======= ======= Matcher Template Project Result ================ ======== ======= ======= files AB BC B irrelevant-files AB BC B ================ ======== ======= ======= So with the "combine" method, it's always possible to further restrict where the job runs, but never to expand it. As to your question about what if something appears in both -- in my "combine" proposal, I would continue to treat them independently, so if a project-local variant has an "irrelevant-files:" matcher, and a project-template variant has a "files:" matcher, then the usual nonsense would happen: they'd both have to match. So a job with "files: tests/" and "irrelevant-files: docs/" would never run because it's impossible to satisfy both. That actually matches some current behavior we have -- a job must match a "real" job definition as well as at least one project-template or project-local job definition, so a "job" with "files: tests/" and a project-template with "irrelevant-files: docs/" will behave the same way today. However, if we go with the "overwrite" proposal, since we're altering the behavior of the non-branch matchers anyway, that might be a fine time to pair them up and treat them as a unit. Then, whichever is the latest would win. So a job with "files" and a project-template with "irrelevant-files" would end up only having an irrelevant-files attribute. Then a further project-local stanza with only "files" would eliminate the irrelevant-files attribute leaving only the files value. Okay, let's summarize: Proposal 1: All project-template and project-local job variants matching the item's branch must also match the item. * Files and irrelevant-files on project-template and project stanzas are essentially combined in a set intersection. * It's possible to further reduce the scope of jobs, but not expand. * Files and irrelevant-files are still independent matchers, and if both are present, both must match. * It's not possible to alter a job attribute by adding a project-local variant with only a files matcher (it would cause the whole job to run or not run). But it's still possible to do that in the main job definition itself. Proposal 2: Files and irrelevant-files are treated as overwriteable attributes and evaluated after branch-matching variants are combined. * Files and irrelevant-files are overwritten, so the last value encountered when combining all the matching variants (looking only at branches) wins. * Files and irrelevant-files will be treated as a pair, so that if "irrelevant-files" appears, it will erase a previous "files" attribute. * It's possible to both reduce and expand the scope of jobs, but the user may need to manually copy values from a parent or other variant in order to do so. * It will no longer be possible to alter a job attribute by adding a variant with only a files matcher -- in all cases files and irrelevant-files are used solely to determine whether the job is run, not to determine whether to apply a variant. I think both would be good solutions to the problem. The key points for me are whether we want to keep the "alter a job attribute with variant with a files matcher" functionality (the "rebuild_index" example from above), and whether the additional control of overwriting the matchers (at the cost of redundancy in configuration) is preferable to combining the matchers. -Jim

1 1

Zuul Logo
by James Cole 30 Apr '18

30 Apr '18

Hello Zuul Enthusiasts, I’m James, a graphic designer with the OpenStack Foundation, and I’ve been working with Jim and the Zuul-core team to design an official Zuul logo. As you probably know, the goal has been to create something unique that can be used across a variety of mediums, such as an avatar on GitHub, on the website, as badge, and on other types of Zuul communications. After a few iterations, the team is leaning towards the concept featured in the attached PDF. The symbol in the logo was inspired by the geometry of the gate in Ghostbusters, since Zuul focuses on project gating and the name “Zuul” originated from the Zuul the Gatekeeper in the movie. More generally, the logo consists of a triangular, linework shape with the Zuul word mark placed underneath. The logo can be used a number of different ways and still maintain legibility at smaller sizes. The mockups in the attached PDF show a few different use cases, and offers some supporting typography and color information at the end. In case the email attachment did not make it to you, here is a link to view the document on Dropbox <https://www.dropbox.com/s/vv4yszyev1q9pbs/Zuul_Logo.pdf?dl=0>. We need to finalize something soon, since Zuul will be featured at the OpenStack Summit in May. So, we would love to hear what you think about this idea, if there is a detail we missed, or if you have any other feedback. Thank you! James Cole Graphic Designer OpenStack Foundation

10 22

Overriding project-templates
by corvus＠inaugust.com 30 Apr '18

30 Apr '18

Hi, We tried to make the Zuul v3 configuration language as intuitive as possible, and incorporated a lot that we learned from our years running Zuul v2. One thing that we didn't anticipate was how folks would end up wanting to use a job in both project-templates *and* local project stanzas. Essentially, we had assumed that if you wanted to control how a job was run, you would add it to a project stanza directly rather that use a project-template. It's easy to do so if you use one or the other. However, it turns out there are lots of good reasons to use both. For example, in a project-template we may want to establish a recommended way to run a job, or that a job should always be run with a set of related jobs. Yet a project may still want to indicate that job should only run on certain changes in that specific repo. To be very specific -- a very commonly expressed frustration is that a project can't specify a "files" or "irrelevant-files" matcher to override a job that appears in a project-template. Reconciling those is difficult, largely because once Zuul decides to run a job (for example, by a declaration in a project-template) it is impossible to dissuade it from running that job by adding any extra configuration to a project. We need to tread carefully when fixing this, because quite a number of related concepts could be affected. For instance, we need to preserve branch independence (a change to stop running a job in one branch shouldn't affect others). And we need to preserve the ability for job variants to layer on to each other (a project-local variant should still be able to alter a variant in a project-template). I propose that we remedy this by making a small change to how Zuul determines that a job should run: When a job appears multiple times on a project (for instance if it appears in a project-template and also on the project itself), all of the project-local variants which match the item's branch must also match the item in order for the job to run. In other words, if a job appears in a project-template used by a project and on the project, then both must match. This effectively causes the "files" and "irrelevant-files" attributes on all of the project-local job definitions matching a given branch to be combined. The combination of multiple files matchers behaves as a union, and irrelevant-files matchers as an intersection. ================ ======== ======= ======= Matcher Template Project Result ================ ======== ======= ======= files AB BC ABC irrelevant-files AB BC B ================ ======== ======= ======= I believe this will address the shortcoming identified above, but before we get too far in implementing it, I'd like to ask folks to take a moment and evaluate whether it will address the issues you've seen, or if you foresee any problems which I haven't anticipated. Thanks, Jim

1 0

Zuul project update 2018-04-30
by corvus＠inaugust.com 30 Apr '18

30 Apr '18

Hi, This past week has seen the following: We released Zuul 3.0.2. Note that we added a dependency on fb-re2, a python wrapper for the re2 library, which is now a binary dependency. The long-term plan is to start moving all of our regular expression parsing to re2, mostly because it is resistant to denial-of-service attacks. Clint has some patches in progress to do that. But in the mean time, any new uses of regular expressions should use re2 so that we don't create additional debt. We plan on issuing a Nodepool release this week with several accumulated bugfixes and improvements. Tobiash improved the merger so that it is more able to recover from an unclean shutdown. He also has a change [1] in review to add regular expression support to the project stanza so that you can, for example, specify that a job should run on all projects. Last week (and over the weekend) we merged my work to reduce memory consumption for configuration changes. It's now reasonable to consider configuration changes as not any more expensive than normal changes. The CPU time needed to calculate configuration changes is also reduced. This week I plan on returning to reviewing our outstanding specs. Tristan has been working on adding MQTT schema documentation. He also has been working on adding OpenAPI documentation but couldn't find a sphinx doc generator or how to embed swagger-ui in zuul-web. It sounds like we could use some investigation in to how to best handle documentation for the REST API. He also has a patch [2] to support secure diskimage env-vars in nodepool. Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email Thanks, Jim [1] https://review.openstack.org/535713 [2] https://review.openstack.org/564687

1 0

Proposed updates to website to empower our non-coder friends
by Monty Taylor 26 Apr '18

26 Apr '18

Hey everybody! As part of the work with the Foundation staff on getting Zuul set up, we've chatted a bit about how to manage the website content. Obviously from the dev side, we find it important to be able to edit content like source code, propose those changes to gerrit and to review each other's changes. There are, however, folks out there who would like to help us do things like marketing who are not crazy about the concept of editing raw html. SO - we went and did a little bit of exploration to see what we could find. Actually, what really happened is that we agreed we'd just go with the html thing we have currently for now and circle back later to figure out a CMS + code review system that would work for everyone. But then I had a weekend and a personal project that had pretty much the same set of requirements. End result - what I'd like to propose is that we adopt a combination of using gatsby [1] as a static site generator and netlify-cms [2] as a web-based CMS interface for editing content and submitting changes to gerrit. Both gatsby and netlify-cms are completely client-side - there are no server side components to run. I found them by looking at https://headlesscms.org/ and https://www.staticgen.com/ gatsby ------ gatsby is a Javascript/React-based static site generator that allows managing content in markdown files. It has netlify-cms plugin support, and the build toolchain works very similarly to the new javascript toolchain for zuul-dashboard. I've got a first rough-draft of this part done that I'll push up for review today or tomorrow. netlify-cms ----------- netlify-cms is a Javascript/React-based CMS for static site generators. It has support for either pushing changes directly in to git or for submitting them as proposed changes. It currently basically only supports github - but the github interface is broken out into a driver layer. In fact, they have two different options- one that talks to github directly and one that talks to a netlify git-gateway service, so the plugin layer isn't just theoretical. We'll need to write a gerrit driver, but that shouldn't be terribly hard. Proposed Plan ------------- We can keep this pretty simple. Step one- migrate existing site from static html to gatsby + zuul jobs. Step two - write gerrit driver for netlify-cms. Step three - add netlify-cms to website. Why not use ... --------------- There are many options out there for static site generation. I like gatsby because it's pure-javascript rather than having a go, ruby or python component. I've personally come to really like keeping web things in javascript ... especially since our zuul-dashboard is doing javascript build tooling now too. There is also tight integration support with netlify-cms (which is also a good choice for the same reason) - and it was super easy for me to get up and going with one of their examples. [1] https://www.gatsbyjs.org/ [2] https://www.netlifycms.org/

3 3

Zuul project update 2018-04-23
by corvus＠inaugust.com 25 Apr '18

25 Apr '18

Hi, This past week has seen the following: Shrews started testing Ansible 2.5 and has related changes up for review[1]. I made a video[2] to upload to the website which explains the gating algorithm. I expect it to be merged soon. I've also resumed work on the configuration loading refactor which will allow us to use less memory for dynamic configuration changes. I plan on continuing that this week. Tobiash has been improving scalability with parent-change-enqueued event and many github app installations[3]. Part or that stack improves github testing. Next week, he plans on working on weak job dependencies (jobs that still run even if their parents are skipped -- so that Zuul can decide that an image build job does or doesn't need to run), along with finishing up GitHub branch status protection handling. Clarkb has (with Shrews) made zuul-web more asynchronous. He points out that we've merged a number of bugfixes, including updates to zuul-changes and encrypt_secret. We've alo removed the zuul-cloner command. He also asks whether it's time to make another release. TristanC is working on `allowed-triggers` and `allowed-reporters` tenant settings to better support a multi-tenant environment with a multitude of connections. The nodepool driver work mentioned last week has merged. Another change[4] is proposed to make the OpenStack NodeLauncher generic. Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email Thanks, Jim [1] https://review.openstack.org/#/q/topic:zuul-ansible-2.5 [2] https://review.openstack.org/562005 [3] https://review.openstack.org/#/q/topic:parent-change-enqueued+(status:open+… [4] https://review.openstack.org/535555

2 1

Re: Zuul + OpenStack Foundation update
by Lauren Sell 24 Apr '18

24 Apr '18

Hi everyone, Thanks very much Jim for providing the update on our plans heading into the Vancouver Summit. I want to sincerely apologize for the miscommunication around the board meeting. We presented an update on making Zuul an independent project at the Gold Member meeting yesterday and again at the Board Meeting today, but unfortunately the entire presentation today was scheduled during the executive session and not able to be heard by the public. It was a definitely oversight and miscommunication. I want to share the deck we presented here and am happy to do a call for those who want to run through it, or simply chat about it in IRC. We welcome any feedback on the plans: https://docs.google.com/presentation/d/1iR_Q0ntV5BAs1TNjN-b1JBnsfahb2hI4IxM… The primary feedback we received during the gold member meeting was trying to understand where Zuul would fit under the OpenStack Foundation (ie. is it still part of OpenStack, or an adjacent project with independent governance?) and where Zuul fits in the CI/CD landscape (ie. what’s competitive and complementary). For people who are newer to the space, there has been a lot more activity recently around XCI and the OpenCI initiative and of course existing tools like Jenkins and new ones like Spinnaker, and they need help parsing what’s an effort by a group of people versus what’s actually software and when you would use which software :) Hope that is useful, and again, apologies for the miscommunication. I’m now subscribed to the mailing list and am happy to discuss here :) Thanks, Lauren PS If you can’t view the deck in google docs, I’m happy to send over a PDF. I just didn’t want to send an attachment to the mailing list... >> Begin forwarded message: >> >> From: corvus(a)inaugust.com <mailto:corvus@inaugust.com> (James E. Blair) >> Subject: Zuul + OpenStack Foundation update >> Date: April 11, 2018 at 5:28:08 PM CDT >> To: Zuul Discussion List <zuul-discuss(a)lists.zuul-ci.org <mailto:zuul-discuss@lists.zuul-ci.org>> >> >> Hi, >> >> Several of us on the zuul-core team have been working with the OpenStack >> Foundation staff on establishing and launching Zuul as a top-level >> project under the Foundation's umbrella. >> >> As a free/open-source project, and one that follows OpenStack's "four >> opens" we generally strive to use public channels for collaboration. >> However, when making a change like this, that is not always practical, >> especially in the early stages. I think we'll get there soon, but in >> the mean time, I ask you to accept Monty, Clark, Jeremy and I as >> delegates of the project to work with the Foundation on this. >> >> I'll try to keep everyone updated as best as possible with what we're >> working on, though sometimes those updates may need to be vague. Please >> feel free to contact me or the others privately if you have specific >> concerns you want to ensure we address. Here are updates from our most >> recent discussions: >> >> * There will be discussion about Zuul at the OpenStack board of >> directors meeting on April 24. That will be a phone call to which >> anyone is welcome to listen. >> >> * The Foundation will promote Zuul at the upcoming OpenStack Summit in >> Vancouver. It will be officially announced as a Foundation project on >> the keynote stage and via press release. >> >> * We are working with the Foundation's design team on a logo. We want >> to have something that can be adapted to work as an avatar (i.e., on >> GitHub), on the website, as a badge, and on slide decks. >> >> * The Foundation marketing team has begun a review of the website >> content and will be assisting with changes. Included in this is an >> effort to collect user stories for Zuul. >> >> * We're looking into implementing a system to manage the website content >> which is both Gerrit-friendly and has a workflow suitable for those >> unaccustomed to using Git. Monty has volunteered to research and >> report back with some options to explore. >> >> -Jim >> >> _______________________________________________ >> Zuul-discuss mailing list >> Zuul-discuss(a)lists.zuul-ci.org <mailto:Zuul-discuss@lists.zuul-ci.org> >> http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-discuss >

1 0

Simplified DSL
by Tristan Cacqueray 21 Apr '18

21 Apr '18

Hello folks, Some users may find job written in Ansible hard to reproduce, and I was wondering if we could add a new "exec" job attribute that would simply be shell task to be executed on a single node. This would let us write zuul.yaml file more similar to other in-repo ci system. For example a zuul.yaml could be: - job: name: my-job exec: - mvn build - mvn test - project: check: jobs: - my-job gate: jobs: - my-job Moreover we could go a step further and streamline that full description in a single stanza: - ci-job: - mvn build - mvn test Which could be translated into a unique job name attached to the check and gate pipeline by default. What do you think? -Tristan

8 9

Question regarding using etcd instead of Zookeeper
by Tristan Cacqueray 21 Apr '18

21 Apr '18

Hello folks, From a Kubernetes operator point of view, the Zookeeper component sounds redundant with the already running etcd service. Moreover etcd seems more active and it shows better overall performance. So my question is why Zookeeper was picked in the first place, and would it be possible to support/migrate to etcd in the future? Thanks in advance, -Tristan

9 11

Reverse of required-projects.override-checkout?
by Mike Pagel 20 Apr '18

20 Apr '18

Hi all, I am using zuul now for a while at work and I am struggling to properly configure the following use-case. Hope I don't miss the obvious. I have two repositories A and B. A is an internal, B is an external project from another department. We define our jobs in A's zuul.yaml. B is using different branch names (e.g., because they deliver to other teams as well). A/zuul.yaml (master branch): # build the "master-a" branch of B for A's "master": job: required-projects: name: B override-checkout: master-a Similarly, when we have a maintenance branch "v1.0" in A, B might call it "v1.0-a": A/zuul.yaml (v1.0 branch): # build the "v1.0-a" branch of B for A's "v1.0": job: required-projects: name: B override-checkout: v1.0-a Now, without duplication of these job definitions, how would I run the correct job variant when checking a change in B, let's say in the "v1.0-a" branch, where I cannot fall back to "master" as default branch? AnsibleJob.resolveBranch() has some logic, but I think that's still the forward reference. Could I use job.branches for this, as in: A/zuul.yaml (v1.0 branch): # build this "v1.0" branch in A if a change occurred in (any) "v1.0-a" branch: job: branches: v1.0-a required-projects: name: B override-checkout: v1.0-a I see code in and calls to model.Job.changeMatches() that could suggest this behavior. Or would it try to als check out A's "v1.0-a" branch, which is non-existent? Any hints are greatly appreciated. Mike

2 3

GitHub OAuth application
by Tristan Cacqueray 18 Apr '18

18 Apr '18

Hello folks, Would it be possible to adapt the GitHub driver to use user's access instead of the Zuul unique app key/api token? Ideally, users would register on the scheduler and get redirected to grant access to their GitHub identity through an OAuth app. Then zuul could manage the project's hook. That would be the easiest workflow from an user point of view. Though this would require some sort of user management. Another solution would be to create the access_token out of bound and extend the driver configuration to support using multiple tokens to be used per projects or oganizations. If this can be implemented without a major rewrite of the driver, that sounds like something worthy as it would enable new use cases and it may mitigate rate limiting issue for large deployment. Regards, -Tristan

2 2

Zuul project update 2018-04-16
by corvus＠inaugust.com 16 Apr '18

16 Apr '18

Hi, Last week, we agreed to suspend weekly IRC meetings and instead try to conduct more of our business over email, or in ad-hoc IRC meetings as needed. In an attempt to keep the cadence provided by the meetings, we're going to send out a weekly email update. Hopefully this will help us all keep up with what's going on in the project. Please feel free to reply to this email with further discussion. Last week, we released Zuul 3.0.1. Shrews fixed a common bug seen in long streaming involving multi-byte characters across read boundaries, and is continuing work on making zuul-web more async. Also in progress: improvements to the zuul-from-scratch document. [1] I posted a spec about container support [2]. It's more of a starting point for discussion than a complete plan at this point. It's collected a number of comments so far, and will certainly be revised several times before it is finalized. This week, I plan to work on resuming the work I started before the 3.0.0 release to reduce the memory used by dynamic configuration changes. Mordred has switched us to using nodeenv for javascript dependencies which makes it easier to run the tests locally and means we can use the standard tox-py35 job. He also upgraded webpack to version 4. This week has us moving the dashboard from angularjs v2 to angular.io v5. [3] Tristan has updated the MQTT driver which should be ready now and updated several jobs in zuul-jobs (ansible-lint, review, spec and upload-to-galaxy). In nodepool, the run_handler refactor (which is intended to make adding more drivers easier) is ready for review. He also updated zuul-changes.py to support the new zuul-web api routes. Tobiash fixed a number of bugs, and has patches up to refactor the executor's governor into a sensor API for easier extension. He added support for postgres (we run tests on PG and mysql now), and regex matching of github status. This week he plans on trying out Ansible 2.5, improving scalability of parent-change-enqueued event across many github app installations, and adding tests to status protection handling. Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email Thanks, Jim [1] https://review.openstack.org/#/q/status:open+project:openstack-infra/zuul+b… [2] https://review.openstack.org/560136 [3] https://review.openstack.org/#/q/status:open+project:openstack-infra/zuul+b…

1 0

Next Zuul release (3.0.1?)
by corvus＠inaugust.com 11 Apr '18

11 Apr '18

Hi, We have some mildly-urgent bugfixes merged or ready to merge in Zuul, so I'd like to make a release sometime in the next few days. By mildly-urgent, I mean that the sooner we release the fixes, the less problems people will have in the future (either by creating jobs relying on wrong behavior, or system configurations which are problematic). The items I have on the list are: * Roles checkout fix (merged) * Github3.py version bump (ready to merge) * Postgress fixes/support (merging as I write this) We've also merged some other bugfixes, and may slip some others in during the remaining time. Are there any other items of similar urgency to the three above that we need to release ASAP? Once we're satisfied (probably tomorrow?) I'll release 3.0.1. I think that since we're only focusing on bugfixes, the micro version bump is appropriate. I don't think Nodepool needs a corresponding release. -Jim

4 3

Zuul + OpenStack Foundation update
by corvus＠inaugust.com 11 Apr '18

11 Apr '18

Hi, Several of us on the zuul-core team have been working with the OpenStack Foundation staff on establishing and launching Zuul as a top-level project under the Foundation's umbrella. As a free/open-source project, and one that follows OpenStack's "four opens" we generally strive to use public channels for collaboration. However, when making a change like this, that is not always practical, especially in the early stages. I think we'll get there soon, but in the mean time, I ask you to accept Monty, Clark, Jeremy and I as delegates of the project to work with the Foundation on this. I'll try to keep everyone updated as best as possible with what we're working on, though sometimes those updates may need to be vague. Please feel free to contact me or the others privately if you have specific concerns you want to ensure we address. Here are updates from our most recent discussions: * There will be discussion about Zuul at the OpenStack board of directors meeting on April 24. That will be a phone call to which anyone is welcome to listen. * The Foundation will promote Zuul at the upcoming OpenStack Summit in Vancouver. It will be officially announced as a Foundation project on the keynote stage and via press release. * We are working with the Foundation's design team on a logo. We want to have something that can be adapted to work as an avatar (i.e., on GitHub), on the website, as a badge, and on slide decks. * The Foundation marketing team has begun a review of the website content and will be assisting with changes. Included in this is an effort to collect user stories for Zuul. * We're looking into implementing a system to manage the website content which is both Gerrit-friendly and has a workflow suitable for those unaccustomed to using Git. Monty has volunteered to research and report back with some options to explore. -Jim

1 0

Deprecation notice: Remove support for extensions as lists in zuul-jobs/roles/stage-output - change https://review.openstack.org/540890
by Andreas Jaeger 11 Apr '18

11 Apr '18

Andrea Frittoli has updated roles/stage-output in zuul-jobs to handle dicts instead of lists for extensions - and updated all usage in OpenStack itself. His review https://review.openstack.org/540890 removes now handling of lists. Please check that none of your playbooks and jobs uses extensions with stage-output and update them. I put a WIP on the change and will remove it in two weeks so that we can merge, please update your files as needed and feel free to review the change, Andreas -- Andreas Jaeger aj(a){suse.com,opensuse.org} Twitter: jaegerandi SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126

1 0

Spec for using containers as Zuul build resources
by corvus＠inaugust.com 11 Apr '18

11 Apr '18

Hi, We've all discussed using containers in Zuul for some time, but it's been difficult to get on the same page. Partly because of differing use cases and assumptions, and partly because Zuul brings some unique things to the table which need to be integrated into the container mindset. I've uploaded the first draft of a spec which I hope will address this and set a path for us to successfully integrate containers: https://review.openstack.org/560136 It's long on exposition and short on work items at this point. I think first it's useful for us to look at it from a distance and see if it handles the use cases we think we need to cover, and if the general approaches to workflow will work. Then, we should focus on the details, and expand it to ensure that we describe the actual processes we're going to implement. And finally, it should grow a list of work items so we can get things done. If you haven't reviewed a spec like this before, know this: it's a very iterative process. Please leave detailed inline review comments in Gerrit, where we may engage in a back-and-forth discussion for a while. If it gets awkward to have that discussion in Gerrit, we can move it to the mailing list, but in general, it's preferable to keep the discussion in Gerrit where the context and history are all together. After some discussion, I will update the spec with suggested changes and we'll go through the process again. The goal is to achieve consensus during the review process and merge a polished result, not to merge the change ASAP. This also establishes a specs section of the Zuul developer documentation. We had a brief discussion in an earlier IRC meeting about how to handle specs now that openstack-infra/infra-specs is no longer appropriate. I think putting them directly in the Zuul repo is worth trying. The main danger there is accumulating too much documentation which does not reflect the current state of the project, but this includes big red warnings, so maybe that will help. It also looks like a significant amount of this work may end up happening in the nodepool repo as opposed to zuul, but as long as we don't have a dedicated place for cross-project specs, I think the zuul repo is a reasonable place to start. -Jim

2 1

Zuul meeting time
by Tobias.Henkel＠bmw.de 09 Apr '18

09 Apr '18

Hi zuul community, Currently the zuul meeting is at 22UTC which is currently 23:00 localtime or soon 0:00 within summer time at my location. That makes it impossible for me to participate regularly. However I’d really like to attend the zuul meeting. So I’d like to ask zuul folks about your opinion about shifting it around 20ish UTC. I wouldn’t have any strong opinion about the day. Thanks and kind regards Tobias

9 15

Fedora packages
by Tristan Cacqueray 02 Apr '18

02 Apr '18

Greeting folks, I have created a Nodepool package[0] in Fedora and it is proposed for f27 and f28 updates here[1] and there[2]. Please leave a +1 if the package works for you. The package contains service files, default configurations and man pages. The drivers are splitted in sub packages so that the openstacksdk is only pulled by the nodepool-driver-openstack package. A quick start install doc using dnf could then be reduced to: dnf install nodepool-launcher nodepool-driver-static systemctl start nodepool-launcher Regarding a Zuul package, it doesn't seems feasible at the moment since Fedora ships Ansible version 2.4 and 2.5. Though perhaps with modularity we can have a Zuul module that depends on Ansible 2.3. Regards, -Tristan [0]: https://apps.fedoraproject.org/packages/nodepool [2]: https://bodhi.fedoraproject.org/updates/FEDORA-2018-d2da815b3e [3]: https://bodhi.fedoraproject.org/updates/FEDORA-2018-75021b046f

2 1

Software Factory 3.0 released
by Tristan Cacqueray 30 Mar '18

30 Mar '18

Greeting folks, We are pleased to announce the release of Software Factory version 3.0: http://www.softwarefactory-project.io/releases/3.0 It is a software development forge that integrates the last version of Zuul and Nodepool along with a few patches that couldn't make it into the upstream release: * Ansible-2.4 support: https://review.openstack.org/535839 * Dynamic tenant configuration: https://review.openstack.org/535878 * MQTT reporter: https://review.openstack.org/535543 * Job and Projects dashboard: https://review.openstack.org/550979 * Broken config protection: https://review.openstack.org/535511 * Zookeeper auto-retry: https://review.openstack.org/535537 * OpenContainer Nodepool provider: https://review.openstack.org/535556 * Kubernetes Nodepool provider: https://review.openstack.org/535557 Best regards, The Software Factory Team

3 3

Next steps after 3.0.0 release
by corvus＠inaugust.com 29 Mar '18

29 Mar '18

Hi, Now that 3.0.0 is released[1] (yay!), it's a good time to go over what we need to do next. The first order of business is to land the changes to add reno: https://review.openstack.org/536861 https://review.openstack.org/262597 You should read the docs that those changes introduce (they'll be in the developer docs for future reference), but the key bit is: When adding a noteworthy feature, fixing a noteworthy bug or introducing a behavior change that a user or operator should know about, it is a good idea to add a release note to the same patch. Every release from this point out will have release notes included in the documentation. We relaxed our review standards significantly during the feature branch phase, but we've been returning to normal for the past few months. At this point, the master branches of Nodepool and Zuul should always be kept working and ready for release. Gating usually handles the technical side of that, but don't lose track of the intent when it comes to other aspects. Please make sure that changes have documentation if they affect a user-visible aspect of the system, have a release note if the change is noteworthy, and have tests if they alter behavior. If a change is missing these things, please ask about it. These rules don't need to be too rigid, but if we flex them, we should be in the habit of discussing it and establishing that there's a good reason. We need to be careful about making changes to the zuul-jobs and zuul-base-jobs repos now. We'll write up a policy and include it in the docs for those repos, but the important thing is that we can't use new Zuul features in those repos until we've given folks a chance to upgrade (so probably some number of weeks after the feature appears in a release). And we can't remove features from Zuul until we've removed their use from zuul-jobs, and given folks who might be running a fork of zuul-jobs time to update. I expect us to release Zuul and Nodepool much more often now. Any time we make a significant change and establish that the software is working as intended, we should release. We've assigned some very specific meaning to version numbers recently, but don't think that will continue. The versions we release will be what they are depending on what has landed when we release. Thanks everyone for your work on the release, and I look forward to reviewing all the patches that are waiting in the wings! -Jim [1] http://lists.zuul-ci.org/pipermail/zuul-announce/2018-March/000006.html

1 0

Docs and git repos on zuul-ci.org
by corvus＠inaugust.com 28 Mar '18

28 Mar '18

Hi, We now have all of the Zuul documentation hosted on the Zuul website[1], so folks may want to start referencing those URLs, rather than docs.openstack.org. We also have the Zuul git repos mirrored under the Zuul domain at [2]. That page shows only the Zuul-related git repos. Additionally, if you're referencing the zuul-jobs or zuul-base-jobs repos, you may want to switch your connection to use git.zuul-ci.org instead. Note that if you do that, the project name will be "git.zuul-ci.org/zuul-jobs" -- with no "openstack-infra" prefixed to the name, as there are no organization prefixes on repositories mirrored at git.zuul-ci.org. [1] https://zuul-ci.org/docs/ [2] http://git.zuul-ci.org/ -Jim

2 2