April 2018 - Zuul-discuss - lists.opendev.org

Spec for a tenant-scoped admin web API
by Matthieu Huin 27 Sep '18

27 Sep '18

Hello, Following Jim's example with the containers spec, I've opened a spec review in the zuul repository: https://review.openstack.org/562321 I've been toying around with adding a few "admin" endpoints to zuul and nodepool for some time now [1][2]. Using Software Factory's authorization engine to limit access to these endpoints to specific users, and consuming germqtt's event feed to react to comments on a review, I demonstrated PoCs for features that I think would be of interest to Zuul users in general: * Ability to re-enqueue builds from zuul's web UI * Ability to trigger a autohold by commenting "autohold <job> on <tenant>" on gerrit review The aforementioned patches were meant as experiments; now I would like to go further and work to have this integrated into Zuul in a way that would be tenant-scoped. Before I rework these extensively I seek your guidance and opinions on how to proceed with this; hence this spec review. There are surely things I have overlooked, and my proposal to use groups to implement policies might need discussing. Please let me know what you think, also if it is the right place for the spec (the review on infra-specs is still open) MHU [1] https://review.openstack.org/536319 [2] https://review.openstack.org/539004

1 1

Nodepool driver interface
by David Shrewsbury 13 Sep '18

13 Sep '18

Greetings all, We just merged the last piece needed [1] for all of Tristan's various experimental nodepool drivers. This has reminded me of an idea that I've had in the back of my mind for a while now. Currently, there are reviews up for Kubernetes, Amazon EC2, Ansible, OpenContainer, VMWare, and Azure. I'd like for us to at least discuss the possibility of a pluggable driver system so that the drivers can be developed independently from nodepool, rather than in-tree to nodepool. Why? A. Having them in-tree leads to the assumption that they are tested. We do not have access to all of these services, so actively testing them within nodepool is not (currently) possible. B. Nodepool reviewers will be responsible for maintaining the drivers. I, myself, do not have experience with these services, nor access to them, nor the desire or time to learn some of them. I would venture a guess that most of us share some of these deficiencies. This puts us in a spot of just guessing at fixes, or just ignoring them hoping someone else will pick up the ball. C. With plugins, users could get driver updates/fixes without waiting for a nodepool release. I think keeping the OpenStack driver in-tree makes sense because we actively test and use it. I can even get on board with keeping the static driver too. But the others, to me, would be best kept out of tree *unless* we can add access to those services to our Zuul environment and actively test them as well (that at least takes care of the A reason above, and most of B if we have excellent tests setup). Thoughts? [1] https://review.openstack.org/535555 -- David Shrewsbury (Shrews)

7 9

Zuul Logo
by James Cole 01 May '18

01 May '18

Hello Zuul Enthusiasts, I’m James, a graphic designer with the OpenStack Foundation, and I’ve been working with Jim and the Zuul-core team to design an official Zuul logo. As you probably know, the goal has been to create something unique that can be used across a variety of mediums, such as an avatar on GitHub, on the website, as badge, and on other types of Zuul communications. After a few iterations, the team is leaning towards the concept featured in the attached PDF. The symbol in the logo was inspired by the geometry of the gate in Ghostbusters, since Zuul focuses on project gating and the name “Zuul” originated from the Zuul the Gatekeeper in the movie. More generally, the logo consists of a triangular, linework shape with the Zuul word mark placed underneath. The logo can be used a number of different ways and still maintain legibility at smaller sizes. The mockups in the attached PDF show a few different use cases, and offers some supporting typography and color information at the end. In case the email attachment did not make it to you, here is a link to view the document on Dropbox <https://www.dropbox.com/s/vv4yszyev1q9pbs/Zuul_Logo.pdf?dl=0>. We need to finalize something soon, since Zuul will be featured at the OpenStack Summit in May. So, we would love to hear what you think about this idea, if there is a detail we missed, or if you have any other feedback. Thank you! James Cole Graphic Designer OpenStack Foundation

10 22

Overriding project-templates
by corvus＠inaugust.com 30 Apr '18

30 Apr '18

Hi, We tried to make the Zuul v3 configuration language as intuitive as possible, and incorporated a lot that we learned from our years running Zuul v2. One thing that we didn't anticipate was how folks would end up wanting to use a job in both project-templates *and* local project stanzas. Essentially, we had assumed that if you wanted to control how a job was run, you would add it to a project stanza directly rather that use a project-template. It's easy to do so if you use one or the other. However, it turns out there are lots of good reasons to use both. For example, in a project-template we may want to establish a recommended way to run a job, or that a job should always be run with a set of related jobs. Yet a project may still want to indicate that job should only run on certain changes in that specific repo. To be very specific -- a very commonly expressed frustration is that a project can't specify a "files" or "irrelevant-files" matcher to override a job that appears in a project-template. Reconciling those is difficult, largely because once Zuul decides to run a job (for example, by a declaration in a project-template) it is impossible to dissuade it from running that job by adding any extra configuration to a project. We need to tread carefully when fixing this, because quite a number of related concepts could be affected. For instance, we need to preserve branch independence (a change to stop running a job in one branch shouldn't affect others). And we need to preserve the ability for job variants to layer on to each other (a project-local variant should still be able to alter a variant in a project-template). I propose that we remedy this by making a small change to how Zuul determines that a job should run: When a job appears multiple times on a project (for instance if it appears in a project-template and also on the project itself), all of the project-local variants which match the item's branch must also match the item in order for the job to run. In other words, if a job appears in a project-template used by a project and on the project, then both must match. This effectively causes the "files" and "irrelevant-files" attributes on all of the project-local job definitions matching a given branch to be combined. The combination of multiple files matchers behaves as a union, and irrelevant-files matchers as an intersection. ================ ======== ======= ======= Matcher Template Project Result ================ ======== ======= ======= files AB BC ABC irrelevant-files AB BC B ================ ======== ======= ======= I believe this will address the shortcoming identified above, but before we get too far in implementing it, I'd like to ask folks to take a moment and evaluate whether it will address the issues you've seen, or if you foresee any problems which I haven't anticipated. Thanks, Jim

1 0

Zuul project update 2018-04-30
by corvus＠inaugust.com 30 Apr '18

30 Apr '18

Hi, This past week has seen the following: We released Zuul 3.0.2. Note that we added a dependency on fb-re2, a python wrapper for the re2 library, which is now a binary dependency. The long-term plan is to start moving all of our regular expression parsing to re2, mostly because it is resistant to denial-of-service attacks. Clint has some patches in progress to do that. But in the mean time, any new uses of regular expressions should use re2 so that we don't create additional debt. We plan on issuing a Nodepool release this week with several accumulated bugfixes and improvements. Tobiash improved the merger so that it is more able to recover from an unclean shutdown. He also has a change [1] in review to add regular expression support to the project stanza so that you can, for example, specify that a job should run on all projects. Last week (and over the weekend) we merged my work to reduce memory consumption for configuration changes. It's now reasonable to consider configuration changes as not any more expensive than normal changes. The CPU time needed to calculate configuration changes is also reduced. This week I plan on returning to reviewing our outstanding specs. Tristan has been working on adding MQTT schema documentation. He also has been working on adding OpenAPI documentation but couldn't find a sphinx doc generator or how to embed swagger-ui in zuul-web. It sounds like we could use some investigation in to how to best handle documentation for the REST API. He also has a patch [2] to support secure diskimage env-vars in nodepool. Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email Thanks, Jim [1] https://review.openstack.org/535713 [2] https://review.openstack.org/564687

1 0

Proposed updates to website to empower our non-coder friends
by Monty Taylor 27 Apr '18

27 Apr '18

Hey everybody! As part of the work with the Foundation staff on getting Zuul set up, we've chatted a bit about how to manage the website content. Obviously from the dev side, we find it important to be able to edit content like source code, propose those changes to gerrit and to review each other's changes. There are, however, folks out there who would like to help us do things like marketing who are not crazy about the concept of editing raw html. SO - we went and did a little bit of exploration to see what we could find. Actually, what really happened is that we agreed we'd just go with the html thing we have currently for now and circle back later to figure out a CMS + code review system that would work for everyone. But then I had a weekend and a personal project that had pretty much the same set of requirements. End result - what I'd like to propose is that we adopt a combination of using gatsby [1] as a static site generator and netlify-cms [2] as a web-based CMS interface for editing content and submitting changes to gerrit. Both gatsby and netlify-cms are completely client-side - there are no server side components to run. I found them by looking at https://headlesscms.org/ and https://www.staticgen.com/ gatsby ------ gatsby is a Javascript/React-based static site generator that allows managing content in markdown files. It has netlify-cms plugin support, and the build toolchain works very similarly to the new javascript toolchain for zuul-dashboard. I've got a first rough-draft of this part done that I'll push up for review today or tomorrow. netlify-cms ----------- netlify-cms is a Javascript/React-based CMS for static site generators. It has support for either pushing changes directly in to git or for submitting them as proposed changes. It currently basically only supports github - but the github interface is broken out into a driver layer. In fact, they have two different options- one that talks to github directly and one that talks to a netlify git-gateway service, so the plugin layer isn't just theoretical. We'll need to write a gerrit driver, but that shouldn't be terribly hard. Proposed Plan ------------- We can keep this pretty simple. Step one- migrate existing site from static html to gatsby + zuul jobs. Step two - write gerrit driver for netlify-cms. Step three - add netlify-cms to website. Why not use ... --------------- There are many options out there for static site generation. I like gatsby because it's pure-javascript rather than having a go, ruby or python component. I've personally come to really like keeping web things in javascript ... especially since our zuul-dashboard is doing javascript build tooling now too. There is also tight integration support with netlify-cms (which is also a good choice for the same reason) - and it was super easy for me to get up and going with one of their examples. [1] https://www.gatsbyjs.org/ [2] https://www.netlifycms.org/

3 3

Zuul project update 2018-04-23
by corvus＠inaugust.com 26 Apr '18

26 Apr '18

Hi, This past week has seen the following: Shrews started testing Ansible 2.5 and has related changes up for review[1]. I made a video[2] to upload to the website which explains the gating algorithm. I expect it to be merged soon. I've also resumed work on the configuration loading refactor which will allow us to use less memory for dynamic configuration changes. I plan on continuing that this week. Tobiash has been improving scalability with parent-change-enqueued event and many github app installations[3]. Part or that stack improves github testing. Next week, he plans on working on weak job dependencies (jobs that still run even if their parents are skipped -- so that Zuul can decide that an image build job does or doesn't need to run), along with finishing up GitHub branch status protection handling. Clarkb has (with Shrews) made zuul-web more asynchronous. He points out that we've merged a number of bugfixes, including updates to zuul-changes and encrypt_secret. We've alo removed the zuul-cloner command. He also asks whether it's time to make another release. TristanC is working on `allowed-triggers` and `allowed-reporters` tenant settings to better support a multi-tenant environment with a multitude of connections. The nodepool driver work mentioned last week has merged. Another change[4] is proposed to make the OpenStack NodeLauncher generic. Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email Thanks, Jim [1] https://review.openstack.org/#/q/topic:zuul-ansible-2.5 [2] https://review.openstack.org/562005 [3] https://review.openstack.org/#/q/topic:parent-change-enqueued+(status:open+… [4] https://review.openstack.org/535555

2 1

Re: Zuul + OpenStack Foundation update
by Lauren Sell 25 Apr '18

25 Apr '18

Hi everyone, Thanks very much Jim for providing the update on our plans heading into the Vancouver Summit. I want to sincerely apologize for the miscommunication around the board meeting. We presented an update on making Zuul an independent project at the Gold Member meeting yesterday and again at the Board Meeting today, but unfortunately the entire presentation today was scheduled during the executive session and not able to be heard by the public. It was a definitely oversight and miscommunication. I want to share the deck we presented here and am happy to do a call for those who want to run through it, or simply chat about it in IRC. We welcome any feedback on the plans: https://docs.google.com/presentation/d/1iR_Q0ntV5BAs1TNjN-b1JBnsfahb2hI4IxM… The primary feedback we received during the gold member meeting was trying to understand where Zuul would fit under the OpenStack Foundation (ie. is it still part of OpenStack, or an adjacent project with independent governance?) and where Zuul fits in the CI/CD landscape (ie. what’s competitive and complementary). For people who are newer to the space, there has been a lot more activity recently around XCI and the OpenCI initiative and of course existing tools like Jenkins and new ones like Spinnaker, and they need help parsing what’s an effort by a group of people versus what’s actually software and when you would use which software :) Hope that is useful, and again, apologies for the miscommunication. I’m now subscribed to the mailing list and am happy to discuss here :) Thanks, Lauren PS If you can’t view the deck in google docs, I’m happy to send over a PDF. I just didn’t want to send an attachment to the mailing list... >> Begin forwarded message: >> >> From: corvus(a)inaugust.com <mailto:corvus@inaugust.com> (James E. Blair) >> Subject: Zuul + OpenStack Foundation update >> Date: April 11, 2018 at 5:28:08 PM CDT >> To: Zuul Discussion List <zuul-discuss(a)lists.zuul-ci.org <mailto:zuul-discuss@lists.zuul-ci.org>> >> >> Hi, >> >> Several of us on the zuul-core team have been working with the OpenStack >> Foundation staff on establishing and launching Zuul as a top-level >> project under the Foundation's umbrella. >> >> As a free/open-source project, and one that follows OpenStack's "four >> opens" we generally strive to use public channels for collaboration. >> However, when making a change like this, that is not always practical, >> especially in the early stages. I think we'll get there soon, but in >> the mean time, I ask you to accept Monty, Clark, Jeremy and I as >> delegates of the project to work with the Foundation on this. >> >> I'll try to keep everyone updated as best as possible with what we're >> working on, though sometimes those updates may need to be vague. Please >> feel free to contact me or the others privately if you have specific >> concerns you want to ensure we address. Here are updates from our most >> recent discussions: >> >> * There will be discussion about Zuul at the OpenStack board of >> directors meeting on April 24. That will be a phone call to which >> anyone is welcome to listen. >> >> * The Foundation will promote Zuul at the upcoming OpenStack Summit in >> Vancouver. It will be officially announced as a Foundation project on >> the keynote stage and via press release. >> >> * We are working with the Foundation's design team on a logo. We want >> to have something that can be adapted to work as an avatar (i.e., on >> GitHub), on the website, as a badge, and on slide decks. >> >> * The Foundation marketing team has begun a review of the website >> content and will be assisting with changes. Included in this is an >> effort to collect user stories for Zuul. >> >> * We're looking into implementing a system to manage the website content >> which is both Gerrit-friendly and has a workflow suitable for those >> unaccustomed to using Git. Monty has volunteered to research and >> report back with some options to explore. >> >> -Jim >> >> _______________________________________________ >> Zuul-discuss mailing list >> Zuul-discuss(a)lists.zuul-ci.org <mailto:Zuul-discuss@lists.zuul-ci.org> >> http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-discuss >

1 0

Simplified DSL
by Tristan Cacqueray 22 Apr '18

22 Apr '18

Hello folks, Some users may find job written in Ansible hard to reproduce, and I was wondering if we could add a new "exec" job attribute that would simply be shell task to be executed on a single node. This would let us write zuul.yaml file more similar to other in-repo ci system. For example a zuul.yaml could be: - job: name: my-job exec: - mvn build - mvn test - project: check: jobs: - my-job gate: jobs: - my-job Moreover we could go a step further and streamline that full description in a single stanza: - ci-job: - mvn build - mvn test Which could be translated into a unique job name attached to the check and gate pipeline by default. What do you think? -Tristan

8 9

Question regarding using etcd instead of Zookeeper
by Tristan Cacqueray 21 Apr '18

21 Apr '18

Hello folks, From a Kubernetes operator point of view, the Zookeeper component sounds redundant with the already running etcd service. Moreover etcd seems more active and it shows better overall performance. So my question is why Zookeeper was picked in the first place, and would it be possible to support/migrate to etcd in the future? Thanks in advance, -Tristan

9 11