- Zuul-discuss - lists.opendev.org

Elevator Pitch for Zuul
by Greg Dekoenigsberg 27 Sep '19

27 Sep '19

"Speculative gating saved our project." --Johannes Foufas, Volvo The most important question around Zuul is: why? Why, among all of the other CI/CD projects of various descriptions, should anyone choose Zuul? You generally get a very short time to explain to users why they should bother with a new technology, and the answer "Project X Uses It," while important, is seldom sufficient. I'd like to propose a 50-word elevator pitch for Zuul that focuses on its key differentiator: multi-project gating. It goes like this: "Complexity doesn't go away; it just moves. In the age of monolithic applications, complexity lived in the single application. Now, in the age of microservices, complexity lives in the relationships between applications. Zuul is the essential CI/CD project for the age of microservices, because it provides predictive gating between multiple projects." I believe that good marketing consists of two pieces: simplifying a story to the resonant essence, and then amplifying that story as much as possible. If we can agree that this simple story resonates, maybe we can figure out how to get this story in front of as many people as possible. --g

8 8

Gerrit support in master unstable
by corvus＠inaugust.com 19 Sep '19

19 Sep '19

Hi, We recently merged a bunch of changes which altered the Gerrit driver in master, and unfortunately real-world testing has exposed some errors which were not caught in either manual or automated testing. If you use Gerrit, particularly if you have it configured with an HTTP password, you may wish to avoid running master and stick with 3.10.2 for the moment until we fix those problems. I do not think they are very complicated and believe that we can merge fixes to them in not much more time than it would take to revert the changes, but if I'm wrong about that, we'll revert the stack and make another attempt. -Jim

1 0

Proposed new project: zuul-registry
by corvus＠gnu.org 17 Sep '19

17 Sep '19

Hi, We broke new ground with speculative container images, but in using this feature over the past few months, we have found some rough edges that I think can be smoothed by implementing our own container image registry server. Hear me out. :) There are two container image registries involved in the system: the intermediate registry and the buildset registry. The intermediate registry is a long-lived service which holds images after builds complete in order to transfer them between buildsets. In other words, when we build an image, we put it in the intermediate registry, and if, later, a change arrives which depends on that image, we pull it from that registry, and place it in the buildset registry. The buildset registry is short-lived, and is meant to supply images that we have local builds of if they appear in the dependency chain, or supply them from upstream dockerhub otherwise. Ideally the requirements for the intermediate registry are: * Authenticated write access * Anonymous read access * Support for more than one registry namespace (e.g., docker hub, quay, etc) Only "authenticated write access" is met by the current docker registry server -- and that's by putting the entire thing behind authentication -- read and write. In addition, we have found that the docker registry service seems to have problems working with a swift backend -- occasionally it writes zero-byte image objects, and the existing registry server logging isn't giving us any clues why. Ideally the requirements for the buildset registry are: * Authenticated access * Pass-through support for images not present * Support for more than one registry namespace Only "authenticated access" is met by the current docker registry; we accomplish the second by running a second instance and configuring them both as mirrors. I believe that if we make our own registry software, we can satisfy all of the requirements. The two registry servers have differing needs, but a single piece of software should be able to satisfy both by enabling or disabling certain options. It will be easy for us to configure whether read access requires authentication. We should be able to emulate different registry servers by observing the "Host" header (or, failing that, run on different ports). We can implement our own logic to perform pass-through requests automatically if an image is not present locally. And finally, we should be able to make uploading to swift more robust (though, obviously that alone is not a compelling reason to do this). I have made an initial implementation in Python which performs the basic functions of a registry (push and pull work). At this point, I have enough of an understanding of the protocol to feel that the remaining goals are achievable with a reasonable amount of effort. If folks don't object, I'd like to create a new repository ("zuul-registry" perhaps?) to hold this and continue further development. I would consider this an optional part of Zuul which operators could choose to deploy if they have a speculative image pipeline. -Jim

4 4

Open Infrastructure Summit Shanghai: Forum Submissions Open
by Jimmy McArthur 16 Sep '19

16 Sep '19

Hello Everyone! We are now accepting Forum [1] submissions for the 2019 Open Infrastructure Summit in Shanghai [2]. Please submit your ideas through the Summit CFP tool [3] through September20th. Don't forget to put your brainstorming etherpad up on the Shanghai Forum page [4]. This is not a classic conference track with speakers and presentations. OSF community members (participants in development teams, operators, working groups, SIGs, and other interested individuals) discuss the topics they want to cover and get alignment on and we welcome your participation. The Forum is your opportunity to help shape the development of future project releases. More information about the Forum [1]. Keep in mind, Forum submissions are for discussions, not presentations. The timeline for submissions is as follows: Sep 4th | Formal topic submission tool opens: https://cfp.openstack.org. Sep 20th | Deadline for proposing Forum topics. Scheduling committee meeting to make draft agenda. Sep 30th | Draft Forum schedule published. Crowd sourced session conflict detection. Forum promotion begins. Oct 7th | Scheduling committee final meeting Oct 14th | Forum schedule final Nov 4-6| Forum Time! If you have questions or concerns, please reach out to speakersupport(a)openstack.org <mailto:speakersupport@openstack.org>. Cheers, Jimmy [1] https://wiki.openstack.org/wiki/Forum [2] https://www.openstack.org/summit/shanghai-2019/ [3] https://cfp.openstack.org [4] https://wiki.openstack.org/wiki/Forum/Shanghai2019

1 1

Can we start adding ansible-molecule files for testing zuul roles?
by Sorin Sbarnea 06 Sep '19

06 Sep '19

As this question raised on irc I think is time to get some feedback on it. Molecule is the official Ansible role testing framework which allows testing of Ansible roles (or playbooks) in isolation by taking care care of provisioning and cleanup of the required resources. Most common use-case is to use docker containers to test, where you can start running your role code in a couple of seconds, being able to run in parallel lots of platforms. Mainly as a role developer (or even occasional contributor), it makes extremely easy to run tests locally*, before even creating a change-request. So this opens few sub-questions: a) can we allow people to add molecule files, even if we do not run molecule in check/gates? -- there is value in this, even without jobs. b) can/should we add molecule jobs for these so we can assure they keep running? c) would this create duplication between what is tested as standalone zuul jobs and what what is tested via molecule ones? Is this a reason to worry or not? These roles are used on multiple zuul server so is in our interest to make them as reusable as possible. Maybe on opendev.org instance we do not have the resources to test them on all possible platforms, but this does not mean that other zuul instances cannot run these tests. In addition to making a role developer life easier by allowing it to run ansible code in a say similar to running unittests, we can even see other benefits: uploaded CRs would be of better quality and we will also lower zuul load, improving the build-response-time even for users that do not test their changes locally before uploading them. * Developer can still define DOCKER_HOST=ssh://host and fully avoid the need to install docker locally. So the term "locally" is a bit relaxed, as you can offload the work to external systems. Is more of running tests on my own hardware, but isolated, so I would not mess my localhost. Thanks Sorin Sbarnea

1 0

Root Ara report removed from Zuul Jobs
by Clark Boylan 06 Sep '19

06 Sep '19

Hello Everyone, We have removed the top level Zuul job Ara reports from our Zuul jobs. This was done to reduce the total number of objects we are uploading to our swift/ceph object stores as some clouds have indicated the total object volume is a bit high. Analysis showed that Ara represented a significant chunk of that data. We did not remove that information though. Zuul's build dashboard is able to render a similar report for builds. You can find that by clicking on the "Console" tab of a build. For exampe, here is one for a nova tox job: http://zuul.openstack.org/build/8e581b24d38b4e5c8ff046be081c4525/console We hope this makes our log storage easier to support while still providing the information you need to debug your jobs. Note jobs that run a nested ara-report are not affected by this. I think TripleO, OSA, and others do this. Thank you, Clark

1 0

Report from Gerrit User Summit
by corvus＠inaugust.com 04 Sep '19

04 Sep '19

Hi, Monty and I attended the Gerrit User Summit and hackathon last week. Zuul came up quite a lot, so I wanted to highlight some things here. For a fuller report of the event, see my mail to the OpenStack-Infra mailing list here: http://lists.openstack.org/pipermail/openstack-infra/2019-September/006475.… The following is an extract of the Zuul-relevant parts from that: Zuul ---- Zuul is happily used at Volvo by the propulsion team at Volvo (currently v2, working on moving to v3) [2]. Other teams are looking into it. The Gerrit maintainers are interested in using Zuul to run Gerrit's upstream CI system. Monty and I plan on helping to implement that. We spoke at length with Edwin and Alice who are largely driving the development of the new "checks" API in Gerrit. It is partially implemented now and operational in upstream Gerrit. As written, we would have some difficulty using it effectively with Zuul. However, with Zuul as a use case, some further changes can be made so that I think it will integrate quite well, and with more work could be a quite nice integration. At a very high level, a "checker" in Gerrit represents a single pass/fail result from a CI system or code analyzer, and must be configured on the project in advance by an administrator. Since we want Zuul to report the result of each job it runs on a change, and we don't know that set of jobs until we start, the current implementation doesn't fit the model very well. For the moment, we can use the checks API to report the overall buildset result, but not the jobs. We can, of course, still use Gerrit change messages to report the results of individual jobs just as we do now. But ideally, we'd like to take full advantage of the structured data and reporting the checks API provides. To that end, I've offered to write a design document[1] describing an implementation of support for "sub-checks" -- an idea which appeared in the original checks API design as a potential follow-up. Sub-checks would simply be structured data about individual jobs which are reported along with the overall check result. With this in place, Zuul could get out of the business of leaving comments with links to logs, as each sub-check would support its own pass/fail, duration, and log url. Later, we could extend this to support reporting artifact locations as well, so that within Gerrit, we would see links to the log URL and docs preview sites, etc. There is an opportunity to do some cross-repo testing between Zuul and Gerrit as we work on this. Upstream Gerrit's Gerrit does not have the SSH event stream available, so before we can do any work against it, we need an alternative. I think the best way forward is to implement partial (experimental) support for the checks API, so that we can at least use it to trigger and report on changes, get OpenDev's Zuul added as a checker, and then work on implementing sub-checks in upstream Gerrit and then Zuul. Conclusion ---------- I'm sure I'm leaving stuff out, so feel free to prompt me with questions. In general we got a lot of work done and I think we're set up very well for future collaboration. -Jim [1] https://gerrit-review.googlesource.com/Documentation/dev-contributing.html#… [2] https://model-engineers.com/en/company/references/success-stories/volvo-car…

1 0

Zuul posts non hiddable Gerrit comments
by Vitaliy Lotorev 04 Sep '19

04 Sep '19

Hi, Starting with at least v2.15 Gerrit is able to differentiate bot comments and hide them using 'Show comments' trigger on web UI. This is achived either using tagged comments or robot comments. A while ago I filed an issue about Zuul not using these technics [1] (ticket has links for Gerrit docs with tagged and robot comments). This results that Zuul comments are not hiddable in Gerrit. AFAIK, sending tagged comments via SSH to Gerrit is easy (one-line of code at [2]). I could try providing a patch for tagged comments. What Zuul maintainers think about adding support for bot comments? Should not be sending tagged comments done only if Gerrit version >= 2.15? [1] https://storyboard.openstack.org/#!/story/2005661 [2] https://opendev.org/zuul/zuul/src/branch/master/zuul/driver/gerrit/gerritco…

4 4

review.opendev.org Outage September 16 to Perform Project Renames
by Clark Boylan 04 Sep '19

04 Sep '19

Hello, We will be taking a Gerrit outage of about an hour on September 16 at 14:00 UTC to perform project renames. Please let us know if this scheduling does not work for some reason. We have tried to schedule this for a quiet time at the end of OpenStack's Train release cycle. Also, if you'd like to rename a project, now is the time to start prepping for that. Feel free to ask us any questions you have or bring up your concerns with us. Thank you for your patience, Clark

1 0

Zuul at the Open Source Summit Europe; Zuul swag
by Matthieu Huin 30 Aug '19

30 Aug '19

Hello list, I am pleased to announce that I will be giving a presentation on Zuul with Fabien Boucher at the upcoming Open Source Summit in Lyon, France! The description of the session can be found here: https://osseu19.sched.com/event/TLMB/stop-merging-broken-code-an-introducti… This summit is right before the Shanghai Summit (October 28-30). Feel free to advertise our session! It would be nice to have some Zuul swag before the OSS to hand out and leave around; especially stickers as they were a huge hit at FOSDEM this year. Who should I contact about this? -- Matthieu Huin Senior Software Developper Red Hat <https://www.redhat.com> <https://red.ht/sig>

3 3