September 2019 - Zuul-discuss - lists.opendev.org

Increasing project visibility
by Jeremy Stanley 05 Feb '20

05 Feb '20

With AnsibleFest just around the corner, I'm looking forward to seeing lots of you again. As many on this list are probably already aware, there will be a Zuul booth and I expect to spend a fair amount of time there talking to new folks about what Zuul is and why I think it's so amazing. I hear there will be some talks there which mention Zuul as well, so hopefully that helps spread the word to even more attendees. This got me thinking about how growing Zuul depends on growing our community around it, but to do that people need to know it exists in the first place. Speaking engagements and talking to folks at conferences are great ways, we have a slick looking Web site, some members of the community write articles or get interviewed about cool things they've done with Zuul, we're featured prominently in OSF marketing materials and newsletters... but what else can we do to get out the message, and how can we tell what's working to reach our future community and what isn't? Let's see what more we can come up with! Who's got suggestions? -- Jeremy Stanley

9 23

Fetch roles improvement to enable running zuul-jobs on kubectl connection
by Tristan Cacqueray 24 Jan '20

24 Jan '20

Hello folks, I have been working on supporting kubectl connection for zuul-jobs. The issue is that Ansible synchronize module doesn't work[0]. One solution would be to include the fix in Zuul library[1], but that's not ideal because zuul's synchronize would be slightly different. Alternatively we can synchronize files using oc rsync[2] command but that can't be done from untrusted context. Thus we need to adapt every fetch roles to not perform the synchronize task, and instead rely on the generic fetch-output role. As a first step I added a zuul_use_fetch_output toggle to enable the new behavior, but we should consider enforcing the fetch-output job and make it the default behavior. Indeed It is much easier to copy or link file to ~/zuul-output and let the base job retrieve and publish instead of having each fetch roles bother with running synchronize to zuul.executor.work_root or log_root. A zuul_use_fetch_output toggle implementation is proposed[3] along with a third-party-ci build that demonstrates a full tox-py36[4] job running in a container. Regards, -Tristan [0]: https://github.com/ansible/ansible/pull/62107 [1]: https://review.opendev.org/681553 [2]: https://review.opendev.org/#/c/631402/6/roles/prepare-workspace-openshift/t… [3]: https://review.opendev.org/#/q/topic:zuul-jobs-with-kubectl [4]: https://review.opendev.org/682049

3 2

Developing a Gitlab driver
by Paul Albertella 30 Sep '19

30 Sep '19

Hi, I saw the @ZuulCI tweet about developing a Gitlab driver for Zuul (and responded as @reiterative) and would love to know more, as this is something that I had already been contemplating. I'm not a Zuul user yet, but it has been recommended by some colleagues and I'm keen to explore its possibilities on the projects that I'm involved in, which use Gitlab. Has any work already been done on a Gitlab driver, which could provide a starting point? Or is the existing GitHub driver the best starting point? Cheers, Paul

4 3

Elevator Pitch for Zuul
by Greg Dekoenigsberg 27 Sep '19

27 Sep '19

"Speculative gating saved our project." --Johannes Foufas, Volvo The most important question around Zuul is: why? Why, among all of the other CI/CD projects of various descriptions, should anyone choose Zuul? You generally get a very short time to explain to users why they should bother with a new technology, and the answer "Project X Uses It," while important, is seldom sufficient. I'd like to propose a 50-word elevator pitch for Zuul that focuses on its key differentiator: multi-project gating. It goes like this: "Complexity doesn't go away; it just moves. In the age of monolithic applications, complexity lived in the single application. Now, in the age of microservices, complexity lives in the relationships between applications. Zuul is the essential CI/CD project for the age of microservices, because it provides predictive gating between multiple projects." I believe that good marketing consists of two pieces: simplifying a story to the resonant essence, and then amplifying that story as much as possible. If we can agree that this simple story resonates, maybe we can figure out how to get this story in front of as many people as possible. --g

8 8

Gerrit support in master unstable
by corvus＠inaugust.com 19 Sep '19

19 Sep '19

Hi, We recently merged a bunch of changes which altered the Gerrit driver in master, and unfortunately real-world testing has exposed some errors which were not caught in either manual or automated testing. If you use Gerrit, particularly if you have it configured with an HTTP password, you may wish to avoid running master and stick with 3.10.2 for the moment until we fix those problems. I do not think they are very complicated and believe that we can merge fixes to them in not much more time than it would take to revert the changes, but if I'm wrong about that, we'll revert the stack and make another attempt. -Jim

1 0

Proposed new project: zuul-registry
by corvus＠gnu.org 18 Sep '19

18 Sep '19

Hi, We broke new ground with speculative container images, but in using this feature over the past few months, we have found some rough edges that I think can be smoothed by implementing our own container image registry server. Hear me out. :) There are two container image registries involved in the system: the intermediate registry and the buildset registry. The intermediate registry is a long-lived service which holds images after builds complete in order to transfer them between buildsets. In other words, when we build an image, we put it in the intermediate registry, and if, later, a change arrives which depends on that image, we pull it from that registry, and place it in the buildset registry. The buildset registry is short-lived, and is meant to supply images that we have local builds of if they appear in the dependency chain, or supply them from upstream dockerhub otherwise. Ideally the requirements for the intermediate registry are: * Authenticated write access * Anonymous read access * Support for more than one registry namespace (e.g., docker hub, quay, etc) Only "authenticated write access" is met by the current docker registry server -- and that's by putting the entire thing behind authentication -- read and write. In addition, we have found that the docker registry service seems to have problems working with a swift backend -- occasionally it writes zero-byte image objects, and the existing registry server logging isn't giving us any clues why. Ideally the requirements for the buildset registry are: * Authenticated access * Pass-through support for images not present * Support for more than one registry namespace Only "authenticated access" is met by the current docker registry; we accomplish the second by running a second instance and configuring them both as mirrors. I believe that if we make our own registry software, we can satisfy all of the requirements. The two registry servers have differing needs, but a single piece of software should be able to satisfy both by enabling or disabling certain options. It will be easy for us to configure whether read access requires authentication. We should be able to emulate different registry servers by observing the "Host" header (or, failing that, run on different ports). We can implement our own logic to perform pass-through requests automatically if an image is not present locally. And finally, we should be able to make uploading to swift more robust (though, obviously that alone is not a compelling reason to do this). I have made an initial implementation in Python which performs the basic functions of a registry (push and pull work). At this point, I have enough of an understanding of the protocol to feel that the remaining goals are achievable with a reasonable amount of effort. If folks don't object, I'd like to create a new repository ("zuul-registry" perhaps?) to hold this and continue further development. I would consider this an optional part of Zuul which operators could choose to deploy if they have a speculative image pipeline. -Jim

4 4

Open Infrastructure Summit Shanghai: Forum Submissions Open
by Jimmy McArthur 17 Sep '19

17 Sep '19

Hello Everyone! We are now accepting Forum [1] submissions for the 2019 Open Infrastructure Summit in Shanghai [2]. Please submit your ideas through the Summit CFP tool [3] through September20th. Don't forget to put your brainstorming etherpad up on the Shanghai Forum page [4]. This is not a classic conference track with speakers and presentations. OSF community members (participants in development teams, operators, working groups, SIGs, and other interested individuals) discuss the topics they want to cover and get alignment on and we welcome your participation. The Forum is your opportunity to help shape the development of future project releases. More information about the Forum [1]. Keep in mind, Forum submissions are for discussions, not presentations. The timeline for submissions is as follows: Sep 4th | Formal topic submission tool opens: https://cfp.openstack.org. Sep 20th | Deadline for proposing Forum topics. Scheduling committee meeting to make draft agenda. Sep 30th | Draft Forum schedule published. Crowd sourced session conflict detection. Forum promotion begins. Oct 7th | Scheduling committee final meeting Oct 14th | Forum schedule final Nov 4-6| Forum Time! If you have questions or concerns, please reach out to speakersupport(a)openstack.org <mailto:speakersupport@openstack.org>. Cheers, Jimmy [1] https://wiki.openstack.org/wiki/Forum [2] https://www.openstack.org/summit/shanghai-2019/ [3] https://cfp.openstack.org [4] https://wiki.openstack.org/wiki/Forum/Shanghai2019

1 1

Can we start adding ansible-molecule files for testing zuul roles?
by Sorin Sbarnea 06 Sep '19

06 Sep '19

As this question raised on irc I think is time to get some feedback on it. Molecule is the official Ansible role testing framework which allows testing of Ansible roles (or playbooks) in isolation by taking care care of provisioning and cleanup of the required resources. Most common use-case is to use docker containers to test, where you can start running your role code in a couple of seconds, being able to run in parallel lots of platforms. Mainly as a role developer (or even occasional contributor), it makes extremely easy to run tests locally*, before even creating a change-request. So this opens few sub-questions: a) can we allow people to add molecule files, even if we do not run molecule in check/gates? -- there is value in this, even without jobs. b) can/should we add molecule jobs for these so we can assure they keep running? c) would this create duplication between what is tested as standalone zuul jobs and what what is tested via molecule ones? Is this a reason to worry or not? These roles are used on multiple zuul server so is in our interest to make them as reusable as possible. Maybe on opendev.org instance we do not have the resources to test them on all possible platforms, but this does not mean that other zuul instances cannot run these tests. In addition to making a role developer life easier by allowing it to run ansible code in a say similar to running unittests, we can even see other benefits: uploaded CRs would be of better quality and we will also lower zuul load, improving the build-response-time even for users that do not test their changes locally before uploading them. * Developer can still define DOCKER_HOST=ssh://host and fully avoid the need to install docker locally. So the term "locally" is a bit relaxed, as you can offload the work to external systems. Is more of running tests on my own hardware, but isolated, so I would not mess my localhost. Thanks Sorin Sbarnea

1 0

Root Ara report removed from Zuul Jobs
by Clark Boylan 06 Sep '19

06 Sep '19

Hello Everyone, We have removed the top level Zuul job Ara reports from our Zuul jobs. This was done to reduce the total number of objects we are uploading to our swift/ceph object stores as some clouds have indicated the total object volume is a bit high. Analysis showed that Ara represented a significant chunk of that data. We did not remove that information though. Zuul's build dashboard is able to render a similar report for builds. You can find that by clicking on the "Console" tab of a build. For exampe, here is one for a nova tox job: http://zuul.openstack.org/build/8e581b24d38b4e5c8ff046be081c4525/console We hope this makes our log storage easier to support while still providing the information you need to debug your jobs. Note jobs that run a nested ara-report are not affected by this. I think TripleO, OSA, and others do this. Thank you, Clark

1 0

Report from Gerrit User Summit
by corvus＠inaugust.com 05 Sep '19

05 Sep '19

Hi, Monty and I attended the Gerrit User Summit and hackathon last week. Zuul came up quite a lot, so I wanted to highlight some things here. For a fuller report of the event, see my mail to the OpenStack-Infra mailing list here: http://lists.openstack.org/pipermail/openstack-infra/2019-September/006475.… The following is an extract of the Zuul-relevant parts from that: Zuul ---- Zuul is happily used at Volvo by the propulsion team at Volvo (currently v2, working on moving to v3) [2]. Other teams are looking into it. The Gerrit maintainers are interested in using Zuul to run Gerrit's upstream CI system. Monty and I plan on helping to implement that. We spoke at length with Edwin and Alice who are largely driving the development of the new "checks" API in Gerrit. It is partially implemented now and operational in upstream Gerrit. As written, we would have some difficulty using it effectively with Zuul. However, with Zuul as a use case, some further changes can be made so that I think it will integrate quite well, and with more work could be a quite nice integration. At a very high level, a "checker" in Gerrit represents a single pass/fail result from a CI system or code analyzer, and must be configured on the project in advance by an administrator. Since we want Zuul to report the result of each job it runs on a change, and we don't know that set of jobs until we start, the current implementation doesn't fit the model very well. For the moment, we can use the checks API to report the overall buildset result, but not the jobs. We can, of course, still use Gerrit change messages to report the results of individual jobs just as we do now. But ideally, we'd like to take full advantage of the structured data and reporting the checks API provides. To that end, I've offered to write a design document[1] describing an implementation of support for "sub-checks" -- an idea which appeared in the original checks API design as a potential follow-up. Sub-checks would simply be structured data about individual jobs which are reported along with the overall check result. With this in place, Zuul could get out of the business of leaving comments with links to logs, as each sub-check would support its own pass/fail, duration, and log url. Later, we could extend this to support reporting artifact locations as well, so that within Gerrit, we would see links to the log URL and docs preview sites, etc. There is an opportunity to do some cross-repo testing between Zuul and Gerrit as we work on this. Upstream Gerrit's Gerrit does not have the SSH event stream available, so before we can do any work against it, we need an alternative. I think the best way forward is to implement partial (experimental) support for the checks API, so that we can at least use it to trigger and report on changes, get OpenDev's Zuul added as a checker, and then work on implementing sub-checks in upstream Gerrit and then Zuul. Conclusion ---------- I'm sure I'm leaving stuff out, so feel free to prompt me with questions. In general we got a lot of work done and I think we're set up very well for future collaboration. -Jim [1] https://gerrit-review.googlesource.com/Documentation/dev-contributing.html#… [2] https://model-engineers.com/en/company/references/success-stories/volvo-car…

1 0