January 2019 - Zuul-discuss - lists.opendev.org

Speculative docker images
by corvus＠inaugust.com 22 Mar '19

22 Mar '19

Hi, As you know, with respect to git repositories, Zuul frequently operates well into the future, allowing jobs to build software with changes that haven't landed yet under the assurance that they will. As more of us have incorporated container image building into our processes, we find that we want the same thing there -- that not only would our git repositories represent the future that Zuul sees, but also our container image registries. As a concrete and simple example, we have an integration job called 'zuul-quick-start' which uses both Nodepool and Zuul container images. Those images themselves depend on build images created from the openstack-infra/system-config repository, which they use via a "FROM" line in Dockerfiles, and extending the zuul-quick-start job to be able to rebuild and use those builder images in the case of a Depends-On change is too complex to consider. In particular, the "FROM" line is problematic, as the only way to realistically use a speculative image as a base image is to alter the Dockerfile or make the image available in a registry. Therefore, this is a proposal for a system for building speculative images which can then be used by further speculative images. Every item in a Zuul pipeline has its own future state which is related to, but separate from all the other items. It needs to be able to build an image for a project and store it in a registry under the final published name and tag, without affecting any of the changes ahead of it. Therefore, each item needs its own image registry. In the case of a single, simple job such as zuul-quick-start, we can just run a docker registry on the node on which the job is running. In the case of a more complicated system where we want to build an image once and use it in multiple dependent jobs in a job graph, we will start an image registry at the root of the job graph, pause the job, and the dependent jobs will use it. Now that we have the registry for the change, we need to have that registry reflect the state of any changes ahead. We need to know when the change ahead has finished building its images so that we can then add those images into our change's registry. To accomplish this, we can extend Zuul so that jobs have a provides/requires interface. If our job requires docker images produced by a job in a dependent change, we can say: - job: name: zuul-quick-start requires: docker-images - job: name: system-config-build-image-python-builder provides: docker-images Zuul will see that zuul-quick-start should not start until all of the jobs for all of the changes ahead which provide "docker-images" have completed. Then zuul-quick-start will be able to retrieve images from the preceding changes' jobs and add them to its own registry. In the gate, the behavior of this is obvious: if a change has a job with a requirement, iterate over all changes ahead of it in the shared queue and search for jobs on those changes which provide that requirement. If there are none, or all of them have completed or paused, then that job is ready to start. Otherwise, wait. In an independent pipeline, we do include dependent changes ahead in the shared queue, but they are not live -- they don't run any actual jobs. So in that case, we're going to have to peek outside our shared queue and see if our dependent changes show up anywhere else in the same pipeline, and if they do, wait for their jobs. If they don't show up, then we know they completed earlier, and it's safe to start. There's one more thing about this change that will help us later: a job that "provides" something should be able to pass data to a job which "requires" it. So our image build jobs can use zuul_return to provide information about which images they built so that it's easier for later jobs to import those images into their registries. Once again, the independent pipelines make this tricky, as since those jobs may have run some time in the past, the scheduler may have forgotten the data they returned. However, once the SQL database is required, we can rely on that to fill in any missing data. That's the only change we need to Zuul itself to support this system. The rest is all playbooks and roles. That brings us to the challenge of how to actually transfer these artifacts between items. I can think of two approaches. They are not mutually exclusive -- we can implement both if we want. They both have pros and cons. But I would like input from Zuul operators (including openstack-infra/opendev) as to which they would prefer, and I'll implement that approach in OpenDev for Zuul's own upstream use. If we have a split decision, someone else can implement the second approach. I think we can support both in zuul-jobs, but I'd like us to run the most widely applicable approach upstream. Alternative 1: Use the Logserver ================================ In the case of a simple job, at the end of the job, or in the case of a complex job graph, a child job dependent on the completion of the image build process (so we don't have to wait for all the jobs to complete), we will export the images to the filesystem and upload their layers (minus any layers which are already on Docker Hub, in order to save space) to the logserver. This next bit is a little complicated, but remember that this will all be encapsulated in roles so we only have to implement it once; the user interface will resemble "roles: [fetch-dependent-images]". At the end of the image build job, for every image that we built: Export it to a directory on the filesystem. For each layer in the image: Perform a HEAD request against the Docker Hub registry to see if the layer exists in the target repository. If that exists, layer exists in docker, remove from filesystem. rsync image directories to logserver. zuul_return image location on logserver. At the start of a job which requires images: For each previous image (as provided by zuul_return): Rsync image directory from logserver to filesystem. For each layer in image manifest: If layer is missing from filesystem: Get the layer from Docker Hub. Import image into local registry. The advantages of this approach: * No external operational dependencies (i.e., you don't have to run a docker registry). The disadvantages of this approach: * Images can't easily be fetched and used for debugging. * Implementation is slightly more complex. Alternative 2: Use a Central Registry ===================================== In this approach, we would run a central registry to store speculative images. It would not store final images at all, and every image uploaded to it would be tagged only with a tag like "pipeline_check_change_1234_original_tag" to uniquely identify its build context. At the end of the image build job, for every image that we built: Copy it from the per-change registry to the central registry, and tag it with pipeline and change id. zuul_return image location and tag. At the start of a job which requires images: For each previous image (as provided by zuul_return): Copy it from the central registry to the local registry, and tag it with the original tag. The advantages of this approach: * Developers can fetch images from the central registry by pulling them with the crazy tag. * Implementation is a little easier, and uses more standardized tools. The disadvantages of this approach: * An external service is required. * We will need to prune old images from the registry. Conclusion ========== I don't think this system will take long to implement with either of the alternatives. Please let me know what you think of the system, and of the storage/transfer alternatives. -Jim

4 4

Parameterized Build
by Tristan Cacqueray 21 Mar '19

21 Mar '19

Hello, It seems like it would help the migration of Jenkins users if they could have a similar feature[0] in Zuul to manually trigger a job with custom parameters. One solution is to create a fake change or branch to trigger a job, but perhaps Zuul can offer a better user experience. In particular, the Jenkins implementation offers a nice form to set the parameters. I started to implement this feature with a jobtrigger driver[1] along with a web page[2] to expose the job's variables but James asked to describe the use case first. Our current use case is for QA/QE engineers who would like to run a single (usualy long running) job with custom parameters such as the product version, the system version, ... With Jenkins, we can have lots of jobs grouped as multijob running in different phases, and the parameterized build is a very convenient way to run a single job, sometime using a parameter to add a breakpoint and get access to the environment. Another solution would be to build on top of the zuul-runner to provide a similar workflow, but it seems like more efficient if we could re-use the zuul-scheduler and let it request/cleanup the nodes. In summary, could we adjust the architecture to support executing an individual job instead of an entire job graph and could we support a zuul-web trigger? Regards, -Tristan [0]: https://wiki.jenkins.io/display/JENKINS/Parameterized+Build [1]: https://review.openstack.org/555153 [2]: https://review.openstack.org/554839

3 9

Zuul 2019 planning
by Lauren Sell 05 Feb '19

05 Feb '19

Hi everyone, We’d like to work with the Zuul community on 2019 planning so we can best support and promote the project this year. If the mailing list is the best way to start the conversation, we can use the below etherpad, or have a discussion on this thread. We’re also happy to host a quick call if folks want to brainstorm in a high bandwidth setting. The main things we’re looking to document are the goals for the community this year (for example, attract more contributors or document more users in production or make Zuul easier to try out and use…) along with any key milestones, features, topics, etc. that we’ll want to communicate. Please take a look at the etherpad and add your thoughts over the next week: https://etherpad.openstack.org/p/zuul-2019-planning <https://etherpad.openstack.org/p/zuul-2019-planning> Thanks! Lauren

2 2

Open Infrastructure Summit Denver - Community Voting Open
by Claire Massey 31 Jan '19

31 Jan '19

Hi everyone, Community voting for the Open Infrastructure Summit Denver CFP sessions is open! You can VOTE HERE <https://www.openstack.org/summit/denver-2019/vote-for-speakers>, but what does that mean? Now that the Call for Presentations (CFP) has closed, all submissions are available for community vote and input. After community voting closes, the volunteer Programming Committee members will receive the presentations to review and determine the final selections for Summit schedule. While community votes are meant to help inform the decision, Programming Committee members are expected to exercise judgment in their area of expertise and help ensure diversity of sessions and speakers. View full details of the session selection process here <https://www.openstack.org/summit/denver-2019/call-for-presentations/selecti…>. In order to vote, you need an OSF community membership. If you do not have an account, please create one by going to openstack.org/join <http://openstack.org/join>. If you need to reset your password, you can do that here <https://www.openstack.org/Security/lostpassword>. Hurry, voting closes Monday, February 4 at 11:59pm Pacific Time (Tuesday, February 5 at 7:59 UTC). Continue to visit https://www.openstack.org/summit/denver-2019 <https://www.openstack.org/summit/denver-2019> for all Summit-related information. REGISTER Register for the Summit <https://openinfrasummitandptgdenver.eventbrite.com/> before prices increase in late February! VISA APPLICATION PROCESS Make sure to secure your Visa soon. More information <https://www.openstack.org/summit/denver-2019/travel/#travel-support> about the Visa application process. TRAVEL SUPPORT PROGRAM February 27 is the last day to submit applications. Please submit your applications <https://www.openstack.org/summit/denver-2019/travel/#travel-support> by 11:59pm Pacific Time (February 28 at 7:59am UTC). If you have any questions, please email summit(a)openstack.org <mailto:summit@openstack.org>. Cheers, Claire

1 0

Using nodepool v3 with Jenkins
by Darragh Bailey 31 Jan '19

31 Jan '19

Hi, Hoping that we can make use of nodepool v3 with Jenkins to solve some of our existing problems, and at the same time move a bit closer to upgrading from zuul v2 -> v3. Going to outline our current problem, my initial thinking that resulted in some of my questions last night in IRC and some further thinking on it that may provide a way forward for us. Hoping that someone can chime in and identify if this is realistic or if we're heading down the wrong path before getting to far. Keep in mind that I'm only starting with nodepool, so I'm unsure if I even understand some of the concepts. e.g. does nodepool get a request for a particular type of node or is it infered by there being none of that type available in the queues in zookeeper? Current problem: We have some resources that are limited and are not directly accessible as workers for Jenkins, and so need a way to manage the allocation of resources to certain jobs. Some of these resources are provided by teams and some are from a set of resources managed by the CI team. So we probably have ~4 sets of these types of resources. At the moment this is done using some (too much) hackier with groovy, the Global Event Listener Plugin (GELP), and Lockable Resources plugin. The GELP allows us to hook into various points in Jenkins, such as when the job is scheduled but not yet running Jenkins (QUEUED). Unfortunately there doesn't appear to be any way with the gearman protocol to inspect what job is going to be assigned before it is picked up, so we can only do manipulation after the job has been scheduled and before it starts running. The data required to provide the job with the details required to use the limited resource is retrieved from vault and injected right before the job gets fully launched on Jenkins. This is done by creating lockable resource entries dynamically based on the number of the resources available, and then catching the event on when Jenkins allows the job needing this to start and injecting the required data into the environment. Unfortunately this means the queuing for the jobs requiring these limited resources occurs within Jenkins instead of Zuul because the job has still be scheduled even if it is not yet running. We've run into a number of problems resulting in workers getting allocated for jobs that don't get to start, and taking them away from jobs that could be run, and effectively lightly DOSing the system. Initial thoughts: So enter nodepool and the hope that if we move the management of both these resources and the workers out of Jenkins and into something a little more flexible, it can prevent the job from getting scheduled in the first place if not enough of the limited resources are available, and allow jobs that do not need these to be scheduled because a worker isn't getting assigned to the job that is queued waiting. Initially I was thinking of having two resources, making the one representing the limited resource a fake resource of some kind that Jenkins doesn't connect to, but as it appears that it's a requirement of the Jenkins Nodepool agent that nodes returned must be accessible to Jenkins, that doesn't seem to fly. Next idea was whether it would be possible to have a static pool and a another driver that could grab a node from the static pool if a request for the limited resource node type came in, which prompted some of my questions in IRC last night. Further thinking done: After spending a bit more time with the static nodepool driver, and depending on how it works (there are lots of gaps in my understanding), I'm wondering if what we need is to create a custom driver and place all of our static nodes in a internal pool inside of this driver. Additionally query our vault instance for a list of these limited resource groups and build some pools of those internally as well. And assign labels as part of the requirements for the jobs using these limited resources. Then instead of immediate registering all of the nodes with zookeeper, instead depending on the node label request that comes in, populate zookeeper with an entry containing suitable labels. If it is one requiring a limited resource identified by a name such as teamA-resources, the driver code can test if there are any not in use and if so create a node entry in zookeeper that contains labels "teamA-resources" and "teamA-resource-5". The second label can be used in Jenkins by our GELP code to determine which set of data to inject into the environment of the job scheduled on that node. Obviously there is still some hackier in there, but does a custom driver building on the idea of combining some of the ideas of the static driver along with the dynamic drivers where instead of launching a new node, it simply creates a node entry on demand using both a static node and details on what resource is required? Possibly I'm misunderstanding how nodepool works in this case, and maybe it just populates a queue in zookeeper and tries to keep some in there, which may mean that what needs to happen is that one entry allocating a node to each of the sets of limited resources needs to be added as entries to zookeeper queues by the driver with further entries added if there are enough resources available as those queues are consumed? Which is a bit different, but seems like it would still work to limit the amount of damage where multiple jobs were launched and required more of the limited resource than was available. Currently can end up seeing 3 queue instances for a single limited resource queued up in Jenkins, so as we migrate another case were this will be needed we could end up seeing a queue of 9-10. Seems nodepool might limit it to only losing one per team, until we can migrate to zuulv3 and use some of the mutex capabilities to handle this a bit better. Thoughts? Is this the best way to dig up as not digging isn't an option here... -- Darragh Bailey "Nothing is foolproof to a sufficiently talented fool"

2 4

Zuul jobs for container publishing
by corvus＠inaugust.com 31 Jan '19

31 Jan '19

Hi, We recently added a new pipeline to OpenStack's Zuul, and three new jobs and roles to the zuul-jobs repo to support container publishing. Zuul users may be interested in adopting both the pipeline and the jobs into their own workflows. The new pipeline is the "promote" pipeline. It runs jobs on changes after they merge, but unlike the "post" pipeline, it does not run with the resulting branch state after the merge -- it runs in the context of the change. This means that it's not suitable for building release artifacts, but that's not its purpose. Its purpose is to promote artifacts previously built in the gate pipeline to production. That is safe to do because anything built in the gate pipeline is built with the full future state of in-flight changes to all related projects and branches (even if it's built before they land). We build a lot of artifacts in the gate pipeline now, and then throw them away only to build them again in post. The promote pipeline is an attempt to avoid that waste and instead preserve artifacts built in gate and directly publish them -- but only if and when the changes for which they were built are merged. The first functional use of this system is in building container images and publishing them on Docker Hub. We have begun using this in openstack-infra to build container images of third-party applications, but this will work just as well and as easily for building images of applications in our infrastructure (for example, we are also starting to use this to publish Zuul images). To implement this, we wrote three jobs which are available now from zuul-jobs (and therefore can be used in any project or job): * build-docker-image [1] * upload-docker-image [2] * promote-docker-image [3] The first job should run in the check pipeline and simply runs "docker build" on the arguments passed to it. It is used to verify that the build succeeds. The second job should run in the gate pipeline. It uploads the image to Docker Hub into the final repository location, but it only does so with a single tag with the form "change_123456" where '123456' is the Gerrit change number for the change under consideration. This will mean that docker pulls won't accidentally fetch the image yet. But the image is staged in Docker Hub and is ready to be promoted if the change passes all its tests and merges. The third job runs in the promote pipeline. This job, unlike the others, only performs Docker Hub API calls and requires no build resources. Therefore it is a zero-node job in Zuul -- a job which only runs on the Zuul executor. This job finds a previously uploaded image with a "change_" tag, and re-tags it with any specified tags ("latest" by default, but you can also tag version numbers or anything else). It also cleans up unused "change_" tags so they don't clutter up the repository. The nice thing about zero-node jobs is that they start and finish very quickly (since they don't have to deal with resource allocation and contention). With this system, an image will be fully published within about a minute after the change merges. A complete example .zuul.yaml might look like this: - secret: name: myproject-dockerhub data: username: dockerhubuser password: !encrypted/pkcs1-oaep - DFlbrDM5eUMptMGIVMXV1g455xOJLi92UYF08Z2/JlIGu3t6v052o9FKlVyj1ZmpXs5+2 JTa5jHkLTvTsYs9fCaNcQc2nmViCyWNlbOMzjB17uiZOaYFNs1sMqZcUZbGEz7Y8ds6Qq NBXI10jWFPTah4QxUuBvUbT3vmjnUToCzexl5ZGhKgijcnROWfUsnlCdugpgoNIcPsUki zty5FotDihnrC8n8vIomVK6EClY38ty97pLrADzFDd+Cos/OUlvi2xooUhzx8Bn020rJA lqEU5v8LGXp5QkHx0MSDx6JY6KppJ/4p/yM+4By6l+A20zdcimxmgiNc9rMWPwDj7xsao m7NAZWmWqOO0Xkhgt6WOfugwgt9X46sgs2+yDEfbnI5ok8uRbAB/4FWj/KdpyXwhcf+O2 wEfhxLwDbAoGONQPjb4YcZmCXtmR7Qe5t+n2jyczWXvrbaBDUQP5a+YtVNN/xhmQ7D740 POlxv7bLxJAixzqaQ3d8Rz9ZEv6zzRuhWph32UQtZ1JxSNww+EvmXm2eEi2Q2z6pT1Cx/ j2OrFyA2GL/UJOVb15VHKF6bgHPHWJtpjPFhqdcvBhVute4BWB+KPcWH+y+apHN1enK3H tNJO9iqm34nKwSuj5ExmFw50LtwR5/9FyRuRPq/vBL+8y82v8FDmeYsBeobn5M= - job: name: myproject-build-image parent: build-docker-image description: Build Docker images. vars: &image_vars docker_images: - context: . repository: myproject/imagename - job: name: myproject-upload-image parent: upload-docker-image description: Build Docker images and upload to Docker Hub. secrets: - name: docker_credentials secret: myproject-dockerhub pass-to-parent: true vars: *image_vars - job: name: myproject-promote-image parent: promote-docker-image description: Promote previously uploaded Docker images. secrets: - name: docker_credentials secret: myproject-dockerhub pass-to-parent: true vars: *image_vars - project: check: jobs: - myproject-build-image gate: jobs: - myproject-upload-image promote: jobs: - myproject-promote-image And the pipeline definition looks like: - pipeline: name: promote description: | This pipeline runs jobs that operate after each change is merged in order to promote artifacts generated in the gate pipeline. manager: supercedent precedence: low post-review: True trigger: gerrit: - event: change-merged success: gerrit: {} mysql: failure: gerrit: {} mysql: If you find that you need jobs which behave slightly differently, you may be able to inherit from these jobs and add pre or post playbooks. Or if you need something particularly complicated, you can re-implement the jobs using the underlying roles in zuul-jobs, which have the same names an the jobs which invoke them. [1] https://zuul-ci.org/docs/zuul-jobs/jobs.html#job-build-docker-image [2] https://zuul-ci.org/docs/zuul-jobs/jobs.html#job-upload-docker-image [3] https://zuul-ci.org/docs/zuul-jobs/jobs.html#job-promote-docker-image [4] https://zuul-ci.org/docs/zuul/user/encryption.html -Jim

2 1

[summit] Open Infrastructure Summit Shanghai - Dates and Location
by Claire Massey 31 Jan '19

31 Jan '19

Hello Zuul team - We are excited to announce that following Denver, the Open Infrastructure Summit will be traveling to Shanghai, China the week of November 4, 2019. The Summit will be held at the Shanghai Expo Centre. Registration will be available soon. Sign up for updates[1] and reach out to summit(a)openstack.org with any questions. [1] https://www.openstack.org/summit/shanghai-2019 Cheers, Claire

1 0

Zuul project update 2019-01-28
by corvus＠inaugust.com 28 Jan '19

28 Jan '19

Hi, Over the past two weeks, Tristan has been working on the topic:freeze_job and got the "zuul-runner execute" command to run a zuul-job locally, as well as a buildset endpoint and a buildset page to show all the build result for a change. I've been working on image building jobs, and will continue to work on improvements to those this week (see two recent emails to list). Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email -Jim

1 0

Configurable autohold list of allowed build results.
by Gabriele Cerami 24 Jan '19

24 Jan '19

Hi, autohold currently allows users to examine a live environment to proceed with post-mortem examination. In some occasions, trying to find the root cause of a bug by just examining the logs and the timeline of the failed environment may prove too difficult, and the preferable way may be to catch the failure at its root exactly when the root cause happens. In a mechanism similar to breakpoints, we may want to stop the build at a certain point, the proceed manually to prod for weak spots. The only way to get this behaviour from zuul currently is to request an autohold, and add a step that injects a fake failure to the build. With [1] I'd like to suggest an alternative method, the list of build results that are allowed to put a node in hold state can be passed as argument, and not just hardcoded. In this way, the build may be configured to be brought forward up to a certain known to work point, without having to expressly fail, then begin the step by step manually debugging on the live environment. It may be best suitable for non voting, experimental jobs, or during the development of new jobs, but a useful option nonetheless. Any feedback welcome. Thanks. [1] https://review.openstack.org/632498

2 1

CFP Open Until January 23, Open Infrastructure Summit in Denver
by Claire Massey 22 Jan '19

22 Jan '19

Hi everyone, FYI - the CFP is now open for the first Open Infrastructure Summit <https://www.openstack.org/summit/denver-2019/> (formerly the OpenStack Summit) which will be held in Denver, Colorado April 29 - May 1, 2019. Wednesday, *January 23* is the deadline to Submit presentations <https://www.openstack.org/summit/denver-2019/call-for-presentations>. The Open Infrastructure Summit is organized by OSF and designed to be a place where open source infrastructure communities can come together and collaborate in the open. Zuul will have a large and prominent presence at the event so please submit talks to the CFP and plan to attend! SUBMIT YOUR PRESENTATION <https://www.openstack.org/summit/denver-2019/call-for-presentations> Important info: Based on previous Program Committee and attendee feedback, we have added / updated three Tracks: Security, Getting Started, and Open Development (previously Open Source Community). You can find the Track descriptions here <https://www.openstack.org/summit/denver-2019/summit-categories>. All of the OSF pilot projects —including Airship, Kata Containers, StarlingX and Zuul — will be front and center alongside other open source communities like Ansible, Cloud Foundry, Docker, Kubernetes, and many more. The Open Infrastructure Summit (formerly the OpenStack Summit), has evolved to recognize our diverse audience, and to signal to the market that the event is relevant for all IT infrastructure decision makers. If you’re interested in influencing the Summit content, apply to be a Programming Committee member <https://openstackfoundation.formstack.com/forms/openstackdenver2019_program…>*, where you can also find a full list of time requirements and expectations. Nominations will close on January 4, 2019. The content submission process for the Forum and Project Teams Gathering will be managed separately in the upcoming months. *OSF Staff will serve as the Programming Committee for the Getting Started Track. Denver Summit registration and sponsor sales are currently open. Learn more <https://www.openstack.org/summit/denver-2019> and email summit(a)openstack.org <mailto:summit@openstack.org> with any questions. Please email speakersupport(a)openstack.org <mailto:speakersupport@openstack.org> with any questions or feedback. Thanks, Claire

1 1