- Zuul-discuss - lists.opendev.org

[all] Denver Open Infrastructure Summit Community Contributor Awards!
by Kendall Nelson 06 Feb '19

06 Feb '19

Hello Everyone! As we approach the Summit (still a ways away thankfully), its time to kick off the Community Contributor Award nominations[1]! For those of you that have never heard of the CCA, I'll briefly explain what they are :) We all know people in our communities that do the dirty jobs, we all know people that will bend over backwards trying to help someone new, we all know someone that is a savant in some area of the code we could never hope to understand. These people rarely get the … [View More]

1 0

Zuul 2019 planning
by Lauren Sell 05 Feb '19

05 Feb '19

Hi everyone, We’d like to work with the Zuul community on 2019 planning so we can best support and promote the project this year. If the mailing list is the best way to start the conversation, we can use the below etherpad, or have a discussion on this thread. We’re also happy to host a quick call if folks want to brainstorm in a high bandwidth setting. The main things we’re looking to document are the goals for the community this year (for example, attract more contributors or document more … [View More]

2 2

Superuser Award nominations open until March 22
by Claire Massey 04 Feb '19

04 Feb '19

Hey everyone The Superuser Award nominations are now open for the Denver Open Infrastructure Summit. March 22 is the deadline for submissions. The Superuser Awards recognize teams using open infrastructure to meaningfully improve business and differentiate in a competitive industry, while also contributing back to the open-source community. The awards will be given out during the Denver Summit. *This is a great opportunity to nominate companies that are using Zuul!* You can learn more … [View More]

1 0

Open Infrastructure Summit Denver - Community Voting Open
by Claire Massey 31 Jan '19

31 Jan '19

Hi everyone, Community voting for the Open Infrastructure Summit Denver CFP sessions is open! You can VOTE HERE <https://www.openstack.org/summit/denver-2019/vote-for-speakers>, but what does that mean? Now that the Call for Presentations (CFP) has closed, all submissions are available for community vote and input. After community voting closes, the volunteer Programming Committee members will receive the presentations to review and determine the final selections for Summit schedule. … [View More]

1 0

Using nodepool v3 with Jenkins
by Darragh Bailey 31 Jan '19

31 Jan '19

Hi, Hoping that we can make use of nodepool v3 with Jenkins to solve some of our existing problems, and at the same time move a bit closer to upgrading from zuul v2 -> v3. Going to outline our current problem, my initial thinking that resulted in some of my questions last night in IRC and some further thinking on it that may provide a way forward for us. Hoping that someone can chime in and identify if this is realistic or if we're heading down the wrong path before getting to far. Keep … [View More]in mind that I'm only starting with nodepool, so I'm unsure if I even understand some of the concepts. e.g. does nodepool get a request for a particular type of node or is it infered by there being none of that type available in the queues in zookeeper? Current problem: We have some resources that are limited and are not directly accessible as workers for Jenkins, and so need a way to manage the allocation of resources to certain jobs. Some of these resources are provided by teams and some are from a set of resources managed by the CI team. So we probably have ~4 sets of these types of resources. At the moment this is done using some (too much) hackier with groovy, the Global Event Listener Plugin (GELP), and Lockable Resources plugin. The GELP allows us to hook into various points in Jenkins, such as when the job is scheduled but not yet running Jenkins (QUEUED). Unfortunately there doesn't appear to be any way with the gearman protocol to inspect what job is going to be assigned before it is picked up, so we can only do manipulation after the job has been scheduled and before it starts running. The data required to provide the job with the details required to use the limited resource is retrieved from vault and injected right before the job gets fully launched on Jenkins. This is done by creating lockable resource entries dynamically based on the number of the resources available, and then catching the event on when Jenkins allows the job needing this to start and injecting the required data into the environment. Unfortunately this means the queuing for the jobs requiring these limited resources occurs within Jenkins instead of Zuul because the job has still be scheduled even if it is not yet running. We've run into a number of problems resulting in workers getting allocated for jobs that don't get to start, and taking them away from jobs that could be run, and effectively lightly DOSing the system. Initial thoughts: So enter nodepool and the hope that if we move the management of both these resources and the workers out of Jenkins and into something a little more flexible, it can prevent the job from getting scheduled in the first place if not enough of the limited resources are available, and allow jobs that do not need these to be scheduled because a worker isn't getting assigned to the job that is queued waiting. Initially I was thinking of having two resources, making the one representing the limited resource a fake resource of some kind that Jenkins doesn't connect to, but as it appears that it's a requirement of the Jenkins Nodepool agent that nodes returned must be accessible to Jenkins, that doesn't seem to fly. Next idea was whether it would be possible to have a static pool and a another driver that could grab a node from the static pool if a request for the limited resource node type came in, which prompted some of my questions in IRC last night. Further thinking done: After spending a bit more time with the static nodepool driver, and depending on how it works (there are lots of gaps in my understanding), I'm wondering if what we need is to create a custom driver and place all of our static nodes in a internal pool inside of this driver. Additionally query our vault instance for a list of these limited resource groups and build some pools of those internally as well. And assign labels as part of the requirements for the jobs using these limited resources. Then instead of immediate registering all of the nodes with zookeeper, instead depending on the node label request that comes in, populate zookeeper with an entry containing suitable labels. If it is one requiring a limited resource identified by a name such as teamA-resources, the driver code can test if there are any not in use and if so create a node entry in zookeeper that contains labels "teamA-resources" and "teamA-resource-5". The second label can be used in Jenkins by our GELP code to determine which set of data to inject into the environment of the job scheduled on that node. Obviously there is still some hackier in there, but does a custom driver building on the idea of combining some of the ideas of the static driver along with the dynamic drivers where instead of launching a new node, it simply creates a node entry on demand using both a static node and details on what resource is required? Possibly I'm misunderstanding how nodepool works in this case, and maybe it just populates a queue in zookeeper and tries to keep some in there, which may mean that what needs to happen is that one entry allocating a node to each of the sets of limited resources needs to be added as entries to zookeeper queues by the driver with further entries added if there are enough resources available as those queues are consumed? Which is a bit different, but seems like it would still work to limit the amount of damage where multiple jobs were launched and required more of the limited resource than was available. Currently can end up seeing 3 queue instances for a single limited resource queued up in Jenkins, so as we migrate another case were this will be needed we could end up seeing a queue of 9-10. Seems nodepool might limit it to only losing one per team, until we can migrate to zuulv3 and use some of the mutex capabilities to handle this a bit better. Thoughts? Is this the best way to dig up as not digging isn't an option here... -- Darragh Bailey "Nothing is foolproof to a sufficiently talented fool" [View Less]

2 4

Zuul jobs for container publishing
by corvus＠inaugust.com 31 Jan '19

31 Jan '19

Hi, We recently added a new pipeline to OpenStack's Zuul, and three new jobs and roles to the zuul-jobs repo to support container publishing. Zuul users may be interested in adopting both the pipeline and the jobs into their own workflows. The new pipeline is the "promote" pipeline. It runs jobs on changes after they merge, but unlike the "post" pipeline, it does not run with the resulting branch state after the merge -- it runs in the context of the change. This means that it's not … [View More]suitable for building release artifacts, but that's not its purpose. Its purpose is to promote artifacts previously built in the gate pipeline to production. That is safe to do because anything built in the gate pipeline is built with the full future state of in-flight changes to all related projects and branches (even if it's built before they land). We build a lot of artifacts in the gate pipeline now, and then throw them away only to build them again in post. The promote pipeline is an attempt to avoid that waste and instead preserve artifacts built in gate and directly publish them -- but only if and when the changes for which they were built are merged. The first functional use of this system is in building container images and publishing them on Docker Hub. We have begun using this in openstack-infra to build container images of third-party applications, but this will work just as well and as easily for building images of applications in our infrastructure (for example, we are also starting to use this to publish Zuul images). To implement this, we wrote three jobs which are available now from zuul-jobs (and therefore can be used in any project or job): * build-docker-image [1] * upload-docker-image [2] * promote-docker-image [3] The first job should run in the check pipeline and simply runs "docker build" on the arguments passed to it. It is used to verify that the build succeeds. The second job should run in the gate pipeline. It uploads the image to Docker Hub into the final repository location, but it only does so with a single tag with the form "change_123456" where '123456' is the Gerrit change number for the change under consideration. This will mean that docker pulls won't accidentally fetch the image yet. But the image is staged in Docker Hub and is ready to be promoted if the change passes all its tests and merges. The third job runs in the promote pipeline. This job, unlike the others, only performs Docker Hub API calls and requires no build resources. Therefore it is a zero-node job in Zuul -- a job which only runs on the Zuul executor. This job finds a previously uploaded image with a "change_" tag, and re-tags it with any specified tags ("latest" by default, but you can also tag version numbers or anything else). It also cleans up unused "change_" tags so they don't clutter up the repository. The nice thing about zero-node jobs is that they start and finish very quickly (since they don't have to deal with resource allocation and contention). With this system, an image will be fully published within about a minute after the change merges. A complete example .zuul.yaml might look like this: - secret: name: myproject-dockerhub data: username: dockerhubuser password: !encrypted/pkcs1-oaep - DFlbrDM5eUMptMGIVMXV1g455xOJLi92UYF08Z2/JlIGu3t6v052o9FKlVyj1ZmpXs5+2 JTa5jHkLTvTsYs9fCaNcQc2nmViCyWNlbOMzjB17uiZOaYFNs1sMqZcUZbGEz7Y8ds6Qq NBXI10jWFPTah4QxUuBvUbT3vmjnUToCzexl5ZGhKgijcnROWfUsnlCdugpgoNIcPsUki zty5FotDihnrC8n8vIomVK6EClY38ty97pLrADzFDd+Cos/OUlvi2xooUhzx8Bn020rJA lqEU5v8LGXp5QkHx0MSDx6JY6KppJ/4p/yM+4By6l+A20zdcimxmgiNc9rMWPwDj7xsao m7NAZWmWqOO0Xkhgt6WOfugwgt9X46sgs2+yDEfbnI5ok8uRbAB/4FWj/KdpyXwhcf+O2 wEfhxLwDbAoGONQPjb4YcZmCXtmR7Qe5t+n2jyczWXvrbaBDUQP5a+YtVNN/xhmQ7D740 POlxv7bLxJAixzqaQ3d8Rz9ZEv6zzRuhWph32UQtZ1JxSNww+EvmXm2eEi2Q2z6pT1Cx/ j2OrFyA2GL/UJOVb15VHKF6bgHPHWJtpjPFhqdcvBhVute4BWB+KPcWH+y+apHN1enK3H tNJO9iqm34nKwSuj5ExmFw50LtwR5/9FyRuRPq/vBL+8y82v8FDmeYsBeobn5M= - job: name: myproject-build-image parent: build-docker-image description: Build Docker images. vars: &image_vars docker_images: - context: . repository: myproject/imagename - job: name: myproject-upload-image parent: upload-docker-image description: Build Docker images and upload to Docker Hub. secrets: - name: docker_credentials secret: myproject-dockerhub pass-to-parent: true vars: *image_vars - job: name: myproject-promote-image parent: promote-docker-image description: Promote previously uploaded Docker images. secrets: - name: docker_credentials secret: myproject-dockerhub pass-to-parent: true vars: *image_vars - project: check: jobs: - myproject-build-image gate: jobs: - myproject-upload-image promote: jobs: - myproject-promote-image And the pipeline definition looks like: - pipeline: name: promote description: | This pipeline runs jobs that operate after each change is merged in order to promote artifacts generated in the gate pipeline. manager: supercedent precedence: low post-review: True trigger: gerrit: - event: change-merged success: gerrit: {} mysql: failure: gerrit: {} mysql: If you find that you need jobs which behave slightly differently, you may be able to inherit from these jobs and add pre or post playbooks. Or if you need something particularly complicated, you can re-implement the jobs using the underlying roles in zuul-jobs, which have the same names an the jobs which invoke them. [1] https://zuul-ci.org/docs/zuul-jobs/jobs.html#job-build-docker-image [2] https://zuul-ci.org/docs/zuul-jobs/jobs.html#job-upload-docker-image [3] https://zuul-ci.org/docs/zuul-jobs/jobs.html#job-promote-docker-image [4] https://zuul-ci.org/docs/zuul/user/encryption.html -Jim [View Less]

2 1

[summit] Open Infrastructure Summit Shanghai - Dates and Location
by Claire Massey 30 Jan '19

30 Jan '19

Hello Zuul team - We are excited to announce that following Denver, the Open Infrastructure Summit will be traveling to Shanghai, China the week of November 4, 2019. The Summit will be held at the Shanghai Expo Centre. Registration will be available soon. Sign up for updates[1] and reach out to summit(a)openstack.org with any questions. [1] https://www.openstack.org/summit/shanghai-2019 Cheers, Claire

1 0

Zuul project update 2019-01-28
by corvus＠inaugust.com 28 Jan '19

28 Jan '19

Hi, Over the past two weeks, Tristan has been working on the topic:freeze_job and got the "zuul-runner execute" command to run a zuul-job locally, as well as a buildset endpoint and a buildset page to show all the build result for a change. I've been working on image building jobs, and will continue to work on improvements to those this week (see two recent emails to list). Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email -Jim

1 0

Configurable autohold list of allowed build results.
by Gabriele Cerami 24 Jan '19

24 Jan '19

Hi, autohold currently allows users to examine a live environment to proceed with post-mortem examination. In some occasions, trying to find the root cause of a bug by just examining the logs and the timeline of the failed environment may prove too difficult, and the preferable way may be to catch the failure at its root exactly when the root cause happens. In a mechanism similar to breakpoints, we may want to stop the build at a certain point, the proceed manually to prod for weak spots. The … [View More]

2 1

CFP Open Until January 23, Open Infrastructure Summit in Denver
by Claire Massey 22 Jan '19

22 Jan '19

Hi everyone, FYI - the CFP is now open for the first Open Infrastructure Summit <https://www.openstack.org/summit/denver-2019/> (formerly the OpenStack Summit) which will be held in Denver, Colorado April 29 - May 1, 2019. Wednesday, *January 23* is the deadline to Submit presentations <https://www.openstack.org/summit/denver-2019/call-for-presentations>. The Open Infrastructure Summit is organized by OSF and designed to be a place where open source infrastructure communities … [View More]

1 1