- Zuul-discuss - lists.opendev.org

swift upload jobs + ara-report
by Mohammed Naser 26 Aug '19

26 Aug '19

Hi everyone, The ara-report role combined with swift upload is causing a huge amount of objects generated, which I think need to be optimized in order for the role to be more usable. So, given that we're one of the clouds that OpenDev uses to publish job results, I've managed to pull some general information from that object storage cluster.. for yesterday (a sunday), there was 180k PUTs for *static* ARA assets .. and that's on a quiet day. Let's take a more busy day, Thursday, that resulted … [View More]

4 3

Tristan is a Zuul Maintainer
by corvus＠inaugust.com 22 Aug '19

22 Aug '19

Hi, I'm pleased to announce that Tristan Cacqueray (tristanC) has been added[1] to the ranks of Zuul Maintainers. Congratulations Tristan, and thank you for all your hard work! For more information about the process and the qualifications and responsibilities of maintainers, please see [2]. -Jim [1] https://review.opendev.org/677308 [2] https://zuul-ci.org/docs/zuul/governance.html#zuul-maintainers

1 0

Shanghai Forum Selection Committee
by Jimmy McArthur 21 Aug '19

21 Aug '19

Hello Zuul! The Forum in Shanghai is rapidly approaching. We would love 1 volunteer from Zuul for the Forum Selection Committee. Ideally, the volunteer would already be serving in some capacity in a governance role for your project. For information on the Summit in Shanghai: https://www.openstack.org/summit/shanghai-2019/ For more information on the Forum, please see:https://wiki.openstack.org/wiki/Forum Please reach out to myself orknelson(a)openstack.org if you're interested. … [View More]

1 0

Proposed behavior change on tags feature (passing it to Ansible)
by Sorin Sbarnea 13 Aug '19

13 Aug '19

It raised to my concern that sometimes we need to pass some tags (or skip-tags) to ansible playbooks or roles. At this moment there is no support for this in zuul but there is an older CR that would add it: https://review.opendev.org/#/c/575672/ (will need few changes) Since that review was created we already got a "tags" feature in zuul, which was initially purposed for categorization on jobs. I would like to redefine the meaning of current "tags" feature and document the fact that these … [View More]

3 2

Enabling new log features
by corvus＠inaugust.com 13 Aug '19

13 Aug '19

Hi, We recently made a release of Zuul with a bunch of new functionality for handling logs in-browser. The idea is that when Zuul reports the completion of a job, the URL it uses will be a link to the Zuul web application. There is a page for each build where a summary of information about the build is displayed, a tree-view of all the log files stored for the build, along with inline rendering of any text files therein, and a complete structured rendering of the tasks in the job. Note that … [View More]in all cases these features require an existing SQL reporter configuration. Here's an example in use: https://review.opendev.org/#/c/675413/ There are a few steps you may need to take to upgrade existing installations to use these new features (and they may vary depending on your log storage). Here's a brief outline: Set up CORS: static log server ------------------------------ If you are simply uploading your logs with the "upload-logs" role, then you will need to make sure that the webserver which serves these logs adds an Access-Control-Allow-Origin header. If you are using Apache, you should make sure there is a line like this in the VirtualHost section: Header set Access-Control-Allow-Origin "*" If you are using nginx, make sure this appears in the location section: add_header 'Access-Control-Allow-Origin' '*'; Set up CORS: Swift ------------------ If you are using the upload-logs-swift role, it will create containers with the necessary settings automatically. However, some of these settings were only recently added to the role, so if you have a container which was created before August 1, you will need to make some updates. Run this script in an environment with openstacksdk installed, and a valid clouds.yaml: http://paste.openstack.org/raw/755695/ That will cause the Access-Control-Allow-Origin header to be set on responses which arrive with an Origin header. Test CORS --------- You can check whether the settings above are working correctly by visiting the build page for any build. Below the summary table, you should see a header that says "Results" and below that, one or more rows for the hosts used in the build with some icons and numbers. If that shows up, then the log server CORS setup is correct. Add the generate-zuul-manifest role ----------------------------------- Many of the new features require a new manifest file be uploading along with the logs. Add the generate-zuul-manifest[1] role to your base post playbook. This should run on localhost immediately before the log upload role. Once that is in place, you should begin to see "Zuul Manifest" appear under the "Artifacts" header in the build page. Remove obsolete static generation roles --------------------------------------- If you are using swift for log storage, you may already be using the htmlify-logs role[2]. This is obsoleted by the inline text log viewer, so now is a good time to remove it. Enable build page reporting --------------------------- Once that is all working, you can update your tenant configuration to set: report-build-page: true This will cause Zuul to report its internal build page URL rather than the log url (or the success- or failure-url of the job)[3]. I think that about covers it. Let me know if you have any questions. -Jim [1] https://zuul-ci.org/docs/zuul-jobs/log-roles.html#role-generate-zuul-manife… [2] https://zuul-ci.org/docs/zuul-jobs/log-roles.html#role-htmlify-logs [3] https://zuul-ci.org/docs/zuul/admin/tenants.html#attr-tenant.report-build-p… [View Less]

1 0

Shanghai Summit Schedule is Live!
by Jimmy McArthur 07 Aug '19

07 Aug '19

Hi everyone, The agenda for the Open Infrastructure Summit <https://www.openstack.org/summit/shanghai-2019/summit-schedule> (formerly the OpenStack Summit) is now live! If you need a reason to join the Summit in Shanghai, November 4-6, here’s what you can expect: * Breakout sessions spanning 30+ open source projects from technical community leaders and organizations including ARM, WalmartLabs, China Mobile, China Railway, Shanghai Electric Power Company, China UnionPay, … [View More]

1 0

Switch to ansible 2.8 for default version
by Paul Belanger 29 Jul '19

29 Jul '19

Greetings, For the last month[1] zuul.ansible.com has been using ansible 2.8 as the default version of ansible for jobs run by Zuul. Today we've upgraded to ansible 2.8.3 which reminded me to send this post. So far, we've yet to see any jobs failures related to ansible 2.8 however our job are not that complex. I'd ask if any zuul operators are using ansible 2.8 for jobs, and experiences so far? I would be great if we could bump the default for Zuul to 2.8, but would like a larger sample size. … [View More]

1 1

July 21 Deadline: Join OSF to Vote in the 2020 Foundation Board Elections
by Jeremy Stanley 17 Jul '19

17 Jul '19

As contributors to Zuul, a confirmed project supported by the OpenStack Foundation (OSF), we want to remind everyone of a benefit of joining the Foundation as an individual member: participating in the annual Foundation Board of Directors election. The OSF board represents all of the projects[1], not just OpenStack, and so it’s important for every community to actively participate, as both voters and candidates, in this election process. If you are not already an OpenStack Foundation member, … [View More]

1 0

Community Voting is now Open!
by Jimmy McArthur 16 Jul '19

16 Jul '19

Community voting for the Open Infrastructure Summit Shanghai sessions is open! You can VOTE HERE <https://www.openstack.org/summit/shanghai-2019/vote-for-speakers>, but what does that mean? Now that the Call for Presentations has closed, all submissions are available for community vote and input. After community voting closes, the volunteer Programming Committee members will receive the results to review to help them determine the final selections for Summit schedule. While … [View More]

1 0

Testing in zuul-jobs
by corvus＠inaugust.com 15 Jul '19

15 Jul '19

Hi, I've been working on the testing situation in Zuul jobs, and I think it's getting to a good place. The overall goal is to make it easier to have test jobs that exercise roles in the zuul-jobs repo. In turn, I think that will make it easier for reviewers to accept new roles and patches. The first change to support this is that we now have in-repo configuration of what jobs run on changes to zuul-jobs. We couldn't have this before because we don't want to impose any special requirements … [View More]on systems that use zuul-jobs. Jobs that test zuul-jobs may need to specify nodesets, secrets, etc., not to mention the actual project stanza to indicate what jobs run. To solve this, the "zuul" tenant of the OpenDev Zuul loads *extra* configuration from the zuul-jobs repo which no other Zuul system or tenant should do. In the extra configuration, we can add jobs for the repo itself. This is in the zuul-tests.d/ directory of the repo: https://opendev.org/zuul/zuul-jobs/src/branch/master/zuul-tests.d In that directory, I have organized the test jobs in the same way I have organized them in the documentation. I think we should continue that as we add new jobs. Next, I've added a playbook that we can use to get to something like 80% test coverage fairly quickly. It simply includes a named role. Since many of our roles just install (and possibly run) software, this test provides significant coverage at almost no cost. To use this role, just run the "test-playbooks/simple-role-test.yaml" playbook and pass the "role_name" variable. For example: https://opendev.org/zuul/zuul-jobs/src/commit/6cec9080a047f549578c7f8291aff… If you need to do anything other than simply including a role (for example, testing how multiple roles interact, or performing validation after the role runs), you should probably make a dedicated playbook for the job. I think that starting now, we should require at least the simple-role-test playbook for every role in the repo (unless there is some good reason it won't work). Some roles have special handling for different platforms and therefore need to be tested on each. Some notable examples include many of the roles which typically appear in base jobs. I have created a script in tools/update-test-platforms.py which will look for jobs with the tags "all-platforms" or "all-platforms-multinode" and it will automatically create (or delete) identical jobs for each of the platforms that are available in OpenDev. Of course, if you don't need the whole set (perhaps you only need to test on one or two specific platforms), you can still do the same thing manually. The script is located here: https://opendev.org/zuul/zuul-jobs/src/commit/6cec9080a047f549578c7f8291aff… And here is an example job which uses it: https://opendev.org/zuul/zuul-jobs/src/commit/6cec9080a047f549578c7f8291aff… It adds every job to the check and gate pipeline too. The next change which is relevant is that Zuul now automatically runs jobs when their configurations change. This means that we can have many, many jobs in the zuul-jobs repo (i.e., perhaps one job per role), each of which has a file matcher for the role directory (and possibly test playbook), and these jobs will only run on changes to those files or changes to their own job definitions. As a result, we should expect to have lots of jobs in this repo, but expect very few of them to run on any given change. The names of these jobs should start with the prefix "zuul-jobs-test". For example, "zuul-jobs-test-install-pdk-dependencies" tests the "install-pdk-dependencies" role. Thanks to David Simard who set up the original versions of the base role tests along with a fine example of a well-tested standalone role ("upload-git-mirror"). He showed us the way, and the changes I have made are simply to facilitate that test design. I've started working on adding test jobs for existing roles in the repo. There are quite a lot, and I would appreciate help on that. We're in no hurry, so if you can spare a minute every now and then to add a new job, that would be great. Also, if someone is proposing a change to an untested role, see if you can add a quick test job for it at the same time. Thanks, Jim [View Less]

1 0

Re: zuul-base-jobs: Changing git connection
by Andreas Jaeger 15 Jul '19

15 Jul '19

On 15/07/2019 16.53, Andreas Jaeger wrote: > On 01/07/2019 16.14, Andreas Jaeger wrote: >> We are deprecating the git.zuul-ci.org server, it redirects already to >> opendev.org. Thus, there's a change ready for zuul-base-jobs to switch >> to the new URL. This means that everybody using zuul-base jobs, needs to >> add a connection to "opendev.org" to their zuul config (if not already >> existing), otherwise your jobs will break. >> >> The change for … [View More]

2 1

OpenInfra Days Vietnam 2019 (Hanoi) - Call For Presentations
by Trinh Nguyen 12 Jul '19

12 Jul '19

Hello, Hope you're doing well :) The OpenInfra Days Vietnam 2019 [1] is looking for speakers in many different topics (e.g., container, CI, deployment, edge computing, etc.). If you would love to have a taste of Hanoi, the capital of Vietnam, please join us this one-day event and submit your presentation [2]. *- Date:* 24 AUGUST 2019 *- Location:* INTERCONTINENTAL HANOI LANDMARK72, HANOI, VIETNAM Especially this time, we're honored to have the Upstream Institute Training [3] hosted by the … [View More]

1 1

Zuul moved to zuul OpenDev Zuul Tenant
by corvus＠inaugust.com 12 Jul '19

12 Jul '19

Hi, In the OpenDev installation of Zuul (which is the Zuul that Zuul itself uses for gating), I have moved all of the Zuul projects to their own tenant (named "zuul"). If you want to see the status, you won't find it on zuul.openstack.org anymore. The new url is: https://zuul.opendev.org/t/zuul/status Also, if you have been using any dev releases (or our completely unadvertised javascript content tarball releases), new versions are no longer published to tarballs.openstack.org, but are … [View More]

1 0

Design for mirrors in zuul-jobs
by corvus＠inaugust.com 09 Jul '19

09 Jul '19

Hi, When we started zuul-jobs, we had the idea that being able to use local mirrors or proxies instead of remote package mirrors, etc, is a good practice that we'd like to share with the wider community. However, our implementation in the configure-mirrors role (and others) was inherited from and built around the systems we had been running in OpenDev (and OpenStack infra), and ended up encoding a number of assumptions which don't generally hold. We'd like to replace it with a more generic … [View More]

1 0

Fwd: [OpenStack Foundation] [Shanghai Summit] CFP Deadline TONIGHT
by Jimmy McArthur 02 Jul '19

02 Jul '19

Deadlines! Submit! > *From:* Ashlee Ferguson <ashlee(a)openstack.org> > *Date:* July 2, 2019 at 2:17 PM > *To:* foundation(a)lists.openstack.org, > openstack-discuss(a)lists.openstack.org, community(a)lists.openstack.org, > kata-dev(a)lists.katacontainers.io > *Subject:* [OpenStack Foundation] [Shanghai Summit] CFP Deadline TONIGHT > Hi everyone, > > The Shanghai Summit Call for Presentations [1] deadline is TONIGHT, > July 2 at 11:59 pm PT (July 3, 2019 … [View More]

1 0

Assigning nodeset/nodes dynamically based in user who made the pull request in github
by Kasibhatla Sarma 30 Jun '19

30 Jun '19

Hi Team, I am trying to see if there are any options using which I can select the nodeset/nodes dynamically. For example, assume that I have two users in two different regions (us-west and us-east) Based on who made the pull request in GitHub, we would like to run the check/gate pipelines on specific nodes. For example, - nodeset: name: emea nodes: - name: node_emea label: centos76 groups: - name: emea nodes: - node_emea The first job returns the location of the user who made a pull request … [View More]

5 8

CFP Deadline for Open Infrastructure Summit Shanghai
by Jimmy McArthur 26 Jun '19

26 Jun '19

Hi Everyone! The July 2 deadline to submit a presentation [1] for the Open Infrastructure Summit [2] in Shanghai is in less than one week! Submit your session today and join the global community in Shanghai, November 4-6, 2019. Sessions will be presented in both Mandarin and English, so you may submit your presentation in either language. Submit your presentations, panels, and hands-on workshops [3] before July 2 at 11:59 pm PT (July 3, 2019 at 15:00 China Standard Time). Tracks [4]: … [View More]

1 0

Feedback for Zuul web UI
by Paul Belanger 24 Jun '19

24 Jun '19

Greetings, I wanted to share some feedback I've gotten over the last few weeks about the Zuul web UI, I am mostly doing so, because I am struggling to make progress on fixing these issue myself (I don't have my JS experience). Here are a few recent examples: Builds page ----------- * Lack of pagination support. Currently, there is no easy way to scroll through the history of the builds UI. Right now, it is hardcoded to a single page, of 50 results. Users would like the ability to go … [View More]

2 1

Zuul autohold changes
by David Shrewsbury 17 Jun '19

17 Jun '19

Hello, I've mentioned this info in the IRC channel, but thought it would be a good idea to mention this on the mailing list as well. I have a series of changes [1] up that significantly change the way Zuul deals with autohold requests. The tl;dr of it is that we now store those requests in ZooKeeper rather than in memory. Advantages of this change are: 1. Easily identify which nodes are held for a request and to easily delete those held nodes when the request is deleted. 2. … [View More]

2 1

Reviews for the Bitbucket Server driver
by Mark Meyer 14 Jun '19

14 Jun '19

Hi, I'd like you to review the Bitbucket Server driver for Zuul. We now have all the components in place. Basic testing shows that it's working and my team at work has been using it for a while in an experimental setting. Now we need to go the next steps and have it refined. *Not Bitbucket Cloud Thanks, mark -- Mark Meyer mark(a)ofosos.org

1 0

Slack connectivity
by Mark Meyer 13 Jun '19

13 Jun '19

Hey list, I'd like to gauge interest here. So you want connectivity options for slack? I'm thinking of having this as part of zuul to trigger jobs and report results, out alternatively as an external bit that runs on the scheduler node. We already have some code and some ideas. But I don't feel like walking that path on my own. What so you think? Cheers, mark -- Mark Meyer mark(a)ofosos.org

4 3

GitHub API ReadTimeout Errors
by Paul Belanger 12 Jun '19

12 Jun '19

Greetings, We (ansible team) are running into a issue using the zuul github driver, please see the following pastebin: http://paste.openstack.org/show/752756/ As you can see, we don't seem to be getting reliable network access to api.github.com. I am currently trying to figure out if this is a network issue with our control plane provider (vexxhost) or something related to GitHub side (eg: bad load balancer). However, what it does result in, is Zuul missing events from GitHub and not … [View More]

2 3

Shanghai PTG Changes
by Kendall Nelson 12 Jun '19

12 Jun '19

Hello All, After Denver we were able to take time to reflect on the improvements we can make now that the PTG will occur immediately following the summit for the near future. While Shanghai will have its own set of variables, it's still good to reevaluate how we allocate time for groups and how we structure the week overall. tldr; - Onboarding is moving into the PTG for this round (updates stay a part of the Summit) - You can still do regular PTG stuff (or both onboarding and regular PTG … [View More]stuff) - PTG slots can be as short as 1/4 of a day - More shared space at the Shanghai venue, less dedicated space - New breakdown: 1.5 days of Forum and 3.5 days of PTG - Survey will be out in a few weeks for requesting PTG space We'll have our traditional project team meetings at the PTG in Shanghai as the default format, that won't change. However, we know many of you don't expect to have all your regulars attend the PTG in Shanghai. To combat this and still help project teams make use of the PTG in the most effective way possible we are encouraging teams that want to meet but might not have all the people they need to have technical discussions to meet anyway and instead focus on a more thorough onboarding of our Chinese contributors. Project teams could also do a combination of the two, spend an hour and a half on onboarding (or however much time you see fit) and then have your regular technical discussions after. Project Updates will still be a part of the Summit like normal, its just the onboardings that will be compacted into the PTG for Shanghai. We are making PTG days more granular as well and will have the option to request 1/4 day slots in an effort to leave less empty space in the schedule. So if you are only doing onboarding, you probably only need 1/4 to 1/2 of a day. If you are doing just your regular technical discussions and still need three days, thats fine too. The venue itself (similar to Denver) will have a few large rooms for bigger teams to meet, however, most teams will meet in shared space. For those teams meeting to have only technical discussions and for teams that have larger groups, we will try to prioritize giving them their own dedicated space. For the shared spaces, we will add to the PTGbot more clearly defined locations within the shared space so its easier to find teams meeting there. I regret to inform you that, again, projection will be a very limited commodity. Yeah.. please don't shoot the messenger. Due to using mainly shared space, projection is just something we are not able to offer. The other change I haven't already mentioned is that we are going to have the PTG start a half day early. Instead of only being 3 days like in Denver, we are going to add more time to the PTG and subtract a half day from the Forum. Basically the breakdown will be 1.5 Forum and 3.5 PTG with the Summit overlapping the first two days. I will be sending the PTG survey out to PTLs/Project Leads in a couple weeks with a few changes. -Kendall Nelson (diablo_rojo) [View Less]

1 0

Zuul project update 2019-06-10
by corvus＠inaugust.com 10 Jun '19

10 Jun '19

Hi, Last week we merged the Tenant-scoped admin web API spec: https://zuul-ci.org/docs/zuul/developer/specs/tenant-scoped-admin-web-API.h… We can start reviewing the implementation for that now (status:open topic:zuul_admin_web). David is finishing up a series which stores autohold information in Zookeeper to improve the experience around that. Tobias has continued his work on adding event IDs to log messages. Clark is moving us to node version 10 across all of our javascript jobs. … [View More]

1 0

Toggle host-key-checking per label
by Paul Belanger 05 Jun '19

05 Jun '19

Greetings, I'd like to get a few eyes on https://review.opendev.org/663378/ please, as I'm currently blocked on doing some network automation with nodepool / zuul. Due to issues with a network appliance, we have a use case where an interface comes online but there is no default gateway assigned (apparently this needs to be done by a human). In this case, we actually want nodepool to launch the node, but not scan for hostkeys. This is okay for us, because zuul doesn't directly manage this … [View More]

1 0

Different merge-mode for different pipeline?
by Nick Hwangbo 30 May '19

30 May '19

Hello, Is it possible to specify different merge-modes for different pipelines for the same repository? For example, is something like this possible in a zuul.yaml configuration: - job: name: test run: playbooks/test.yaml - project: merge-mode: cherry-pick check: jobs: - test - project: merge-mode: merge-resolve gate: jobs: - test The check pipeline would use cherry-pick as a merge mode, while the gate pipeline would use merge-resolve? Thanks, Nick

2 2

Zuul project update 2019-05-29
by corvus＠inaugust.com 29 May '19

29 May '19

Hi, It's been a while since I've sent one of these out, largely due to a busy schedule involving launching OpenDev, vacation, conferences, and associated travel. Now seems a good time to try to resync. First specs: I think the tenant-scoped admin web API spec is ready for final approval. I think everyone who has helped refine that has had their issues addressed, so we probably just need folks to go over it once again and give it the OK. I'd like to merge that next Friday, June 7 unless … [View More]

1 0

Gerrit Downtime May 31, 2019 beginning 1500UTC
by Clark Boylan 28 May '19

28 May '19

Hello everyone, We'll be taking a short (hopefully no longer than an hour) Gerrit downtime so that we can rename some projects this Friday, May 31, 2019 beginning at 1500UTC. Some of these renames are cleanups after the great OpenDev git migration and others aren't, but all of them require us to stop Gerrit to update the database and contents on disk. If you'd like specifics on what repos are being renamed or what the process looks like, you can follow along at: https://wiki.openstack.org/… [View More]

1 0

AMQP driver
by Tristan Cacqueray 27 May '19

27 May '19

Hello, There is an initial AMQP driver proposed waiting for reviews[0]. The use-case is to integrate Zuul with fedmsg/umb[1] which is used for artifacts testing. The goal is to run Zuul jobs for the different testing phases that happen after a change is merged, namely on koji builds and on composition events. I guess this goes beyond Zuul's git driven model, and the AMQP trigger would need to tell the jobs where the external provided artifacts are. So could Zuul be extended to support such a … [View More]

2 2

periodic jobs (timer tigger) and job.files attribute
by Paul Belanger 25 May '19

25 May '19

Greetings, Over the last few days I've been working to create a job that is shared between check / gate pipelines and periodic pipeline. As an example, using the following job stanza: - job: name: ansible-test-network-integration-vyos parent: ansible-network-vyos-appliance pre-run: playbooks/ansible-test-network-integration-base/pre.yaml run: playbooks/ansible-test-network-integration-base/run.yaml required-projects: - name: github.com/ansible/ansible … [View More] timeout: 3600 vars: ansible_test_network_integration: vyos files: - ^lib/ansible/modules/network/vyos/.* - ^lib/ansible/module_utils/network/vyos/.* - ^lib/ansible/plugins/connection/network_cli.py - ^test/integration/targets/vyos_.* you can see we are using the job.files attribute, this works as expects when we open a PR in github against github.com/ansible/ansible. Here is the pipeline stanza: - project-template: name: ansible-test-network-integration periodic: jobs: - ansible-test-network-integration-vyos third-party-check: jobs: - ansible-test-network-integration-vyos However, the issue is, current periodic jobs are not running, because from what I see the job.files attribute does not match. So, based on a suggestion from Spamaps I tried the following: - project-template: name: ansible-test-network-integration periodic: jobs: - ansible-test-network-integration-vyos third-party-check: jobs: - ansible-test-network-integration-vyos: files: [] which didn't work, I even tried: - project-template: name: ansible-test-network-integration periodic: jobs: - ansible-test-network-integration-vyos third-party-check: jobs: - ansible-test-network-integration-vyos: files: - ^.* and no luck. The best I can guess, is that jobs enqueued into the periodic pipeline via timer trigger don't seem to have files attribute for the change? Which basically prevent any job.files match possible. My question here is, how could we allow this? Does it make sense if we have job.files set to an empty list, remove any existing job.files defined? Or, if periodic pipelines don't present files in the change object, could we just drop the job.files match process? I know that I could remove the job.files from the job stanza, and move them into the pipeline stanza, however in this case I would not like to do that for reasons. I also know I could create another job.name, maybe prefaces with periodic, again I'd prefer not to do that too. However, if there is no way to fix this based on above, I'll have to do one of these solutions. Thanks, Paul [View Less]

3 3

ansible version for linting?
by Andreas Jaeger 24 May '19

24 May '19

Which ansible version should we use for linting in repos like openstack/project-config, openstack/openstack-zuul-jobs and zuul/zuul-jobs? OpenDev runs ansible 2.7 which is the default for Zuul - but Zuul supports 2.5 (as deprecated) and 2.6 and soon 2.8 as well... So, do we lint with the oldest version? The newest? The default? right now we use ansible 2.5.x for linting. I proposed the first changes to go to 2.7 - but then thought further and wrote this email: https://review.opendev.org/#/… [View More]

1 1

Ansible 2.8 support
by Paul Belanger 13 May '19

13 May '19

Greetings, Hopefully this week ansible 2.8.0 will be released, yay. As a results, Tobias as already done most of the work[1] to get Zuul ready to support it. I'd like to ask for humans to review the patch, the main change here for zuul is to our console stream plugin. Ansible 2.8 now includes rescued / ignored in the results output. Also, facts are now disabled by default when using gather_facts. I think it would be awesome if we could land and do a zuul release just after ansible 2.8 was … [View More]

1 1

Zuul's Roadmap and Backlog
by Tristan Cacqueray 13 May '19

13 May '19

Hello folks, I don't know if this subject has been discussed at the last PTG, so here is a thread about it. There is a growing backlog of bug fixes and features, and it's a bit frustrating not knowing when or if a patch is going to be approved. For the 3.0 release, there was a board with the list of stories and this was very helpful in communicating what was going to be included and what should have been reviewed. Would it be possible to use such board for the next releases? As an example, … [View More]

5 9

Job doesn't enter the gate pipeline
by Vitaliy Lotorev 11 May '19

11 May '19

Hi, I'm trying to setup Zuul with in-house Gerrit. check jobs from independent pipeline are started in predictable way (on new PS and 'recheck' comment). The issue is I can't start jobs in buildroot_gate (dependent) pipeline. I simplified pipeline to this one: - pipeline: name: buildroot_gate manager: dependent require: gerrit: open: true trigger: gerrit: - event: comment-added comment: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*gate … [View More] success: gerrit: Verified: 1 mysql: failure: gerrit: Verified: -1 mysql: I.e. for debug purpose I'd like buildroot_gate pipeline to get started on 'gate' comment. Once I add 'gate' comment to PS Zuul on Gerrit Scheduler successfully detects this and reports the as follows: scheduler_1 | 'url': ' https://mycompanygerrit/c/production/vip1-boards/sd-fc/+/12134'}] scheduler_1 | 2019-05-09 20:53:58,502 DEBUG zuul.Scheduler: Change <Change 0x7f18e2104c50 production/vip1-boards/sd-fc 12$ 34,12> has been updated, clearing dependent change caches scheduler_1 | 2019-05-09 20:53:58,502 DEBUG zuul.GerritConnection: Scheduling event from gerrit: <GerritTriggerEvent comm$ nt-added mycompanygerrit/production/vip1-boards/sd-fc master 12134,12 Verified:0, Code-Review:1> scheduler_1 | 2019-05-09 20:53:58,502 DEBUG zuul.Scheduler: Run handler awake scheduler_1 | 2019-05-09 20:53:58,502 DEBUG zuul.Scheduler: Fetching trigger event scheduler_1 | 2019-05-09 20:53:58,503 DEBUG zuul.Scheduler: Processing trigger event <GerritTriggerEvent comment-added ge$ rit.elvees.com/production/vip1-boards/sd-fc master 12134,12 Verified:0, Code-Review:1> scheduler_1 | 2019-05-09 20:53:58,503 DEBUG zuul.Pipeline.nto7.buildroot_gate: Event <GerritTriggerEvent comment-added ge$ rit.elvees.com/production/vip1-boards/sd-fc master 12134,12 Verified:0, Code-Review:1> for change <Change 0x7f18e2104c50 p$ oduction/vip1-boards/sd-fc 12134,12> matched <GerritEventFilter types: comment-added ignore_deletes: True comments: (?i)^($ atch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*gate> in pipeline <DependentPipelineManager buildroot_gate> scheduler_1 | 2019-05-09 20:53:58,503 DEBUG zuul.Pipeline.nto7.buildroot_gate: Considering adding change <Change 0x7f18e2$ 04c50 production/vip1-boards/sd-fc 12134,12> scheduler_1 | 2019-05-09 20:53:58,503 DEBUG zuul.Pipeline.nto7.buildroot_gate: Change <Change 0x7f18e2104c50 production/v$ p1-boards/sd-fc 12134,12> can not merge, ignoring scheduler_1 | 2019-05-09 20:53:58,503 DEBUG zuul.Pipeline.nto7.buildroot_gate: Change <Change 0x7f18e2104c50 production/v$ p1-boards/sd-fc 12134,12> is not ready to be enqueued, ignoring scheduler_1 | 2019-05-09 20:53:58,503 DEBUG zuul.Pipeline.nto7.check: Starting queue processor: check scheduler_1 | 2019-05-09 20:53:58,503 DEBUG zuul.Pipeline.nto7.check: Finished queue processor: check (changed: False) scheduler_1 | 2019-05-09 20:53:58,504 DEBUG zuul.Pipeline.nto7.gate: Starting queue processor: gate scheduler_1 | 2019-05-09 20:53:58,504 DEBUG zuul.Pipeline.nto7.gate: Finished queue processor: gate (changed: False) scheduler_1 | 2019-05-09 20:53:58,504 DEBUG zuul.Pipeline.nto7.buildroot_gate: Starting queue processor: buildroot_gate scheduler_1 | 2019-05-09 20:53:58,504 DEBUG zuul.Pipeline.nto7.buildroot_gate: Finished queue processor: buildroot_gate (c hanged: False) Please, note Scheduler logs "can not merge, ignoring", "is not ready to be enqueued, ignoring". How could I understand the reason why it can't be merged? This PS it definitely mergable manually: I could vote +2 and Sumbit the PS. -- Regards, Vitaliy [View Less]

2 2

Correct way to trigger a periodic job manually
by Ian Wienand 08 May '19

08 May '19

Hello, The manual says to trigger a periodic job against master you'd do [1] $ zuul enqueue-ref --tenant openstack --trigger timer --pipeline periodic --project openstack/example_project --ref refs/heads/master Which looks about right to me, but this sets a newrev of 00..000 which is not correct; e.g. [2]. Tracing back that job to find the cause of the RETRY_LIMIT on ze06 I see: --- 2019-05-06 07:20:50,875 DEBUG zuul.AnsibleJob: [build: b1387c9741134f83a6f11e1cad0386f0] Checking out … [View More]master 2019-05-06 07:20:50,877 ERROR zuul.AnsibleJob: [build: b1387c9741134f83a6f11e1cad0386f0] Exception while executing job Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/git/refs/symbolic.py", line 305, in set_reference obj = self.repo.rev_parse(ref + "^{}") # optionally deref tags File "/usr/local/lib/python3.5/dist-packages/git/repo/fun.py", line 213, in rev_parse obj = name_to_object(repo, rev[:start]) File "/usr/local/lib/python3.5/dist-packages/git/repo/fun.py", line 147, in name_to_object raise BadName(name) gitdb.exc.BadName: Ref 'master' did not resolve to an object During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/zuul/executor/server.py", line 790, in execute self._execute() File "/usr/local/lib/python3.5/dist-packages/zuul/executor/server.py", line 914, in _execute repo.checkout(selected_ref) File "/usr/local/lib/python3.5/dist-packages/zuul/merger/merger.py", line 369, in checkout File "/usr/local/lib/python3.5/dist-packages/git/refs/symbolic.py", line 308, in set_reference raise ValueError("Could not extract object from %s" % ref) ValueError: Could not extract object from master --- I set newref manually to the git master HEAD change-id, and it did work -- but it looks different to jobs actually triggered by the periodic driver itself. For example [3] is the job I triggered, and [4] is one from the periodic driver -- [4] does not have a newref. I feel like this used to work, probably because when I look back I wrote that bit of documentation [5] ;) I guess something changed; perhaps it is the client setting newref to 0000.000 by default instead of something like None? Any tips on what should I update the instructions to say now? -i [1] https://zuul-ci.org/docs/zuul/admin/client.html#manual-enqueue-examples [2] http://zuul.openstack.org/build/b1387c9741134f83a6f11e1cad0386f0 [3] http://zuul.openstack.org/build/8781cd38097b437d90ba5a9a5505633c [4] http://zuul.openstack.org/build/d30f960f7ca04b33af1eb5b7513b1b66 [5] https://review.opendev.org/#/c/518662/ [View Less]

2 2

exclude-unprotected-branches equivalent for Gerrit connection
by Brendan Jackman 07 May '19

07 May '19

Hi all, In my org's Gerrit setup, refs under personal/$username/ are a free-for-all. I have seen the same in other organisations too. This leads to the problem that is solved for Github by exclude-protected-branches; those branches eventually end up with broken Zuul configs (actually case that brought this to my attention is that I cannot delete a job that is no longer used, because the config in a personal/ branch refers to it). It seems like a simple solution would be to add an option … [View More]

2 2

Inheritance confusion, docs needed?
by Michael Johnson 06 May '19

06 May '19

As a developer creating Zuul jobs, one of the areas i find confusing is the inheritance of variables. I am not sure if this is an enhancement style request or just a missing section in the documentation. When you use host-vars, the top level variables are "merged" with the parent job variables, however the nested variables, such as "devstack_localrc" are not merged, they are "replaced". As an example is the opendev Zuul jobs (I think this applies to non-OpenStack jobs as well): When you use … [View More]

2 1

Zuul working group session at OpenInfra Summit
by corvus＠inaugust.com 29 Apr '19

29 Apr '19

Hi, If you're at the OpenInfra Summit and would like to discuss what is or isn't working about sharing jobs between Zuul installations, please join the working group session on Tuesday: https://www.openstack.org/summit/denver-2019/summit-schedule/events/23736/z… -Jim

1 0

Celebrating Zuul's OSF Top Level Project Confirmation
by Clark Boylan 28 Apr '19

28 Apr '19

Zuul was confirmed by the OSF board as a top level OSF project today in Denver. To celebrate we'd like to meetup for arcade games, drinks, and food at FTW Denver Monday night. This is doubling as the OpenDev Infra team gathering, but considering the overlap between Zuul and Infra I figure that won't be an issue. If you'd like to join us we will be gathering at 6:45pm Monday in front of the big blue bear statue at the convention center. If it is raining, we'll likely meet up just inside the … [View More]

1 0

Tagging beta / rc releases
by Paul Belanger 24 Apr '19

24 Apr '19

Greetings, I wanted to raise the topic of maybe using beta / rc tags to publish pre-release versions of zuul / nodepool to pypi. Currently, before each official release we usually select a git SHA to be installed onto zuul.opendev.org, to ensure we can run a scale without any issues. Then a few days later, if things are well we tag a new release or work to address the problem. Then restart the process until things look good. I really like this approach, as currently there is no larger zuul … [View More]

3 8

Migrating git repos to OpenDev
by corvus＠inaugust.com 21 Apr '19

21 Apr '19

Hi, As discussed in November[1], the OpenStack project infrastructure is being rebranded as "OpenDev" to better support a wider community of projects. We are nearly ready to perform the part of this transition with the largest impact: moving the authoritative git repositories for existing projects. In this email, I'd like to introduce the new hosting system we are preparing, discuss the transition, and invite projects to work with us on the logistics of the change. Gerrit ====== Gerrit is … [View More]the core of our system and it will remain so in OpenDev. As part of this move, we will rename the gerrit server from review.openstack.org to review.opendev.org. As part of the transition, we will automatically merge appropriate changes to all branches of all repositories updating .gitreview and Zuul configuration files. Any further changes (README files, etc.) we expect to be made by individual project contributors. Repository Browsing =================== Currently our canonical *public* repository system is the cgit server at https://git.openstack.org/ (and git.airshipit.org, git.starlingx.io, and git.zuul-ci.org) This is a load balanced cluster of several servers which is designed to handle all the public git repository traffic, as it scales much better than Gerrit (and has a more friendly domain name). >From a technical standpoint, it's excellent, but its usability could be improved. Therefore, as part of this transition, we will replace the cgit servers with a new system based on Gitea. Gitea is a complete development collaboration system, but it's very flexible and will allow us to disable components which we aren't using. We will operate it in a read-only configuration where it will act as the public mirror for Gerrit. The advantages it has over the current system are: * Shorter domain name in project URLs: https://git.openstack.org/openstack/nova vs https://opendev.org/openstack/nova * Clone and browsing URLs are the same (with cgit, the browsing URL has an extra path component) * More visually pleasing code browsing * Integrated code searching * Ability to highlight multiple lines in links When we perform the transition we will install redirects from git.openstack.org (and the other git sites) to opendev.org, and will maintain those redirects for the foreseeable future. We will construct them so that even existing deep links to individual files in individual commits to cgit will redirect to the correct location on opendev.org. This system is up and running now with a live mirror of data from Gerrit, and you can start testing it out today at https://opendev.org/ Please let us know if you encounter any problems. If you would like to read more about the design of this system and the transition, see the infra-spec[2]. GitHub ====== Currently all OpenStack projects are replicated to GitHub. We do not plan on changing that during the transition, however, any projects outside of the openstack*/ namespaces will not automatically be replicated to GitHub, and we do not plan on adding that in the future. We do, however, support projects using Zuul to run post-merge jobs to push updates to GitHub or any other third-party mirrors with their own credentials. We will be happy to work with anyone interested in that to help set up jobs to do so. We are adopting this approach so that individual projects can have more control over how they are represented in social media, and to give us more flexibility in supporting our own organizational namespaces on OpenDev without assuming they map directly to GitHub. Eventually we plan on moving the OpenStack project to that system as well and retiring direct replication from Gerrit to GitHub completely. But we will defer that work until after this transition. Logistics ========= We can prepare much of the system in advance (as we have for the hosting system on opendev.org) but the actual transition and renaming of the Gerrit server will need to happen at once during an outage window. We need to schedule that outage and begin preparing for it. Since all of the project git URLs are going to change (to replace git.openstack.org with opendev.org and review.openstack.org with review.opendev.org) we can additionally take the opportunity to reorganize projects into different organizations. For example, during the transition we will rename Zuul, and it's associated projects, from the "openstack-infra" org to "zuul". So their new names will be "zuul/zuul", "zuul/nodepool", etc. This is an excellent time for the rest of the OpenStack Foundation pilot projects to do the same. If the OpenStack project desires this, it would also be a good time to move unofficial projects out of the openstack/ namespace. Therefore, we need your help: Action Items ============ We need each of the following projects: * OpenStack * Airship * StarlingX * Zuul To nominate a single point of contact to work with us on the transition. It would be helpful for that person to attend the next (and possibly next several) openstack infra team meetings in IRC [3]. We will work with those people on scheduling the transition, as well as finalizing the list of projects which should be renamed as part of the transition. If you manage an unofficial project and would like to take the opportunity to move or rename your project, please add it to this ethercalc[4]. [1] http://lists.openstack.org/pipermail/openstack-dev/2018-November/136403.html [2] http://specs.openstack.org/openstack-infra/infra-specs/specs/opendev-gerrit… [3] http://eavesdrop.openstack.org/#Project_Infrastructure_Team_Meeting [4] https://ethercalc.openstack.org/opendev-transition [View Less]

5 6

Post OpenDev Repo Migration Issues
by Clark Boylan 21 Apr '19

21 Apr '19

Hello, There are two quick things we want to point out that we've noticed after the OpenDev Gerrit manuevers. First, any commit Depends-On footers for unmerged changes will need to use https://review.opendev.org urls instead of https://review.openstack.org urls. This is due to a limitation in Zuul and how it sees the Gerrit server. Any Depends-On footers that use https://review.openstack.org are currently non-functional and they will not enforce the dependency relationship. Second, while we … [View More]

3 2

Should we drop our custom Ansible plugins?
by corvus＠inaugust.com 15 Apr '19

15 Apr '19

Hi, When we created Zuul v3, we faced a fairly unique situation -- we needed to run untrusted Ansible playbooks without compromising the host running Ansible (i.e., the Zuul executor). To address this, we created a set of Ansible plugins which wrap the upstream Ansible plugins and block actions which we consider dangerous. Broadly speaking, there are two classes of actions we might want to block: 1) Actions which may expose private information stored on the executor (including information … [View More]used by Zuul itself or other jobs), or otherwise compromise the security of the executor itself (so that future jobs can not be trusted). 2) Actions which may allow a user to take advantage of privileges afforded to the executor host itself. For example, if the executor is behind a network firewall and therefore can access unprotected network services. We added bubblewrap as a second layer of defense against the first class. It has been very effective -- our custom plugins have had several vulnerabilities, all of which were ultimately unexploitable due to the use of bubblewrap. Bubblewrap doesn't address the second class. Here's more background on the bubblewrap addition: https://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3-execut… Maintaining the Ansible plugins is a lot of work, and it's very easy to get wrong. For each new Ansible version, we need to look for any changes to module parameters and make corresponding updates. All of our security issues so far have been due to things we missed here. Meanwhile, these custom plugins increase the area where playbooks which may otherwise work can not be used with Zuul (for example, because they use a lookup plugin which writes to the executor). This makes Zuul more difficult to use. Therefore, I propose that we drop support for the custom plugins and rely entirely on bubblewrap to protect us from the first class of issues. It will mean less work for us, make Zuul more useful, and, at least with regard to the first class of problems, I don't think Zuul will be less secure. That brings me to the second class. I have a feeling (which is unsubstantiated), that there are probably enough holes in our plugins which make the second class of actions possible too. For example, we allow use of the URI module, but that means that any playbook can hit any HTTP URL accessible by the executor. While it may be difficult for a user to make a connection to a mysql server, it's not outside the realm of possibility. Dropping support for our custom plugins would mean that Zuul has no defense against this class of action. We would need to document that and ensure that users know that if their executors run untrusted code that they should have adequate application or network level protections in place. In short: don't put an executor inside the squishy center of your security perimeter. Before we do this, we need to make sure that Zuul itself doesn't impose any constraints that contradict this. The executor only uses Gearman to communicate with the rest of Zuul, and it can adequately be secured. Before we add any features which require the executor to communicate via ZooKeeper, we should make sure that we support zookeeper auth. What do folks think? Is this a reasonable change to make, or should we continue supporting the custom plugins? -Jim [View Less]

7 19

Upgrading the lists.(openstack|airshipit|starlingx|zuul-ci).org server Friday April 12
by Clark Boylan 12 Apr '19

12 Apr '19

It is that time of the Ubuntu LTS cycle again and we need to upgrade our mailman mailing list server. We'd like to do that this Friday, April 12. We expect the upgrade to begin at about 17:00UTC and result in a 30-45 minute outage. The reason for the extended outage is that we will be upgrading the server in place to preserve its mail reputation. Thankfully email is a persistent system and your clients should queue up email sent until the server is back up again and accepting smtp connections. … [View More]

1 1

New Zuul job to replicate a project's git repository to a remote git server
by David Moreau Simard 11 Apr '19

11 Apr '19

Hi, It is now possible for projects to replicate their git repository to a custom location by inheriting from the 'upload-git-mirror' job provided by Zuul. This job wraps around the 'upload-git-mirror' Ansible role that is part of the zuul-jobs library [1]. In order to use this job, you must supply a secret in the following format: === - secret: name: <name of your secret> data: user: <ssh user of the remote git server> host: <address of the remote git … [View More]server> host_key: <ssh host key of the remote git server> ssh_key: <private key to authenticate with the remote git server> === The 'host_key' parameter can be retrieved from your known_hosts file or with a command like 'ssh-keyscan -H <host>' or 'ssh-keyscan -t rsa <host>'. For example, the 'host_key' when pushing to GitHub would be, on a single line: github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ== The 'ssh_key' parameter should be encrypted before being committed to the git repository. Zuul provides a tool for easily encrypting files such as SSH private keys and you can find more information about it in the documentation [2]. For example, encrypting a key for the "openstack/ara" project would look like this: === zuul/tools/encrypt_secret.py --infile /home/dmsimard/.ssh/ara_git_key --tenant openstack https://zuul.openstack.org openstack/ara === You would then use the secret in a job inheriting from 'upload-git-mirror' as such: === - job: name: <project>-upload-git-mirror parent: upload-git-mirror description: Mirrors openstack/<project> to neworg/<project> vars: git_mirror_repository: neworg/<project> secrets: - name: git_mirror_credentials secret: <name of your secret> pass-to-parent: true === Finally, the job must be set to run in your project's 'post' pipeline which is triggered every time a new commit is merged to the repository: === - project: check: jobs: # [...] gate: jobs: # [...] post: jobs: - <project>-upload-git-mirror === Note that the replication would only begin *after* the change has merged, meaning that merging the addition of the post job would not trigger the post job itself immediately. The post job will only trigger the next time that a commit is merged. [1]: https://zuul-ci.org/docs/zuul-jobs/general-roles.html#role-upload-git-mirror [2]: https://zuul-ci.org/docs/zuul/user/encryption.html David Moreau Simard dmsimard = [irc, github, twitter] [View Less]

1 0

Update on the tenant-scoped admin REST API for Zuul
by Matthieu Huin 11 Apr '19

11 Apr '19

Hello, I'd like to introduce the applications of the spec referred to in the title [1]. I have worked in parallel on a possible implementation of said spec, backend-side [2]. Since it is not necessarily trivial to evaluate or test by oneself, I have also tried to put together a proof of concept for the GUI using a keycloak instance as the JWT provider, with -almost- everything set up together with docker, so that people can play around with the feature. [3] It requires a few manual steps in … [View More]

1 0

Proposal to create zuul-operator project
by Monty Taylor 08 Apr '19

08 Apr '19

Hey everybody, It seems that kubernetes operators are all the rage these days, and in the last 2 weeks I've had multiple conversations that have all come back around to someone wanting a zuul operator to exist. SO ... I would like to propose that we create a zuul-operator project as part of Zuul. Given our close ties with Ansible, I think we should use ansible-operator [0] to write it. Since we have zuul in production on both vanilla k8s and on OpenShift, I think we should set up gate … [View More]

9 23

New driver for Nodepool
by Alexander Braverman Masis 03 Apr '19

03 Apr '19

Hello, I'm interested in writing a new driver for Nodepool based on Linchpin[1]. Should I contribute it to zuul core as a built in driver, or add a configuration option so the driver can be developed separately and run from out-of-tree? Thanks, [1] https://github.com/CentOS-PaaS-SIG/linchpin -- Alex Braverman Masis

2 1

PTG Team Photos
by Kendall Nelson 29 Mar '19

29 Mar '19

Hello! If your team is interested in having a team photo taken, here is the signup[1]! There are slots Thursday and Friday from 10:00 AM to 4:30 PM. The location is TBD but will likely be close to where registration will be. I'll send an email out the day before with a reminder of your time slot and an exact location. -Kendall (diablo_rojo) [1] https://docs.google.com/spreadsheets/d/1DgsRHVWW2YLv7ewfX0M21zWJRf4wUfPG4ff…

1 0

Zuul project update 2019-03-25
by corvus＠inaugust.com 25 Mar '19

25 Mar '19

Hi, Since the last update, Tristan has updated the zookeeper auth patch based on recent comments, rebased the zuul-runner patch on the multi-ansible stack, and Investigated how to enable the executor service to use a COE. I expect we'll have a good discussion about that once Tristan's findings are ready. Fabian worked on a Elasticsearch Zuul reporter to enable indexation of build and buildset data. It enable building analytics dashboards on top of Kibana for instance. I reviewed and helped … [View More]

1 0

nodepoolcmd + secure.conf
by Paul Belanger 25 Mar '19

25 Mar '19

Greetings, I wanted to discuss maybe reverting the following change[1]. On friday I updated our local nodepool.yaml configuration to move zookeeper-servers section into secure.conf. This was mostly to prepare for the upcoming patch adding SSL support for zookeeper. However, because 480310 removed the default value of the -s flag, I now find myself having to type the following everytime I want to interact with nodepool: $ nodepool -s /etc/nodepool/secure.conf list $ nodepool -s /etc/… [View More]

2 2

Speculative docker images
by corvus＠inaugust.com 22 Mar '19

22 Mar '19

Hi, As you know, with respect to git repositories, Zuul frequently operates well into the future, allowing jobs to build software with changes that haven't landed yet under the assurance that they will. As more of us have incorporated container image building into our processes, we find that we want the same thing there -- that not only would our git repositories represent the future that Zuul sees, but also our container image registries. As a concrete and simple example, we have an … [View More]integration job called 'zuul-quick-start' which uses both Nodepool and Zuul container images. Those images themselves depend on build images created from the openstack-infra/system-config repository, which they use via a "FROM" line in Dockerfiles, and extending the zuul-quick-start job to be able to rebuild and use those builder images in the case of a Depends-On change is too complex to consider. In particular, the "FROM" line is problematic, as the only way to realistically use a speculative image as a base image is to alter the Dockerfile or make the image available in a registry. Therefore, this is a proposal for a system for building speculative images which can then be used by further speculative images. Every item in a Zuul pipeline has its own future state which is related to, but separate from all the other items. It needs to be able to build an image for a project and store it in a registry under the final published name and tag, without affecting any of the changes ahead of it. Therefore, each item needs its own image registry. In the case of a single, simple job such as zuul-quick-start, we can just run a docker registry on the node on which the job is running. In the case of a more complicated system where we want to build an image once and use it in multiple dependent jobs in a job graph, we will start an image registry at the root of the job graph, pause the job, and the dependent jobs will use it. Now that we have the registry for the change, we need to have that registry reflect the state of any changes ahead. We need to know when the change ahead has finished building its images so that we can then add those images into our change's registry. To accomplish this, we can extend Zuul so that jobs have a provides/requires interface. If our job requires docker images produced by a job in a dependent change, we can say: - job: name: zuul-quick-start requires: docker-images - job: name: system-config-build-image-python-builder provides: docker-images Zuul will see that zuul-quick-start should not start until all of the jobs for all of the changes ahead which provide "docker-images" have completed. Then zuul-quick-start will be able to retrieve images from the preceding changes' jobs and add them to its own registry. In the gate, the behavior of this is obvious: if a change has a job with a requirement, iterate over all changes ahead of it in the shared queue and search for jobs on those changes which provide that requirement. If there are none, or all of them have completed or paused, then that job is ready to start. Otherwise, wait. In an independent pipeline, we do include dependent changes ahead in the shared queue, but they are not live -- they don't run any actual jobs. So in that case, we're going to have to peek outside our shared queue and see if our dependent changes show up anywhere else in the same pipeline, and if they do, wait for their jobs. If they don't show up, then we know they completed earlier, and it's safe to start. There's one more thing about this change that will help us later: a job that "provides" something should be able to pass data to a job which "requires" it. So our image build jobs can use zuul_return to provide information about which images they built so that it's easier for later jobs to import those images into their registries. Once again, the independent pipelines make this tricky, as since those jobs may have run some time in the past, the scheduler may have forgotten the data they returned. However, once the SQL database is required, we can rely on that to fill in any missing data. That's the only change we need to Zuul itself to support this system. The rest is all playbooks and roles. That brings us to the challenge of how to actually transfer these artifacts between items. I can think of two approaches. They are not mutually exclusive -- we can implement both if we want. They both have pros and cons. But I would like input from Zuul operators (including openstack-infra/opendev) as to which they would prefer, and I'll implement that approach in OpenDev for Zuul's own upstream use. If we have a split decision, someone else can implement the second approach. I think we can support both in zuul-jobs, but I'd like us to run the most widely applicable approach upstream. Alternative 1: Use the Logserver ================================ In the case of a simple job, at the end of the job, or in the case of a complex job graph, a child job dependent on the completion of the image build process (so we don't have to wait for all the jobs to complete), we will export the images to the filesystem and upload their layers (minus any layers which are already on Docker Hub, in order to save space) to the logserver. This next bit is a little complicated, but remember that this will all be encapsulated in roles so we only have to implement it once; the user interface will resemble "roles: [fetch-dependent-images]". At the end of the image build job, for every image that we built: Export it to a directory on the filesystem. For each layer in the image: Perform a HEAD request against the Docker Hub registry to see if the layer exists in the target repository. If that exists, layer exists in docker, remove from filesystem. rsync image directories to logserver. zuul_return image location on logserver. At the start of a job which requires images: For each previous image (as provided by zuul_return): Rsync image directory from logserver to filesystem. For each layer in image manifest: If layer is missing from filesystem: Get the layer from Docker Hub. Import image into local registry. The advantages of this approach: * No external operational dependencies (i.e., you don't have to run a docker registry). The disadvantages of this approach: * Images can't easily be fetched and used for debugging. * Implementation is slightly more complex. Alternative 2: Use a Central Registry ===================================== In this approach, we would run a central registry to store speculative images. It would not store final images at all, and every image uploaded to it would be tagged only with a tag like "pipeline_check_change_1234_original_tag" to uniquely identify its build context. At the end of the image build job, for every image that we built: Copy it from the per-change registry to the central registry, and tag it with pipeline and change id. zuul_return image location and tag. At the start of a job which requires images: For each previous image (as provided by zuul_return): Copy it from the central registry to the local registry, and tag it with the original tag. The advantages of this approach: * Developers can fetch images from the central registry by pulling them with the crazy tag. * Implementation is a little easier, and uses more standardized tools. The disadvantages of this approach: * An external service is required. * We will need to prune old images from the registry. Conclusion ========== I don't think this system will take long to implement with either of the alternatives. Please let me know what you think of the system, and of the storage/transfer alternatives. -Jim [View Less]

4 4

Parameterized Build
by Tristan Cacqueray 21 Mar '19

21 Mar '19

Hello, It seems like it would help the migration of Jenkins users if they could have a similar feature[0] in Zuul to manually trigger a job with custom parameters. One solution is to create a fake change or branch to trigger a job, but perhaps Zuul can offer a better user experience. In particular, the Jenkins implementation offers a nice form to set the parameters. I started to implement this feature with a jobtrigger driver[1] along with a web page[2] to expose the job's variables but … [View More]

3 9

Re: Zuul 3.7.0 released
by Jimmy McArthur 20 Mar '19

20 Mar '19

Congrats to the whole Zuul community :) > James E. Blair <mailto:corvus@inaugust.com> > March 20, 2019 at 3:33 PM > The Zuul team is pleased to announce the release of Zuul 3.7.0. > > Download it from PyPI: https://pypi.org/project/zuul > > The release notes, included below, are also available in the > documentation with hyperlink references: > > https://zuul-ci.org/docs/zuul/releasenotes.html#relnotes-3-7-0 > > Upgrade Notes > ============= > &… [View More]

1 0

Re: Migrating git repos to OpenDev
by corvus＠inaugust.com 20 Mar '19

20 Mar '19

corvus(a)inaugust.com (James E. Blair) writes: > Hi, > > As discussed in November[1], the OpenStack project infrastructure is > being rebranded as "OpenDev" to better support a wider community of > projects. > > We are nearly ready to perform the part of this transition with the > largest impact: moving the authoritative git repositories for existing > projects. Working with the folks representing each of the affected OSF projects, we've scheduled the migration for … [View More]

1 0

BoF/WG Space at Open Infrastructure Summit Denver
by Jimmy McArthur 19 Mar '19

19 Mar '19

Does the Zuul Community have any interest in BoF or Working Group space in Denver? For those unfamiliar, Working Groups are informal groups of people sharing a common interest for a task or code subsystem. Birds of a Feather is an effort to interest new community members to your project through a presentation and Q&A format. Each session type is 40 minutes. Cheers, Jimmy

4 11

Zuul Board Approval Presentation
by Jimmy McArthur 18 Mar '19

18 Mar '19

Hi All - I'm working with Zuul on their pending project confirmation on behalf of the OpenStack Foundation. As you may or may not know, Zuul will need to do a presentation in front of the OpenStack Foundation Board. I've started an etherpad [1] with a rough outline, but ultimately this should be a community driven effort. If you are able to fill in some blanks, we welcome your contribution! Cheers, Jimmy [1] https://etherpad.openstack.org/p/Zuul_2019_OSF_Project_Confirmation_Process

1 0

Zuul generic jobs
by Tomasz Szandała 14 Mar '19

14 Mar '19

Greetings, I have a project, that requires compilation for different targets. Same commands, but each one has to be done separately. I have seen how OpenStack does it: https://github.com/openstack-infra/openstack-zuul-jobs/blob/master/zuul.d/j… so we define dozens of jobs separately even if they differ only by single element. My idea was to define job compilation, in which it is Ansible playbook that determines which target it should choose, based on job name: (compilation.yaml - playbook) - … [View More]

7 9

Zuul project update 2019-03-11
by corvus＠inaugust.com 13 Mar '19

13 Mar '19

Hi, Since the last email update, we merged the Multi-Ansible spec: https://zuul-ci.org/docs/zuul/developer/specs/multiple-ansible-versions.html Tristan worked on adding a job filter to the trigger event, as discussed previously on this list. The webtrigger patch is now based on that. Tristan also continued working on zuul-runner and a set of patches to support nodes with python3 only. I finally finished the changes related to buildset docker registries (also mentioned previously on this … [View More]

5 5

404 - Tenant does not exist
by Tomasz Szandała 10 Mar '19

10 Mar '19

Greetings, I have set up zuul v3 using docker-compose, so on my host I can see following containers running: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b47be7107613 zuul/zuul-web "/usr/bin/dumb-init …" 12 hours ago Up 18 minutes 0.0.0.0:9000->9000/tcp docker_web_1 5b4fcd2a2238 zuul/zuul-executor "/usr/bin/dumb-init …" 12 hours ago Up 18 minutes … [View More]

1 1

Re: [all] [forum] Forum Submissions are open!
by Kendall Nelson 27 Feb '19

27 Feb '19

Hello :) On Wed, Feb 27, 2019 at 1:26 PM Chris Morgan <mihalis68(a)gmail.com> wrote: > I think the issue is that forum submissions building on what gets > discussed in Berlin can't be expected to be finalised whilst attendees to > the berlin meetup are still traveling. It's not that Erik can't pull these > things together, in fact he's an old hand at this, it's more that this > process isn't reasonable if there's so little time to collate what we learn > in Berlin and … [View More]feed it forward to Denver. Frankly it sounds like because the > planning committee needs 5 weeks, Erik can have two days. Seem unfair. > Honestly, the decision process doesn't take much time, aside from organizing a time that all 10 people can meet across x timezones (a thing unto itself). Its the community feedback period, giving people enough time to secure travel approval from their management, loading the sessions into the actual schedule app, and other print deadlines that force us to have everything set this far out. I will definitely help the ops community in whatever way I can! Do you have remote attendance set up for the meetup? > Chris > > On Wed, Feb 27, 2019 at 2:29 PM Kendall Nelson <kennelson11(a)gmail.com> > wrote: > >> Another- nother thought: You could take a look at what is submitted by >> project teams closer to the deadline and see if your ideas might fit well >> with theirs since they are looking for feedback from operators anyway. In >> the past I have always hoped for more engagement in the forum sessions I've >> submitted but only ever had one or two operators able to join us. >> >> -Kendall (diablo_rojo) >> >> On Wed, Feb 27, 2019 at 11:14 AM Kendall Nelson <kennelson11(a)gmail.com> >> wrote: >> >>> Hello :) >>> >>> On Wed, Feb 27, 2019 at 11:08 AM Jimmy McArthur <jimmy(a)openstack.org> >>> wrote: >>> >>>> Erik, >>>> >>>> I definitely understand the timeline is tight. One of the reasons that >>>> we publish the schedule so early is to enable community members to plan >>>> their schedule early, especially as there is more overlap with the main >>>> Summit Schedule in Denver. Additionally, travel approval is often >>>> predicated upon someone showing they're leading/moderating a session. >>>> >>>> Before publishing the schedule, we print a draft Forum schedule for >>>> community feedback and start promotion of the schedule, which we have to >>>> put up on the OpenStack website and apps at 5 weeks out. Extending the date >>>> beyond the 10th won't give the Forum Selection Committee enough time to >>>> complete those tasks. >>>> >>>> I think if the Ops team can come up with some high level discussion >>>> topics, we'll be happy to put some holds in the Forum schedule for >>>> Ops-specific content. diablo_rojo has also offered to attend some of the >>>> Ops sessions remotely as well, if that would help you all shape some things >>>> into actual sessions. >>>> >>> >>> I'm definitely happy to help as much as I can. If you'll have something >>> set up that I can call into (zoom, webex, bluejeans, hangout, whatever), I >>> definitely will. I could also read through etherpads you take notes in and >>> help summarize things into forum proposals. >>> >>> Another thing to note is that whatever you/we submit, it doesn't have to >>> be award winning :) Its totally possible to change session descriptions and >>> edit who the speaker is later. >>> >>> Other random thought, I know Sean McGinnis has attended a lot of the >>> Operators stuff in the past so maybe he could help narrow things down too? >>> Not to sign him up for more work, but I know he's written a forum propsal >>> or two in the past ;) >>> >>> >>>> >>>> I wish I could offer a further extension, but extending it another week >>>> would push too far into the process. >>>> >>>> Cheers, >>>> Jimmy >>>> >>>> Erik McCormick <emccormick(a)cirrusseven.com> >>>> February 27, 2019 at 12:43 PM >>>> >>>> Jimmy, >>>> >>>> I won't even get home until the 10th much less have time to follow up >>>> with anyone. The formation of those sessions often come from >>>> discussions spawned at the meetup and expanded upon later with folks >>>> who could not attend. Could we at least get until 3/17? I understand >>>> your desire to finalize the schedule, but 6 weeks out should be more >>>> than enough time, no? >>>> >>>> Thanks, >>>> Erik >>>> >>>> Jimmy McArthur <jimmy(a)openstack.org> >>>> February 27, 2019 at 12:04 PM >>>> >>>> Hi Erik, >>>> >>>> We are able to extend the deadline to 11:59PM Pacific, March 10th. >>>> That should give the weekend to get any additional stragglers in and still >>>> allow the Forum Programming Committee enough time to manage the rest of the >>>> approval and publishing process in time for people's travel needs, etc... >>>> >>>> For the Ops Meetup specifically, I'd suggest going a bit broader with >>>> the proposals and offering to fill in the blanks later. For example, if >>>> something comes up and everyone agrees it should go to the Forum, just >>>> submit before the end of the Ops session. Kendall or myself would be happy >>>> to help you add details a bit later in the process, should clarification be >>>> necessary. We typically have enough spots for the majority of proposed >>>> Forum sessions. That's not a guarantee, but food for thought. >>>> >>>> Cheers, >>>> Jimmy >>>> >>>> >>>> _______________________________________________ >>>> Airship-discuss mailing list >>>> Airship-discuss(a)lists.airshipit.org >>>> http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss >>>> >>>> Erik McCormick <emccormick(a)cirrusseven.com> >>>> February 27, 2019 at 11:31 AM >>>> Would it be possible to push the deadline back a couple weeks? I expect >>>> there to be a few session proposals that will come out of the Ops Meetup >>>> which ends the day before the deadline. It would be helpful to have a >>>> little time to organize and submit things afterwards. >>>> >>>> Thanks, >>>> Erik >>>> >>>> Jimmy McArthur <jimmy(a)openstack.org> >>>> February 27, 2019 at 10:40 AM >>>> Hi Everyone - >>>> >>>> A quick reminder that we are accepting Forum [1] submissions for the >>>> 2019 Open Infrastructure Summit in Denver [2]. Please submit your ideas >>>> through the Summit CFP tool [3] through March 8th. Don't forget to put >>>> your brainstorming etherpad up on the Denver Forum page [4]. >>>> >>>> This is not a classic conference track with speakers and presentations. >>>> OSF community members (participants in development teams, operators, >>>> working groups, SIGs, and other interested individuals) discuss the topics >>>> they want to cover and get alignment on and we welcome your participation. >>>> The Forum is your opportunity to help shape the development of future >>>> project releases. More information about the Forum [1]. >>>> >>>> If you have questions or concerns, please reach out to >>>> speakersupport(a)openstack.org. >>>> >>>> Cheers, >>>> Jimmy >>>> >>>> [1] https://wiki.openstack.org/wiki/Forum >>>> [2] https://www.openstack.org/summit/denver-2019/ >>>> [3] https://www.openstack.org/summit/denver-2019/call-for-presentations >>>> [4] https://wiki.openstack.org/wiki/Forum/Denver2019 >>>> ___________________________________________ >>>> >>>> >>> Hopefully that helps! >>> >>> -Kendall (diablo_rojo) >>> >> > > -- > Chris Morgan <mihalis68(a)gmail.com> > - Kendall Nelson (diablo_rojo) [View Less]

1 0

Re: [all] [forum] Forum Submissions are open!
by Jimmy McArthur 27 Feb '19

27 Feb '19

Erik, I definitely understand the timeline is tight. One of the reasons that we publish the schedule so early is to enable community members to plan their schedule early, especially as there is more overlap with the main Summit Schedule in Denver. Additionally, travel approval is often predicated upon someone showing they're leading/moderating a session. Before publishing the schedule, we print a draft Forum schedule for community feedback and start promotion of the schedule, which we … [View More]have to put up on the OpenStack website and apps at 5 weeks out. Extending the date beyond the 10th won't give the Forum Selection Committee enough time to complete those tasks. I think if the Ops team can come up with some high level discussion topics, we'll be happy to put some holds in the Forum schedule for Ops-specific content. diablo_rojo has also offered to attend some of the Ops sessions remotely as well, if that would help you all shape some things into actual sessions. I wish I could offer a further extension, but extending it another week would push too far into the process. Cheers, Jimmy > Erik McCormick <mailto:emccormick@cirrusseven.com> > February 27, 2019 at 12:43 PM > Jimmy, > > I won't even get home until the 10th much less have time to follow up > with anyone. The formation of those sessions often come from > discussions spawned at the meetup and expanded upon later with folks > who could not attend. Could we at least get until 3/17? I understand > your desire to finalize the schedule, but 6 weeks out should be more > than enough time, no? > > Thanks, > Erik > Jimmy McArthur <mailto:jimmy@openstack.org> > February 27, 2019 at 12:04 PM > Hi Erik, > > We are able to extend the deadline to 11:59PM Pacific, March 10th. > That should give the weekend to get any additional stragglers in and > still allow the Forum Programming Committee enough time to manage the > rest of the approval and publishing process in time for people's > travel needs, etc... > > For the Ops Meetup specifically, I'd suggest going a bit broader with > the proposals and offering to fill in the blanks later. For example, > if something comes up and everyone agrees it should go to the Forum, > just submit before the end of the Ops session. Kendall or myself > would be happy to help you add details a bit later in the process, > should clarification be necessary. We typically have enough spots for > the majority of proposed Forum sessions. That's not a guarantee, but > food for thought. > > Cheers, > Jimmy > > > _______________________________________________ > Airship-discuss mailing list > Airship-discuss(a)lists.airshipit.org > http://lists.airshipit.org/cgi-bin/mailman/listinfo/airship-discuss > Erik McCormick <mailto:emccormick@cirrusseven.com> > February 27, 2019 at 11:31 AM > Would it be possible to push the deadline back a couple weeks? I > expect there to be a few session proposals that will come out of the > Ops Meetup which ends the day before the deadline. It would be helpful > to have a little time to organize and submit things afterwards. > > Thanks, > Erik > > Jimmy McArthur <mailto:jimmy@openstack.org> > February 27, 2019 at 10:40 AM > Hi Everyone - > > A quick reminder that we are accepting Forum [1] submissions for the > 2019 Open Infrastructure Summit in Denver [2]. Please submit your > ideas through the Summit CFP tool [3] through March 8th. Don't forget > to put your brainstorming etherpad up on the Denver Forum page [4]. > > This is not a classic conference track with speakers and > presentations. OSF community members (participants in development > teams, operators, working groups, SIGs, and other interested > individuals) discuss the topics they want to cover and get alignment > on and we welcome your participation. The Forum is your opportunity > to help shape the development of future project releases. More > information about the Forum [1]. > > If you have questions or concerns, please reach out to > speakersupport(a)openstack.org <mailto:speakersupport@openstack.org>. > > Cheers, > Jimmy > > [1] https://wiki.openstack.org/wiki/Forum > [2] https://www.openstack.org/summit/denver-2019/ > [3] https://www.openstack.org/summit/denver-2019/call-for-presentations > [4] https://wiki.openstack.org/wiki/Forum/Denver2019 > ___________________________________________ > [View Less]

2 2

Re: [all] [forum] Forum Submissions are open!
by Jimmy McArthur 27 Feb '19

27 Feb '19

Hi Erik, We are able to extend the deadline to 11:59PM Pacific, March 10th. That should give the weekend to get any additional stragglers in and still allow the Forum Programming Committee enough time to manage the rest of the approval and publishing process in time for people's travel needs, etc... For the Ops Meetup specifically, I'd suggest going a bit broader with the proposals and offering to fill in the blanks later. For example, if something comes up and everyone agrees it … [View More]

1 0

[all] [forum] Forum Submissions are open!
by Jimmy McArthur 27 Feb '19

27 Feb '19

Hi Everyone - A quick reminder that we are accepting Forum [1] submissions for the 2019 Open Infrastructure Summit in Denver [2]. Please submit your ideas through the Summit CFP tool [3] through March 8th. Don't forget to put your brainstorming etherpad up on the Denver Forum page [4]. This is not a classic conference track with speakers and presentations. OSF community members (participants in development teams, operators, working groups, SIGs, and other interested individuals) … [View More]

1 0

OpenInfra Day Vietnam 2019 - pre-CFP
by Trinh Nguyen 27 Feb '19

27 Feb '19

Hello, This is the pre-calling for presentations at the OpenInfra Day in Vietnam this year. If you love to visit Hanoi <https://en.wikipedia.org/wiki/Hanoi>, the capital of Vietnam, and share your passion for the Open Infrastructure of any topic (container, CI, deployment, edge computing, etc.), please let me know by replying to this email. Below is the tentative information of the event: - Date: 31 August 2019 - Location: Hanoi, Vietnam We are working with the OpenStack Foundation to … [View More]

1 0

Zuul project update 2019-02-25
by corvus＠inaugust.com 25 Feb '19

25 Feb '19

Matthieu is working on authenticated web API, and thus enabling tenant- or project-scoped privileged actions. The spec needs feedback, especially the authorization definition in Zuul: https://review.openstack.org/562321 There is also a review topic with WIP implementation of the current state of the spec: https://review.openstack.org/#/q/topic:zuul_admin_web Tristan worked on web and AMQP triggers: https://review.openstack.org/#/q/topic:webtrigger https://review.openstack.org/#/q/… [View More]

1 0

Escaping text in Zuul inventory file
by corvus＠inaugust.com 24 Feb '19

24 Feb '19

Hi, Recently Zuul was modified to include the change message (i.e., commit message in Gerrit, PR message in GitHub) as a Zuul variable: https://review.openstack.org/631207 Ansible applies jinja templating to the inventory file (which we rely on for variables in many jobs). If the change message incidentally included jinja escapes, then Ansible may fail to run due to syntax errors or undefined variables. To correct this, we agreed that allowing templating of the change message was not … [View More]useful and should be disabled. An initial approach using "{% raw %}" / "{% endraw %}" was deemed too fragile (a change message with "{% endraw %}" in it would defeat the fix). So we have merged this change which uses an Ansible-specific YAML tag, "!unsafe", to mark the string as untrusted input which should not be templated: https://review.openstack.org/633930 This appears to be an elegant and complete solution. Yet before we release the next version of Zuul with this, I think we should consider possible ramifications a bit more. I can think of two things that including YAML tags in the inventory *might* cause problems with, and I'd like input on whether folks think these could be actual problems, or are, instead, easily overcome. And of course, if you think of other potential problems. First, the YAML tag may make it more difficult for users who perform automatic processing of inventory files collected after the job runs. I know of know examples of that myself, and it seems like it should be safe for us to say that if you're going to parse an Ansible YAML inventory file, you should be using a YAML parser that understands Ansible YAML tags. But perhaps someone knows of why this would be an undue hardship. The second is really a special case of the first. As our build page in the Zuul web UI continues to become more sophisticated, we might want to have it parse the inventory file much as it is now currently parsing the job-output.json file. I don't recall any discussions of this, but if we think it's likely, we should probably make sure that any Javascript YAML parser we might use could handle the tag. If we think that these issues, or any others that folks may think of, make it worth avoiding the !unsafe tag, we could solve the problem by base64 encoding the change message. Ansible has convenient jinja filters for base64 decoding, and in fact, Ansible modules such as slurp provide data in base64-encoded form. It would be easy for a job which needed to inspect the change message to do so by running it through the b64decode filter first. However, if we think these issues are not blockers, the !unsafe tag presents the simplest and most human-readable approach. -Jim [View Less]

6 7

[all] Forum Submissions are Open!
by Kendall Nelson 22 Feb '19

22 Feb '19

Hello Everyone! We are now accepting Forum [1] submissions for the 2019 Open Infrastructure Summit in Denver [2]. Please submit your ideas through the Summit CFP tool [3] through March 8th. Don't forget to put your brainstorming etherpad up on the Denver Forum page [4]. This is not a classic conference track with speakers and presentations. OSF community members (participants in development teams, operators, working groups, SIGs, and other interested individuals) discuss the topics they … [View More]

1 0

Using add-build-sshkey with static node
by Tristan Cacqueray 21 Feb '19

21 Feb '19

Hello, The add-build-sshkey isn't great with static node because the key added to the authorized_keys isn't removed if the remove-build-sshkey doesn't run. This results in key being leaked in authorized_keys without bound. Thus I proposed to add an option to cleanup keys before adding a new one: https://review.openstack.org/632620 . However this isn't great for static node that can be used by more than one job concurrently (multi-slot). Though, multi-slot already needs special trick to … [View More]

1 0

Zuul talks at Open Infrastructure Summit Denver in April
by Claire Massey 20 Feb '19

20 Feb '19

Hey Zuul community, The agenda is now live for the Open Infrastructure Summit and there are quite a few Zuul talks - congratulations to all of the speakers! View the agenda for the Open Infrastructure Summit Denver <https://www.openstack.org/summit/denver-2019/summit-schedule#day=2019-04-29>. Zuul sessions include: Moving your testing infrastructure from Jenkins to Zuul Unleashing the power of Fabric: Orchestrating new performance features for SR-IOV VNFs Getting a Neural Network Up … [View More]

1 0

Zuul/Nodepool tracing
by Simon.Westphahl＠bmw.de 20 Feb '19

20 Feb '19

Hi, recently I was getting sick of grepping through the logs of scheduler, executor and nodepool to debug an issue. My main problem was that there was no single identifier that I could use to find all logs that are related to a specific change. You have to look for the change/PR to find the matching 'Change' object. Then use the found object id to continue searching for the node request id, and so on. I decided to try to add tracing[0] to Zuul, as I was hoping that would simplify my use-case.… [View More]

4 4

[all] Denver Forum Brainstorming
by Kendall Nelson 20 Feb '19

20 Feb '19

Hello Everyone! Welcome to the topic selection process for our Forum in Denver! This is not a classic conference track with speakers and presentations. OSF community members (participants in development teams, operators, working groups, SIGs, and other interested individuals) discuss the topics they want to cover and get alignment on and we welcome your participation. The Forum is your opportunity to help shape the development of future project releases. For OpenStack Denver marks the … [View More]beginning of Train’s release cycle, where ideas and requirements will be gathered. We should come armed with feedback from the upcoming Stein release if at all possible. We aim to ensure the broadest coverage of topics that will allow for multiple parts of the community getting together to discuss key areas within our community/projects. For OSF Projects (StarlingX, Zuul, Airship, Kata Containers) As a refresher, the idea is to gather ideas and requirements for your project’s upcoming release. Look to https://wiki.openstack.org/wiki/Forum for an idea of how to structure fishbowls and discussions for your project. The idea is to ensure the broadest coverage of topics, while allowing for the project community to discuss critical areas of concern. To make sure we are presenting the best topics for discussion, we have asked representatives of each of your projects to help us out in the Forum selection process. There are two stages to the brainstorming: 1.If you haven’t already, its encouraged that you set up an etherpad with your team and start discussing ideas you'd like to talk about at the Forum and work out which ones to submit. 2. On the 22nd of February, we will open up a more formal web-based tool for you to submit abstracts for the most popular sessions that came out of your brainstorming. Make an etherpad and add it to the list at: https://wiki.openstack.org/wiki/Forum/Denver2018 <https://wiki.openstack.org/wiki/Forum/Berlin2018> This is your opportunity to think outside the box and talk with other projects, groups, and individuals that you might not see during Summit sessions. Look for interested parties to collaborate with and share your ideas. Examples of typical sessions that make for a great Forum: - Strategic, whole-of-community discussions, to think about the big picture, including beyond just one release cycle and new technologies - eg Making OpenStack One Platform for containers/VMs/Bare Metal (Strategic session) the entire community congregates to share opinions on how to make OpenStack achieve its integration engine goal - Cross-project sessions, in a similar vein to what has happened at past forums, but with increased emphasis on issues that are of relevant to all areas of the community - eg Rolling Upgrades at Scale (Cross-Project session) – the Large Deployments Team collaborates with Nova, Cinder and Keystone to tackle issues that come up with rolling upgrades when there’s a large number of machines. - Project-specific sessions, where community members most interested in a specific project can discuss their experience with the project over the last release and provide feedback, collaborate on priorities, and present or generate 'blue sky' ideas for the next release - eg Neutron Pain Points (Project-Specific session) – Co-organized by neutron developers and users. Neutron developers bring some specific questions about implementation and usage. Neutron users bring feedback from the latest release. All community members interested in Neutron discuss ideas about the future. Think about what kind of session ideas might end up as: Project-specific, cross-project or strategic/whole-of-community discussions. There'll be more slots for the latter two, so do try and think outside the box! This part of the process is where we gather broad community consensus - in theory the second part is just about fitting in as many of the good ideas into the schedule as we can. Further details about the forum can be found at: https://wiki.openstack.org/wiki/Forum Thanks all! Kendall Nelson, on behalf of the OpenStack Foundation, User Committee & Technical Committee [View Less]

1 0

Job at periodic pipeline with "files" section
by Felix Enrique Llorente Pastora 19 Feb '19

19 Feb '19

Hi All, Here at tripleo CI we have try to run a tripleo job with "files" section from parent and it does not run at periodic, after doing some testing with zuul quickstart looks like it does not work, kind of make sense. Have open a bug here https://storyboard.openstack.org/#!/story/2005040, but don't know if it make sense to fix this. BR -- Quique Llorente Openstack TripleO CI

2 4

zuul-preview - a helper service for JAMstack preview sites
by Monty Taylor 14 Feb '19

14 Feb '19

Heya, One of the cool things we do in Zuul jobs are our static html preview sites. We use this for zuul-dashboard preview, as well as storyboard, docs builds and others. Recently some of us in OpenDev land have been working with the OpenStack Foundation folks to support using netlify-cms with gerrit, which lets people who would prefer to edit web content via vim/emacs via git collaborate with those who would prefer to do such a thing in a visual CMS-like manner. It's really cool - … [View More]

8 13

Zuul-config reload
by Tomasz Szandała 13 Feb '19

13 Feb '19

Greetings, how can I reload main configuration repository? It seems like my commits to Gerrit's zuul-config still requires to restart zuul's docker (I am using docker-compose setup). Kind regards, Tomasz

2 3

An URLTrigger driver for Zuul
by Fabien Boucher 11 Feb '19

11 Feb '19

Hello, I would like to propose a solution to trigger jobs based on the state of an external resource accessible at an URL. The use case is quite common and actually Jenkins propose this kind of feature via the URLTrigger plugin [0]. I proposed two implementations of an URLTrigger driver for Zuul: - A trigger driver [1]: It is quite similar to the timer driver. The CronTrigger is used to provide the resource check rate. At each loop it checks the resource state and keep the state in a cache. … [View More]

3 3

Zuul project update 2019-02-11
by corvus＠inaugust.com 11 Feb '19

11 Feb '19

Hi, Tristan has: Worked on config introspection and build a dashboard to list configuration relationship and identify unused items: https://review.openstack.org/#/q/topic:config-introspection Worked on displaying the job's hierarchy: https://review.openstack.org/#/q/topic:jobs-hierarchy Worked on improving the build/buildset zuul-web interface: https://review.openstack.org/#/q/topic:web-build Paul has been working on allowing job.run to be a list of playbooks: … [View More]

1 0

[all] Denver Open Infrastructure Summit Community Contributor Awards!
by Kendall Nelson 06 Feb '19

06 Feb '19

Hello Everyone! As we approach the Summit (still a ways away thankfully), its time to kick off the Community Contributor Award nominations[1]! For those of you that have never heard of the CCA, I'll briefly explain what they are :) We all know people in our communities that do the dirty jobs, we all know people that will bend over backwards trying to help someone new, we all know someone that is a savant in some area of the code we could never hope to understand. These people rarely get the … [View More]

1 0

Zuul 2019 planning
by Lauren Sell 05 Feb '19

05 Feb '19

Hi everyone, We’d like to work with the Zuul community on 2019 planning so we can best support and promote the project this year. If the mailing list is the best way to start the conversation, we can use the below etherpad, or have a discussion on this thread. We’re also happy to host a quick call if folks want to brainstorm in a high bandwidth setting. The main things we’re looking to document are the goals for the community this year (for example, attract more contributors or document more … [View More]

2 2

Superuser Award nominations open until March 22
by Claire Massey 04 Feb '19

04 Feb '19

Hey everyone The Superuser Award nominations are now open for the Denver Open Infrastructure Summit. March 22 is the deadline for submissions. The Superuser Awards recognize teams using open infrastructure to meaningfully improve business and differentiate in a competitive industry, while also contributing back to the open-source community. The awards will be given out during the Denver Summit. *This is a great opportunity to nominate companies that are using Zuul!* You can learn more … [View More]

1 0

Open Infrastructure Summit Denver - Community Voting Open
by Claire Massey 31 Jan '19

31 Jan '19

Hi everyone, Community voting for the Open Infrastructure Summit Denver CFP sessions is open! You can VOTE HERE <https://www.openstack.org/summit/denver-2019/vote-for-speakers>, but what does that mean? Now that the Call for Presentations (CFP) has closed, all submissions are available for community vote and input. After community voting closes, the volunteer Programming Committee members will receive the presentations to review and determine the final selections for Summit schedule. … [View More]

1 0

Using nodepool v3 with Jenkins
by Darragh Bailey 31 Jan '19

31 Jan '19

Hi, Hoping that we can make use of nodepool v3 with Jenkins to solve some of our existing problems, and at the same time move a bit closer to upgrading from zuul v2 -> v3. Going to outline our current problem, my initial thinking that resulted in some of my questions last night in IRC and some further thinking on it that may provide a way forward for us. Hoping that someone can chime in and identify if this is realistic or if we're heading down the wrong path before getting to far. Keep … [View More]in mind that I'm only starting with nodepool, so I'm unsure if I even understand some of the concepts. e.g. does nodepool get a request for a particular type of node or is it infered by there being none of that type available in the queues in zookeeper? Current problem: We have some resources that are limited and are not directly accessible as workers for Jenkins, and so need a way to manage the allocation of resources to certain jobs. Some of these resources are provided by teams and some are from a set of resources managed by the CI team. So we probably have ~4 sets of these types of resources. At the moment this is done using some (too much) hackier with groovy, the Global Event Listener Plugin (GELP), and Lockable Resources plugin. The GELP allows us to hook into various points in Jenkins, such as when the job is scheduled but not yet running Jenkins (QUEUED). Unfortunately there doesn't appear to be any way with the gearman protocol to inspect what job is going to be assigned before it is picked up, so we can only do manipulation after the job has been scheduled and before it starts running. The data required to provide the job with the details required to use the limited resource is retrieved from vault and injected right before the job gets fully launched on Jenkins. This is done by creating lockable resource entries dynamically based on the number of the resources available, and then catching the event on when Jenkins allows the job needing this to start and injecting the required data into the environment. Unfortunately this means the queuing for the jobs requiring these limited resources occurs within Jenkins instead of Zuul because the job has still be scheduled even if it is not yet running. We've run into a number of problems resulting in workers getting allocated for jobs that don't get to start, and taking them away from jobs that could be run, and effectively lightly DOSing the system. Initial thoughts: So enter nodepool and the hope that if we move the management of both these resources and the workers out of Jenkins and into something a little more flexible, it can prevent the job from getting scheduled in the first place if not enough of the limited resources are available, and allow jobs that do not need these to be scheduled because a worker isn't getting assigned to the job that is queued waiting. Initially I was thinking of having two resources, making the one representing the limited resource a fake resource of some kind that Jenkins doesn't connect to, but as it appears that it's a requirement of the Jenkins Nodepool agent that nodes returned must be accessible to Jenkins, that doesn't seem to fly. Next idea was whether it would be possible to have a static pool and a another driver that could grab a node from the static pool if a request for the limited resource node type came in, which prompted some of my questions in IRC last night. Further thinking done: After spending a bit more time with the static nodepool driver, and depending on how it works (there are lots of gaps in my understanding), I'm wondering if what we need is to create a custom driver and place all of our static nodes in a internal pool inside of this driver. Additionally query our vault instance for a list of these limited resource groups and build some pools of those internally as well. And assign labels as part of the requirements for the jobs using these limited resources. Then instead of immediate registering all of the nodes with zookeeper, instead depending on the node label request that comes in, populate zookeeper with an entry containing suitable labels. If it is one requiring a limited resource identified by a name such as teamA-resources, the driver code can test if there are any not in use and if so create a node entry in zookeeper that contains labels "teamA-resources" and "teamA-resource-5". The second label can be used in Jenkins by our GELP code to determine which set of data to inject into the environment of the job scheduled on that node. Obviously there is still some hackier in there, but does a custom driver building on the idea of combining some of the ideas of the static driver along with the dynamic drivers where instead of launching a new node, it simply creates a node entry on demand using both a static node and details on what resource is required? Possibly I'm misunderstanding how nodepool works in this case, and maybe it just populates a queue in zookeeper and tries to keep some in there, which may mean that what needs to happen is that one entry allocating a node to each of the sets of limited resources needs to be added as entries to zookeeper queues by the driver with further entries added if there are enough resources available as those queues are consumed? Which is a bit different, but seems like it would still work to limit the amount of damage where multiple jobs were launched and required more of the limited resource than was available. Currently can end up seeing 3 queue instances for a single limited resource queued up in Jenkins, so as we migrate another case were this will be needed we could end up seeing a queue of 9-10. Seems nodepool might limit it to only losing one per team, until we can migrate to zuulv3 and use some of the mutex capabilities to handle this a bit better. Thoughts? Is this the best way to dig up as not digging isn't an option here... -- Darragh Bailey "Nothing is foolproof to a sufficiently talented fool" [View Less]

2 4

Zuul jobs for container publishing
by corvus＠inaugust.com 31 Jan '19

31 Jan '19

Hi, We recently added a new pipeline to OpenStack's Zuul, and three new jobs and roles to the zuul-jobs repo to support container publishing. Zuul users may be interested in adopting both the pipeline and the jobs into their own workflows. The new pipeline is the "promote" pipeline. It runs jobs on changes after they merge, but unlike the "post" pipeline, it does not run with the resulting branch state after the merge -- it runs in the context of the change. This means that it's not … [View More]suitable for building release artifacts, but that's not its purpose. Its purpose is to promote artifacts previously built in the gate pipeline to production. That is safe to do because anything built in the gate pipeline is built with the full future state of in-flight changes to all related projects and branches (even if it's built before they land). We build a lot of artifacts in the gate pipeline now, and then throw them away only to build them again in post. The promote pipeline is an attempt to avoid that waste and instead preserve artifacts built in gate and directly publish them -- but only if and when the changes for which they were built are merged. The first functional use of this system is in building container images and publishing them on Docker Hub. We have begun using this in openstack-infra to build container images of third-party applications, but this will work just as well and as easily for building images of applications in our infrastructure (for example, we are also starting to use this to publish Zuul images). To implement this, we wrote three jobs which are available now from zuul-jobs (and therefore can be used in any project or job): * build-docker-image [1] * upload-docker-image [2] * promote-docker-image [3] The first job should run in the check pipeline and simply runs "docker build" on the arguments passed to it. It is used to verify that the build succeeds. The second job should run in the gate pipeline. It uploads the image to Docker Hub into the final repository location, but it only does so with a single tag with the form "change_123456" where '123456' is the Gerrit change number for the change under consideration. This will mean that docker pulls won't accidentally fetch the image yet. But the image is staged in Docker Hub and is ready to be promoted if the change passes all its tests and merges. The third job runs in the promote pipeline. This job, unlike the others, only performs Docker Hub API calls and requires no build resources. Therefore it is a zero-node job in Zuul -- a job which only runs on the Zuul executor. This job finds a previously uploaded image with a "change_" tag, and re-tags it with any specified tags ("latest" by default, but you can also tag version numbers or anything else). It also cleans up unused "change_" tags so they don't clutter up the repository. The nice thing about zero-node jobs is that they start and finish very quickly (since they don't have to deal with resource allocation and contention). With this system, an image will be fully published within about a minute after the change merges. A complete example .zuul.yaml might look like this: - secret: name: myproject-dockerhub data: username: dockerhubuser password: !encrypted/pkcs1-oaep - DFlbrDM5eUMptMGIVMXV1g455xOJLi92UYF08Z2/JlIGu3t6v052o9FKlVyj1ZmpXs5+2 JTa5jHkLTvTsYs9fCaNcQc2nmViCyWNlbOMzjB17uiZOaYFNs1sMqZcUZbGEz7Y8ds6Qq NBXI10jWFPTah4QxUuBvUbT3vmjnUToCzexl5ZGhKgijcnROWfUsnlCdugpgoNIcPsUki zty5FotDihnrC8n8vIomVK6EClY38ty97pLrADzFDd+Cos/OUlvi2xooUhzx8Bn020rJA lqEU5v8LGXp5QkHx0MSDx6JY6KppJ/4p/yM+4By6l+A20zdcimxmgiNc9rMWPwDj7xsao m7NAZWmWqOO0Xkhgt6WOfugwgt9X46sgs2+yDEfbnI5ok8uRbAB/4FWj/KdpyXwhcf+O2 wEfhxLwDbAoGONQPjb4YcZmCXtmR7Qe5t+n2jyczWXvrbaBDUQP5a+YtVNN/xhmQ7D740 POlxv7bLxJAixzqaQ3d8Rz9ZEv6zzRuhWph32UQtZ1JxSNww+EvmXm2eEi2Q2z6pT1Cx/ j2OrFyA2GL/UJOVb15VHKF6bgHPHWJtpjPFhqdcvBhVute4BWB+KPcWH+y+apHN1enK3H tNJO9iqm34nKwSuj5ExmFw50LtwR5/9FyRuRPq/vBL+8y82v8FDmeYsBeobn5M= - job: name: myproject-build-image parent: build-docker-image description: Build Docker images. vars: &image_vars docker_images: - context: . repository: myproject/imagename - job: name: myproject-upload-image parent: upload-docker-image description: Build Docker images and upload to Docker Hub. secrets: - name: docker_credentials secret: myproject-dockerhub pass-to-parent: true vars: *image_vars - job: name: myproject-promote-image parent: promote-docker-image description: Promote previously uploaded Docker images. secrets: - name: docker_credentials secret: myproject-dockerhub pass-to-parent: true vars: *image_vars - project: check: jobs: - myproject-build-image gate: jobs: - myproject-upload-image promote: jobs: - myproject-promote-image And the pipeline definition looks like: - pipeline: name: promote description: | This pipeline runs jobs that operate after each change is merged in order to promote artifacts generated in the gate pipeline. manager: supercedent precedence: low post-review: True trigger: gerrit: - event: change-merged success: gerrit: {} mysql: failure: gerrit: {} mysql: If you find that you need jobs which behave slightly differently, you may be able to inherit from these jobs and add pre or post playbooks. Or if you need something particularly complicated, you can re-implement the jobs using the underlying roles in zuul-jobs, which have the same names an the jobs which invoke them. [1] https://zuul-ci.org/docs/zuul-jobs/jobs.html#job-build-docker-image [2] https://zuul-ci.org/docs/zuul-jobs/jobs.html#job-upload-docker-image [3] https://zuul-ci.org/docs/zuul-jobs/jobs.html#job-promote-docker-image [4] https://zuul-ci.org/docs/zuul/user/encryption.html -Jim [View Less]

2 1

[summit] Open Infrastructure Summit Shanghai - Dates and Location
by Claire Massey 31 Jan '19

31 Jan '19

Hello Zuul team - We are excited to announce that following Denver, the Open Infrastructure Summit will be traveling to Shanghai, China the week of November 4, 2019. The Summit will be held at the Shanghai Expo Centre. Registration will be available soon. Sign up for updates[1] and reach out to summit(a)openstack.org with any questions. [1] https://www.openstack.org/summit/shanghai-2019 Cheers, Claire

1 0

Zuul project update 2019-01-28
by corvus＠inaugust.com 28 Jan '19

28 Jan '19

Hi, Over the past two weeks, Tristan has been working on the topic:freeze_job and got the "zuul-runner execute" command to run a zuul-job locally, as well as a buildset endpoint and a buildset page to show all the build result for a change. I've been working on image building jobs, and will continue to work on improvements to those this week (see two recent emails to list). Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email -Jim

1 0

Configurable autohold list of allowed build results.
by Gabriele Cerami 24 Jan '19

24 Jan '19

Hi, autohold currently allows users to examine a live environment to proceed with post-mortem examination. In some occasions, trying to find the root cause of a bug by just examining the logs and the timeline of the failed environment may prove too difficult, and the preferable way may be to catch the failure at its root exactly when the root cause happens. In a mechanism similar to breakpoints, we may want to stop the build at a certain point, the proceed manually to prod for weak spots. The … [View More]

2 1

CFP Open Until January 23, Open Infrastructure Summit in Denver
by Claire Massey 22 Jan '19

22 Jan '19

Hi everyone, FYI - the CFP is now open for the first Open Infrastructure Summit <https://www.openstack.org/summit/denver-2019/> (formerly the OpenStack Summit) which will be held in Denver, Colorado April 29 - May 1, 2019. Wednesday, *January 23* is the deadline to Submit presentations <https://www.openstack.org/summit/denver-2019/call-for-presentations>. The Open Infrastructure Summit is organized by OSF and designed to be a place where open source infrastructure communities … [View More]

1 1

Two new Zuul ideas pass-to-parent: true and named project groups
by Monty Taylor 21 Jan '19

21 Jan '19

Hey all! I had an idea for a zuul feature to make the containers jobs (and jobs like them) easier. And then one that also came to mind to make some of our central jobs easier to deal with in openstack. First idea: Right now, secrets, for good reason, are only usable by jobs in their own repo. However, for use-cases like the containers job, or publishing to pypi, or readthedocs even - it's hard to have a job in zuul-jobs that is usable for doing those tasks because they all need secrets. … [View More] The idea is to add a flag to the secret request, like this: - job: name: system-config-build-image parent: build-container-image abstract: true allowed-projects: openstack-infra/system-config - job: name: system-config-upload-image parent: system-config-build-image secrets: name: credentials secret: system-config-dockerhub allowed-projects: openstack-infra/system-config secrets: name: credentials secret: system-config-dockerhub pass-to-parent: true By passing the flag, we tell zuul to pass the secret variable file to every playbook in the inheritance chain above the job (but still not child jobs). Obviously this requires that the user of the job trust the parent job. In fact, maybe we even add a "warning" - so that if someone submits a job config to zuul with a pass-to-parent flag, it leaves an inline comment on the change (without a negative vote) warning about the implications of choosing to do that - that way reviewers can be sure they understand that they're trusting someone upstream from them. Second idea: Somewhere - maybe in config repo - but maybe in main.yaml - or maybe it's even just a normal global object - we allow for the creation of arbitrary named groups of projects. So - for instance, if we go for config-projects, something like: - project-group: name: openstack projects: - openstack/nova - openstack/swift - project-group: name: opendev projects: - openstack-infra/system-config - openstack-infra/bindep Then, we allow project-group entries to be used in allowed-projects, like this: - job: name: publish-openstack-sphinx-docs parent: publish-openstack-sphinx-docs-base final: true pre-run: post-run: - playbooks/sphinx/post.yaml allowed-projects: - project-group: openstack This would allow people (say AJaeger, or whoever) to maintain a list of "official" projects that can publish docs to docs.openstack.org ... but let any of those projects add the job, so that we didn't have to put the jobs into project pipelines in project-config. We could do a similar thing for container jobs. Say put this in system-config: - job: name: opendev-upload-image parent: opendev-build-image secrets: name: credentials secret: opendev-dockerhub allowed-projects: - project-group: opendev Then in bindep we have: - job: name: bindep-upload-image parent: opendev-upload-image final: true vars: &jinja-init_vars images: - target: bindepo repository: opendevorg/bindep The more I type things, the more I think there's no need to restrict the creation of named project groups. I could see openstack projects, like say kolla, wanting to create a project-group in one of their repos, so that they could put a secret into a job in a central repo for their dockerhub org, but use the job they make in each of their individual repos. Some of this could be done with tenants - but those are a bit more heavy-weight - and we already see that inside of OpenStack there are teams with many repos. There's also some similarity in problem solved between the two - both make for better reuse of jobs that needs privileges. But I think it's two different use-cases. One is about facilitating writing jobs in zuul-jobs (or other generic job locations) that need secrets to be passed to them but are otherwise generic and parameterizable. The second is about having discreet jobs that have specific secrets that should only be used in a subset of projects - but where it's possible that a given subset of projects might get access to a group of discreet credentialed jobs and managing membership in the group makes more logical sense than managing project-pipeline assignment. It also allows for having child-jobs of discreet credentialed jobs (like the container example) Anywho - what do people think? Monty [View Less]

4 4

Infra team upgrading review.openstack.org Gerrit from 2.13.9 to 2.13.12 January 18 at about 1700UTC
by Clark Boylan 17 Jan '19

17 Jan '19

We will be performing a minor Gerrit upgrade to version 2.13.12 tomorrow (January 18, 2019) at 1700UTC. We've tested this upgrade on our dev server, https://review-dev.openstack.org, and expect it to be a quick upgrade. Any outage shouldn't last more than 10 minutes. We will let our configuration management tooling manage the upgrade so we won't have an exact time, but will try to get it as close to 1700UTC as possible. Feel free to test out the new version on the dev server. We are happy to … [View More]

1 0

Discrepancies in zuul-jobs
by Matthieu Huin 17 Jan '19

17 Jan '19

Hello, There seems to be a few discrepancies in the way some roles are implemented in zuul-jobs. The main problems I have encountered are: [1] discrepancies in dependencies management: for example the upload-to-pypi job assumes twine is installed on the build host, and will fail if twine is missing; whereas the npm-related jobs install nodejs and npm in the playbooks. [2] discrepancies in OS support: some of the npm-related jobs assume they're running on Debian or Ubuntu (explicit call to apt … [View More]

5 6

Zuul featured in Open Infrastructure Community Newsletter
by Claire Massey 17 Jan '19

17 Jan '19

Hi everyone, Today, OSF distributed the next Open Infrastructure Community Newsletter. The bi-weekly newsletters provide a digest of the latest developments and activities across open infrastructure projects, events and users. This week, we highlighted the latest with *Zuul* as well as a brief update about the Open Infrastructure Summit and additional OSF project news. You can checkout the full newsletter on Superuser [1], and if you are interested in receiving the upcoming newsletters, you … [View More]

1 0

Is it possible to support merge-mode: rebase in zuul merger?
by Yang, Feng 3. (NSB - CN/Hangzhou) 15 Jan '19

15 Jan '19

Hello Zuul experts, Happy New Year! I didn’t find clues in the historical mail list. So please allow me to ask here. We are using zuul v2 for ~2 years, recently we encountered a special case: merged codes in zuul merger is NOT the same with gerrit rebase codes. On zuul side, we are using default mode ‘git merge -s resolve’ On Gerrit repo side, we are using “rebase if necessary” strategy. Then we are thinking if it is possible to add a new merge-mode: rebase into zuul merger. Did you plan to … [View More]

4 4

Zuul project update 2019-01-14
by corvus＠inaugust.com 15 Jan '19

15 Jan '19

Hi, Today I approved our formal governance document[1], after receiving votes from all of the current maintainers. Thanks, everyone, for helping to create that! David is working on nodepool image build timeouts[2]. We made a Zuul release last week, and are about to make a Nodepool release. Please help write next week's email by updating this etherpad: https://etherpad.openstack.org/p/zuul-update-email [1] https://zuul-ci.org/docs/zuul/governance.html [2] https://review.openstack.org/629923 -Jim

1 0

Zubbi 1.0.0 released
by Felix.FS.Schmidt＠bmw.de 14 Jan '19

14 Jan '19

Hello, With the start of the new year we are happy to announce that Zubbi 1.0.0 is now available on PyPI: https://pypi.org/project/zubbi/. The new Zubbi version provides support for Gerrit repositories. As opposed to GitHub, Zubbi yet doesn't support event handling for Gerrit. Thus, Gerrit repositories can only be scraped periodically. During the OpenStack summit, the question for a test instance came up. So we deployed a Zubbi demo showing the jobs and roles from the openstack-infra/zuul-… [View More]

1 0

Zuul project update 2019-01-07
by corvus＠inaugust.com 08 Jan '19

08 Jan '19

Hi, Welcome back from the holidays! Fabian has been working on a driver for Pagure (https://pagure.io/ ) and now has a usable implementation here: https://review.openstack.org/#/c/604404/ I have continued work on artifact URL storage in the db in service of building a 'promote' pipeline; I should be able to have jobs and roles demonstrating it soon. I'd like to call everyone's attention to three items in review with wide-ranging impact; it would be good for as many folks as possible to … [View More]

2 1

AW: AW: Different queues across branches?
by Mike Pagel 29 Dec '18

29 Dec '18

Happy holidays to everyone! I wrote: > In that case however, having multiple queue names is an > erroneous condition. Just wanted to wrap this up. I realize that raising an exception (or similar behavior) to complain about this condition will break existing installations, which is not acceptable. In particular not for this kind of thing. Nevertheless the discussion helped me to understand the behavior in my setup. Thanks a lot, Mike -----Ursprüngliche Nachricht----- Von: Mike Pagel … [View More]

1 0

Different queues across branches?
by Mike Pagel 20 Dec '18

20 Dec '18

Hey zuulers, we recently had to create a maintenance branch across all repositories required to build the project. Since the two branches no longer affect each other in terms of a green build I intended to decouple the gating process of master and branch by changing the gate's queue name in the branch. I was expecting the branch gates to be executed in the new queue and the master gates being executed in the original one. However, it looks like all gates are still happening in the same … [View More]

3 8

How to define zuul.executor.inventory_file on a localhost
by Akhil Jayakumar (akjayaku) 19 Dec '18

19 Dec '18

Hi All, Our openstack cinder ci jobs are failing due to below error. Do you know where we need to define zuul.executor variable “inventory_file”? Appreciate your help. 2018-12-18 06:48:58.993454 | TASK [log-inventory : Copy ansible inventory to logs dir] 2018-12-18 06:48:59.045029 | primary | ERROR 2018-12-18 06:48:59.045544 | primary | { 2018-12-18 06:48:59.045637 | primary | "msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute '… [View More]

3 3

Zuul project update 2018-12-17
by corvus＠inaugust.com 18 Dec '18

18 Dec '18

Hi, The past two weeks in the Zuul project have seen: David discovered an issue in node deletion that was leaving empty ZooKeeper nodes and added code to nodepool to clean those up automatically. He also fixed a race in the test_handler_poll_session_expired nodepool test. Tobias worked on spec for scale out scheduler (what we've been informally calling Zuul v4 -- where we likely connect all components via Zookeeper and remove our use of Gearman). He also fixed an issue with github status … [View More]

3 2

Creation of OSF Project Confirmation Guidelines
by Claire Massey 13 Dec '18

13 Dec '18

Hi everyone, As you know, Zuul is a pilot project supported by the OpenStack Foundation (OSF). As such, we wanted to make sure you’re aware of an effort recently kicked off a sub group of the OSF Board of Directors to draft guidelines[1] for confirming pilot projects as full, top-level, open infrastructure project of the Foundation. The current pilot projects that will be eligible for confirmation review in 2019-2020 are Airship, Kata Containers, StarlingX and Zuul. Currently the OpenStack … [View More]

1 0