- Zuul-discuss - lists.opendev.org

swift upload jobs + ara-report
by Mohammed Naser 26 Aug '19

26 Aug '19

Hi everyone, The ara-report role combined with swift upload is causing a huge amount of objects generated, which I think need to be optimized in order for the role to be more usable. So, given that we're one of the clouds that OpenDev uses to publish job results, I've managed to pull some general information from that object storage cluster.. for yesterday (a sunday), there was 180k PUTs for *static* ARA assets .. and that's on a quiet day. Let's take a more busy day, Thursday, that resulted in 375,000 PUTs for static ARA assets. I think over the long time, this won't be sustainable, given an average of 200k, we're storing millions of objects per week that are being duplicated. While storage is cheap, the large number of objects starts presenting challenges (as an operator). Sure, we can decide to solve the server-side bits, but I think Zuul being a bit nicer against the clouds they talk to is a good thing (given there is actual costs that come from storage). I would like to propose that we somehow stop pushing static assets of ARA into object storage and keep them stored in some common location that ARA references to. I think this would *massively* decrease the load that is put when using Swift to publish logs. I'm open to other ideas as well. Thanks, Mohammed -- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser(a)vexxhost.com W. http://vexxhost.com

4 3

Tristan is a Zuul Maintainer
by corvus＠inaugust.com 21 Aug '19

21 Aug '19

Hi, I'm pleased to announce that Tristan Cacqueray (tristanC) has been added[1] to the ranks of Zuul Maintainers. Congratulations Tristan, and thank you for all your hard work! For more information about the process and the qualifications and responsibilities of maintainers, please see [2]. -Jim [1] https://review.opendev.org/677308 [2] https://zuul-ci.org/docs/zuul/governance.html#zuul-maintainers

1 0

Shanghai Forum Selection Committee
by Jimmy McArthur 21 Aug '19

21 Aug '19

Hello Zuul! The Forum in Shanghai is rapidly approaching. We would love 1 volunteer from Zuul for the Forum Selection Committee. Ideally, the volunteer would already be serving in some capacity in a governance role for your project. For information on the Summit in Shanghai: https://www.openstack.org/summit/shanghai-2019/ For more information on the Forum, please see:https://wiki.openstack.org/wiki/Forum Please reach out to myself orknelson(a)openstack.org if you're interested. Volunteers should respond on or before September 2, 2019. Cheers, Jimmy

1 0

Proposed behavior change on tags feature (passing it to Ansible)
by Sorin Sbarnea 13 Aug '19

13 Aug '19

It raised to my concern that sometimes we need to pass some tags (or skip-tags) to ansible playbooks or roles. At this moment there is no support for this in zuul but there is an older CR that would add it: https://review.opendev.org/#/c/575672/ (will need few changes) Since that review was created we already got a "tags" feature in zuul, which was initially purposed for categorization on jobs. I would like to redefine the meaning of current "tags" feature and document the fact that these would also be passed to ansible. I do not think that categorization conflicts with passing tags to ansible but is better to ask. If we do this we will be able to use "tags" and "skip-tags" just the same way we do in Ansible and avoid the case where our params are different than ansible ones. Cheers Sorin Sbarnea

3 2

Enabling new log features
by corvus＠inaugust.com 13 Aug '19

13 Aug '19

Hi, We recently made a release of Zuul with a bunch of new functionality for handling logs in-browser. The idea is that when Zuul reports the completion of a job, the URL it uses will be a link to the Zuul web application. There is a page for each build where a summary of information about the build is displayed, a tree-view of all the log files stored for the build, along with inline rendering of any text files therein, and a complete structured rendering of the tasks in the job. Note that in all cases these features require an existing SQL reporter configuration. Here's an example in use: https://review.opendev.org/#/c/675413/ There are a few steps you may need to take to upgrade existing installations to use these new features (and they may vary depending on your log storage). Here's a brief outline: Set up CORS: static log server ------------------------------ If you are simply uploading your logs with the "upload-logs" role, then you will need to make sure that the webserver which serves these logs adds an Access-Control-Allow-Origin header. If you are using Apache, you should make sure there is a line like this in the VirtualHost section: Header set Access-Control-Allow-Origin "*" If you are using nginx, make sure this appears in the location section: add_header 'Access-Control-Allow-Origin' '*'; Set up CORS: Swift ------------------ If you are using the upload-logs-swift role, it will create containers with the necessary settings automatically. However, some of these settings were only recently added to the role, so if you have a container which was created before August 1, you will need to make some updates. Run this script in an environment with openstacksdk installed, and a valid clouds.yaml: http://paste.openstack.org/raw/755695/ That will cause the Access-Control-Allow-Origin header to be set on responses which arrive with an Origin header. Test CORS --------- You can check whether the settings above are working correctly by visiting the build page for any build. Below the summary table, you should see a header that says "Results" and below that, one or more rows for the hosts used in the build with some icons and numbers. If that shows up, then the log server CORS setup is correct. Add the generate-zuul-manifest role ----------------------------------- Many of the new features require a new manifest file be uploading along with the logs. Add the generate-zuul-manifest[1] role to your base post playbook. This should run on localhost immediately before the log upload role. Once that is in place, you should begin to see "Zuul Manifest" appear under the "Artifacts" header in the build page. Remove obsolete static generation roles --------------------------------------- If you are using swift for log storage, you may already be using the htmlify-logs role[2]. This is obsoleted by the inline text log viewer, so now is a good time to remove it. Enable build page reporting --------------------------- Once that is all working, you can update your tenant configuration to set: report-build-page: true This will cause Zuul to report its internal build page URL rather than the log url (or the success- or failure-url of the job)[3]. I think that about covers it. Let me know if you have any questions. -Jim [1] https://zuul-ci.org/docs/zuul-jobs/log-roles.html#role-generate-zuul-manife… [2] https://zuul-ci.org/docs/zuul-jobs/log-roles.html#role-htmlify-logs [3] https://zuul-ci.org/docs/zuul/admin/tenants.html#attr-tenant.report-build-p…

1 0

Shanghai Summit Schedule is Live!
by Jimmy McArthur 06 Aug '19

06 Aug '19

Hi everyone, The agenda for the Open Infrastructure Summit <https://www.openstack.org/summit/shanghai-2019/summit-schedule> (formerly the OpenStack Summit) is now live! If you need a reason to join the Summit in Shanghai, November 4-6, here’s what you can expect: * Breakout sessions spanning 30+ open source projects from technical community leaders and organizations including ARM, WalmartLabs, China Mobile, China Railway, Shanghai Electric Power Company, China UnionPay, Haitong Securities Company, CERN, and more. * Project updates and onboarding from OSF projects: Airship, Kata Containers, OpenStack, StarlingX, and Zuul. * Join collaborative sessions at the Forum <https://wiki.openstack.org/wiki/Forum>, where open infrastructure operators and upstream developers will gather to jointly chart the future of open source infrastructure, discussing topics ranging from upgrades to networking models and how to get started contributing. * Get hands on training around open source technologies directly from the developers and operators building the software. Now what? * Register before prices increase <https://www.openstack.org/summit/shanghai-2019> on August 14 at 11:59pm PT (August 15 at 2:59pm China Standard Time). * Recruiting new talent? Pitching a new product? Enhance the visibility of your organization by sponsoring the Summit <https://t.e2ma.net/click/aq0kpb/ygpkur/q287bl>! Questions? Reach out to summit(a)openstack.org Cheers, Jimmy

1 0

Switch to ansible 2.8 for default version
by Paul Belanger 29 Jul '19

29 Jul '19

Greetings, For the last month[1] zuul.ansible.com has been using ansible 2.8 as the default version of ansible for jobs run by Zuul. Today we've upgraded to ansible 2.8.3 which reminded me to send this post. So far, we've yet to see any jobs failures related to ansible 2.8 however our job are not that complex. I'd ask if any zuul operators are using ansible 2.8 for jobs, and experiences so far? I would be great if we could bump the default for Zuul to 2.8, but would like a larger sample size. I've also proposed a change to opendev base-test job[2] to allow us to start doing some testing using the zuul tenant on opendev.org. [1] https://github.com/ansible/ansible-zuul-jobs/pull/155 [2] https://review.opendev.org/673012/ Paul

1 1

July 21 Deadline: Join OSF to Vote in the 2020 Foundation Board Elections
by Jeremy Stanley 17 Jul '19

17 Jul '19

As contributors to Zuul, a confirmed project supported by the OpenStack Foundation (OSF), we want to remind everyone of a benefit of joining the Foundation as an individual member: participating in the annual Foundation Board of Directors election. The OSF board represents all of the projects[1], not just OpenStack, and so it’s important for every community to actively participate, as both voters and candidates, in this election process. If you are not already an OpenStack Foundation member, please consider joining by July 21 to have a voice in the 2020 election. It’s free to join and only takes a few minutes to complete the form. Join here: https://www.openstack.org/join/register/?membership-type=foundation Time sensitive! In order to be eligible to vote in the January 2020 Individual Board of Directors election you must join as an Individual Foundation Member no later than this Sunday, July 21, 2019 at 11:59pm PT: https://www.openstack.org/election/2020-individual-director-election/ The OpenStack Foundation Board of Directors provides strategic and financial oversight of Foundation resources and staff. The 24 person Board is composed of directors elected by the Individual Foundation Members (8), directors elected by the Gold Members (8) and directors appointed by the Platinum Members (8). Individual Member Directors are there to represent the Individual Members of the Foundation. These Directors act as the link between the thousands of members of the Foundation and the Board, and are not representing the companies for which they work. Meet the current Board of Directors, and learn more here: https://www.openstack.org/foundation/board-of-directors/ https://www.openstack.org/foundation/ For further OpenStack Foundation updates and news, please sign up for the Foundation mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/foundation You can also select the type of content you receive from the OpenStack Foundation here: https://www.openstack.org/community/email-signup [1] We’re aware that some of the language about the election on openstack.org is still “openstack project” specific, and we’ll be updating that prior to the 2020 elections to more accurately reflect that the OSF Board represents all OSF projects. Please don’t let this dissuade you from joining, if you’re interested in voting! -- Jeremy Stanley on behalf of the OSF staff

1 0

Community Voting is now Open!
by Jimmy McArthur 16 Jul '19

16 Jul '19

Community voting for the Open Infrastructure Summit Shanghai sessions is open! You can VOTE HERE <https://www.openstack.org/summit/shanghai-2019/vote-for-speakers>, but what does that mean? Now that the Call for Presentations has closed, all submissions are available for community vote and input. After community voting closes, the volunteer Programming Committee members will receive the results to review to help them determine the final selections for Summit schedule. While community votes are meant to help inform the decision, Programming Committee members are expected to exercise judgment in their area of expertise and help ensure diversity of sessions and speakers. View full details of the session selection process <https://cfp.openstack.org/#selection-process>. In order to vote, you need an OSF community membership. If you do not have an account, please create one by going to openstack.org/join <https://www.openstack.org/join>. If you need to reset your password, you can do that here <https://www.openstack.org/Security/lostpassword>. Hurry, voting closes Monday, July 22 at 11:59pmPacific Time (Tuesday, July 23 at 6:59 UTC). Continue to visit https://www.openstack.org/summit/shanghai-2019 for all Summit-related information. REGISTER Register for the Summit <https://www.openstack.org/summit/shanghai-2019> before prices increase on August 7th! VISA APPLICATION PROCESS Make sure to secure your Visa soon. More information <https://www.openstack.org/summit/shanghai-2019/travel/#travel-support> about the Visa application process. TRAVEL SUPPORT PROGRAM August 8th is the last day to submit applications. Please submit your applications <https://openstackfoundation.formstack.com/forms/travelsupportshanghai> by 11:59pm Pacific Time (August 9th at 6:59am UTC). If you have any questions, please email summit(a)openstack.org <mailto:summit@openstack.org>. Cheers, Jimmy

1 0

Testing in zuul-jobs
by corvus＠inaugust.com 15 Jul '19

15 Jul '19

Hi, I've been working on the testing situation in Zuul jobs, and I think it's getting to a good place. The overall goal is to make it easier to have test jobs that exercise roles in the zuul-jobs repo. In turn, I think that will make it easier for reviewers to accept new roles and patches. The first change to support this is that we now have in-repo configuration of what jobs run on changes to zuul-jobs. We couldn't have this before because we don't want to impose any special requirements on systems that use zuul-jobs. Jobs that test zuul-jobs may need to specify nodesets, secrets, etc., not to mention the actual project stanza to indicate what jobs run. To solve this, the "zuul" tenant of the OpenDev Zuul loads *extra* configuration from the zuul-jobs repo which no other Zuul system or tenant should do. In the extra configuration, we can add jobs for the repo itself. This is in the zuul-tests.d/ directory of the repo: https://opendev.org/zuul/zuul-jobs/src/branch/master/zuul-tests.d In that directory, I have organized the test jobs in the same way I have organized them in the documentation. I think we should continue that as we add new jobs. Next, I've added a playbook that we can use to get to something like 80% test coverage fairly quickly. It simply includes a named role. Since many of our roles just install (and possibly run) software, this test provides significant coverage at almost no cost. To use this role, just run the "test-playbooks/simple-role-test.yaml" playbook and pass the "role_name" variable. For example: https://opendev.org/zuul/zuul-jobs/src/commit/6cec9080a047f549578c7f8291aff… If you need to do anything other than simply including a role (for example, testing how multiple roles interact, or performing validation after the role runs), you should probably make a dedicated playbook for the job. I think that starting now, we should require at least the simple-role-test playbook for every role in the repo (unless there is some good reason it won't work). Some roles have special handling for different platforms and therefore need to be tested on each. Some notable examples include many of the roles which typically appear in base jobs. I have created a script in tools/update-test-platforms.py which will look for jobs with the tags "all-platforms" or "all-platforms-multinode" and it will automatically create (or delete) identical jobs for each of the platforms that are available in OpenDev. Of course, if you don't need the whole set (perhaps you only need to test on one or two specific platforms), you can still do the same thing manually. The script is located here: https://opendev.org/zuul/zuul-jobs/src/commit/6cec9080a047f549578c7f8291aff… And here is an example job which uses it: https://opendev.org/zuul/zuul-jobs/src/commit/6cec9080a047f549578c7f8291aff… It adds every job to the check and gate pipeline too. The next change which is relevant is that Zuul now automatically runs jobs when their configurations change. This means that we can have many, many jobs in the zuul-jobs repo (i.e., perhaps one job per role), each of which has a file matcher for the role directory (and possibly test playbook), and these jobs will only run on changes to those files or changes to their own job definitions. As a result, we should expect to have lots of jobs in this repo, but expect very few of them to run on any given change. The names of these jobs should start with the prefix "zuul-jobs-test". For example, "zuul-jobs-test-install-pdk-dependencies" tests the "install-pdk-dependencies" role. Thanks to David Simard who set up the original versions of the base role tests along with a fine example of a well-tested standalone role ("upload-git-mirror"). He showed us the way, and the changes I have made are simply to facilitate that test design. I've started working on adding test jobs for existing roles in the repo. There are quite a lot, and I would appreciate help on that. We're in no hurry, so if you can spare a minute every now and then to add a new job, that would be great. Also, if someone is proposing a change to an untested role, see if you can add a quick test job for it at the same time. Thanks, Jim

1 0

Re: zuul-base-jobs: Changing git connection
by Andreas Jaeger 15 Jul '19

15 Jul '19

On 15/07/2019 16.53, Andreas Jaeger wrote: > On 01/07/2019 16.14, Andreas Jaeger wrote: >> We are deprecating the git.zuul-ci.org server, it redirects already to >> opendev.org. Thus, there's a change ready for zuul-base-jobs to switch >> to the new URL. This means that everybody using zuul-base jobs, needs to >> add a connection to "opendev.org" to their zuul config (if not already >> existing), otherwise your jobs will break. >> >> The change for this is: >> https://review.opendev.org/667813 >> >> I've marked the change as WIP, I'll un-WIP it after the 15th of July, > > We're merging this now, in case you need to update your config, Since a question came up: You need to change the baseurl from https://git.zuul-ci.org/ to https://opendev.org/zuul Andreas -- Andreas Jaeger aj(a)suse.com Twitter: jaegerandi SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg GF: Nils Brauckmann, Felix Imendörffer, Enrica Angelone, HRB 247165 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126

2 1

OpenInfra Days Vietnam 2019 (Hanoi) - Call For Presentations
by Trinh Nguyen 12 Jul '19

12 Jul '19

Hello, Hope you're doing well :) The OpenInfra Days Vietnam 2019 [1] is looking for speakers in many different topics (e.g., container, CI, deployment, edge computing, etc.). If you would love to have a taste of Hanoi, the capital of Vietnam, please join us this one-day event and submit your presentation [2]. *- Date:* 24 AUGUST 2019 *- Location:* INTERCONTINENTAL HANOI LANDMARK72, HANOI, VIETNAM Especially this time, we're honored to have the Upstream Institute Training [3] hosted by the OpenStack Foundation on the next day (25 August 2019). [1] http://day.vietopeninfra.org/ [2] https://forms.gle/iiRBxxyRv1mGFbgi7 [3] https://docs.openstack.org/upstream-training/upstream-training-content.html See you in Hanoi! Bests, On behalf of the VietOpenInfra Group. -- *Trinh Nguyen* *www.edlab.xyz <https://www.edlab.xyz>*

1 1

Zuul moved to zuul OpenDev Zuul Tenant
by corvus＠inaugust.com 12 Jul '19

12 Jul '19

Hi, In the OpenDev installation of Zuul (which is the Zuul that Zuul itself uses for gating), I have moved all of the Zuul projects to their own tenant (named "zuul"). If you want to see the status, you won't find it on zuul.openstack.org anymore. The new url is: https://zuul.opendev.org/t/zuul/status Also, if you have been using any dev releases (or our completely unadvertised javascript content tarball releases), new versions are no longer published to tarballs.openstack.org, but are now at: https://tarballs.opendev.org/zuul/zuul/ Zuul maintainers are now completely responsible for the pipeline definitions and trusted jobs that are part of the zuul tenant. These are located in the zuul/project-config project: https://opendev.org/zuul/project-config If that project hasn't been on your radar, now is a good time to notice it. Several of the release and publication jobs have changed as part of the move, so keep an eye out for unexpected problems there. Some of them now use the promote pipeline, so we should end up with faster publication. There is one immediate behavior change: the zuul tenant lacks the 'clean-check' requirement that the openstack tenant had. This means that you can approve a change to enqueue it into gate before the check results have returned. Feel free to use this to your advantage -- we trust the gate. :) -Jim

1 0

Design for mirrors in zuul-jobs
by corvus＠inaugust.com 09 Jul '19

09 Jul '19

Hi, When we started zuul-jobs, we had the idea that being able to use local mirrors or proxies instead of remote package mirrors, etc, is a good practice that we'd like to share with the wider community. However, our implementation in the configure-mirrors role (and others) was inherited from and built around the systems we had been running in OpenDev (and OpenStack infra), and ended up encoding a number of assumptions which don't generally hold. We'd like to replace it with a more generic system which can accommodate a wider range of schemes. I've proposed a system at: https://review.opendev.org/669948 which I think will achieve that. Essentially, each type of mirror that the roles in zuul-jobs may expect to know about would be a key in a dictionary, and under that key are data specific to that mirror. So the configure-mirrors role would, if it's being run on a Fedora system, check the 'fedora' key of the mirror_info variable, and if it finds it, use that URL to configure Fedora package mirrors. The container roles would likewise use the same mirror_info variable to see if there is a "dockerhub" entry in the dictionary. Please take a moment and review that change (which is documentation only at this point) and see if it accommodates the use cases you're aware of, and suggest any changes that may help us avoid backing into another corner in the future. Thanks, Jim

1 0

Fwd: [OpenStack Foundation] [Shanghai Summit] CFP Deadline TONIGHT
by Jimmy McArthur 02 Jul '19

02 Jul '19

Deadlines! Submit! > *From:* Ashlee Ferguson <ashlee(a)openstack.org> > *Date:* July 2, 2019 at 2:17 PM > *To:* foundation(a)lists.openstack.org, > openstack-discuss(a)lists.openstack.org, community(a)lists.openstack.org, > kata-dev(a)lists.katacontainers.io > *Subject:* [OpenStack Foundation] [Shanghai Summit] CFP Deadline TONIGHT > Hi everyone, > > The Shanghai Summit Call for Presentations [1] deadline is TONIGHT, > July 2 at 11:59 pm PT (July 3, 2019 at 15:00 China Standard Time)! > > Submit your presentations, panels, and Hands-on Workshops for the Open > Infrastructure Summit [2] by the end of today, and join the global > community in Shanghai, November 4-6, 2019. Sessions will be presented > in both Mandarin and English, so you may submit your presentation in > either language. > > Tracks [4]: > 5G, NFV & Edge > AI, Machine Learning & HPC > CI/CD > Container Infrastructure > Getting Started > Hands-on Workshops > Open Development > Private & Hybrid Cloud > Public Cloud > Security > > Other Helpful Shanghai Summit & PTG Information-- > > * Register now [5] before the early bird registration deadline in > early August (USD or RMB options available) > * Apply for Travel Support [6] before August 8. More information here [7]. > * Interested in sponsoring the Summit? [8]. > * The content submission process for the Forum and Project > Teams Gathering will be managed separately in the upcoming months. > > We look forward to your submissions! > > Cheers, > Ashlee > > [1] https://cfp.openstack.org/ > [2] https://www.openstack.org/summit/shanghai-2019/ > [3] https://cfp.openstack.org/ > [4] https://www.openstack.org/summit/shanghai-2019/summit-categories/ > [5] https://www.openstack.org/summit/shanghai-2019/ > [6] https://openstackfoundation.formstack.com/forms/travelsupportshanghai > [7] https://www.openstack.org/summit/shanghai-2019/travel/ > [8] https://www.openstack.org/summit/shanghai-2019/sponsors/ > > > > > _______________________________________________ > Foundation mailing list > Foundation(a)lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/foundation

1 0

Assigning nodeset/nodes dynamically based in user who made the pull request in github
by Kasibhatla Sarma 30 Jun '19

30 Jun '19

Hi Team, I am trying to see if there are any options using which I can select the nodeset/nodes dynamically. For example, assume that I have two users in two different regions (us-west and us-east) Based on who made the pull request in GitHub, we would like to run the check/gate pipelines on specific nodes. For example, - nodeset: name: emea nodes: - name: node_emea label: centos76 groups: - name: emea nodes: - node_emea The first job returns the location of the user who made a pull request (say zuul_location) - job: name: check_location run: playbooks/check_location.yaml parent: base I would like to use the location in the nodeset as something like this.. - job: name: pullrequest_label run: playbooks/pullrequest_label.yaml dependencies: - check_location parent: base vars: location: "{{ zuul_location }}" nodeset: "{{ zuul_location }}" I would like to know if there are ways to achieve this Thanks, Sarma K

5 8

CFP Deadline for Open Infrastructure Summit Shanghai
by Jimmy McArthur 26 Jun '19

26 Jun '19

Hi Everyone! The July 2 deadline to submit a presentation [1] for the Open Infrastructure Summit [2] in Shanghai is in less than one week! Submit your session today and join the global community in Shanghai, November 4-6, 2019. Sessions will be presented in both Mandarin and English, so you may submit your presentation in either language. Submit your presentations, panels, and hands-on workshops [3] before July 2 at 11:59 pm PT (July 3, 2019 at 15:00 China Standard Time). Tracks [4]: Container Infrastructure Hands-on Workshops AI, Machine Learning & HPC Private & Hybrid Cloud Public Cloud 5G, NFV & Edge Open Development Getting Started CI/CD Security Upcoming Shanghai Summit Deadlines * Register now [5] before the early bird registration deadline in early August (USD or RMB options available) * Apply for Travel Support [6] before August 8. For more information on the Travel Support Program, go here [7]. * Interested in sponsoring the Summit? [8]. * The content submission process for the Forum and Project Teams Gathering will be managed separately in the upcoming months. We look forward to your submissions! Cheers, Jimmy [1] https://cfp.openstack.org/ [2] https://www.openstack.org/summit/shanghai-2019/ [3] https://cfp.openstack.org/ [4] https://www.openstack.org/summit/shanghai-2019/summit-categories/ [5] https://www.openstack.org/summit/shanghai-2019/ [6] https://openstackfoundation.formstack.com/forms/travelsupportshanghai [7] https://www.openstack.org/summit/shanghai-2019/travel/ [8] https://www.openstack.org/summit/shanghai-2019/sponsors/

1 0

Feedback for Zuul web UI
by Paul Belanger 24 Jun '19

24 Jun '19

Greetings, I wanted to share some feedback I've gotten over the last few weeks about the Zuul web UI, I am mostly doing so, because I am struggling to make progress on fixing these issue myself (I don't have my JS experience). Here are a few recent examples: Builds page ----------- * Lack of pagination support. Currently, there is no easy way to scroll through the history of the builds UI. Right now, it is hardcoded to a single page, of 50 results. Users would like the ability to go further backwards in time, even listing more then 50 results per page. * More filters. It has been asked, for the ability to add filtering for start / end time ranges. This would allow a user to create a weekly report email, linking to the results of the week. The context is periodic jobs. Voting / non-voting jobs has also been requested. * Regex of job name filters. We currently have a lot of jobs which append the version of python onto a job name. For example, ansible-test-network-integration-vyos-python37, ansible-test-network-integration-vyos-python36, etc. A few users have requested the ability to use a single regex for job name filter. Buildset page ------------- * Add link to builds page of jobs that ran. Some users would like to see a link create, which auto creates the build page filters for a change that was build. This is because, in our case we have some jobs to do not report back to github, and depending on PR to list jobs isn't 100%. I'm happy to give additional information if needed, or also happy to write the code, howerver I'd likely need some hand holding for the first little bit. Thanks! Paul

2 1

Zuul autohold changes
by David Shrewsbury 17 Jun '19

17 Jun '19

Hello, I've mentioned this info in the IRC channel, but thought it would be a good idea to mention this on the mailing list as well. I have a series of changes [1] up that significantly change the way Zuul deals with autohold requests. The tl;dr of it is that we now store those requests in ZooKeeper rather than in memory. Advantages of this change are: 1. Easily identify which nodes are held for a request and to easily delete those held nodes when the request is deleted. 2. Supports future scheduler scale-out. 3. Hold requests are kept across Zuul restarts. This work conflicts with Matthew's work on the web API [2] as it adds new commands for deleting autohold requests, and getting autohold detail information. I fully expect his work to merge first, so my series of changes will likely need rework after that merge. I bring this up because I'm about to be AFK for a couple of weeks for shoulder surgery (such fun), so if those merges happen while I'm out, I'm hoping someone can volunteer to fix any merge conflicts or other minor changes that may be needed. Since this all started because of a hand-wavey comment from Monty, I volunteer him. :) :) :) j/k [1] https://review.opendev.org/#/q/topic:autohold-revamp [2] https://zuul-ci.org/docs/zuul/developer/specs/tenant-scoped-admin-web-API.h… -Shrews

2 1

Reviews for the Bitbucket Server driver
by Mark Meyer 13 Jun '19

13 Jun '19

Hi, I'd like you to review the Bitbucket Server driver for Zuul. We now have all the components in place. Basic testing shows that it's working and my team at work has been using it for a while in an experimental setting. Now we need to go the next steps and have it refined. *Not Bitbucket Cloud Thanks, mark -- Mark Meyer mark(a)ofosos.org

1 0

Slack connectivity
by Mark Meyer 12 Jun '19

12 Jun '19

Hey list, I'd like to gauge interest here. So you want connectivity options for slack? I'm thinking of having this as part of zuul to trigger jobs and report results, out alternatively as an external bit that runs on the scheduler node. We already have some code and some ideas. But I don't feel like walking that path on my own. What so you think? Cheers, mark -- Mark Meyer mark(a)ofosos.org

4 3

GitHub API ReadTimeout Errors
by Paul Belanger 12 Jun '19

12 Jun '19

Greetings, We (ansible team) are running into a issue using the zuul github driver, please see the following pastebin: http://paste.openstack.org/show/752756/ As you can see, we don't seem to be getting reliable network access to api.github.com. I am currently trying to figure out if this is a network issue with our control plane provider (vexxhost) or something related to GitHub side (eg: bad load balancer). However, what it does result in, is Zuul missing events from GitHub and not properly enqueue PRs into the pipelines. I understand there is only so much we might be able to do regarding a network issue, but I'd be interested to hear if there are idea of how we could tolerate them more. I also wonder if we should be reporting these issues back to the users PR too. My main goal is to minimize the amount of humans that _think_ zuul is to blame for tests not being run on their PRs, when in fact it is something else. - Paul

2 3

Shanghai PTG Changes
by Kendall Nelson 11 Jun '19

11 Jun '19

Hello All, After Denver we were able to take time to reflect on the improvements we can make now that the PTG will occur immediately following the summit for the near future. While Shanghai will have its own set of variables, it's still good to reevaluate how we allocate time for groups and how we structure the week overall. tldr; - Onboarding is moving into the PTG for this round (updates stay a part of the Summit) - You can still do regular PTG stuff (or both onboarding and regular PTG stuff) - PTG slots can be as short as 1/4 of a day - More shared space at the Shanghai venue, less dedicated space - New breakdown: 1.5 days of Forum and 3.5 days of PTG - Survey will be out in a few weeks for requesting PTG space We'll have our traditional project team meetings at the PTG in Shanghai as the default format, that won't change. However, we know many of you don't expect to have all your regulars attend the PTG in Shanghai. To combat this and still help project teams make use of the PTG in the most effective way possible we are encouraging teams that want to meet but might not have all the people they need to have technical discussions to meet anyway and instead focus on a more thorough onboarding of our Chinese contributors. Project teams could also do a combination of the two, spend an hour and a half on onboarding (or however much time you see fit) and then have your regular technical discussions after. Project Updates will still be a part of the Summit like normal, its just the onboardings that will be compacted into the PTG for Shanghai. We are making PTG days more granular as well and will have the option to request 1/4 day slots in an effort to leave less empty space in the schedule. So if you are only doing onboarding, you probably only need 1/4 to 1/2 of a day. If you are doing just your regular technical discussions and still need three days, thats fine too. The venue itself (similar to Denver) will have a few large rooms for bigger teams to meet, however, most teams will meet in shared space. For those teams meeting to have only technical discussions and for teams that have larger groups, we will try to prioritize giving them their own dedicated space. For the shared spaces, we will add to the PTGbot more clearly defined locations within the shared space so its easier to find teams meeting there. I regret to inform you that, again, projection will be a very limited commodity. Yeah.. please don't shoot the messenger. Due to using mainly shared space, projection is just something we are not able to offer. The other change I haven't already mentioned is that we are going to have the PTG start a half day early. Instead of only being 3 days like in Denver, we are going to add more time to the PTG and subtract a half day from the Forum. Basically the breakdown will be 1.5 Forum and 3.5 PTG with the Summit overlapping the first two days. I will be sending the PTG survey out to PTLs/Project Leads in a couple weeks with a few changes. -Kendall Nelson (diablo_rojo)

1 0

Zuul project update 2019-06-10
by corvus＠inaugust.com 10 Jun '19

10 Jun '19

Hi, Last week we merged the Tenant-scoped admin web API spec: https://zuul-ci.org/docs/zuul/developer/specs/tenant-scoped-admin-web-API.h… We can start reviewing the implementation for that now (status:open topic:zuul_admin_web). David is finishing up a series which stores autohold information in Zookeeper to improve the experience around that. Tobias has continued his work on adding event IDs to log messages. Clark is moving us to node version 10 across all of our javascript jobs. Mark Meyer has been refining his patch series to add a Bitbucker driver. I've been (and will continue to) work on moving Zuul into its own tenant in the OpenDev Zuul. That will require some changes to where we publish branch-tip builds. Help write the next update email by updating https://etherpad.openstack.org/p/zuul-update-email Thanks, Jim

1 0

Toggle host-key-checking per label
by Paul Belanger 05 Jun '19

05 Jun '19

Greetings, I'd like to get a few eyes on https://review.opendev.org/663378/ please, as I'm currently blocked on doing some network automation with nodepool / zuul. Due to issues with a network appliance, we have a use case where an interface comes online but there is no default gateway assigned (apparently this needs to be done by a human). In this case, we actually want nodepool to launch the node, but not scan for hostkeys. This is okay for us, because zuul doesn't directly manage this node (due to connection being network_cli), we actually create a nodeset with another node (fedora) that zuul can still reach then use nested ansible to provision the network appliance. We cannot move the network node into its own pool, because we cannot split labels multinodes across pools, we get node_failures back from nodepool. I'm open to other ideas / solutions to work around this use-case, but believe adding a host-key-checking toggle at the label level, should work. - Paul

1 0

Different merge-mode for different pipeline?
by Nick Hwangbo 29 May '19

29 May '19

Hello, Is it possible to specify different merge-modes for different pipelines for the same repository? For example, is something like this possible in a zuul.yaml configuration: - job: name: test run: playbooks/test.yaml - project: merge-mode: cherry-pick check: jobs: - test - project: merge-mode: merge-resolve gate: jobs: - test The check pipeline would use cherry-pick as a merge mode, while the gate pipeline would use merge-resolve? Thanks, Nick

2 2

Zuul project update 2019-05-29
by corvus＠inaugust.com 29 May '19

29 May '19

Hi, It's been a while since I've sent one of these out, largely due to a busy schedule involving launching OpenDev, vacation, conferences, and associated travel. Now seems a good time to try to resync. First specs: I think the tenant-scoped admin web API spec is ready for final approval. I think everyone who has helped refine that has had their issues addressed, so we probably just need folks to go over it once again and give it the OK. I'd like to merge that next Friday, June 7 unless anyone thinks we need more time. Please take a look: https://review.opendev.org/562321 The operator spec should probably be a priority for us too, based on the amount of interest here (and the POC implementation Tristan has waiting in the wings). This one is still in the design phase, so please add your comments now to make sure it takes the right shape: https://review.opendev.org/659180 Other activity: Tristan has updated the freeze-job stack again, and started work on a new "cleanup" phase which we've previously discussed adding. Shrews has cleaned up image build leaks in Nodepool's builder, improved Nodepool unit tests and tooling around detecting memory leaks. He ported the OpenDev Nodepool to use jemalloc -- it seems that our general recommendation for operating large Zuul systems is to use that. He's currently working on improving the Zuul autohold experience. We may want to look into whether we should line that up behind the tenant web admin API to avoid conflicts. Tobias has been adding event IDs to log messages and generally improving the logging system to account for that kind of extra data; this should help trace events through the various services. In the GitHub driver, he's adding support for CODEOWNERS files, and a squash-merge option. And he's working on support for smart reconfiguration (like full reconfiguration but only for tenants that changed their configs). I've been working on a job to test the new Docker registry roles. It's raising some issues about how we should do testing of zuul-jobs repos going forward, and it's looking like we may need a new repo to hold these test jobs. I'll propose a change to create that after I get the first draft of this job finished. Help write the next update email by updating https://etherpad.openstack.org/p/zuul-update-email Thanks, Jim

1 0

Gerrit Downtime May 31, 2019 beginning 1500UTC
by Clark Boylan 28 May '19

28 May '19

Hello everyone, We'll be taking a short (hopefully no longer than an hour) Gerrit downtime so that we can rename some projects this Friday, May 31, 2019 beginning at 1500UTC. Some of these renames are cleanups after the great OpenDev git migration and others aren't, but all of them require us to stop Gerrit to update the database and contents on disk. If you'd like specifics on what repos are being renamed or what the process looks like, you can follow along at: https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Upcoming_Project_… https://etherpad.openstack.org/p/project-renames-2019-05-31 As always feel free to bring up questions or concerns and we'll do our best to answer/address them. Thank you for your patience and sorry for the interruption, Clark

1 0

AMQP driver
by Tristan Cacqueray 26 May '19

26 May '19

Hello, There is an initial AMQP driver proposed waiting for reviews[0]. The use-case is to integrate Zuul with fedmsg/umb[1] which is used for artifacts testing. The goal is to run Zuul jobs for the different testing phases that happen after a change is merged, namely on koji builds and on composition events. I guess this goes beyond Zuul's git driven model, and the AMQP trigger would need to tell the jobs where the external provided artifacts are. So could Zuul be extended to support such a use-case? Regards, -Tristan [0]: https://review.opendev.org/#/q/topic:amqp-trigger [1]: https://pagure.io/fedora-ci/messages

2 2

periodic jobs (timer tigger) and job.files attribute
by Paul Belanger 24 May '19

24 May '19

Greetings, Over the last few days I've been working to create a job that is shared between check / gate pipelines and periodic pipeline. As an example, using the following job stanza: - job: name: ansible-test-network-integration-vyos parent: ansible-network-vyos-appliance pre-run: playbooks/ansible-test-network-integration-base/pre.yaml run: playbooks/ansible-test-network-integration-base/run.yaml required-projects: - name: github.com/ansible/ansible timeout: 3600 vars: ansible_test_network_integration: vyos files: - ^lib/ansible/modules/network/vyos/.* - ^lib/ansible/module_utils/network/vyos/.* - ^lib/ansible/plugins/connection/network_cli.py - ^test/integration/targets/vyos_.* you can see we are using the job.files attribute, this works as expects when we open a PR in github against github.com/ansible/ansible. Here is the pipeline stanza: - project-template: name: ansible-test-network-integration periodic: jobs: - ansible-test-network-integration-vyos third-party-check: jobs: - ansible-test-network-integration-vyos However, the issue is, current periodic jobs are not running, because from what I see the job.files attribute does not match. So, based on a suggestion from Spamaps I tried the following: - project-template: name: ansible-test-network-integration periodic: jobs: - ansible-test-network-integration-vyos third-party-check: jobs: - ansible-test-network-integration-vyos: files: [] which didn't work, I even tried: - project-template: name: ansible-test-network-integration periodic: jobs: - ansible-test-network-integration-vyos third-party-check: jobs: - ansible-test-network-integration-vyos: files: - ^.* and no luck. The best I can guess, is that jobs enqueued into the periodic pipeline via timer trigger don't seem to have files attribute for the change? Which basically prevent any job.files match possible. My question here is, how could we allow this? Does it make sense if we have job.files set to an empty list, remove any existing job.files defined? Or, if periodic pipelines don't present files in the change object, could we just drop the job.files match process? I know that I could remove the job.files from the job stanza, and move them into the pipeline stanza, however in this case I would not like to do that for reasons. I also know I could create another job.name, maybe prefaces with periodic, again I'd prefer not to do that too. However, if there is no way to fix this based on above, I'll have to do one of these solutions. Thanks, Paul

3 3

ansible version for linting?
by Andreas Jaeger 23 May '19

23 May '19

Which ansible version should we use for linting in repos like openstack/project-config, openstack/openstack-zuul-jobs and zuul/zuul-jobs? OpenDev runs ansible 2.7 which is the default for Zuul - but Zuul supports 2.5 (as deprecated) and 2.6 and soon 2.8 as well... So, do we lint with the oldest version? The newest? The default? right now we use ansible 2.5.x for linting. I proposed the first changes to go to 2.7 - but then thought further and wrote this email: https://review.opendev.org/#/c/659810/ https://review.opendev.org/#/c/659811/ Andreas -- Andreas Jaeger aj(a){suse.com,opensuse.org} Twitter: jaegerandi SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 -- Andreas Jaeger aj(a){suse.com,opensuse.org} Twitter: jaegerandi SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126

1 1

Ansible 2.8 support
by Paul Belanger 13 May '19

13 May '19

Greetings, Hopefully this week ansible 2.8.0 will be released, yay. As a results, Tobias as already done most of the work[1] to get Zuul ready to support it. I'd like to ask for humans to review the patch, the main change here for zuul is to our console stream plugin. Ansible 2.8 now includes rescued / ignored in the results output. Also, facts are now disabled by default when using gather_facts. I think it would be awesome if we could land and do a zuul release just after ansible 2.8 was released. Would go well with out story of cross project collaboration. [1] https://review.opendev.org/631933

1 1

Zuul's Roadmap and Backlog
by Tristan Cacqueray 12 May '19

12 May '19

Hello folks, I don't know if this subject has been discussed at the last PTG, so here is a thread about it. There is a growing backlog of bug fixes and features, and it's a bit frustrating not knowing when or if a patch is going to be approved. For the 3.0 release, there was a board with the list of stories and this was very helpful in communicating what was going to be included and what should have been reviewed. Would it be possible to use such board for the next releases? As an example, here is a not exhaustive list of features with unknown status. Wouldn't it help to pin them to a release so that reviewer/contributor could help efficiently? * topic:github-review-reporter * topic:zubbi-integration * topic:pagure-driver * topic:web-openapi * topic:zuul_admin_web * topic:scale-out-scheduler * topic:freeze_job Regards, -Tristan

5 9

Job doesn't enter the gate pipeline
by Vitaliy Lotorev 11 May '19

11 May '19

Hi, I'm trying to setup Zuul with in-house Gerrit. check jobs from independent pipeline are started in predictable way (on new PS and 'recheck' comment). The issue is I can't start jobs in buildroot_gate (dependent) pipeline. I simplified pipeline to this one: - pipeline: name: buildroot_gate manager: dependent require: gerrit: open: true trigger: gerrit: - event: comment-added comment: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*gate success: gerrit: Verified: 1 mysql: failure: gerrit: Verified: -1 mysql: I.e. for debug purpose I'd like buildroot_gate pipeline to get started on 'gate' comment. Once I add 'gate' comment to PS Zuul on Gerrit Scheduler successfully detects this and reports the as follows: scheduler_1 | 'url': ' https://mycompanygerrit/c/production/vip1-boards/sd-fc/+/12134'}] scheduler_1 | 2019-05-09 20:53:58,502 DEBUG zuul.Scheduler: Change <Change 0x7f18e2104c50 production/vip1-boards/sd-fc 12$ 34,12> has been updated, clearing dependent change caches scheduler_1 | 2019-05-09 20:53:58,502 DEBUG zuul.GerritConnection: Scheduling event from gerrit: <GerritTriggerEvent comm$ nt-added mycompanygerrit/production/vip1-boards/sd-fc master 12134,12 Verified:0, Code-Review:1> scheduler_1 | 2019-05-09 20:53:58,502 DEBUG zuul.Scheduler: Run handler awake scheduler_1 | 2019-05-09 20:53:58,502 DEBUG zuul.Scheduler: Fetching trigger event scheduler_1 | 2019-05-09 20:53:58,503 DEBUG zuul.Scheduler: Processing trigger event <GerritTriggerEvent comment-added ge$ rit.elvees.com/production/vip1-boards/sd-fc master 12134,12 Verified:0, Code-Review:1> scheduler_1 | 2019-05-09 20:53:58,503 DEBUG zuul.Pipeline.nto7.buildroot_gate: Event <GerritTriggerEvent comment-added ge$ rit.elvees.com/production/vip1-boards/sd-fc master 12134,12 Verified:0, Code-Review:1> for change <Change 0x7f18e2104c50 p$ oduction/vip1-boards/sd-fc 12134,12> matched <GerritEventFilter types: comment-added ignore_deletes: True comments: (?i)^($ atch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*gate> in pipeline <DependentPipelineManager buildroot_gate> scheduler_1 | 2019-05-09 20:53:58,503 DEBUG zuul.Pipeline.nto7.buildroot_gate: Considering adding change <Change 0x7f18e2$ 04c50 production/vip1-boards/sd-fc 12134,12> scheduler_1 | 2019-05-09 20:53:58,503 DEBUG zuul.Pipeline.nto7.buildroot_gate: Change <Change 0x7f18e2104c50 production/v$ p1-boards/sd-fc 12134,12> can not merge, ignoring scheduler_1 | 2019-05-09 20:53:58,503 DEBUG zuul.Pipeline.nto7.buildroot_gate: Change <Change 0x7f18e2104c50 production/v$ p1-boards/sd-fc 12134,12> is not ready to be enqueued, ignoring scheduler_1 | 2019-05-09 20:53:58,503 DEBUG zuul.Pipeline.nto7.check: Starting queue processor: check scheduler_1 | 2019-05-09 20:53:58,503 DEBUG zuul.Pipeline.nto7.check: Finished queue processor: check (changed: False) scheduler_1 | 2019-05-09 20:53:58,504 DEBUG zuul.Pipeline.nto7.gate: Starting queue processor: gate scheduler_1 | 2019-05-09 20:53:58,504 DEBUG zuul.Pipeline.nto7.gate: Finished queue processor: gate (changed: False) scheduler_1 | 2019-05-09 20:53:58,504 DEBUG zuul.Pipeline.nto7.buildroot_gate: Starting queue processor: buildroot_gate scheduler_1 | 2019-05-09 20:53:58,504 DEBUG zuul.Pipeline.nto7.buildroot_gate: Finished queue processor: buildroot_gate (c hanged: False) Please, note Scheduler logs "can not merge, ignoring", "is not ready to be enqueued, ignoring". How could I understand the reason why it can't be merged? This PS it definitely mergable manually: I could vote +2 and Sumbit the PS. -- Regards, Vitaliy

2 2

Correct way to trigger a periodic job manually
by Ian Wienand 07 May '19

07 May '19

Hello, The manual says to trigger a periodic job against master you'd do [1] $ zuul enqueue-ref --tenant openstack --trigger timer --pipeline periodic --project openstack/example_project --ref refs/heads/master Which looks about right to me, but this sets a newrev of 00..000 which is not correct; e.g. [2]. Tracing back that job to find the cause of the RETRY_LIMIT on ze06 I see: --- 2019-05-06 07:20:50,875 DEBUG zuul.AnsibleJob: [build: b1387c9741134f83a6f11e1cad0386f0] Checking out master 2019-05-06 07:20:50,877 ERROR zuul.AnsibleJob: [build: b1387c9741134f83a6f11e1cad0386f0] Exception while executing job Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/git/refs/symbolic.py", line 305, in set_reference obj = self.repo.rev_parse(ref + "^{}") # optionally deref tags File "/usr/local/lib/python3.5/dist-packages/git/repo/fun.py", line 213, in rev_parse obj = name_to_object(repo, rev[:start]) File "/usr/local/lib/python3.5/dist-packages/git/repo/fun.py", line 147, in name_to_object raise BadName(name) gitdb.exc.BadName: Ref 'master' did not resolve to an object During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/zuul/executor/server.py", line 790, in execute self._execute() File "/usr/local/lib/python3.5/dist-packages/zuul/executor/server.py", line 914, in _execute repo.checkout(selected_ref) File "/usr/local/lib/python3.5/dist-packages/zuul/merger/merger.py", line 369, in checkout File "/usr/local/lib/python3.5/dist-packages/git/refs/symbolic.py", line 308, in set_reference raise ValueError("Could not extract object from %s" % ref) ValueError: Could not extract object from master --- I set newref manually to the git master HEAD change-id, and it did work -- but it looks different to jobs actually triggered by the periodic driver itself. For example [3] is the job I triggered, and [4] is one from the periodic driver -- [4] does not have a newref. I feel like this used to work, probably because when I look back I wrote that bit of documentation [5] ;) I guess something changed; perhaps it is the client setting newref to 0000.000 by default instead of something like None? Any tips on what should I update the instructions to say now? -i [1] https://zuul-ci.org/docs/zuul/admin/client.html#manual-enqueue-examples [2] http://zuul.openstack.org/build/b1387c9741134f83a6f11e1cad0386f0 [3] http://zuul.openstack.org/build/8781cd38097b437d90ba5a9a5505633c [4] http://zuul.openstack.org/build/d30f960f7ca04b33af1eb5b7513b1b66 [5] https://review.opendev.org/#/c/518662/

2 2

exclude-unprotected-branches equivalent for Gerrit connection
by Brendan Jackman 07 May '19

07 May '19

Hi all, In my org's Gerrit setup, refs under personal/$username/ are a free-for-all. I have seen the same in other organisations too. This leads to the problem that is solved for Github by exclude-protected-branches; those branches eventually end up with broken Zuul configs (actually case that brought this to my attention is that I cannot delete a job that is no longer used, because the config in a personal/ branch refers to it). It seems like a simple solution would be to add an option under tenant.untrusted-projects.<project> that can be used to filter the branches considered by Zuul, by their name. I think the most obvious interface for that would be like job.files & job.irrelevant-files, i.e. a regex or list thereof, with a negative complement option alongside it. I think it could then behave much like exclude-unprotected-branches in that it works at the source connection level. Although I've mentioned Gerrit, I don't see why it shouldn't be independent of the actual source connection driver. This seems suspiciously obvious so I suspect I may be rather naive in suggesting it! Has it been discussed before, does it present more complications than I can see? I'm not too sure how this would related to tags. What does Zuul do with those? I didn't give the proposed option a name because I don't know if what I really want is a ref filter or a branch filter. In fact, I face a certain degree of pain because a couple the projects my Zuul deployment checks out have a lot of tags[1] so maybe this could kill two birds with one stone. Cheers, Brendan [1] $ git ls-remote --tags --refs git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git | wc -l 2763 This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed (‘intended recipient’). Accordingly the dissemination, distribution, copying or other use of this email or any of its content by any person other than the intended recipient may constitute a breach of civil or criminal law and is strictly prohibited. If you are not the intended recipient, please let us know as soon as possible and remove this email from your system. Company Reg No: 06855929 | Blu Wireless Technology Ltd, 5th Floor, 1 Temple Way, Bristol, BS2 0BY | Registered in England & Wales

2 2

Inheritance confusion, docs needed?
by Michael Johnson 06 May '19

06 May '19

As a developer creating Zuul jobs, one of the areas i find confusing is the inheritance of variables. I am not sure if this is an enhancement style request or just a missing section in the documentation. When you use host-vars, the top level variables are "merged" with the parent job variables, however the nested variables, such as "devstack_localrc" are not merged, they are "replaced". As an example is the opendev Zuul jobs (I think this applies to non-OpenStack jobs as well): When you use host-vars, the top level variables are "merged" with the parent job variables, however the nested variables, such as "devstack_localrc" are not merged, they are "replaced". host-vars: controller: devstack_localrc: DATABASE_USER: a-user In this example snippet, the variables at the devstack_localrc level are "merged", but the variables at the "DATABASE_USER" level are replaced and not inherited from the parent. Should Zuul be recursively "merging" all the way down or should we document how/which variables are inherited from the parent and which are not? The reason I'm asking is as our job definitions get more complicated (multi-node), I find myself copying variables from the parent jobs that I would really like to just inherit. Michael

2 1

Zuul working group session at OpenInfra Summit
by corvus＠inaugust.com 29 Apr '19

29 Apr '19

Hi, If you're at the OpenInfra Summit and would like to discuss what is or isn't working about sharing jobs between Zuul installations, please join the working group session on Tuesday: https://www.openstack.org/summit/denver-2019/summit-schedule/events/23736/z… -Jim

1 0

Celebrating Zuul's OSF Top Level Project Confirmation
by Clark Boylan 28 Apr '19

28 Apr '19

Zuul was confirmed by the OSF board as a top level OSF project today in Denver. To celebrate we'd like to meetup for arcade games, drinks, and food at FTW Denver Monday night. This is doubling as the OpenDev Infra team gathering, but considering the overlap between Zuul and Infra I figure that won't be an issue. If you'd like to join us we will be gathering at 6:45pm Monday in front of the big blue bear statue at the convention center. If it is raining, we'll likely meet up just inside the convention center building next to the statue to stay dry. Hope to see you there and congratulations Zuul! Clark

1 0

Tagging beta / rc releases
by Paul Belanger 24 Apr '19

24 Apr '19

Greetings, I wanted to raise the topic of maybe using beta / rc tags to publish pre-release versions of zuul / nodepool to pypi. Currently, before each official release we usually select a git SHA to be installed onto zuul.opendev.org, to ensure we can run a scale without any issues. Then a few days later, if things are well we tag a new release or work to address the problem. Then restart the process until things look good. I really like this approach, as currently there is no larger zuul and helps to catch any issues before our users install the released version. However, I would like to propose we make a minor change, is when we select to git SHA, is to also tag it using a pre-release version number (eg: beta / rc). By having it pre-release, (eg: pip install --pre zuul) would allow my current tooling (today we only installed from pypi, not git) to pick up the change and also provide some feedback. Our deployment tooling is setup to only use releases from pypi (wheels) in an effort to try not to carry any patches downstream. We could also switch to pip install via git, however this would add some extra overhead to production servers to compile wheels. - Paul

3 8

Migrating git repos to OpenDev
by corvus＠inaugust.com 21 Apr '19

21 Apr '19

Hi, As discussed in November[1], the OpenStack project infrastructure is being rebranded as "OpenDev" to better support a wider community of projects. We are nearly ready to perform the part of this transition with the largest impact: moving the authoritative git repositories for existing projects. In this email, I'd like to introduce the new hosting system we are preparing, discuss the transition, and invite projects to work with us on the logistics of the change. Gerrit ====== Gerrit is the core of our system and it will remain so in OpenDev. As part of this move, we will rename the gerrit server from review.openstack.org to review.opendev.org. As part of the transition, we will automatically merge appropriate changes to all branches of all repositories updating .gitreview and Zuul configuration files. Any further changes (README files, etc.) we expect to be made by individual project contributors. Repository Browsing =================== Currently our canonical *public* repository system is the cgit server at https://git.openstack.org/ (and git.airshipit.org, git.starlingx.io, and git.zuul-ci.org) This is a load balanced cluster of several servers which is designed to handle all the public git repository traffic, as it scales much better than Gerrit (and has a more friendly domain name). >From a technical standpoint, it's excellent, but its usability could be improved. Therefore, as part of this transition, we will replace the cgit servers with a new system based on Gitea. Gitea is a complete development collaboration system, but it's very flexible and will allow us to disable components which we aren't using. We will operate it in a read-only configuration where it will act as the public mirror for Gerrit. The advantages it has over the current system are: * Shorter domain name in project URLs: https://git.openstack.org/openstack/nova vs https://opendev.org/openstack/nova * Clone and browsing URLs are the same (with cgit, the browsing URL has an extra path component) * More visually pleasing code browsing * Integrated code searching * Ability to highlight multiple lines in links When we perform the transition we will install redirects from git.openstack.org (and the other git sites) to opendev.org, and will maintain those redirects for the foreseeable future. We will construct them so that even existing deep links to individual files in individual commits to cgit will redirect to the correct location on opendev.org. This system is up and running now with a live mirror of data from Gerrit, and you can start testing it out today at https://opendev.org/ Please let us know if you encounter any problems. If you would like to read more about the design of this system and the transition, see the infra-spec[2]. GitHub ====== Currently all OpenStack projects are replicated to GitHub. We do not plan on changing that during the transition, however, any projects outside of the openstack*/ namespaces will not automatically be replicated to GitHub, and we do not plan on adding that in the future. We do, however, support projects using Zuul to run post-merge jobs to push updates to GitHub or any other third-party mirrors with their own credentials. We will be happy to work with anyone interested in that to help set up jobs to do so. We are adopting this approach so that individual projects can have more control over how they are represented in social media, and to give us more flexibility in supporting our own organizational namespaces on OpenDev without assuming they map directly to GitHub. Eventually we plan on moving the OpenStack project to that system as well and retiring direct replication from Gerrit to GitHub completely. But we will defer that work until after this transition. Logistics ========= We can prepare much of the system in advance (as we have for the hosting system on opendev.org) but the actual transition and renaming of the Gerrit server will need to happen at once during an outage window. We need to schedule that outage and begin preparing for it. Since all of the project git URLs are going to change (to replace git.openstack.org with opendev.org and review.openstack.org with review.opendev.org) we can additionally take the opportunity to reorganize projects into different organizations. For example, during the transition we will rename Zuul, and it's associated projects, from the "openstack-infra" org to "zuul". So their new names will be "zuul/zuul", "zuul/nodepool", etc. This is an excellent time for the rest of the OpenStack Foundation pilot projects to do the same. If the OpenStack project desires this, it would also be a good time to move unofficial projects out of the openstack/ namespace. Therefore, we need your help: Action Items ============ We need each of the following projects: * OpenStack * Airship * StarlingX * Zuul To nominate a single point of contact to work with us on the transition. It would be helpful for that person to attend the next (and possibly next several) openstack infra team meetings in IRC [3]. We will work with those people on scheduling the transition, as well as finalizing the list of projects which should be renamed as part of the transition. If you manage an unofficial project and would like to take the opportunity to move or rename your project, please add it to this ethercalc[4]. [1] http://lists.openstack.org/pipermail/openstack-dev/2018-November/136403.html [2] http://specs.openstack.org/openstack-infra/infra-specs/specs/opendev-gerrit… [3] http://eavesdrop.openstack.org/#Project_Infrastructure_Team_Meeting [4] https://ethercalc.openstack.org/opendev-transition

5 6

Post OpenDev Repo Migration Issues
by Clark Boylan 20 Apr '19

20 Apr '19

Hello, There are two quick things we want to point out that we've noticed after the OpenDev Gerrit manuevers. First, any commit Depends-On footers for unmerged changes will need to use https://review.opendev.org urls instead of https://review.openstack.org urls. This is due to a limitation in Zuul and how it sees the Gerrit server. Any Depends-On footers that use https://review.openstack.org are currently non-functional and they will not enforce the dependency relationship. Second, while we have fairly robust http(s) redirects in place for old urls (including those from cgit to gitea) any ssh repo urls will need to be updated to use the new canonical paths if a repo has moved. We are unable to redirect incoming ssh requests against Gerrit. If you need to know what the canonical path is go to the corresponding https repo url and the resulting destination is the canonical location. Again thank you for your patience, Clark

3 2

Should we drop our custom Ansible plugins?
by corvus＠inaugust.com 15 Apr '19

15 Apr '19

Hi, When we created Zuul v3, we faced a fairly unique situation -- we needed to run untrusted Ansible playbooks without compromising the host running Ansible (i.e., the Zuul executor). To address this, we created a set of Ansible plugins which wrap the upstream Ansible plugins and block actions which we consider dangerous. Broadly speaking, there are two classes of actions we might want to block: 1) Actions which may expose private information stored on the executor (including information used by Zuul itself or other jobs), or otherwise compromise the security of the executor itself (so that future jobs can not be trusted). 2) Actions which may allow a user to take advantage of privileges afforded to the executor host itself. For example, if the executor is behind a network firewall and therefore can access unprotected network services. We added bubblewrap as a second layer of defense against the first class. It has been very effective -- our custom plugins have had several vulnerabilities, all of which were ultimately unexploitable due to the use of bubblewrap. Bubblewrap doesn't address the second class. Here's more background on the bubblewrap addition: https://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3-execut… Maintaining the Ansible plugins is a lot of work, and it's very easy to get wrong. For each new Ansible version, we need to look for any changes to module parameters and make corresponding updates. All of our security issues so far have been due to things we missed here. Meanwhile, these custom plugins increase the area where playbooks which may otherwise work can not be used with Zuul (for example, because they use a lookup plugin which writes to the executor). This makes Zuul more difficult to use. Therefore, I propose that we drop support for the custom plugins and rely entirely on bubblewrap to protect us from the first class of issues. It will mean less work for us, make Zuul more useful, and, at least with regard to the first class of problems, I don't think Zuul will be less secure. That brings me to the second class. I have a feeling (which is unsubstantiated), that there are probably enough holes in our plugins which make the second class of actions possible too. For example, we allow use of the URI module, but that means that any playbook can hit any HTTP URL accessible by the executor. While it may be difficult for a user to make a connection to a mysql server, it's not outside the realm of possibility. Dropping support for our custom plugins would mean that Zuul has no defense against this class of action. We would need to document that and ensure that users know that if their executors run untrusted code that they should have adequate application or network level protections in place. In short: don't put an executor inside the squishy center of your security perimeter. Before we do this, we need to make sure that Zuul itself doesn't impose any constraints that contradict this. The executor only uses Gearman to communicate with the rest of Zuul, and it can adequately be secured. Before we add any features which require the executor to communicate via ZooKeeper, we should make sure that we support zookeeper auth. What do folks think? Is this a reasonable change to make, or should we continue supporting the custom plugins? -Jim

7 19

Upgrading the lists.(openstack|airshipit|starlingx|zuul-ci).org server Friday April 12
by Clark Boylan 12 Apr '19

12 Apr '19

It is that time of the Ubuntu LTS cycle again and we need to upgrade our mailman mailing list server. We'd like to do that this Friday, April 12. We expect the upgrade to begin at about 17:00UTC and result in a 30-45 minute outage. The reason for the extended outage is that we will be upgrading the server in place to preserve its mail reputation. Thankfully email is a persistent system and your clients should queue up email sent until the server is back up again and accepting smtp connections. This means the outage shouldn't be very noticeable. Finally, for list admins, there are new DMARC moderation action settings. We ask that you don't change these settings and instead work with us if you need to address DMARC problems. Our current preference is that we pass email through unmodified so that the signatures still validate. Thank you all for your patience and feel free to ask us questions, Clark

1 1

New Zuul job to replicate a project's git repository to a remote git server
by David Moreau Simard 11 Apr '19

11 Apr '19

Hi, It is now possible for projects to replicate their git repository to a custom location by inheriting from the 'upload-git-mirror' job provided by Zuul. This job wraps around the 'upload-git-mirror' Ansible role that is part of the zuul-jobs library [1]. In order to use this job, you must supply a secret in the following format: === - secret: name: <name of your secret> data: user: <ssh user of the remote git server> host: <address of the remote git server> host_key: <ssh host key of the remote git server> ssh_key: <private key to authenticate with the remote git server> === The 'host_key' parameter can be retrieved from your known_hosts file or with a command like 'ssh-keyscan -H <host>' or 'ssh-keyscan -t rsa <host>'. For example, the 'host_key' when pushing to GitHub would be, on a single line: github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ== The 'ssh_key' parameter should be encrypted before being committed to the git repository. Zuul provides a tool for easily encrypting files such as SSH private keys and you can find more information about it in the documentation [2]. For example, encrypting a key for the "openstack/ara" project would look like this: === zuul/tools/encrypt_secret.py --infile /home/dmsimard/.ssh/ara_git_key --tenant openstack https://zuul.openstack.org openstack/ara === You would then use the secret in a job inheriting from 'upload-git-mirror' as such: === - job: name: <project>-upload-git-mirror parent: upload-git-mirror description: Mirrors openstack/<project> to neworg/<project> vars: git_mirror_repository: neworg/<project> secrets: - name: git_mirror_credentials secret: <name of your secret> pass-to-parent: true === Finally, the job must be set to run in your project's 'post' pipeline which is triggered every time a new commit is merged to the repository: === - project: check: jobs: # [...] gate: jobs: # [...] post: jobs: - <project>-upload-git-mirror === Note that the replication would only begin *after* the change has merged, meaning that merging the addition of the post job would not trigger the post job itself immediately. The post job will only trigger the next time that a commit is merged. [1]: https://zuul-ci.org/docs/zuul-jobs/general-roles.html#role-upload-git-mirror [2]: https://zuul-ci.org/docs/zuul/user/encryption.html David Moreau Simard dmsimard = [irc, github, twitter]

1 0

Update on the tenant-scoped admin REST API for Zuul
by Matthieu Huin 11 Apr '19

11 Apr '19

Hello, I'd like to introduce the applications of the spec referred to in the title [1]. I have worked in parallel on a possible implementation of said spec, backend-side [2]. Since it is not necessarily trivial to evaluate or test by oneself, I have also tried to put together a proof of concept for the GUI using a keycloak instance as the JWT provider, with -almost- everything set up together with docker, so that people can play around with the feature. [3] It requires a few manual steps in order to be set up properly though: 1. Build the zuul containers using the code from the patchset: cd zuul/doc/sources/admin/examples && docker-compose up --build 2. log in to the keycloak instance at http://localhost:8282 as admin/admin and create users in the "zuul" realm. The default rules of the PoC expect a user called "venkman", and a user belonging to the "gozerians" group (groups should be created automatically within the zuul realm) 3. The zuul web root page will automatically redirect to keycloak's login page. When back to Zuul, you can check your token and the tenants on which the user is allowed to perform actions, they are logged in the browser's javascript console. If you don't feel like deploying, here is a short screencast of what enqueueing/dequeueing from the GUI could look like: https://softwarefactory-project.io/static/zuul_jwt_demo.mkv Where can you help? The GUI part is in no way production ready, it is tightly coupled to keycloak, and I've started learning React about 2 weeks ago, so the code is terrible and takes a lot of shortcuts. Anybody willing and able to take over is welcome! the spec however is in a state where it is ready for reviews, same for the gerrit topic implementation. Feedback and possibly merging would be greatly appreciated. Cheers, MHU [1] https://review.openstack.org/#/c/562321/ [2] https://review.openstack.org/#/q/topic:zuul_admin_web [3] https://review.openstack.org/#/c/643536/

1 0

Proposal to create zuul-operator project
by Monty Taylor 08 Apr '19

08 Apr '19

Hey everybody, It seems that kubernetes operators are all the rage these days, and in the last 2 weeks I've had multiple conversations that have all come back around to someone wanting a zuul operator to exist. SO ... I would like to propose that we create a zuul-operator project as part of Zuul. Given our close ties with Ansible, I think we should use ansible-operator [0] to write it. Since we have zuul in production on both vanilla k8s and on OpenShift, I think we should set up gate jobs to make sure it works on both. Once we're happy with it - we should publish it to operatorhub. [1] Does anyone object? It not, I'll get a repo started. Monty [0] https://github.com/operator-framework/operator-sdk/blob/master/doc/ansible/… [1] https://www.operatorhub.io/

9 23

New driver for Nodepool
by Alexander Braverman Masis 03 Apr '19

03 Apr '19

Hello, I'm interested in writing a new driver for Nodepool based on Linchpin[1]. Should I contribute it to zuul core as a built in driver, or add a configuration option so the driver can be developed separately and run from out-of-tree? Thanks, [1] https://github.com/CentOS-PaaS-SIG/linchpin -- Alex Braverman Masis

2 1

PTG Team Photos
by Kendall Nelson 28 Mar '19

28 Mar '19

Hello! If your team is interested in having a team photo taken, here is the signup[1]! There are slots Thursday and Friday from 10:00 AM to 4:30 PM. The location is TBD but will likely be close to where registration will be. I'll send an email out the day before with a reminder of your time slot and an exact location. -Kendall (diablo_rojo) [1] https://docs.google.com/spreadsheets/d/1DgsRHVWW2YLv7ewfX0M21zWJRf4wUfPG4ff…

1 0

Zuul project update 2019-03-25
by corvus＠inaugust.com 25 Mar '19

25 Mar '19

Hi, Since the last update, Tristan has updated the zookeeper auth patch based on recent comments, rebased the zuul-runner patch on the multi-ansible stack, and Investigated how to enable the executor service to use a COE. I expect we'll have a good discussion about that once Tristan's findings are ready. Fabian worked on a Elasticsearch Zuul reporter to enable indexation of build and buildset data. It enable building analytics dashboards on top of Kibana for instance. I reviewed and helped merge Tobias's work on multi-ansible. We upgrade OpenDev's Zuul, released 3.7.0, found some bugs, fixed them, and I plan on releasing 3.7.1 today. Next I plan on reviewing the zuul-runner series (as outlined in the previous update email). The remainder of the backlog from the last update email: 1) The zuul-runner series This has been in development for a long time, and has a significant impact on the executor. Now that the multi-ansible series has merged and it has been rebased, let's focus on this next. 2) The web authentication spec This has been in development for a while, and folks working on it recently signaled it's ready for review. This shouldn't conflict with anything ahead of it, so we can review it as time permits. 3) The web authentication series Once the spec lands, we can review the implementation which has been developing in parallel. Help write the next update email by updating https://etherpad.openstack.org/p/zuul-update-email Thanks, Jim

1 0