- Zuul-discuss - lists.opendev.org

Different merge-mode for different pipeline?
by Nick Hwangbo 29 May '19

29 May '19

Hello, Is it possible to specify different merge-modes for different pipelines for the same repository? For example, is something like this possible in a zuul.yaml configuration: - job: name: test run: playbooks/test.yaml - project: merge-mode: cherry-pick check: jobs: - test - project: merge-mode: merge-resolve gate: jobs: - test The check pipeline would use cherry-pick as a merge mode, while the gate pipeline would use merge-resolve? Thanks, Nick

2 2

Zuul project update 2019-05-29
by corvus＠inaugust.com 29 May '19

29 May '19

Hi, It's been a while since I've sent one of these out, largely due to a busy schedule involving launching OpenDev, vacation, conferences, and associated travel. Now seems a good time to try to resync. First specs: I think the tenant-scoped admin web API spec is ready for final approval. I think everyone who has helped refine that has had their issues addressed, so we probably just need folks to go over it once again and give it the OK. I'd like to merge that next Friday, June 7 unless anyone thinks we need more time. Please take a look: https://review.opendev.org/562321 The operator spec should probably be a priority for us too, based on the amount of interest here (and the POC implementation Tristan has waiting in the wings). This one is still in the design phase, so please add your comments now to make sure it takes the right shape: https://review.opendev.org/659180 Other activity: Tristan has updated the freeze-job stack again, and started work on a new "cleanup" phase which we've previously discussed adding. Shrews has cleaned up image build leaks in Nodepool's builder, improved Nodepool unit tests and tooling around detecting memory leaks. He ported the OpenDev Nodepool to use jemalloc -- it seems that our general recommendation for operating large Zuul systems is to use that. He's currently working on improving the Zuul autohold experience. We may want to look into whether we should line that up behind the tenant web admin API to avoid conflicts. Tobias has been adding event IDs to log messages and generally improving the logging system to account for that kind of extra data; this should help trace events through the various services. In the GitHub driver, he's adding support for CODEOWNERS files, and a squash-merge option. And he's working on support for smart reconfiguration (like full reconfiguration but only for tenants that changed their configs). I've been working on a job to test the new Docker registry roles. It's raising some issues about how we should do testing of zuul-jobs repos going forward, and it's looking like we may need a new repo to hold these test jobs. I'll propose a change to create that after I get the first draft of this job finished. Help write the next update email by updating https://etherpad.openstack.org/p/zuul-update-email Thanks, Jim

1 0

Gerrit Downtime May 31, 2019 beginning 1500UTC
by Clark Boylan 28 May '19

28 May '19

Hello everyone, We'll be taking a short (hopefully no longer than an hour) Gerrit downtime so that we can rename some projects this Friday, May 31, 2019 beginning at 1500UTC. Some of these renames are cleanups after the great OpenDev git migration and others aren't, but all of them require us to stop Gerrit to update the database and contents on disk. If you'd like specifics on what repos are being renamed or what the process looks like, you can follow along at: https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Upcoming_Project_… https://etherpad.openstack.org/p/project-renames-2019-05-31 As always feel free to bring up questions or concerns and we'll do our best to answer/address them. Thank you for your patience and sorry for the interruption, Clark

1 0

AMQP driver
by Tristan Cacqueray 26 May '19

26 May '19

Hello, There is an initial AMQP driver proposed waiting for reviews[0]. The use-case is to integrate Zuul with fedmsg/umb[1] which is used for artifacts testing. The goal is to run Zuul jobs for the different testing phases that happen after a change is merged, namely on koji builds and on composition events. I guess this goes beyond Zuul's git driven model, and the AMQP trigger would need to tell the jobs where the external provided artifacts are. So could Zuul be extended to support such a use-case? Regards, -Tristan [0]: https://review.opendev.org/#/q/topic:amqp-trigger [1]: https://pagure.io/fedora-ci/messages

2 2

periodic jobs (timer tigger) and job.files attribute
by Paul Belanger 24 May '19

24 May '19

Greetings, Over the last few days I've been working to create a job that is shared between check / gate pipelines and periodic pipeline. As an example, using the following job stanza: - job: name: ansible-test-network-integration-vyos parent: ansible-network-vyos-appliance pre-run: playbooks/ansible-test-network-integration-base/pre.yaml run: playbooks/ansible-test-network-integration-base/run.yaml required-projects: - name: github.com/ansible/ansible timeout: 3600 vars: ansible_test_network_integration: vyos files: - ^lib/ansible/modules/network/vyos/.* - ^lib/ansible/module_utils/network/vyos/.* - ^lib/ansible/plugins/connection/network_cli.py - ^test/integration/targets/vyos_.* you can see we are using the job.files attribute, this works as expects when we open a PR in github against github.com/ansible/ansible. Here is the pipeline stanza: - project-template: name: ansible-test-network-integration periodic: jobs: - ansible-test-network-integration-vyos third-party-check: jobs: - ansible-test-network-integration-vyos However, the issue is, current periodic jobs are not running, because from what I see the job.files attribute does not match. So, based on a suggestion from Spamaps I tried the following: - project-template: name: ansible-test-network-integration periodic: jobs: - ansible-test-network-integration-vyos third-party-check: jobs: - ansible-test-network-integration-vyos: files: [] which didn't work, I even tried: - project-template: name: ansible-test-network-integration periodic: jobs: - ansible-test-network-integration-vyos third-party-check: jobs: - ansible-test-network-integration-vyos: files: - ^.* and no luck. The best I can guess, is that jobs enqueued into the periodic pipeline via timer trigger don't seem to have files attribute for the change? Which basically prevent any job.files match possible. My question here is, how could we allow this? Does it make sense if we have job.files set to an empty list, remove any existing job.files defined? Or, if periodic pipelines don't present files in the change object, could we just drop the job.files match process? I know that I could remove the job.files from the job stanza, and move them into the pipeline stanza, however in this case I would not like to do that for reasons. I also know I could create another job.name, maybe prefaces with periodic, again I'd prefer not to do that too. However, if there is no way to fix this based on above, I'll have to do one of these solutions. Thanks, Paul

3 3

ansible version for linting?
by Andreas Jaeger 23 May '19

23 May '19

Which ansible version should we use for linting in repos like openstack/project-config, openstack/openstack-zuul-jobs and zuul/zuul-jobs? OpenDev runs ansible 2.7 which is the default for Zuul - but Zuul supports 2.5 (as deprecated) and 2.6 and soon 2.8 as well... So, do we lint with the oldest version? The newest? The default? right now we use ansible 2.5.x for linting. I proposed the first changes to go to 2.7 - but then thought further and wrote this email: https://review.opendev.org/#/c/659810/ https://review.opendev.org/#/c/659811/ Andreas -- Andreas Jaeger aj(a){suse.com,opensuse.org} Twitter: jaegerandi SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 -- Andreas Jaeger aj(a){suse.com,opensuse.org} Twitter: jaegerandi SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126

1 1

Ansible 2.8 support
by Paul Belanger 13 May '19

13 May '19

Greetings, Hopefully this week ansible 2.8.0 will be released, yay. As a results, Tobias as already done most of the work[1] to get Zuul ready to support it. I'd like to ask for humans to review the patch, the main change here for zuul is to our console stream plugin. Ansible 2.8 now includes rescued / ignored in the results output. Also, facts are now disabled by default when using gather_facts. I think it would be awesome if we could land and do a zuul release just after ansible 2.8 was released. Would go well with out story of cross project collaboration. [1] https://review.opendev.org/631933

1 1

Zuul's Roadmap and Backlog
by Tristan Cacqueray 12 May '19

12 May '19

Hello folks, I don't know if this subject has been discussed at the last PTG, so here is a thread about it. There is a growing backlog of bug fixes and features, and it's a bit frustrating not knowing when or if a patch is going to be approved. For the 3.0 release, there was a board with the list of stories and this was very helpful in communicating what was going to be included and what should have been reviewed. Would it be possible to use such board for the next releases? As an example, here is a not exhaustive list of features with unknown status. Wouldn't it help to pin them to a release so that reviewer/contributor could help efficiently? * topic:github-review-reporter * topic:zubbi-integration * topic:pagure-driver * topic:web-openapi * topic:zuul_admin_web * topic:scale-out-scheduler * topic:freeze_job Regards, -Tristan

5 9

Job doesn't enter the gate pipeline
by Vitaliy Lotorev 11 May '19

11 May '19

Hi, I'm trying to setup Zuul with in-house Gerrit. check jobs from independent pipeline are started in predictable way (on new PS and 'recheck' comment). The issue is I can't start jobs in buildroot_gate (dependent) pipeline. I simplified pipeline to this one: - pipeline: name: buildroot_gate manager: dependent require: gerrit: open: true trigger: gerrit: - event: comment-added comment: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*gate success: gerrit: Verified: 1 mysql: failure: gerrit: Verified: -1 mysql: I.e. for debug purpose I'd like buildroot_gate pipeline to get started on 'gate' comment. Once I add 'gate' comment to PS Zuul on Gerrit Scheduler successfully detects this and reports the as follows: scheduler_1 | 'url': ' https://mycompanygerrit/c/production/vip1-boards/sd-fc/+/12134'}] scheduler_1 | 2019-05-09 20:53:58,502 DEBUG zuul.Scheduler: Change <Change 0x7f18e2104c50 production/vip1-boards/sd-fc 12$ 34,12> has been updated, clearing dependent change caches scheduler_1 | 2019-05-09 20:53:58,502 DEBUG zuul.GerritConnection: Scheduling event from gerrit: <GerritTriggerEvent comm$ nt-added mycompanygerrit/production/vip1-boards/sd-fc master 12134,12 Verified:0, Code-Review:1> scheduler_1 | 2019-05-09 20:53:58,502 DEBUG zuul.Scheduler: Run handler awake scheduler_1 | 2019-05-09 20:53:58,502 DEBUG zuul.Scheduler: Fetching trigger event scheduler_1 | 2019-05-09 20:53:58,503 DEBUG zuul.Scheduler: Processing trigger event <GerritTriggerEvent comment-added ge$ rit.elvees.com/production/vip1-boards/sd-fc master 12134,12 Verified:0, Code-Review:1> scheduler_1 | 2019-05-09 20:53:58,503 DEBUG zuul.Pipeline.nto7.buildroot_gate: Event <GerritTriggerEvent comment-added ge$ rit.elvees.com/production/vip1-boards/sd-fc master 12134,12 Verified:0, Code-Review:1> for change <Change 0x7f18e2104c50 p$ oduction/vip1-boards/sd-fc 12134,12> matched <GerritEventFilter types: comment-added ignore_deletes: True comments: (?i)^($ atch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*gate> in pipeline <DependentPipelineManager buildroot_gate> scheduler_1 | 2019-05-09 20:53:58,503 DEBUG zuul.Pipeline.nto7.buildroot_gate: Considering adding change <Change 0x7f18e2$ 04c50 production/vip1-boards/sd-fc 12134,12> scheduler_1 | 2019-05-09 20:53:58,503 DEBUG zuul.Pipeline.nto7.buildroot_gate: Change <Change 0x7f18e2104c50 production/v$ p1-boards/sd-fc 12134,12> can not merge, ignoring scheduler_1 | 2019-05-09 20:53:58,503 DEBUG zuul.Pipeline.nto7.buildroot_gate: Change <Change 0x7f18e2104c50 production/v$ p1-boards/sd-fc 12134,12> is not ready to be enqueued, ignoring scheduler_1 | 2019-05-09 20:53:58,503 DEBUG zuul.Pipeline.nto7.check: Starting queue processor: check scheduler_1 | 2019-05-09 20:53:58,503 DEBUG zuul.Pipeline.nto7.check: Finished queue processor: check (changed: False) scheduler_1 | 2019-05-09 20:53:58,504 DEBUG zuul.Pipeline.nto7.gate: Starting queue processor: gate scheduler_1 | 2019-05-09 20:53:58,504 DEBUG zuul.Pipeline.nto7.gate: Finished queue processor: gate (changed: False) scheduler_1 | 2019-05-09 20:53:58,504 DEBUG zuul.Pipeline.nto7.buildroot_gate: Starting queue processor: buildroot_gate scheduler_1 | 2019-05-09 20:53:58,504 DEBUG zuul.Pipeline.nto7.buildroot_gate: Finished queue processor: buildroot_gate (c hanged: False) Please, note Scheduler logs "can not merge, ignoring", "is not ready to be enqueued, ignoring". How could I understand the reason why it can't be merged? This PS it definitely mergable manually: I could vote +2 and Sumbit the PS. -- Regards, Vitaliy

2 2

Correct way to trigger a periodic job manually
by Ian Wienand 07 May '19

07 May '19

Hello, The manual says to trigger a periodic job against master you'd do [1] $ zuul enqueue-ref --tenant openstack --trigger timer --pipeline periodic --project openstack/example_project --ref refs/heads/master Which looks about right to me, but this sets a newrev of 00..000 which is not correct; e.g. [2]. Tracing back that job to find the cause of the RETRY_LIMIT on ze06 I see: --- 2019-05-06 07:20:50,875 DEBUG zuul.AnsibleJob: [build: b1387c9741134f83a6f11e1cad0386f0] Checking out master 2019-05-06 07:20:50,877 ERROR zuul.AnsibleJob: [build: b1387c9741134f83a6f11e1cad0386f0] Exception while executing job Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/git/refs/symbolic.py", line 305, in set_reference obj = self.repo.rev_parse(ref + "^{}") # optionally deref tags File "/usr/local/lib/python3.5/dist-packages/git/repo/fun.py", line 213, in rev_parse obj = name_to_object(repo, rev[:start]) File "/usr/local/lib/python3.5/dist-packages/git/repo/fun.py", line 147, in name_to_object raise BadName(name) gitdb.exc.BadName: Ref 'master' did not resolve to an object During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/zuul/executor/server.py", line 790, in execute self._execute() File "/usr/local/lib/python3.5/dist-packages/zuul/executor/server.py", line 914, in _execute repo.checkout(selected_ref) File "/usr/local/lib/python3.5/dist-packages/zuul/merger/merger.py", line 369, in checkout File "/usr/local/lib/python3.5/dist-packages/git/refs/symbolic.py", line 308, in set_reference raise ValueError("Could not extract object from %s" % ref) ValueError: Could not extract object from master --- I set newref manually to the git master HEAD change-id, and it did work -- but it looks different to jobs actually triggered by the periodic driver itself. For example [3] is the job I triggered, and [4] is one from the periodic driver -- [4] does not have a newref. I feel like this used to work, probably because when I look back I wrote that bit of documentation [5] ;) I guess something changed; perhaps it is the client setting newref to 0000.000 by default instead of something like None? Any tips on what should I update the instructions to say now? -i [1] https://zuul-ci.org/docs/zuul/admin/client.html#manual-enqueue-examples [2] http://zuul.openstack.org/build/b1387c9741134f83a6f11e1cad0386f0 [3] http://zuul.openstack.org/build/8781cd38097b437d90ba5a9a5505633c [4] http://zuul.openstack.org/build/d30f960f7ca04b33af1eb5b7513b1b66 [5] https://review.opendev.org/#/c/518662/

2 2

exclude-unprotected-branches equivalent for Gerrit connection
by Brendan Jackman 07 May '19

07 May '19

Hi all, In my org's Gerrit setup, refs under personal/$username/ are a free-for-all. I have seen the same in other organisations too. This leads to the problem that is solved for Github by exclude-protected-branches; those branches eventually end up with broken Zuul configs (actually case that brought this to my attention is that I cannot delete a job that is no longer used, because the config in a personal/ branch refers to it). It seems like a simple solution would be to add an option under tenant.untrusted-projects.<project> that can be used to filter the branches considered by Zuul, by their name. I think the most obvious interface for that would be like job.files & job.irrelevant-files, i.e. a regex or list thereof, with a negative complement option alongside it. I think it could then behave much like exclude-unprotected-branches in that it works at the source connection level. Although I've mentioned Gerrit, I don't see why it shouldn't be independent of the actual source connection driver. This seems suspiciously obvious so I suspect I may be rather naive in suggesting it! Has it been discussed before, does it present more complications than I can see? I'm not too sure how this would related to tags. What does Zuul do with those? I didn't give the proposed option a name because I don't know if what I really want is a ref filter or a branch filter. In fact, I face a certain degree of pain because a couple the projects my Zuul deployment checks out have a lot of tags[1] so maybe this could kill two birds with one stone. Cheers, Brendan [1] $ git ls-remote --tags --refs git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git | wc -l 2763 This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed (‘intended recipient’). Accordingly the dissemination, distribution, copying or other use of this email or any of its content by any person other than the intended recipient may constitute a breach of civil or criminal law and is strictly prohibited. If you are not the intended recipient, please let us know as soon as possible and remove this email from your system. Company Reg No: 06855929 | Blu Wireless Technology Ltd, 5th Floor, 1 Temple Way, Bristol, BS2 0BY | Registered in England & Wales

2 2

Inheritance confusion, docs needed?
by Michael Johnson 06 May '19

06 May '19

As a developer creating Zuul jobs, one of the areas i find confusing is the inheritance of variables. I am not sure if this is an enhancement style request or just a missing section in the documentation. When you use host-vars, the top level variables are "merged" with the parent job variables, however the nested variables, such as "devstack_localrc" are not merged, they are "replaced". As an example is the opendev Zuul jobs (I think this applies to non-OpenStack jobs as well): When you use host-vars, the top level variables are "merged" with the parent job variables, however the nested variables, such as "devstack_localrc" are not merged, they are "replaced". host-vars: controller: devstack_localrc: DATABASE_USER: a-user In this example snippet, the variables at the devstack_localrc level are "merged", but the variables at the "DATABASE_USER" level are replaced and not inherited from the parent. Should Zuul be recursively "merging" all the way down or should we document how/which variables are inherited from the parent and which are not? The reason I'm asking is as our job definitions get more complicated (multi-node), I find myself copying variables from the parent jobs that I would really like to just inherit. Michael

2 1

Zuul working group session at OpenInfra Summit
by corvus＠inaugust.com 29 Apr '19

29 Apr '19

Hi, If you're at the OpenInfra Summit and would like to discuss what is or isn't working about sharing jobs between Zuul installations, please join the working group session on Tuesday: https://www.openstack.org/summit/denver-2019/summit-schedule/events/23736/z… -Jim

1 0

Celebrating Zuul's OSF Top Level Project Confirmation
by Clark Boylan 28 Apr '19

28 Apr '19

Zuul was confirmed by the OSF board as a top level OSF project today in Denver. To celebrate we'd like to meetup for arcade games, drinks, and food at FTW Denver Monday night. This is doubling as the OpenDev Infra team gathering, but considering the overlap between Zuul and Infra I figure that won't be an issue. If you'd like to join us we will be gathering at 6:45pm Monday in front of the big blue bear statue at the convention center. If it is raining, we'll likely meet up just inside the convention center building next to the statue to stay dry. Hope to see you there and congratulations Zuul! Clark

1 0

Tagging beta / rc releases
by Paul Belanger 24 Apr '19

24 Apr '19

Greetings, I wanted to raise the topic of maybe using beta / rc tags to publish pre-release versions of zuul / nodepool to pypi. Currently, before each official release we usually select a git SHA to be installed onto zuul.opendev.org, to ensure we can run a scale without any issues. Then a few days later, if things are well we tag a new release or work to address the problem. Then restart the process until things look good. I really like this approach, as currently there is no larger zuul and helps to catch any issues before our users install the released version. However, I would like to propose we make a minor change, is when we select to git SHA, is to also tag it using a pre-release version number (eg: beta / rc). By having it pre-release, (eg: pip install --pre zuul) would allow my current tooling (today we only installed from pypi, not git) to pick up the change and also provide some feedback. Our deployment tooling is setup to only use releases from pypi (wheels) in an effort to try not to carry any patches downstream. We could also switch to pip install via git, however this would add some extra overhead to production servers to compile wheels. - Paul

3 8

Migrating git repos to OpenDev
by corvus＠inaugust.com 21 Apr '19

21 Apr '19

Hi, As discussed in November[1], the OpenStack project infrastructure is being rebranded as "OpenDev" to better support a wider community of projects. We are nearly ready to perform the part of this transition with the largest impact: moving the authoritative git repositories for existing projects. In this email, I'd like to introduce the new hosting system we are preparing, discuss the transition, and invite projects to work with us on the logistics of the change. Gerrit ====== Gerrit is the core of our system and it will remain so in OpenDev. As part of this move, we will rename the gerrit server from review.openstack.org to review.opendev.org. As part of the transition, we will automatically merge appropriate changes to all branches of all repositories updating .gitreview and Zuul configuration files. Any further changes (README files, etc.) we expect to be made by individual project contributors. Repository Browsing =================== Currently our canonical *public* repository system is the cgit server at https://git.openstack.org/ (and git.airshipit.org, git.starlingx.io, and git.zuul-ci.org) This is a load balanced cluster of several servers which is designed to handle all the public git repository traffic, as it scales much better than Gerrit (and has a more friendly domain name). >From a technical standpoint, it's excellent, but its usability could be improved. Therefore, as part of this transition, we will replace the cgit servers with a new system based on Gitea. Gitea is a complete development collaboration system, but it's very flexible and will allow us to disable components which we aren't using. We will operate it in a read-only configuration where it will act as the public mirror for Gerrit. The advantages it has over the current system are: * Shorter domain name in project URLs: https://git.openstack.org/openstack/nova vs https://opendev.org/openstack/nova * Clone and browsing URLs are the same (with cgit, the browsing URL has an extra path component) * More visually pleasing code browsing * Integrated code searching * Ability to highlight multiple lines in links When we perform the transition we will install redirects from git.openstack.org (and the other git sites) to opendev.org, and will maintain those redirects for the foreseeable future. We will construct them so that even existing deep links to individual files in individual commits to cgit will redirect to the correct location on opendev.org. This system is up and running now with a live mirror of data from Gerrit, and you can start testing it out today at https://opendev.org/ Please let us know if you encounter any problems. If you would like to read more about the design of this system and the transition, see the infra-spec[2]. GitHub ====== Currently all OpenStack projects are replicated to GitHub. We do not plan on changing that during the transition, however, any projects outside of the openstack*/ namespaces will not automatically be replicated to GitHub, and we do not plan on adding that in the future. We do, however, support projects using Zuul to run post-merge jobs to push updates to GitHub or any other third-party mirrors with their own credentials. We will be happy to work with anyone interested in that to help set up jobs to do so. We are adopting this approach so that individual projects can have more control over how they are represented in social media, and to give us more flexibility in supporting our own organizational namespaces on OpenDev without assuming they map directly to GitHub. Eventually we plan on moving the OpenStack project to that system as well and retiring direct replication from Gerrit to GitHub completely. But we will defer that work until after this transition. Logistics ========= We can prepare much of the system in advance (as we have for the hosting system on opendev.org) but the actual transition and renaming of the Gerrit server will need to happen at once during an outage window. We need to schedule that outage and begin preparing for it. Since all of the project git URLs are going to change (to replace git.openstack.org with opendev.org and review.openstack.org with review.opendev.org) we can additionally take the opportunity to reorganize projects into different organizations. For example, during the transition we will rename Zuul, and it's associated projects, from the "openstack-infra" org to "zuul". So their new names will be "zuul/zuul", "zuul/nodepool", etc. This is an excellent time for the rest of the OpenStack Foundation pilot projects to do the same. If the OpenStack project desires this, it would also be a good time to move unofficial projects out of the openstack/ namespace. Therefore, we need your help: Action Items ============ We need each of the following projects: * OpenStack * Airship * StarlingX * Zuul To nominate a single point of contact to work with us on the transition. It would be helpful for that person to attend the next (and possibly next several) openstack infra team meetings in IRC [3]. We will work with those people on scheduling the transition, as well as finalizing the list of projects which should be renamed as part of the transition. If you manage an unofficial project and would like to take the opportunity to move or rename your project, please add it to this ethercalc[4]. [1] http://lists.openstack.org/pipermail/openstack-dev/2018-November/136403.html [2] http://specs.openstack.org/openstack-infra/infra-specs/specs/opendev-gerrit… [3] http://eavesdrop.openstack.org/#Project_Infrastructure_Team_Meeting [4] https://ethercalc.openstack.org/opendev-transition

5 6

Post OpenDev Repo Migration Issues
by Clark Boylan 20 Apr '19

20 Apr '19

Hello, There are two quick things we want to point out that we've noticed after the OpenDev Gerrit manuevers. First, any commit Depends-On footers for unmerged changes will need to use https://review.opendev.org urls instead of https://review.openstack.org urls. This is due to a limitation in Zuul and how it sees the Gerrit server. Any Depends-On footers that use https://review.openstack.org are currently non-functional and they will not enforce the dependency relationship. Second, while we have fairly robust http(s) redirects in place for old urls (including those from cgit to gitea) any ssh repo urls will need to be updated to use the new canonical paths if a repo has moved. We are unable to redirect incoming ssh requests against Gerrit. If you need to know what the canonical path is go to the corresponding https repo url and the resulting destination is the canonical location. Again thank you for your patience, Clark

3 2

Should we drop our custom Ansible plugins?
by corvus＠inaugust.com 15 Apr '19

15 Apr '19

Hi, When we created Zuul v3, we faced a fairly unique situation -- we needed to run untrusted Ansible playbooks without compromising the host running Ansible (i.e., the Zuul executor). To address this, we created a set of Ansible plugins which wrap the upstream Ansible plugins and block actions which we consider dangerous. Broadly speaking, there are two classes of actions we might want to block: 1) Actions which may expose private information stored on the executor (including information used by Zuul itself or other jobs), or otherwise compromise the security of the executor itself (so that future jobs can not be trusted). 2) Actions which may allow a user to take advantage of privileges afforded to the executor host itself. For example, if the executor is behind a network firewall and therefore can access unprotected network services. We added bubblewrap as a second layer of defense against the first class. It has been very effective -- our custom plugins have had several vulnerabilities, all of which were ultimately unexploitable due to the use of bubblewrap. Bubblewrap doesn't address the second class. Here's more background on the bubblewrap addition: https://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3-execut… Maintaining the Ansible plugins is a lot of work, and it's very easy to get wrong. For each new Ansible version, we need to look for any changes to module parameters and make corresponding updates. All of our security issues so far have been due to things we missed here. Meanwhile, these custom plugins increase the area where playbooks which may otherwise work can not be used with Zuul (for example, because they use a lookup plugin which writes to the executor). This makes Zuul more difficult to use. Therefore, I propose that we drop support for the custom plugins and rely entirely on bubblewrap to protect us from the first class of issues. It will mean less work for us, make Zuul more useful, and, at least with regard to the first class of problems, I don't think Zuul will be less secure. That brings me to the second class. I have a feeling (which is unsubstantiated), that there are probably enough holes in our plugins which make the second class of actions possible too. For example, we allow use of the URI module, but that means that any playbook can hit any HTTP URL accessible by the executor. While it may be difficult for a user to make a connection to a mysql server, it's not outside the realm of possibility. Dropping support for our custom plugins would mean that Zuul has no defense against this class of action. We would need to document that and ensure that users know that if their executors run untrusted code that they should have adequate application or network level protections in place. In short: don't put an executor inside the squishy center of your security perimeter. Before we do this, we need to make sure that Zuul itself doesn't impose any constraints that contradict this. The executor only uses Gearman to communicate with the rest of Zuul, and it can adequately be secured. Before we add any features which require the executor to communicate via ZooKeeper, we should make sure that we support zookeeper auth. What do folks think? Is this a reasonable change to make, or should we continue supporting the custom plugins? -Jim

7 19

Upgrading the lists.(openstack|airshipit|starlingx|zuul-ci).org server Friday April 12
by Clark Boylan 12 Apr '19

12 Apr '19

It is that time of the Ubuntu LTS cycle again and we need to upgrade our mailman mailing list server. We'd like to do that this Friday, April 12. We expect the upgrade to begin at about 17:00UTC and result in a 30-45 minute outage. The reason for the extended outage is that we will be upgrading the server in place to preserve its mail reputation. Thankfully email is a persistent system and your clients should queue up email sent until the server is back up again and accepting smtp connections. This means the outage shouldn't be very noticeable. Finally, for list admins, there are new DMARC moderation action settings. We ask that you don't change these settings and instead work with us if you need to address DMARC problems. Our current preference is that we pass email through unmodified so that the signatures still validate. Thank you all for your patience and feel free to ask us questions, Clark

1 1

New Zuul job to replicate a project's git repository to a remote git server
by David Moreau Simard 11 Apr '19

11 Apr '19

Hi, It is now possible for projects to replicate their git repository to a custom location by inheriting from the 'upload-git-mirror' job provided by Zuul. This job wraps around the 'upload-git-mirror' Ansible role that is part of the zuul-jobs library [1]. In order to use this job, you must supply a secret in the following format: === - secret: name: <name of your secret> data: user: <ssh user of the remote git server> host: <address of the remote git server> host_key: <ssh host key of the remote git server> ssh_key: <private key to authenticate with the remote git server> === The 'host_key' parameter can be retrieved from your known_hosts file or with a command like 'ssh-keyscan -H <host>' or 'ssh-keyscan -t rsa <host>'. For example, the 'host_key' when pushing to GitHub would be, on a single line: github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ== The 'ssh_key' parameter should be encrypted before being committed to the git repository. Zuul provides a tool for easily encrypting files such as SSH private keys and you can find more information about it in the documentation [2]. For example, encrypting a key for the "openstack/ara" project would look like this: === zuul/tools/encrypt_secret.py --infile /home/dmsimard/.ssh/ara_git_key --tenant openstack https://zuul.openstack.org openstack/ara === You would then use the secret in a job inheriting from 'upload-git-mirror' as such: === - job: name: <project>-upload-git-mirror parent: upload-git-mirror description: Mirrors openstack/<project> to neworg/<project> vars: git_mirror_repository: neworg/<project> secrets: - name: git_mirror_credentials secret: <name of your secret> pass-to-parent: true === Finally, the job must be set to run in your project's 'post' pipeline which is triggered every time a new commit is merged to the repository: === - project: check: jobs: # [...] gate: jobs: # [...] post: jobs: - <project>-upload-git-mirror === Note that the replication would only begin *after* the change has merged, meaning that merging the addition of the post job would not trigger the post job itself immediately. The post job will only trigger the next time that a commit is merged. [1]: https://zuul-ci.org/docs/zuul-jobs/general-roles.html#role-upload-git-mirror [2]: https://zuul-ci.org/docs/zuul/user/encryption.html David Moreau Simard dmsimard = [irc, github, twitter]

1 0

Update on the tenant-scoped admin REST API for Zuul
by Matthieu Huin 11 Apr '19

11 Apr '19

Hello, I'd like to introduce the applications of the spec referred to in the title [1]. I have worked in parallel on a possible implementation of said spec, backend-side [2]. Since it is not necessarily trivial to evaluate or test by oneself, I have also tried to put together a proof of concept for the GUI using a keycloak instance as the JWT provider, with -almost- everything set up together with docker, so that people can play around with the feature. [3] It requires a few manual steps in order to be set up properly though: 1. Build the zuul containers using the code from the patchset: cd zuul/doc/sources/admin/examples && docker-compose up --build 2. log in to the keycloak instance at http://localhost:8282 as admin/admin and create users in the "zuul" realm. The default rules of the PoC expect a user called "venkman", and a user belonging to the "gozerians" group (groups should be created automatically within the zuul realm) 3. The zuul web root page will automatically redirect to keycloak's login page. When back to Zuul, you can check your token and the tenants on which the user is allowed to perform actions, they are logged in the browser's javascript console. If you don't feel like deploying, here is a short screencast of what enqueueing/dequeueing from the GUI could look like: https://softwarefactory-project.io/static/zuul_jwt_demo.mkv Where can you help? The GUI part is in no way production ready, it is tightly coupled to keycloak, and I've started learning React about 2 weeks ago, so the code is terrible and takes a lot of shortcuts. Anybody willing and able to take over is welcome! the spec however is in a state where it is ready for reviews, same for the gerrit topic implementation. Feedback and possibly merging would be greatly appreciated. Cheers, MHU [1] https://review.openstack.org/#/c/562321/ [2] https://review.openstack.org/#/q/topic:zuul_admin_web [3] https://review.openstack.org/#/c/643536/

1 0

Proposal to create zuul-operator project
by Monty Taylor 08 Apr '19

08 Apr '19

Hey everybody, It seems that kubernetes operators are all the rage these days, and in the last 2 weeks I've had multiple conversations that have all come back around to someone wanting a zuul operator to exist. SO ... I would like to propose that we create a zuul-operator project as part of Zuul. Given our close ties with Ansible, I think we should use ansible-operator [0] to write it. Since we have zuul in production on both vanilla k8s and on OpenShift, I think we should set up gate jobs to make sure it works on both. Once we're happy with it - we should publish it to operatorhub. [1] Does anyone object? It not, I'll get a repo started. Monty [0] https://github.com/operator-framework/operator-sdk/blob/master/doc/ansible/… [1] https://www.operatorhub.io/

9 23

New driver for Nodepool
by Alexander Braverman Masis 03 Apr '19

03 Apr '19

Hello, I'm interested in writing a new driver for Nodepool based on Linchpin[1]. Should I contribute it to zuul core as a built in driver, or add a configuration option so the driver can be developed separately and run from out-of-tree? Thanks, [1] https://github.com/CentOS-PaaS-SIG/linchpin -- Alex Braverman Masis

2 1

PTG Team Photos
by Kendall Nelson 28 Mar '19

28 Mar '19

Hello! If your team is interested in having a team photo taken, here is the signup[1]! There are slots Thursday and Friday from 10:00 AM to 4:30 PM. The location is TBD but will likely be close to where registration will be. I'll send an email out the day before with a reminder of your time slot and an exact location. -Kendall (diablo_rojo) [1] https://docs.google.com/spreadsheets/d/1DgsRHVWW2YLv7ewfX0M21zWJRf4wUfPG4ff…

1 0

Zuul project update 2019-03-25
by corvus＠inaugust.com 25 Mar '19

25 Mar '19

Hi, Since the last update, Tristan has updated the zookeeper auth patch based on recent comments, rebased the zuul-runner patch on the multi-ansible stack, and Investigated how to enable the executor service to use a COE. I expect we'll have a good discussion about that once Tristan's findings are ready. Fabian worked on a Elasticsearch Zuul reporter to enable indexation of build and buildset data. It enable building analytics dashboards on top of Kibana for instance. I reviewed and helped merge Tobias's work on multi-ansible. We upgrade OpenDev's Zuul, released 3.7.0, found some bugs, fixed them, and I plan on releasing 3.7.1 today. Next I plan on reviewing the zuul-runner series (as outlined in the previous update email). The remainder of the backlog from the last update email: 1) The zuul-runner series This has been in development for a long time, and has a significant impact on the executor. Now that the multi-ansible series has merged and it has been rebased, let's focus on this next. 2) The web authentication spec This has been in development for a while, and folks working on it recently signaled it's ready for review. This shouldn't conflict with anything ahead of it, so we can review it as time permits. 3) The web authentication series Once the spec lands, we can review the implementation which has been developing in parallel. Help write the next update email by updating https://etherpad.openstack.org/p/zuul-update-email Thanks, Jim

1 0