May 2021 - Zuul-discuss - lists.opendev.org

install and use ansible collections in zuul v3 ansible native job?
by Marios Andreou 23 Nov '21

23 Nov '21

hi folks long story short: can we install and use ansible collections in a zuul v3/ansible native job? In the tripleo-ci squad we are trying to use the tripleo_overcloud_image_build role from the tripleo-operator-ansible collection [1]. We are already using other roles from this collection - e.g. there [2]. This is possible because in that case we have zuul calling bash calling ansible... which means we get to configure collections in the bash bit setting ANSIBLE_COLLECTIONS_PATHS [3] before invoking ansible. For the case I'm having trouble with - what i refer to as ansible native - here https://review.opendev.org/#/c/733818/33/roles/oooci-build-images/tasks/mai… we have a zuul job playbook directly trying to install and use a collection. Currently it fails because instead of collections: (doesn't work " 'collections' is not a valid attribute? [4] ) I am trying to use include_role but then it can't find the collection's modules "ERROR! no action detected in task" [5]. I tried to set it in ansible.cfg but that won't work and anyway it is surely the wrong approach. The one thing left to try is to explicitly require ansible 2.9 for the job - but wanted to reach out and see if anyone else has any problems trying to install and use ansible collections (with modules importantly) as part of a zuul v3/ansible native job. thanks for reading and for any comments, marios [1] https://github.com/openstack/tripleo-operator-ansible/tree/master/roles/tri… [2] https://opendev.org/openstack/tripleo-quickstart-extras/src/commit/55e74bb8… [3] https://opendev.org/openstack/tripleo-quickstart/src/commit/299131bcc6dca86… [4] https://23fd78c0f6b01b9c4209-59c294b8f46eb75cda2bd004e2a1c97f.ssl.cf1.rackc… [5] https://2063b7e2bb67f86f8ec5-87fe67c942382c6ee875c514bffa4f21.ssl.cf2.rackc…

3 8

stage-output role
by James E. Blair 31 May '21

31 May '21

Hi, A while back we had the idea that something like 'stage-output' could be put in a base job post-run playbook, and then users of any job could collect log files by just adding zuul job variables. However, that role has "become: true" set for three tasks (of primary importance: being able to copy system logs) and there are jobs in zuul-jobs which drop sudo at the start of the job. It seems difficult to reconcile these two ideas. Should we add some sort of detection in the stage-output role that is basically: if sudo is available, set become: true for those tasks, otherwise don't? Or should we remove become: true from those tasks and ask users to set become on the entire role invocation if they need it? -Jim

1 1

Possible use of Matrix?
by James E. Blair 29 May '21

29 May '21

Hi, I'm starting this as a new thread to clearly differentiate it from the short term question of "if we evacuate from freenode, where should we go?". I think moving to a different IRC network is important to do in the short term, and I don't want to confuse the issues. At the same time, this whole process has sparked a lot of discussion (mostly elsewhere) about alternative technologies. One that I think shows particular promise is Matrix, and I'd like to find out if other folks feel similarly. Some quick relevant points about Matrix (you can google it for more :): * It's a decentralized chat protocol, so no single entity controls the infrastructure running it. * It is truly open-source and a variety of client interfaces exist. * It optionally supports end-to-end encryption. * There are existing bridges to other communication networks, like Freenode, OFTC. Libera should arrive soon[1]. * A variety of community, outsourced, or self-hosted options exist. * A number of large and small open-source communities use it. Regarding the clients, I'd like to highlight three options: 1) There is a robust plugin for weechat (matrix-weechat). That means folks like me who like to have a small terminal window open for all of their messaging needs can seamlessly transition to using Matrix. 2) There is a very nice webclient (called Element) that looks and behaves like all the other really nice web chat clients; so folks who like user interfaces from this century can dig it. 3) There are mobile clients as well. In all cases, due to the way Matrix works, users can switch between clients or use multiple clients simultaneously, and all of these clients have access to channel history even when disconnected. This is perhaps one of the largest pain points for new users of IRC. The reason that works is that every user (and room) belongs to a "homeserver" and the homeserver handles storing messages and serving them to clients. Homeservers federate with other homeservers to form the larger Matrix network. Any user on any homeserver can talk to any other user or join any room on any other homeserver (assuming permission is granted). Typically, users do not run their own homeservers (unless they are enthusiasts -- the sort of folks who run their own email or web servers). Instead, they would either use a homeserver run by their company, or a project, or use the main public server hosted by the foundation which supports Matrix development: matrix.org. In short, this provides an ideal experience: * No need to run their own infrastructure * Federated, open-source, open protocols (no vendor lock-in) * Ability to communicate with many chat networks at once * Modern UX using more corporate-firewall-friendly connectivity options When you combine this with the IRC bridging support, Matrix makes an ideal interface to all of the IRC networks in addition to the Matrix itself. Okay, here's my vision for the future: First, I'd like to suggest that folks sign up for a matrix.org account and use the IRC bridge to connect to #zuul on freenode. If/when we switch to a different IRC network, just follow us to that network. This is an easy way for us to evaluate Matrix (both the clients and the server network) and decide if we like the technology. Assuming we do, then we should consider eliminating the middle-man, and move from a bridged IRC channel to a native Matrix room. At that point, we will want a homeserver for the room at least. This is where getting feedback early so that we know what to prepare for would be good (do we want a homeserver just to host rooms? or should we also be looking to host user accounts as well?). There are discussions to be had with OpenDev and the Foundation about this, but first we need to know where we want to go. If this sounds like a terrible idea, please let me know. If it sounds worth exploring, then I encourage you to do what I'm doing right now: use Matrix to interact with #zuul on IRC instead of a native IRC client. Try it out and let's see how it works. Here's what you do: 1) Go to https://app.element.io/ and sign up for an account. You know the drill. 2) Once you're logged in, click on the "+" next to "Rooms" and then "Explore public rooms". 3) Under the search bar where it says "Matrix rooms (matrix.org)" click the drop-down and switch it to "Freenode". Type "#zuul" into the search box. Click Join. 4) Now, ideally this would be the end of the instructions, and if we move to OFTC, hopefully it will be; but because of the spam problem on freenode, #zuul requires nick registration. That means we need to tell the matrix.org IRC bridge to use our registered nicks. You'll note that you have been kicked from #zuul and you have an invitation to join a room "Freenode IRC Bridge status". Accept the invitation. 5) Type "!help" to see a list of commands. 6) Type "!nick" followed by your existing Freenode nick. 7) Type "!storepass" followed by the password for your Freenode nick. Note, you're giving your nick password to a bot run by a third party. You'll need to evaluate whether you're okay with that. At the least, use a unique password. 8) Type "!join #zuul" (that should have the same effect as adding the room earlier). I admit, these instructions are weird since I told you to start by doing something that won't work. But I do want to show how simple this should be once we can either relax the nick registration requirement or move off of IRC completely; either way, it's just the first 3 steps. If you want to skip that part, you could initiate a PM to "@appservice-irc:matrix.org" and start at step #5. Each IRC bridge has a different bot running it; to do the same thing for OFTC, you would talk to "@oftc-irc:matrix.org". That's it for now. Please let me know if you have any feedback, and I'd encourage everyone to at least try out element/matrix. I've been using a lot of different chat systems lately, and I'm really pleased with how this is working out. Even if we decide to stay on IRC, I plan to use the Matrix bridge as my primary interface. If you do too, then it will be easy to stay connected with projects on various IRC networks as well as be prepared for any potential moves to Matrix. -Jim [1] https://github.com/matrix-org/matrix-appservice-irc/issues/1324

4 5

OpenDev IRC services are moving to OFTC this weekend
by Jeremy Stanley 26 May '21

26 May '21

As a majority of our constituent projects have voiced a preference for enacting our long-standing evacuation plan, the OpenDev Collaboratory's IRC service bots will be switching from Freenode to the OFTC network this weekend (May 29-30, 2021). We understand this is short notice, but multiple projects have requested that we act quickly. Please expect some gaps in channel logging and notifications from our various bots over the course of the weekend. I have provided a much more detailed writeup to the service-discuss mailing list, and encourage anyone with questions to read it and follow up there if needed. Subsequent updates will be sent only to service-discuss, in order to limit noise for individual project lists and keep further discussion focused in one place as much as possible: http://lists.opendev.org/pipermail/service-discuss/2021-May/000249.html -- Jeremy Stanley

2 1

OpenInfra Live - May 27 at 9am CT (1400 UTC)
by Erin Disney 26 May '21

26 May '21

Hi everyone, This week’s OpenInfra Live episode is brought to you by the Zuul community! Episode: Large Scale Open Source CI Featuring Zuul Date and time: May 27 at 9am CT (1400 UTC) Don’t merge broken code. Infrastructure at scale relies on quality software that is tested before it’s deployed. Operators rely on open source CI systems like Zuul for gating, scaling across organizations and cross-project dependencies. In this OpenInfra Live episode, Jim Blair, Zuul Maintainer and CEO at Acme Gating, and Mohammed Naser, CEO of Vexxhost will provide an overview of Zuul, the open source CI/CD business case and a demo showing what cross project dependencies looks like in production. You can watch us live on: YouTube: https://www.youtube.com/watch?v=yf5iFiCg_Tw <https://www.youtube.com/watch?v=yf5iFiCg_Tw> LinkedIn: https://www.linkedin.com/feed/update/urn:li:ugcPost:6798703707925168128/ <https://www.linkedin.com/feed/update/urn:li:ugcPost:6798703707925168128/> Facebook: https://www.facebook.com/104139126308032/posts/3974109482644291/ <https://www.facebook.com/104139126308032/posts/3974109482644291/> WeChat: recording will be posted on OpenStack WeChat after the live stream Thanks, Erin Erin Disney Event Marketing Open Infrastructure Foundation

1 0

Pipeline run on Gerrit trigger change merge event
by Sankar Palanivel 26 May '21

26 May '21

Hi, We have configured 2 gerrit connections in zuul. We are expecting to have a zuul job of gerrit connection-01 should triggered based on the "change merge" events from gerrit connection-02. Is it even possible to configure in zuul? If so, can you please suggest a way to proceed? Thanks, Sankar Palanivel

2 9

OpenDev Collaboratory IRC services
by Jeremy Stanley 25 May '21

25 May '21

I know lots of people are discussing the recent Freenode IRC upheaval and what it means for their projects. On behalf of the OpenDev sysadmins I've started a thread on the service-discuss(a)lists.opendev.org mailing list to get feedback from communities currently utilizing the OpenDev Collaboratory's IRC bot services in their channels on Freenode (meeting minutes/logging, Gerrit change events, OpenDev service status information, channel operator assistance): http://lists.opendev.org/pipermail/service-discuss/2021-May/000236.html Please follow up there with your thoughts, to assist in our decision making, so we can take the needs of your project into account. Thanks! -- Jeremy Stanley

3 4

GitHub issues with 4.2.0
by Artem Goncharov 25 May '21

25 May '21

Hi all, I had unfortunately no time to investigate it deeper and rolled back to 4.1 (since we deploy zuul in k8 there are currently no old logs). What I have however noted: - gating for GitHub driver doesn’t work for us. The event is empty (https://opendev.org/zuul/zuul/src/branch/master/zuul/driver/github/githubmo… <https://opendev.org/zuul/zuul/src/branch/master/zuul/driver/github/githubmo…> tells event.label is empty). The same was also happening for reviews (https://opendev.org/zuul/zuul/src/branch/master/zuul/driver/github/githubmo… <https://opendev.org/zuul/zuul/src/branch/master/zuul/driver/github/githubmo…> - "because Reviews [{'type': 'approved', 'permission': 'write'}] does not match dict_values([])”) - merger was failing: https://opendev.org/zuul/zuul/src/branch/master/zuul/merger/merger.py#L1032 <https://opendev.org/zuul/zuul/src/branch/master/zuul/merger/merger.py#L1032> - ‘branch’ key is missing. Here I might be giving wrong line reference, since this is purely from memory, but it was definitely complaining to ‘branch' Rollback to 4.1 immediately "fixed" both issues, therefore I do not think this is a problem in the configuration on its own. I would try to deploy some test instance to play with it. This is currently just a feedback to 4.2.0 and potential warning to the ones using Zuul with GitHub Regards, Artem

3 5

help needed to understand failure
by Åsmund Østvold 19 May '21

19 May '21

hi, I have a job that suddenly started to fail at a very strange place in the script. Zuul console log: [...] 2021-05-19 11:12:46.615484 | ubuntu-bionic | + echo 'Cleanup and setup finished' 2021-05-19 11:12:46.615588 | ubuntu-bionic | Cleanup and setup finished 2021-05-19 11:12:46.615596 | ubuntu-bionic | + sleep 10 2021-05-19 11:12:01.254192 | ubuntu-bionic | ERROR 2021-05-19 11:12:01.254539 | ubuntu-bionic | { 2021-05-19 11:12:01.254634 | ubuntu-bionic | "delta": "0:00:10.254252", 2021-05-19 11:12:01.254676 | ubuntu-bionic | "end": "2021-05-19 11:12:54.441609", 2021-05-19 11:12:01.254711 | ubuntu-bionic | "msg": "non-zero return code", 2021-05-19 11:12:01.254745 | ubuntu-bionic | "rc": -15, 2021-05-19 11:12:01.254778 | ubuntu-bionic | "start": "2021-05-19 11:12:44.187357" 2021-05-19 11:12:01.254808 | ubuntu-bionic | } The script(cleanup_and_setup.sh) does the sleep as the last command. The ansible role looks like this: - name: check-format-and-build shell: cmd: | set -x pwd hostname CURRENT_DIR=`pwd` cd {{ zuul.projects['review.resoptima.net/zuul/irma-common-jobs'].src_dir }} file ${CURRENT_DIR} echo "before cleanup_and_setup.sh" ../../resoptima/tools/automation/cleanup_and_setup.sh echo "Finished cleanup_and_setup.sh" sleep 10 file ${CURRENT_DIR} cd ${CURRENT_DIR} cd {{ zuul.project.src_dir }} ./verify-format.sh ./build.sh {{ zuul.tag | default('branch') }} cd ${CURRENT_DIR} cd {{ zuul.projects['review.resoptima.net/zuul/irma-common-jobs'].src_dir }} ../../resoptima/tools/automation/post_cleanup.sh environment: MYGET_NPM_ACCESS_TOKEN:xxxxx [...] I have looked at the log messages from the different processes and can not find any error messages. Does anybody have an idea on what I should investigate? Any new help would be very much appreciated! Best regards, Åsmund

1 2

How to use zuul enqueue-ref?
by Vitaliy Lotorev 15 May '21

15 May '21

Hi, I'm trying to run 'zuul enqueue-ref' to run project in periodic pipeline as it's defined in zuul client docs https://zuul-ci.org/docs/zuul/reference/client.html#manual-enqueue-examples. The project I'm enqueueing runs automatically daily in this pipeline just fine. Command I run in scheduler environment to manually enqueue: zuul enqueue-ref --tenant <mytenant> --trigger timer --pipeline periodic --project <project> --ref refs/for/master On zuul web I see project is enqueued for revision '*Revision *0000000'. This issue is job quickly finishes with an error without actually running playbooks: * Error:* Project <server/project> does not have the default branch master Scheduler logs are (seems nothing useful): 2021-05-15 02:19:42,116 DEBUG zuul.Scheduler: Event <zuul.model.EnqueueEvent object at 0x7f26cc6eb670> for change <Branch 0x7f26cc7616d0 buildroot deletes refs/heads/master from 0000000000000000000000000000000000000000 was directly assigned to pipeline <Scheduler(Thread-1, started daemon 139804113094400)> 2021-05-15 02:19:42,116 DEBUG zuul.Pipeline.nto7.periodic: Considering adding change <Branch 0x7f26cc7616d0 buildroot deletes refs/heads/master from 0000000000000000000000000000000000000000 2021-05-15 02:19:42,116 DEBUG zuul.Pipeline.nto7.periodic: History: [] 2021-05-15 02:19:42,116 DEBUG zuul.Pipeline.nto7.periodic: Dependency graph: OrderedDict() 2021-05-15 02:19:42,116 DEBUG zuul.Pipeline.nto7.periodic: Dynamically created queue <ChangeQueue periodic: buildroot> 2021-05-15 02:19:42,117 DEBUG zuul.Pipeline.nto7.periodic: History after enqueuing changes ahead: [] 2021-05-15 02:19:42,117 INFO zuul.Pipeline.nto7.periodic: Adding change <Branch 0x7f26cc7616d0 solaris/buildroot deletes refs/heads/master from 0000000000000000000000000000000000000000 to queue <ChangeQueue periodic: buildroot> in <Pipeline periodic> 2021-05-15 02:19:42,117 INFO zuul.Pipeline.nto7.periodic: Reporting enqueue, action [] item <QueueItem 56ffcc2c9fef414ca9a44a442e5d65d7 for <Branch 0x7f26cc7616d0 buildroot deletes refs/heads/master from 0000000000000000000000000000000000000000 in periodic> -- Regards, Vitaliy

3 5