January 2021 - Zuul-discuss - lists.opendev.org

install and use ansible collections in zuul v3 ansible native job?
by Marios Andreou 24 Nov '21

24 Nov '21

hi folks long story short: can we install and use ansible collections in a zuul v3/ansible native job? In the tripleo-ci squad we are trying to use the tripleo_overcloud_image_build role from the tripleo-operator-ansible collection [1]. We are already using other roles from this collection - e.g. there [2]. This is possible because in that case we have zuul calling bash calling ansible... which means we get to configure collections in the bash bit setting ANSIBLE_COLLECTIONS_PATHS [3] before invoking ansible. For the case I'm having trouble with - what i refer to as ansible native - here https://review.opendev.org/#/c/733818/33/roles/oooci-build-images/tasks/mai… we have a zuul job playbook directly trying to install and use a collection. Currently it fails because instead of collections: (doesn't work " 'collections' is not a valid attribute? [4] ) I am trying to use include_role but then it can't find the collection's modules "ERROR! no action detected in task" [5]. I tried to set it in ansible.cfg but that won't work and anyway it is surely the wrong approach. The one thing left to try is to explicitly require ansible 2.9 for the job - but wanted to reach out and see if anyone else has any problems trying to install and use ansible collections (with modules importantly) as part of a zuul v3/ansible native job. thanks for reading and for any comments, marios [1] https://github.com/openstack/tripleo-operator-ansible/tree/master/roles/tri… [2] https://opendev.org/openstack/tripleo-quickstart-extras/src/commit/55e74bb8… [3] https://opendev.org/openstack/tripleo-quickstart/src/commit/299131bcc6dca86… [4] https://23fd78c0f6b01b9c4209-59c294b8f46eb75cda2bd004e2a1c97f.ssl.cf1.rackc… [5] https://2063b7e2bb67f86f8ec5-87fe67c942382c6ee875c514bffa4f21.ssl.cf2.rackc…

3 8

[Diversity] Diversity &Inclusion WG Meeting reminder Amy Marrich <amy@demarco.com> Mon, Jan 11, 7:49 AM to foundation, openstack-discuss, kata-dev, starlingx-discuss, airship-discuss, zuul-discuss The Diversity & Inclusion WG invites members of all OSF projects to our meeting today, at 17:00 UTC in the #openstack-diversity channel. The agenda can be found at https://etherpad.openstack.org/p/diversity-wg-agenda. Please feel free to add any other topics you wish to discuss at the meeting. Thanks, Amy (spotz)
by Amy Marrich 01 Feb '21

01 Feb '21

The Diversity & Inclusion WG invites members of all OSF projects to our meeting tomorrow, at 17:00 UTC in the #openstack-diversity channel. The agenda can be found at https://etherpad.openstack.org/p/diversity-wg-agenda. Please feel free to add any other topics you wish to discuss at the meeting. Thanks, Amy (spotz)

1 0

Re: swift-proxy repo creation
by okozachenko 29 Jan '21

29 Jan '21

I suggest that we can agree on the repository name first. I think 2 options for now - `zuul-storage-proxy` which includes any kinds of storage backend which can be configurable by envvar or conf file - `zuul-swift-proxy` which includes only swift proxy. And for other kinds of storage, we will create a new repo. I hope we decide the repo name soon. It also specifies the repo scope. Thank you.

5 7

[openinfra] 2020 Open Infra Foundation Annual Report
by Jimmy McArthur 29 Jan '21

29 Jan '21

Despite being a very different year than most, the Open Infrastructure community, which has over 110,000 community members, made it a productive and successful year. We could not have done this without each and every one of you. One of the biggest milestones that happened in the global community last year was that OpenStack, one of the top three most active open source projects with 15 million cores in production, marked its 10th anniversary in 2020. OpenStack is undoubtedly a success. Millions of CPU cores of computing infrastructure are provided through OpenStack. More than 70 regions of public clouds powered by OpenStack are available for users worldwide. Last year saw our 22nd (on-time) release, with more than 20,000 changes merged in only 5 months. A level of activity that placed OpenStack among the three most active open source development communities in the world. Now, we are seeing users who have been around for years talk about their growing OpenStack production deployments like Workday, LINE, and CERN, as well as new users talk about how they’re running other open source projects in production like Ant Group using Kata Containers, Verizon running StarlingX, and Volvo running Zuul for their autonomous driving software. The Open Infrastructure Foundation would like to extend a huge thanks to the global community for all of the work that went into 2020 and is continuing in 2021 to help people build and operate open infrastructure. Check out the OpenStack community’s achievements in 2020 from the OpenInfra Foundation Annual Report and join us (https://openinfra.dev/join/) [1] to build the next decade of open infrastructure! The 2020 OpenInfra Foundation Annual Report [2] Cheers, Jimmy [1] https://openinfra.dev/join [2] https://www.openstack.org/annual-reports/2020-openstack-foundation-annual-r… Let’s cheer on the new year and to the successful growth of the seeds planted over these past years.

1 0

newbie: wrong comment url posted
by Åsmund Østvold 27 Jan '21

27 Jan '21

hi Again, I am sorry to have to ask but. I have spent the day trying to find the solution. Zuul posts wrong url to logs in Gerrit comments. We get http://zuul-ci.resoptima.net:8000/65/3365/1/..... But we should get: http://zuul-ci.resoptima.local:8000/65/3365/1/.... The host that is running the docker-compose env had originally .net ending. Now it is renamed and I have to resolve the url issue. I have searched all of my setup folders. It has only minor changes to the docker-compose example provided. And I have not explisetly specified zuul-ci.resoptima.net anyware now. In an attempt to fix the issue I have added web.root and web.status_url in etc_zuul/zuul.conf.j2 without it taking effect. I have verified that the new config is present in the docker containers. [web] root=http://zuul-ci.resoptima.local status_url=http://zuul-ci.resoptima.local listen_address=0.0.0.0 Again any help would be appreciated! Even if we are having some small issues like this one I still enjoy Zuul very much! Regards, Åsmund

1 1

Gerrit: tags on changes not merged do not trigger builds
by Åsmund Østvold 27 Jan '21

27 Jan '21

hi, I assume the below request is a bit strange so I will give a short background: Our dev team produces some libraries that are used in different versions in our products. The distribution is done by publishing to https://myget.org/. To enable testing of pre-merge commits our previous build system did get triggered on tags pointing to unmerged gerrit changes. The issue: The tag pipeline defined like this: - pipeline: name: tag description: This pipeline runs jobs in response to any tag event. manager: independent precedence: high post-review: True trigger: gerrit: - event: ref-updated ref: ^refs/tags/.*$ success: mysql: failure: mysql: Our Zuul instans do not start builds when tags are pointing to change that is not merged. Builds do start when tags point to merged commits. I believe the relevant log is: 2020-12-18 10:12:03,656 INFO zuul.Pipeline.resoptima.tag: [e: 4435a9ca5b92495086f3083cf41e62fd] Adding change <Tag 0x7f3174086070 resoptima/irma.databaseextensions creates refs/tags/1.3.0-beta.1 on 8d0bdb76b5122402cc66d9f2b53bdf65d3409e7b> to queue <ChangeQueue tag: resoptima/irma.databaseextensions> in <Pipeline tag> 2020-12-18 10:12:03,657 INFO zuul.Pipeline.resoptima.tag: Reporting enqueue, action [] item <QueueItem 0x7f31542667f0 for <Tag 0x7f3174086070 resoptima/irma.databaseextensions creates refs/tags/1.3.0-beta.1 on 8d0bdb76b5122402cc66d9f2b53bdf65d3409e7b> in tag> 2020-12-18 10:12:03,657 WARNING zuul.GerritConnection: Unable to get change for <GerritTriggerEvent ref-replication-scheduled review.resoptima.net/resoptima/irma.databaseextensions> 2020-12-18 10:12:03,851 INFO zuul.MergeClient: [e: 4435a9ca5b92495086f3083cf41e62fd] Merge <gear.Job 0x7f31542fd8b0 handle: b'H:172.18.0.8:765' name: merger:refstate unique: 54a22a170b1a439c82acbb9948c8a109> complete, merged: False, updated: True, commit: None, branches: [''] 2020-12-18 10:12:22,555 WARNING zuul.GerritConnection: Unable to get change for <GerritTriggerEvent ref-replicated review.resoptima.net/resoptima/irma.databaseextensions> 2020-12-18 10:12:22,555 WARNING zuul.GerritConnection: Unable to get change for <GerritTriggerEvent ref-replication-done review.resoptima.net/resoptima/irma.databaseextensions> Is there something I can do to get this to work? Cheers, Åsmund

4 5

behavior change for pull-from-intermediate-registry
by Paul Belanger 27 Jan '21

27 Jan '21

Greetings, I believe we've encountered a new use case for pull-from-intermediate-registry and would like some feedback. TL;DR: pull-from-intermediate-registry will blindly overwrite images in the buildset-registry, I'd like to expose a way to toggle that feature. Today in zuul.a.c (ansible) we are in the process of migrating a lot of our testing to containers. As such, we have a new project called: https://github.com/ansible/network-ee/ This is the main repo used for building containers, and a job called: network-ee-build-container-image https://github.com/ansible/network-ee/blob/main/.zuul.d/jobs.yaml#L42 Basically, this job run generic enough to run in multiple projects, creating a new container image called: quay.io/ansible/network-ee:latest (there are more images, but we'll use this for the example) In our case, the new use case happens when using depends-on headers with network-ee-build-container-image jobs. We also have an image test job, which create: quay.io/ansible/network-ee-sanity-tests:latest using quay.io/ansible/network-ee:latest above. So for network-ee-sanity-tests job to run, first pause network-ee-build-container-image, then the sanity test job will run. The network-ee-sanity-tests job, parents to our: ansible-buildset-registry-consumer job Which, is basically the same as opendev's version. In this job, is where we run the pull-from-intermediate-registry role, to interact with the previous PR and copy images. The issue, in our case, is because our network-ee-build-container-image job runs in multiple places, it is possible for PR B, which depends on PR A, to have PR Bs quay.io/ansible/network-ee:latest images overwritten by PR A version. Here is an example: In https://dashboard.zuul.ansible.com/t/ansible/buildset/952c22f2df6c4a7181ce5… we have a failing job, network-ee-sanity-test. Looking at the logs, we have used: sha256:803564426f582436b6078d7ae203e2c109f4a47d04e48b78a2498c3755067aa2 https://ce377c2f657a3f4ccc50-902109811c92da02411ac82ade9b6da4.ssl.cf2.rackc… which you can see, is copied in from the intermediate registry: https://ce377c2f657a3f4ccc50-902109811c92da02411ac82ade9b6da4.ssl.cf2.rackc… If you dig more, sha256:8035644 is not produced from: https://dashboard.zuul.ansible.com/t/ansible/build/6bfc5ce56b2241f9bd709e14… our image build job in the buildset, but the parent job located at: https://cabd361c53760bec77a3-269ce6cee969dd0936b97c4252700a78.ssl.cf2.rackc… It took me a while, to fully understand what is happening, so I wanted to share here. In our case, I can think of 2 potential solutions. 1). update our ansible-buildset-registry-consumer job, to expose a flag to toggle the usage of pull-from-intermediate-registry. Given that our test jobs really don't need images from another PR, and we always rebuild, I think this is the safest option. 2). Expose a new flag in pull-from-intermediate-registry to first check if an image exists in the buildset registry, if not, don't copy over it. As it _should_ be a newer image, that was build in the current change, which already includes the changes of the previous depends-on change. Hopefully I explained clearly what is happening, if not, I can share more information. Paul

2 4

swift-proxy repo creation
by okozachenko 26 Jan '21

26 Jan '21

Hi Zuul team. Hope all are safe. I am trying to create a new repo under zuul namespace named swift-proxy. The new repo would include this code. https://review.opendev.org/c/zuul/zuul/+/708233 Let me explain a little about this PS. It is a wsgi proxy to proxy the requests like this; ``` ZUUL_HOST_ADDRESS/{CONTAINER_PREFIX}-{TENANT_NAME}/path -> SWIFT_SERVICE_ENDPOINT/{CONTAINER_PREFIX}-{TENANT_NAME}/path ``` This proxy requires clouds.yaml file and CLOUD_NAME envvar for the swift. Please give the feedback. Thank you. Open10k8s

5 5

Zuul-executor error
by sitakanta hotta 23 Jan '21

23 Jan '21

Here is my configuration. Am I missing host and port? [executor] manage_ansible=false log_config=/etc/zuul/executor-logging.conf job_dir=/var/lib/zuul/builds variables=/opt/project-config/zuul/site-variables.yaml private_key_file=/var/lib/zuul/ssh/nodepool_id_rsa trusted_ro_paths=/etc/openafs:/etc/ssl/certs:/var/lib/zuul/ssh trusted_rw_paths=/afs untrusted_ro_paths=/etc/ssl/certs disk_limit_per_job=5000 I get the following error Traceback (most recent call last): File "/usr/local/bin/zuul-executor", line 8, in <module> sys.exit(main()) File "/usr/local/lib/python3.8/site-packages/zuul/cmd/executor.py", line 113, in main Executor().main() File "/usr/local/lib/python3.8/site-packages/zuul/cmd/__init__.py", line 219, in main self.run() File "/usr/local/lib/python3.8/site-packages/zuul/cmd/executor.py", line 97, in run self.start_log_streamer() File "/usr/local/lib/python3.8/site-packages/zuul/cmd/executor.py", line 57, in start_log_streamer streamer = zuul.lib.log_streamer.LogStreamer( File "/usr/local/lib/python3.8/site-packages/zuul/lib/log_streamer.py", line 166, in __init__ self.server = LogStreamerServer((host, port), File "/usr/local/lib/python3.8/site-packages/zuul/lib/log_streamer.py", line 155, in __init__ super(LogStreamerServer, self).__init__(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/zuul/lib/streamer_utils.py", line 96, in __init__ socketserver.ThreadingTCPServer.__init__(self, *args, **kwargs) File "/usr/local/lib/python3.8/socketserver.py", line 448, in __init__ self.socket = socket.socket(self.address_family, File "/usr/local/lib/python3.8/socket.py", line 231, in __init__ _socket.socket.__init__(self, family, type, proto, fileno) OSError: [Errno 97] Address family not supported by protocol

2 2

Adopting Discourse for mailing list?
by Kenny Ho 22 Jan '21

22 Jan '21

Hi, Just curious, has anyone thought of adopting Discourse ( https://www.discourse.org/) for either Zuul or OpenDev (or both) for hosting mailing lists and forums? I am just wondering since Discourse seems to have more functionalities and it is open source. Some examples of open source organizations using Discourse are Fedora ( https://discussion.fedoraproject.org/) and Mozilla ( https://discourse.mozilla.org/). Regards, Kenny Ho

8 13