- Zuul-discuss - lists.opendev.org

Diversity and Inclusion Meeting Reminder - Tooday
by Amy Marrich 04 Oct '21

04 Oct '21

The Diversity & Inclusion WG invites members of all OIF projects to attend our next meeting Monday October 4th, at 17:00 UTC in the #openinfra- diversity channel on OFTC. The agenda can be found at https://etherpad.openstack.org/p/diversity-wg-agenda. Please feel free to add any topics you wish to discuss at the meeting. Thanks, Amy (apotz)

1 0

Use of "zuul delete-state"
by James E. Blair 27 Sep '21

27 Sep '21

Hi, For those who are tracking master: we ran into some issues recently that required changes to how we store objects in ZooKeeper. If you have run recent commits of Zuul from master and upgrade, you may run into issues starting Zuul because of the change in data format (it may manifest as a crash on startup with a KeyError or something similar). If you do so, run this command to delete the ephemeral data from ZooKeeper: zuul delete-state That will delete all the cached data, job requests, etc, but leave Nodepool and secret key storage intact. It may take a few minutes to run, and the next time you start your scheduler it may take a little longer. Friendly reminder that if you encounter any ZooKeeper-related errors, this command is always there as an escape hatch. -Jim

1 0

"Merge failed" in promote pipeline
by Christian von Schultz 20 Sep '21

20 Sep '21

Hello everyone, I'm trying to make a promote pipeline, using the supercedent pipeline manager. I added a simple job just to see that it works, and it doesn't. The message in Gerrit is "Merge Failed", which I didn't expect from a post-merge pipeline. I thought it would just check out the commit in question and run the job. Do I need to tell it to do that? My pipeline configuration for the promote pipeline is based on this: https://zuul-ci.org/docs/zuul/reference/drivers/gerrit.html#reference-pipel… The project is running with "merge-mode: cherry-pick", but for changes that are already merged, I didn't expect any extra cherry-picking. How is this supposed to work? I'm running Zuul 4.3.0. Scheduler log says: > 2021-09-20 12:44:19,779 DEBUG zuul.Pipeline.embedl-tenant.promote: [e: 6d632d0932ab48189dc4f91098670f68] <QueueItem 98818e7cef9f43ca8aa492387d772bd7 for <Change 0x7fdf74331a90 embedl 1174,2> in promote> is a failing item because ['it has a merge conflict'] Executor log says: > 2021-09-20 12:44:17,771 DEBUG zuul.Merger: [e: 6d632d0932ab48189dc4f91098670f68] Unable to merge {'project': 'embedl', 'connection': 'gerrit', 'merge_mode': 3, 'ref': 'refs/changes/74/1174/2', 'branch': 'work/promotebranch', 'buildset_uuid': 'e2ccdf93250047578dba0ea8adf7b82f', 'number': '1174', 'patchset': '2', 'oldrev': None, 'newrev': None} > Traceback (most recent call last): > File "/usr/local/lib/python3.8/site-packages/zuul/merger/merger.py", line 1000, in _mergeChange > commit = repo.cherryPick(item['ref'], > File "/usr/local/lib/python3.8/site-packages/zuul/merger/merger.py", line 541, in cherryPick > repo.git.cherry_pick("FETCH_HEAD") > File "/usr/local/lib/python3.8/site-packages/git/cmd.py", line 588, in <lambda> > return lambda *args, **kwargs: self._call_process(name, *args, **kwargs) > File "/usr/local/lib/python3.8/site-packages/git/cmd.py", line 1134, in _call_process > return self.execute(call, **exec_kwargs) > File "/usr/local/lib/python3.8/site-packages/git/cmd.py", line 931, in execute > raise GitCommandError(redacted_command, status, stderr_value, stdout_value) > git.exc.GitCommandError: Cmd('git') failed due to: exit code(1) > cmdline: git cherry-pick FETCH_HEAD > stdout: 'Not currently on any branch. > You are currently cherry-picking commit fe48d19. > > nothing to commit, working tree clean' > stderr: 'The previous cherry-pick is now empty, possibly due to conflict resolution. > If you wish to commit it anyway, use: > > git commit --allow-empty > > Otherwise, please use 'git reset'' Best regards, Christian von Schultz

2 1

zuul-jobs testing at an impasse
by Jeremy Stanley 17 Sep '21

17 Sep '21

A quick summary... This week, a new protobuf release dropped Python 2.7 and 3.5 support, causing tox-py27 builds for zuul-jobs changes to fail, raising SyntaxError importing roles.upload-logs-base.library.test_zuul_google_storage_upload. The proposed fix is to use an environment marker to pin to earlier protobuf on Python<3.6 for now: https://review.opendev.org/809460 Another slightly more tricky problem is that this week, a new Virtualenv release bundled a new major version of SetupTools which dropped support for its old use_2to3 option. This has caused problems for a number of very old Python packages, but in our case specifically funcparserlib which was last fully released in 2013. Using latest virtualenv, attempts to install funcparserlib or anything which requires it will just break. We have one document in zuul-jobs (docker-image.rst) which uses blockdiag and seqdiag directives, and that is indirectly responsible for pulling in the funcparserlib package. Thankfully, the funcparserlib maintainers acted quickly and proposed an alpha release which we can explicitly require until the blockdiag project merges a pending pull request to do so itself, and this is proposed already: https://review.opendev.org/809885 Unfortunately, that new alpha release also drops Python 3.5 support, meaning we can't build the zuul-jobs docs (well, docker-image.rst specifically) on Ubuntu Xenial. That might seem like a small thing, we build and publish the zuul-job docs with much newer Python anyway, however the way we've chosen to test the fetch-sphinx-tarball role is to run `tox -e docs` for zuul-jobs on lots of different platforms, one of which is Ubuntu Xenial. Dropping support for Python 3.5 is happening in new releases of projects right and left, and as of January 2022 pip itself intends to make a new release which will no longer work on Python 3.5, so if we intend to continue testing things on Ubuntu Xenial at all, we're almost certainly going to need to work out how to do so with older toolchains which can still work on its Python version. So here's the dilemma, and my question to you. Do we: A. rework the fetch-sphinx-tarball role test to no longer use the docs/requirements.txt in zuul-jobs? B. redo docker-image.rst to switch from blockdiag and seqdiag to something else (another Sphinx plugin, pre-rendered images...)? C. adjust tools/update-test-platforms.py to allow us to exclude specific autogenerated jobs we know can never work and use that to drop the zuul-jobs-test-fetch-sphinx-tarball-ubuntu-xenial? D. remove Ubuntu Xenial testing for zuul-jobs in general? E. some other solution I'm too braindead to come up with on a hot Friday afternoon? Feedback on where I should go from here would be much appreciated! -- Jeremy Stanley

1 1

Mailing lists offline 2021-09-12 for server upgrade
by Jeremy Stanley 12 Sep '21

12 Sep '21

On Sunday, September 12, the OpenDev sysadmins will be performing operating system upgrades for the server hosting the lists.airshipit.org, lists.opendev.org, lists.openstack.org, lists.starlingx.io, and lists.zuul-ci.org sites. Expect extended outages for message delivery and access to archives between 15:00 UTC and 21:00 UTC, though messages sent to the mailing lists at these sites should queue normally and be delivered automatically after maintenance activities have concluded. If you have any questions, feel free to reply to this message, or find us in the #opendev channel on the OFTC IRC network. -- Jeremy Stanley

1 1

Debugging zuul-registry failures when using buildkit
by Ian Wienand 01 Sep '21

01 Sep '21

Change https://review.opendev.org/c/opendev/system-config/+/805933 is using DOCKER_BUILDKIT=1 to build an image (gitea) but has a requirement to pull another image (the new "assets" image, proposed with it's parent https://review.opendev.org/c/opendev/system-config/+/805933/) from the intermediate registry. This reliably fails as you can see from any of the zuul results. The failure is --- root@ubuntu-focal-ovh-bhs1-0026222696:/home/zuul/src/opendev.org/opendev/system-config/docker/gitea# DOCKER_BUILDKIT=1 docker build . ... => ERROR [internal] load metadata for docker.io/opendevorg/assets:latest ------ > [internal] load metadata for docker.io/opendevorg/assets:latest: ------ failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: rpc error: code = Unknown desc = failed commit on ref "config-sha256:c250addbd759142b5feb692703a422348faecf3ddfa2cc8191ab43d265c7bc07": unexpected commit digest sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, expected sha256:c250addbd759142b5feb692703a422348faecf3ddfa2cc8191ab43d265c7bc07: failed precondition --- I enabled debugging on the docker daemon and it produces the following --- Sep 01 03:56:36 ubuntu-focal-ovh-bhs1-0026222696 dockerd[5392]: time="2021-09-01T03:56:36.097177192Z" level=debug msg=resolving host=registry-1.docker.io Sep 01 03:56:36 ubuntu-focal-ovh-bhs1-0026222696 dockerd[5392]: time="2021-09-01T03:56:36.097232892Z" level=debug msg="do request" host=registry-1.docker.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.4.0+unknown request.method=HEAD url="https://registry-1.docker.io/v2/library/golang/manifests/1.16-buster" Sep 01 03:56:36 ubuntu-focal-ovh-bhs1-0026222696 dockerd[5392]: time="2021-09-01T03:56:36.100016964Z" level=debug msg="fetch response received" host="zuul-jobs.buildset-registry:5000" response.header.content-length=921 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Wed, 01 Sep 2021 03:56:36 GMT" response.header.docker-content-digest="sha256:3ae38c942ced144dcbacf8e9895bb553f84de6c8abdfced27c7075f18583f2a4" response.header.server=CherryPy/18.6.0 response.status="200 OK" url="https://zuul-jobs.buildset-registry:5000/v2/opendevorg/assets/manifests/lat…" Sep 01 03:56:36 ubuntu-focal-ovh-bhs1-0026222696 dockerd[5392]: time="2021-09-01T03:56:36.100201992Z" level=debug msg=resolved desc.digest="sha256:3ae38c942ced144dcbacf8e9895bb553f84de6c8abdfced27c7075f18583f2a4" host="zuul-jobs.buildset-registry:5000" Sep 01 03:56:36 ubuntu-focal-ovh-bhs1-0026222696 dockerd[5392]: time="2021-09-01T03:56:36.100454497Z" level=debug msg=fetch digest="sha256:3ae38c942ced144dcbacf8e9895bb553f84de6c8abdfced27c7075f18583f2a4" mediatype=application/vnd.docker.distribution.manifest.v2+json size=921 Sep 01 03:56:36 ubuntu-focal-ovh-bhs1-0026222696 dockerd[5392]: time="2021-09-01T03:56:36.101860659Z" level=debug msg=fetch digest="sha256:c250addbd759142b5feb692703a422348faecf3ddfa2cc8191ab43d265c7bc07" mediatype=application/vnd.docker.container.image.v1+json size=0 --- We see digest="sha256:c250addbd759142b5feb692703a422348faecf3ddfa2cc8191ab43d265c7bc07" has size=0; clarkb helpfully noted that the "unexpected commit digest" e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 is actually the sha256 of an empty string. This seems related. The buildset registry output for this is --- 2021-09-01 04:10:11,794 DEBUG registry.authz: Auth ok read 2021-09-01 04:10:11,794 INFO registry.api: HEAD manifest _local docker/dockerfile 1.3 2021-09-01 04:10:11,794 ERROR registry.api: Manifest docker/dockerfile 1.3 not found 2021-09-01 04:10:11,794 INFO cherrypy.access.140461361540832: ::ffff:172.17.0.1 - - [01/Sep/2021:04:10:11] "HEAD /v2/docker/dockerfile/manifests/1.3?ns=docker.io HTTP/1.1" 404 735 "" "containerd/1.4.0+unknown" 2021-09-01 04:10:12,115 DEBUG registry.authz: Auth ok read 2021-09-01 04:10:12,115 INFO registry.api: HEAD manifest _local library/debian buster-slim 2021-09-01 04:10:12,115 ERROR registry.api: Manifest library/debian buster-slim not found 2021-09-01 04:10:12,116 INFO cherrypy.access.140461361540832: ::ffff:172.17.0.1 - - [01/Sep/2021:04:10:12] "HEAD /v2/library/debian/manifests/buster-slim?ns=docker.io HTTP/1.1" 404 735 "" "containerd/1.4.0+unknown" 2021-09-01 04:10:12,119 DEBUG registry.authz: Auth ok read 2021-09-01 04:10:12,119 INFO registry.api: HEAD manifest _local library/golang 1.16-buster 2021-09-01 04:10:12,120 ERROR registry.api: Manifest library/golang 1.16-buster not found 2021-09-01 04:10:12,120 INFO cherrypy.access.140461361540832: ::ffff:172.17.0.1 - - [01/Sep/2021:04:10:12] "HEAD /v2/library/golang/manifests/1.16-buster?ns=docker.io HTTP/1.1" 404 735 "" "containerd/1.4.0+unknown" 2021-09-01 04:10:12,125 DEBUG registry.authz: Auth ok read 2021-09-01 04:10:12,125 INFO registry.api: HEAD manifest _local opendevorg/assets latest 2021-09-01 04:10:12,126 DEBUG registry.api: Manifest opendevorg/assets latest digest found sha256:3ae38c942ced144dcbacf8e9895bb553f84de6c8abdfced27c7075f18583f2a4 2021-09-01 04:10:12,126 DEBUG registry.api: Retrieved sha256 3ae38c942ced144dcbacf8e9895bb553f84de6c8abdfced27c7075f18583f2a4 2021-09-01 04:10:12,126 INFO cherrypy.access.140461361540832: ::ffff:172.17.0.1 - - [01/Sep/2021:04:10:12] "HEAD /v2/opendevorg/assets/manifests/latest?ns=docker.io HTTP/1.1" 200 - "" "containerd/1.4.0+unknown" 2021-09-01 04:10:12,128 DEBUG registry.authz: Auth ok read 2021-09-01 04:10:12,129 INFO registry.api: GET manifest _local opendevorg/assets latest 2021-09-01 04:10:12,129 DEBUG registry.api: Manifest opendevorg/assets latest digest found sha256:3ae38c942ced144dcbacf8e9895bb553f84de6c8abdfced27c7075f18583f2a4 2021-09-01 04:10:12,129 DEBUG registry.api: Retrieved sha256 3ae38c942ced144dcbacf8e9895bb553f84de6c8abdfced27c7075f18583f2a4 2021-09-01 04:10:12,129 INFO cherrypy.access.140461361540832: ::ffff:172.17.0.1 - - [01/Sep/2021:04:10:12] "GET /v2/opendevorg/assets/manifests/latest?ns=docker.io HTTP/1.1" 200 921 "" "containerd/1.4.0+unknown" --- There is a file on the host --- # cat buildkit/content/blobs/sha256/3ae38c942ced144dcbacf8e9895bb553f84de6c8abdfced27c7075f18583f2a4 { "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "config": { "mediaType": "application/vnd.docker.container.image.v1+json", "digest": "sha256:c250addbd759142b5feb692703a422348faecf3ddfa2cc8191ab43d265c7bc07" }, "layers": [ { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 3122, "digest": "sha256:d400fcf962679c5efdb77b55b187b8afffb53f7342895f0b6f6206cf1fbbac6d" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 1899, "digest": "sha256:1cdb62f96c3d6766a459909f1306f97d174fd4b5fa11a46ecc2f20d8c529db93" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 7671, "digest": "sha256:f7721f02fe31867cdcab1396b7fc132f427ab98fe1f083611b2cc01e3f4fb4a8" } ] --- I note that the "config" here doesn't have a size -- it seems to me from https://docs.docker.com/registry/spec/manifest-v2-2/ it should. So I poked at the original opendev-buildset-registry job that the "assets" image as uploaded to https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/… That shows this has a size "2329" --- 2021-08-31 20:36:33,782 INFO registry.api: Get blob _local opendevorg/assets sha256:c250addbd759142b5feb692703a422348faecf3ddfa2cc8191ab43d265c7bc07 2021-08-31 20:36:33,782 INFO cherrypy.access.140479630381648: ::ffff:172.17.0.1 - - [31/Aug/2021:20:36:33] "GET /v2/opendevorg/assets/blobs/sha256:c250addbd759142b5feb692703a422348faecf3ddfa2cc8191ab43d265c7bc07 HTTP/1.1" 200 2329 "" "containers/5.10.2 (github.com/containers/image)" --- So, I hand edited in this size to the "config" above; to be clear it now looked like ... "config": { "mediaType": "application/vnd.docker.container.image.v1+json", "size": 2329, "digest": "sha256:c250addbd759142b5feb692703a422348faecf3ddfa2cc8191ab43d265c7bc07" }, ... and ... the build started working. Along with the weird hashing of an empty string, this makes more more convinced it has something to do with this size setting. How has this config in the manifest not got a size? Is this really the problem? Is it something we've managed to drop in the intermediate registry, or has docker uploaded it incorrectly? I'd love any suggestions! -i

2 2

[zuul-ci.org] Update Freenode nics
by Jimmy McArthur 30 Aug '21

30 Aug '21

Hi all - I just noticed that the contact nics on the Governance page[1] are still the freenode nics. I'd update, but I'm not sure what to update it all to :) Cheers, Jimmy [1] https://zuul-ci.org/docs/zuul/reference/governance.html

4 4

Static nodes enter infinite 'building' state
by Vitaliy Lotorev 23 Aug '21

23 Aug '21

Hi, I'm using Zuul 4.7.0 and Nodepool 4.2.2. My nodes are all static. Now and then all nodes enter an infinite 'building' state. E.g. here is how building node 0000000113 starts (node_request: 300-0000000094): 2021-08-19 15:28:42,203 DEBUG nodepool.PoolWorker.static-vms-main: [e: 5a36f7ab18e54a11ad728fecab385be6] [node_request: 300-0000000094] Locking request 2021-08-19 15:28:42,219 INFO nodepool.PoolWorker.static-vms-main: [e: 5a36f7ab18e54a11ad728fecab385be6] [node_request: 300-0000000094] Assigning node request <NodeRequest {'state': 'requested', 'state_time': 1629386913.1936047, 'declined_by': [], 'node_types': ['centos7'], 'nodes': [], 'reuse': True, 'requestor': '79d78decd29a', 'requestor_data': {'build_set_uuid': '344e4dec0639481b85d62c0ab6efa0e9', 'tenant_name': 'nto7', 'pipeline_name': 'check', 'job_name': 'buildroot-build'}, 'provider': None, 'relative_priority': 0, 'event_id': '5a36f7ab18e54a11ad728fecab385be6', 'id': '300-0000000094', 'stat': ZnodeStat(czxid=209282, mzxid=209282, ctime=1629386913193, mtime=1629386913193, version=0, cversion=0, aversion=0, ephemeralOwner=72217169448337409, dataLength=400, numChildren=0, pzxid=209282)}> 2021-08-19 15:28:42,219 DEBUG nodepool.driver.NodeRequestHandler[9c4813b37c0b-PoolWorker.static-vms-main-aa0248e60f7d49c9b5065855fe7e0842]: [e: 5a36f7ab18e54a11ad728fecab385be6] [node_request: 300-0000000094] Accepting node request 2021-08-19 15:28:42,253 DEBUG nodepool.driver.NodeRequestHandler[9c4813b37c0b-PoolWorker.static-vms-main-aa0248e60f7d49c9b5065855fe7e0842]: [e: 5a36f7ab18e54a11ad728fecab385be6] [node_request: 300-0000000094] Locked building node 0000000113 for request 2021-08-19 15:28:42,276 DEBUG nodepool.driver.NodeRequestHandler[9c4813b37c0b-PoolWorker.static-vms-main-aa0248e60f7d49c9b5065855fe7e0842]: [e: 5a36f7ab18e54a11ad728fecab385be6] [node_request: 300-0000000094] Waiting for node 0000000113 to be ready 2021-08-19 15:28:42,277 DEBUG nodepool.PoolWorker.static-vms-main: Active requests: ['300-0000000078', '300-0000000079', '300-0000000080', '300-0000000081', '300-0000000082', '300-0000000083', '300-0000000084', '300-0000000085', '300-0000000086', '300-0000000087', '300-0000000088', '300-0000000089', '300-0000000090', '300-0000000091', '300-0000000092', '300-0000000093', '300-0000000094'] After two days node_request: 300-0000000094 still hanging in active requests (node 0000000113 is also seen in web UI in building state). To clean nodepool I have to restart nodepool once a week. Actually this issue has been tracing me for more than a year (so it's not something that appeared in the latest nodepool versions). I hoped recent commit [1] (included in nodepool 4.2.2) might be relevant to fixing my issue but it has not changed anything. [1] https://review.opendev.org/c/zuul/nodepool/+/790674 -- Regards, Vitaliy Lotorev

2 2

Zuul has moved from IRC to Matrix
by James E. Blair 21 Aug '21

21 Aug '21

Hi, The Zuul project has completed its move from IRC to Matrix. If you would like to chat with us, please join our room, #zuul:opendev.org -- you can follow this link for convenience: https://matrix.to/#/#zuul:opendev.org You can read detailed instructions on how to get started using Matrix (it's easy!) here: https://zuul-ci.org/docs/zuul/howtos/matrix.html Those instructions also include information on how to use Matrix to participate in bridged IRC channels, so you can use the same client to participate in communities that use IRC. If you are still in the IRC channel, please go ahead and /part in order to aid us in shutting it down. -Jim

1 0

Moving from IRC to Matrix
by James E. Blair 18 Aug '21

18 Aug '21

Hi, The Zuul community will be moving our official chat platform to Matrix on Saturday, August 21. The room has not been created yet, but when it is, the name will be #zuul:opendev.org We will update the website and instructions in the documentation accordingly over the next few days. Starting next week, we will no longer use the #zuul IRC channel on OFTC, and will encourage anyone remaining in that channel to /part. You don't need to do anything right now -- because Matrix preserves history, you can join the new room at any point in the future and have access to messages sent before you joined, so you won't miss anything by joining after the move. -Jim

1 0