- Zuul-discuss - lists.opendev.org

Zuul scheduler Errors: 'NoneType' object has no attribute 'gauge'
by Sankar Palanivel 12 May '21

12 May '21

Hi Team, We are setting up zuul services on docker containers with openstack backed nodepool and external gerrit instance. When we try to push a change/comment in gerrit, we are getting below errors in scheduler container. It would be helpful if you can suggest a way to fix these errors. 2021-05-11 16:36:59,079 INFO zuul.GerritConnection: [e: 5bb47c4c919748158fd737fbc771f0ff] Updating <Change 0x7f3510395ca0 None 8658,2> 2021-05-11 16:36:59,162 INFO zuul.Pipeline.sample.check: [e: 5bb47c4c919748158fd737fbc771f0ff] Adding change <Change 0x7f3510395ca0 sandbox 8658,2> to queue <ChangeQueue check: sandbox> in <Pipeline check> 2021-05-11 16:36:59,162 INFO zuul.Pipeline.sample.check: Reporting enqueue, action [] item <QueueItem 99080e4bac5f4b36bb83feee1dec557c for <Change 0x7f3510395ca0 sandbox 8658,2> in check> 2021-05-11 16:36:59,569 INFO zuul.MergeClient: [e: 5bb47c4c919748158fd737fbc771f0ff] Merge <gear.Job 0x7f3510326a60 handle: b'H:172.19.0.7:13' name: merger:merge unique: 659107f2f6624a7680be7ac9c4f11956> complete, merged: True, updated: False, commit: d525dc035578c078849cf2a3b324d653d1aef03f, branches: [] 2021-05-11 16:36:59,590 INFO zuul.Pipeline.sample.check: [e: 5bb47c4c919748158fd737fbc771f0ff] Scheduling global repo state for item <QueueItem 99080e4bac5f4b36bb83feee1dec557c for <Change 0x7f3510395ca0 sandbox 8658,2> in check> 2021-05-11 16:36:59,753 INFO zuul.MergeClient: [e: 5bb47c4c919748158fd737fbc771f0ff] Merge <gear.Job 0x7f3510311fd0 handle: b'H:172.19.0.7:14' name: merger:refstate unique: 86f384a2cb3f4181ab6769e6857bc27c> complete, merged: False, updated: True, commit: None, branches: False 2021-05-11 16:36:59,775 INFO zuul.nodepool: [e: 5bb47c4c919748158fd737fbc771f0ff] Submitted node request <NodeRequest 200-0000000006 <NodeSet [<Node None ('infra-build-ubuntu1804-base',):infra-build-ubuntu1804-base>]>> 2021-05-11 16:37:07,288 INFO zuul.nodepool: [e: 5bb47c4c919748158fd737fbc771f0ff] Node request <NodeRequest 200-0000000006 <NodeSet [<Node 0000000008 ('infra-build-ubuntu1804-base',):infra-build-ubuntu1804-base>]>> fulfilled 2021-05-11 16:37:07,298 INFO zuul.nodepool: [e: 5bb47c4c919748158fd737fbc771f0ff] Accepting node request <NodeRequest 200-0000000006 <NodeSet [<Node 0000000008 ('infra-build-ubuntu1804-base',):infra-build-ubuntu1804-base>]>> 2021-05-11 16:37:07,307 INFO zuul.Pipeline.sample.check: [e: 5bb47c4c919748158fd737fbc771f0ff] Completed node request <NodeRequest 200-0000000006 <NodeSet [<Node 0000000008 ('infra-build-ubuntu1804-base',):infra-build-ubuntu1804-base>]>> for job testjob of item <QueueItem 99080e4bac5f4b36bb83feee1dec557c for <Change 0x7f3510395ca0 sandbox 8658,2> in check> with nodes <NodeSet [<Node 0000000008 ('infra-build-ubuntu1804-base',):infra-build-ubuntu1804-base>]> 2021-05-11 16:37:07,318 INFO zuul.nodepool: Setting nodeset <NodeSet [<Node 0000000008 ('infra-build-ubuntu1804-base',):infra-build-ubuntu1804-base>]> in use 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: [e: 5bb47c4c919748158fd737fbc771f0ff] Exception while executing job testjob for change <Change 0x7f3510395ca0 sandbox 8658,2>: 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: Traceback (most recent call last): 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: File "/usr/local/lib/python3.8/site-packages/zuul/manager/__init__.py", line 639, in _executeJobs 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: self.sched.nodepool.useNodeSet( 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: File "/usr/local/lib/python3.8/site-packages/zuul/nodepool.py", line 253, in useNodeSet 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: self.emitStatsResources() 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: File "/usr/local/lib/python3.8/site-packages/zuul/nodepool.py", line 90, in emitStatsResources 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: self.statsd.gauge(key, value, tenant=tenant, resource=resource) 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: AttributeError: 'NoneType' object has no attribute 'gauge' Thanks, Sankar Palanivel

4 5

Zuul on openshift.tv
by James E. Blair 10 May '21

10 May '21

Hi, Just a quick note to let folks know I will be talking about Zuul live on http://openshift.tv this Thursday at 17:00 UTC. -Jim

1 0

Zuul deployment options
by Sankar Palanivel 07 May '21

07 May '21

Hi Team, We are planning to use zuul for some of our projects. Could you please help me out on the following queries? 1. Best way to deploy zuul in production is using QuickStart (using docker-compose) option or install from scratch option? 2. Based on the selection from question 1, how zuul can be scaled for future requirements? 3. I dont find resource requirements (memory/cpu/disk) for any of the zuul services. Please confirm which of the zuul services are (executor, scheduler, nodepool, zookeeper, web, logs, DB) resource intensive? 4. Is it okayto have zookeeper standalone instance for nodepool launcher with openstack provider? or it needs to have a zookeeper in cluster mode? Thanks, Sankar Palanivel

2 5

[Diversity] Diversity & Inclusion WG Meeting reminder
by Amy Marrich 30 Apr '21

30 Apr '21

The Diversity & Inclusion WG invites members of all OSF projects to our meeting Monday May 3rd, at 17:00 UTC in the #openinfra-diversity channel. The agenda can be found at https://etherpad.openstack.org/p/diversity -wg-agenda. Please feel free to add any topics you wish to discuss at the meeting. Thanks, Amy (spotz)

1 0

help to getting speculative merged testing fixed
by Åsmund Østvold 30 Apr '21

30 Apr '21

hi Probably a newbie mistake. I have: - job: name: build-integration-tests parent: base run: playbooks/build_and_integrationtest.yaml required-projects: - resoptima/irma - name: resoptima/tools override-checkout: main - zuul/irma-common-jobs - resoptima/irma.fileimporter In the gate pipeline I get parallel unrelated merged executions, see attached image. I am sure I did something wrong. Any help appreciated! Cheers, Åsmund [image: image.png]

2 2

questions about locally running zuul repo tests
by mikeyang＠andestech.com 27 Apr '21

27 Apr '21

Hi, As part of my eval, I'm looking at the dev process. And we're interested in a Gitea driver (it looks like there was a WIP change<726391:%20more%20work%20towards%20a%20working%20gitea%20driver%20|%20https:…> last year). When I try to run the tests locally with "tox", following the instructions in TESTING.rst, it looks like the Docker Zookeeper instance does not start up properly and the majority of tests fail with a timeout. I tried modifying "tox.ini" in order to open/map port 2181 and I can see that "tox" starts up the zookeeper container, but the tests still timeout. Looking at the code, I see a couple of issues: 1. hostname ... how does tox know the hostname/ip-address of the container (with default bridge networking) 2. certificates ... are these required and how is tox configuring this? Or am I on the wrong path? I see that tools/test-setup-docker.sh seems to have a different way of running the tests. Thanks, - mike. CONFIDENTIALITY NOTICE: This e-mail (and its attachments) may contain confidential and legally privileged information or information protected from disclosure. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein is strictly prohibited. In this case, please immediately notify the sender by return e-mail, delete the message (and any accompanying documents) and destroy all printed hard copies. Thank you for your cooperation. Copyright ANDES TECHNOLOGY CORPORATION - All Rights Reserved.

2 3

Newbie question(s) for eval
by mikeyang＠andestech.com 20 Apr '21

20 Apr '21

Hi, I’m doing an eval of Zuul-CI for my team. I’ve previously used Jenkins and internally/purpose-built CI systems. So Ansible is new to me. I’m using the quick-start tutorial so this is all in the Docker containers and images from the tutorial. I’m running this on a Ubuntu bare-metal machine. And please let me know if there is a more appropriate forum to ask this question or get eval support. So what I tried to do was use a simple job from zuul-jobs. I thought `markdownlint` would be straight-forward. Here’s the rabbit hole that I went down. 1. The quick-start tutorial is using `Ubuntu Focal` images for the `node` 2. I added a second `node` using `Ubuntu Groovy` 3. I can run `testjob` and a custom job (playbook with `shell`s) on the Groovy node 4. I tried to run the `markdownlint` Zuul job on the Groovy node but it fails on the `ensure-nodejs` role * It tries to add `https://deb.nodesource.com/node_6.x groovy main` as a repo * This fails because `dists/groovy` only exists in the 10/12/14/15.x repos * I tried many different ways to assign `node_version` to 15, but the job always fails trying to add `node_6.x` From the Ansible documentation, I see that there are many places in precedence to overwrite a role’s default value, but I have not gotten any of these to work. And I’m unclear about how the precedence extends into Zuul’s `job.vars` and `jobs.extra-vars` (are these equivalent to passing on Ansible’s command line?). Thanks, - mike. CONFIDENTIALITY NOTICE: This e-mail (and its attachments) may contain confidential and legally privileged information or information protected from disclosure. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein is strictly prohibited. In this case, please immediately notify the sender by return e-mail, delete the message (and any accompanying documents) and destroy all printed hard copies. Thank you for your cooperation. Copyright ANDES TECHNOLOGY CORPORATION - All Rights Reserved.

3 5

Nested repo structures
by James E. Blair 20 Apr '21

20 Apr '21

Hi, We were recently presented with a case of a user who uses Gerrit's support for arbitrary repo hierarchies in an unexpected way. Essentially, they have both of the following projects defined in gerrit: * project * project/subproject That is to say, project is not just a pathname, it is a full-blown Gerrit repo. Further, this has nothing to do with git submodules or anything. There are several potential issues in Zuul related to nested repos: * Stats * Secret key storage * Time database * Internal merger repos * Workspace checkouts Stats are good. We already flatten project names there (eg, translate / to _). The secret key storage we are about to move to ZK. We can handle that right now. The time database will be eliminated as part of the v5 work as well, and replaced with SQL. Not a problem. That leaves the git repos. The main issue is that git doesn't deal well with git repos that are cloned into other git repos. We need to keep these separate. The internal merger repos (that is, the ones that users never see) will need a flattened namespace. That should be relatively straightforward since we can take care of that without a user-visible impact. Finally, that leaves the subject of this mail: the user-visible workspace checkouts. Once we get Zuul to a point where it doesn't error-out on a nested repo structure, we still have to decide how to lay it down on disk in the executor workspace. I can see three options: 1) Attempt to lay the repos out in the current fashion, with project/subproject cloned on top of project. I do not think we should do this. While we might be able to do this in the limited context of a workspace checkout (as compared to the internal merger repos), it would still be fraught with problems if, for example, anyone ever did add a "subproject" directory to "project". 2) Allow users to map projects to arbitrary repo checkouts. For this we might add a new mapping to a job like this: - job: checkouts: project: project project/subproject: subproject This would allow users to write out the repos wherever it suits them, whether that's a flat structure as above, or something more complex. They could also move just a single troublesome repo. In fact, you'd really only need the second line of the map above, the first is redundant. 3) Allow users to indicate they want a flat checkout. Given that this is the first use-case for this which we have seen, it's instructive to learn how the users who prompted this discussion are actually using their repos. They want a flat checkout. So they essentially only want the last component of the path to appear in their workspace. This option is a specialized version of the above that's less flexible, but probably simpler to use for this case: - job: flat-checkout: true That would map each project to its last path component. We would need to decide what to do if *that* caused a collision. Presumably we would error out. In this case, the user would have no recourse with Zuul (they would need to change a repo name or stop using a repo in that job). Options 2 and 3 are not mutually exclusive. We could do both so that the typical use would be #3 with #2 as a backup (or for someone with really creative ideas about how to lay down repos). We could also implement one and leave the other as an idea for future expansion. I suspect the internal implementation will be about the same for both. I think I'm inclined to start with #3 just because it concisely enables a specific pattern that may be useful in other contexts. Does anyone have feedback or a preference? -Jim P.S. I suspect we're going to find cases in zuul-jobs where we have assumed project.src_dir == 'src/'+project.canonical_name, but we can and should fix those. We made that a variable for a reason.

4 4

Proposed behavior change on reconfiguration
by James E. Blair 18 Apr '21

18 Apr '21

Hi, We recently merged some changes which improve the consistency of the git repos in different jobs within the same buildset. While it's always the case that every job in a buildset has the same view of the projects involved in its shared queue, projects outside of that queue (including projects involved in providing Zuul roles, playbooks and job required-projects) have not been synchronized, meaning that if a change merged in one of those projects between two jobs in a buildset starting, the earlier job would not see the change and the later job would. It seems more natural for Zuul to fix the state of all of the repos involved in all of the jobs of a buildset, and so the change that we recently merged did this. While this seems like a straightforward change, it had an unanticipated effect on reconfiguration. When Zuul performs a reconfiguration, any jobs that are added to a project are added to currently running buildsets and started; likewise, jobs may be removed from buildsets. Any jobs which are modified but have already run will retain their previous results, but jobs which have not yet run will do so with the new configuration. It turns out that the idea of fixing all of the repos involved in a buildset and also being able to dynamically add, remove, and alter jobs are fundamentally at odds. The simplest example of this is a reconfiguration that adds a playbook to the zuul-jobs repo, while a job on an unrelated repo is running If Zuul has frozen the state of all the repos involved (including zuul-jobs) then the updated version of the job will reference a playbook which will not appear in the previously frozen repo state. In other words, we can freeze both the configuration and the repos, or neither. Earlier in Zuul's development, we felt that the ability to add, remove, or alter jobs while they were running was fairly important. However, Zuul and its usage has evolved considerably since then. I'd like to propose that we continue down the path of freezing both the repo state and the job configuration within a buildset. This means than in the future, if a job is added, removed, or altered during a reconfiguration, it will not take effect for any currently running buildsets, but will take effect for any buildsets started after the reconfiguration (that includes new buildsets created on gate resets). Given the potentially significant change to a long-standing Zuul behavior, I ask that anyone relying on this or who feels that they may be affected by this to please let us know. Thanks, Jim

2 4

[Diversity] D&I WG Office Hours at the PTG
by Amy Marrich 16 Apr '21

16 Apr '21

The Diversity and Inclusion WG has grabbed our usual PTG slot Monday at 16:00UTC and we will be holding office hours to help any project with Inclusive Naming or other D&I related topics. If you'd like us to attend any of your sessions please let me know. If you've been interested in learning more about the WG please come join us! Thanks, Amy (spotz)

1 0

The error was: 'zuul_ssh_key_algorithm' is undefined
by Åsmund Østvold 31 Mar '21

31 Mar '21

Hi, Today a job that has not been changed for a long time( 1000+ executions) failed in a way I can not explain. The below snippet is from the "cleanup" after normal test execution that succeeded. Everything before and after looks to me to not have any problems. [test script output end] 2021-03-29 16:50:00.729468 | ubuntu-bionic | ok: Runtime: 0:02:02.256368 2021-03-29 16:50:00.751939 | 2021-03-29 16:50:00.752085 | PLAY RECAP 2021-03-29 16:50:00.752153 | ubuntu-bionic | ok: 2 changed: 2 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0 2021-03-29 16:50:00.752192 | 2021-03-29 16:50:00.992482 | RUN END RESULT_NORMAL: [trusted : review.resoptima.net/zuul/irma-common-jobs/playbooks/build_and_integrationt…] 2021-03-29 16:50:00.993174 | POST-RUN START: [trusted : review.resoptima.net/zuul/irma_config/playbooks/base/post-ssh.yaml@master] 2021-03-29 16:50:02.365604 | 2021-03-29 16:50:02.365802 | PLAY [all] 2021-03-29 16:50:02.400284 | 2021-03-29 16:50:02.400472 | TASK [remove-build-sshkey : Remove the build SSH key from all nodes] 2021-03-29 16:50:02.591665 | ubuntu-bionic | ERROR 2021-03-29 16:50:02.592037 | ubuntu-bionic | { 2021-03-29 16:50:02.592094 | ubuntu-bionic | "msg": "The task includes an option with an undefined variable. The error was: 'zuul_ssh_key_algorithm' is undefined\n\nThe error appears to be in '/var/lib/zuul/builds/e01e1c4faa834c079a62b6514ed2106f/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/remove-build-sshkey/tasks/main.yaml': line 1, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Remove the build SSH key from all nodes\n ^ here\n" 2021-03-29 16:50:02.592133 | ubuntu-bionic | } 2021-03-29 16:50:02.626484 | 2021-03-29 16:50:02.626654 | PLAY RECAP 2021-03-29 16:50:02.626737 | ubuntu-bionic | ok: 0 changed: 0 unreachable: 0 failed: 1 skipped: 0 rescued: 0 ignored: 0 [normal termination of job] Is this a known issue? As it is < 1/1000 probability of this happening it is not an important issue for me but wanted to check with you anyway. Regards, Åsmund

2 2

[marketing] Zuul 4 & 5 Press Release
by Jimmy McArthur 22 Mar '21

22 Mar '21

Hi Zuul Team - We're gearing up to start promoting the Zuul 4 and 5 release. I know we don't have a definitive timeline yet for the 5 release, but it seems clear that the Zuul community knows what features will land. Is there anyone in the community that could volunteer to put together a blogpost (location tbd - maybe openinfra.dev/blog or superuser). From that, we can extract features for promo and obviously push the blog out to our various marketing channels. Cheers, Jimmy

2 2

Getting a #403 Resource not accessible by integration"
by Cyril Lavier 19 Mar '21

19 Mar '21

Hi. I'm trying to setup a Zuul POC with Github, with 2 projects (one containing the jobs config/playbook and one being an actual project with a simple pipeline with a jsonlint job, to test). My Zuul is connected to Github, it can grab the repositories and receive calls from Github when creating a PR, but Zuul sends this message to the PR : ------------------------------------ Starting check jobs. Warning: Failed to update check run vanilla/check: 403 Resource not accessible by integration ------------------------------------ "vanilla" being the tenant name. In the Zuul WEB UI, the job is linked to the repository but it's still in "queued". I've checked the Github permissions and I don't find anything different than in this howto (https://zuul-ci.org/docs/zuul/howtos/github_setup.html) Also, in the meantime, no nodes are displayed in the "Nodes" section of the webUI, but I declared a static one (with the prerequisites from the doc) in the nodepool config. So I have 2 questions : - Why do I get this 403 error ?, or maybe a way to help me debug (debug logs don't help me a lot) - Why is there no nodes in the "Nodes" section of the webUI ? I assume the job being queued indefinitely may come from either the node not appearing or the 403 error. Thanks.

3 5

Type annotations
by jim＠acmegating.com 17 Mar '21

17 Mar '21

Hi, I'd like to propose that we remove type annotations and checking from our code. I don't think they are providing much value, but they come at a significant cost. They have very rarely (I think perhaps once or twice) caught errors. Meanwhile, we aren't going to relax our test coverage requirements, which should catch most things anyway. On the other hand, they cause the code to be much less readable and I believe that will ultimately lead to more errors. Developer and reviewer fatigue are important things to keep in mind. Some examples where I think annotations are particularly unhelpful: self.on_connect_listeners: List[Callable[[], None]] = [] Or (note the duplication): def __init__(self, ... info: Optional[zuul.model.WebInfo] = None): self.info: Optional[zuul.model.WebInfo] = info We've also had some occurrences where we ended up fighting the annotation system. I've mentioned to several of you that I do very much enjoy the benefits of statically typed languages (I write C code for fun), but I don't think that the type hints in Python are very well executed. Many of us chose to use Python instead of Java specifically for the dynamic typing and as a reaction against the extensive type requirements. The type hinting that is being added to Zuul seems like it carries the costs of Java without the benefits. We started this as a small experiment. We said that we would run the type checker in a mode where it would operate on implicit type information, thereby minimizing the additional cognitive load on authors and reviewers. We then started adding some minimal type information in strategic places to help it out. However, I'm now reviewing changes that are literally 3x larger than they otherwise would be because of all the additional repetitive type information going into them. I'm glad to try experiments like these, but I think this one has grown to the point where we need to decide if we want to go all-in on type annotations or remove them. I feel fairly strongly that the cost to authors and reviewers far outweighs the benefits, and we should make everyone's lives easier by removing them. -Jim

10 18

Dropping gerrit replication trigger event
by Tristan Cacqueray 16 Mar '21

16 Mar '21

Hello folks, After upgrading to Zuul version 4 and Gerrit version 3.2, our scheduler logs are flooded with ref-replication events (about 1GB of debug per hour). It seems like the gerrit driver could be improved to ignore unused events, and the queue processor could be less verbose to reduce the amount of noise. On the other hand, the replication event doesn't seems to be that useful and here is a change to ignore them: https://review.opendev.org/c/zuul/zuul/+/780809 If you do rely on replication event, please raise your voice in the review above. Thanks, -Tristan

2 1

vscode/codium editor for zuul config files
by Sorin Sbarnea 01 Mar '21

01 Mar '21

In case you happen to use vscode or codium, you can now use an extension that verified the syntax of the configuration files and provide auto-complete. Ansible Language extension v0.2.0 adds support for Zuul config files. https://marketplace.visualstudio.com/items?itemName=zbr.vscode-ansible&ssr=… https://open-vsx.org/extension/zbr/vscode-ansible If you like it useful, feel tree to leave a review. PS. Nope, I it does not work with vim, emacs, pycharm, ... Cheers Sorin

1 0

[all] April 2021 PTG Dates & Registration
by Kendall Nelson 01 Mar '21

01 Mar '21

Hello Everyone! I'm sure you all have been anxiously awaiting the announcement of the dates for the next virtual PTG[1]! The PTG will take place April 19-23, 2021! PTG registration is now open[2]. Like last time, it is free, but we will again be using it to communicate details about the event (schedules, passwords, etc), so please take two minutes to register! Early next week we will send out info to mailing lists about signing up teams. Also, the same as last time, we will have an ethercalc signup and a survey to gather some other data about your team. Can't wait to see you all there! -the Kendalls (diablo_rojo & wendallkaters) [1] https://www.openstack.org/ptg/ [2] https://april2021-ptg.eventbrite.com

1 0

Last call for 4.0 deprecations
by jim＠acmegating.com 15 Feb '21

15 Feb '21

Hi, We've merged the SQL database requirement, and as I write this, the ZooKeeper connection requirement patches are working their way through the gate. Those are the last big items blocking the 4.0 release. Before we issue the release, I'd like to ask folks to take a look for anything we've said that we would like to deprecate or remove in a major version bump to make sure we don't miss the opportunity. At this point, I don't think we should entertain last-minute new ideas, but rather, if we put something in the documentation quite some time ago saying that it would be removed in the future, let's make sure we take the opportunity to consider that now. Also, if anyone can think of any other blockers for 4.0, speak now please. :) Otherwise, I think we can cut the release very soon. -Jim

1 1

Scale out scheduler
by jim＠acmegating.com 13 Feb '21

13 Feb '21

Hi, The main blocker for releasing 4.0 at this point is merging the changes that implement the ZooKeeper connection requirement from every Zuul component. Currently the changes that implement those also include some extra changes that refactor the ZooKeeper connection classes. I think that's a good idea, but I'd like to separate concerns and focus on getting the requirements for the 4.0 release taken care of first. On Friday I discussed with some folks an approach to that, and today I finished up a rebase + rework of the series. I uploaded three new versions of these patches: https://review.opendev.org/775459 Mandatory Zookeeper connection for ZuulWeb https://review.opendev.org/775460 Connect executor to Zookeeper https://review.opendev.org/775461 Connect merger to Zookeeper I think they represent the minimum we need to get into 4.0. There is some boilerplate related to ZK connections in each service, but it's easy to reason about for now. I do think we should refactor the ZK connection class, but let's do that on top of these patches. The results will be clear and easy to understand. The original patches also increased the unit test job timeout because the jobs took approximately 25% longer to run. I think perhaps reviewers were under the impression that was necessary due to the new ZooKeeper connections in unit tests. I investigated the issue and found that was not the case. The actual cause was due to extra import statements that were added to support the addition of type annotations. After removing those type annotations and associated import statements, the test runtime remains the same as it currently is. You can read more details about it in the comments of patchset 2 of the last change. It's also worth keeping in mind that extraneous import statements can cause not only unanticipated problems like this, but can often exacerbate the problem of import loops, something we have seen occasionally in Zuul. Keeping additional imports for type annotations to a minimum will be beneficial. I propose the following next steps: 1) Re-propose 775459-775460 as new patchsets on their original changes in order to preserve the review history there. 2) Merge those changes and continue work on releasing 4.0. 3) Meanwhile: refactor the ZooKeeper connection class. 4) Continue work on the remaining scale-out scheduler changes. -Jim

1 0

Nodepool image building
by jim＠acmegating.com 10 Feb '21

10 Feb '21

Hi, I'd like to propose some new features related to Nodepool and image building. The short version is that right now image management is tied fairly closely to diskimage-builder within the nodepool-builder process. This is problematic for folks who want Nodepool image management but can't use DIB (ie, Windows images). It's even problematic for those who do want to use DIB, in that it's difficult to debug image builds. It's worth noting the original Zuulv3 spec had room for more than one nodepool-builder implementation (in an offhand comment, it mentioned that a snapshot-based nodepool-builder process could be created). However, that has not yet come to pass. Over the years, we've speculated about the idea of using Zuul to build images, and I think it's time to see if we can put that into practice. It makes a lot of sense in that we have a ready-made execution environment with Zuul -- build logs and history are well defined and accessible to anyone. It's flexible and can accommodate any image building system (DIB or otherwise). The main downside is the inception aspect, but I'm not worried about that (more later). I think the key requirements are: * Able to create images using any tooling. * Visibility into build history. * Nodepool-launcher should automatically use the "latest" build. Part of this solution is obvious, but I think there are open questions for other parts: * Should the Zuul job be responsible for uploading the image to the cloud? * Should Nodepool be responsible for deleting images? About that downside: clearly there's an issue when bootstrapping a new Zuul system -- how can you run jobs without images? Most new Zuul installations start by using cloud images from the provider. Any Zuul user who wants custom images likely already has some kind of job running on some kind of image anyway. I think it's a natural progression to create a Zuul job to build a custom image and then begin using that custom image in "real" Zuul jobs. Long story short: bootstrap the system with cloud-images. As an OpenDev admin, I think I would be comfortable with a Zuul job that built our DIB images using cloud provider images -- I don't think the differences between those images are so great as to make a DIB job difficult to write. However, if we did use this system, I'm not sure we would ever have to. We have enough clouds with our own images in them already, we could probably use our images to make new ones. In this case: a chicken with no egg. :) Let's work through how this might look. First the easy part, then I'm going to propose some alternate solutions. 1) Build the image: A Zuul job ("build-nodepool-image-ubuntu-focal") is triggered by a timer. It has a mutex to avoid duplicate image builds. It runs DIB in this case but could just as easily run packer or even snapshot the system it's running on (we can add roles in zuul-jobs for common image building systems). 2) Get the image into the cloud(s): a) Our first option is to do this work in Zuul. Have the job (or a dependent job) upload the image to the cloud(s) (if this was a snapshot build, that's a no-op -- it's already there). Advantage: the upload log is visible along with the build log. Disadvantage: if the upload fails, we won't retry like Nodepool does. b) An alternative would be to have Nodepool do this. Nodepool could even query the Zuul API to find the latest build of build-nodepool-image-ubuntu-focal, check the artifacts list, and download the image file from there. After that, it looks a lot like the current nodepool-builder process right after DIB runs (it has a file on disk and it starts to upload it). Advantage: we use all of the Nodepool image lifecycle management. Disadvantage: upload logs are hidden from users (like today). to integrate snapshot builds, we would have to download the snapshot and re-upload it. 3) Get nodepool-launcher to use the new image: a) If we use 2.a, then we have an image in the cloud without a record in Zookeeper. i) We could have nodepool-builder (or something like it) scan the cloud for images matching a certain pattern and create ZK records for them. ii) Or we could just have nodepool-launcher look for that pattern and use the latest image. b) If we use 2b, then launcher will work as it does today with nodepool-builder; no changes are needed. 4) Delete the image when it's no longer needed We should use the n-2 rule we have today so there's always at least 2 images in the cloud for easy rollback. a) If we used 2.a + 3.a.iii then we don't have ZK records. It would probably make sense to handle image deletes in Zuul jobs with this setup. Presumably the jobs would query the cloud API, list images, and delete them according to the age rules. The disadvantage here is we'd be re-implementing a good chuck of nodepool-builder in Ansible modules. It's doable, but may be a bit clunky. b) If we use 2.a + 3.a.i or 2.b + 3.b then we have records in ZK, so nodepool-builder (or something like it) can delete the images more or less how it does today. I think those are most of the choices. Now that I've typed all of that out, I think I really like the system with "b" for all the choices. I'm not too worried about the upload logs being hidden. And the snapshot system is workable, if not ideal. If we really wanted to support snapshots better, then 2.a + 3.a.i + 4.b would probably be best, but at the cost of moving the upload code into zuul-jobs roles. Or we could do the same thing we did last time and implement the "b" solution ("nodepool-zuul-builder") and if someone really wants to do snapshots, then we implement 2.a[snapshot] + 3.a.i + 4.b as ("nodepool-snapshot-builder"). That would probably be fairly efficient as far as code goes. Finally, a word about DIB and the current nodepool-builder: none of this requires that we change anything about the current nodepool-builder process. If we like it, we can keep it exactly as it is. Indeed, in OpenDev's case, we rely very heavily on the cache on the build host, so it's not at all clear that doing builds on throwaway nodes is ideal. Of course, we could do builds on static Zuul nodes and get the benefit of the cache. I think the way to approach this would be to implement the new system, then try using it with DIB, and if we love it and find no significant drawbacks, consider deprecating the current nodepool-builder. If we don't love it, we'll keep it. I'd appreciate feedback on this. I know this is a bit free-form; I'll take the feedback I get and synthesize it into a proper spec. -Jim PS: there's a fairly well-known hack for making non-DIB image builds by replacing DIB with a script that accepts the same arguments. The "b" plan could actually be implemented today with no changes to Nodepool using that same hack. A final implementation probably wouldn't look much different structurally.

5 6

job.provides/requires for periodic pipeline
by Vitaliy Lotorev 10 Feb '21

10 Feb '21

Hi, I have a jobA for project A (job builds images) and jobB for project B (job runs tests for those images). I set both jobs into a nightly periodic pipeline. periodic pipeline has jitter ±5 minutes. jobA and jobB run in pipeline asynchronously. jobB doesn't wait till jobA finishes and fetches latest successful build from jobA which appears to be built 24 hours ago (i.e. jobB tests in images built 24 hours ago). Questions: * Would setting jobA.provides=images and jobB.requires=images order them during one run? I.e both jobA and jobB would enqueue and jobB would wait while jobA finishes. Or should I use job.dependencies? * What are other assumptions for using requires/provides: ** Should be jobs in the same pipeline? ** if job has 'requires' how the job can be debugged separately without running jobs that define 'provides'? -- Regards, Vitaliy

2 3

Type annotations
by Jan.Kubovy＠bmw.de 09 Feb '21

09 Feb '21

…. > I'm not [yet] an expert on type annotations in Python, so I may have to > rely on others who know more details on what we can and should do in the > future, but here are some guidelines that might be a good starting > point: > > * Let's not bend over backwards to make mypy happy. That's not our > goal, so if something is tripping it up but is otherwise not ambiguous > to humans or IDEs, do the minimum work to ignore it. > > * Corollary: do we actually need/want to keep running mypy > automatically in our Zuul jobs? > > * Let's focus on improving interfaces, but not exhaustively typing every > variable. > > * If there are improvements coming in the future (like 3.9) let's do > less now and wait for them. > > And based on those, what do you think about these specific guidelines: > > * Do type method signatures. This is always a good idea without it most of the checking will not work. > * Don't recurse ("List[Callable]", not "List[Callable[[], None]]"). I would not make that a hard rule, more a recommendation. If there Is a benefit in certain cases I would allow it. One can also use type aliases and type vars to make it better usable and look less awkward. > * Avoid typing variable assignment inside of methods. (They should > mostly be inferred, and they do little to improve code comprehension.) If this works reliably, for everyone using it (not sure if this will not break for example usage search). Otherwise I don’t see as much drawback having a list of typed assignments in the init(). But I guess this Is a matter of taste. Jan _______________________________________________ Zuul-discuss mailing list Zuul-discuss(a)lists.zuul-ci.org<mailto:Zuul-discuss@lists.zuul-ci.org> http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-discuss

2 1

Type annotations
by Jan.Kubovy＠bmw.de 08 Feb '21

08 Feb '21

Hi all, my 0.02$ to the discussion: Although I am currently not an active contributor nor a reviewer a year ago I started the work on scale-out-scheduler as a part of my day job. This lead me to introduce the typings in parts of old code and use it in new code. As zuul is a community project I do believe that it relies not only on core developers, who are involved over long period of time but should also be code-friendly to contributors, who want to or are tasked to one-time, short-period jobs. While I do like dynamic languages and understand their advantages, I worked with e.g. with Ruby a decade and half ago and I work with python in other projects too, I do like more an as-complex-as-necessary and as-simple- and as-readable-as-possible code base. IMHO it always depends on a context: A small POC would probably never use type annotation, a huge code base like zuul should IMHO consider it. I do agree that typings add a certain degree verbosity (I was an early adopter of Kotlin - one of the reasons was reduced verbosity, optional type defs) but they also come with certain benefits - better code navigation, support when refactoring, IDE support, etc. Also, I do not understand use of type annotations as an attempt to make python statically typed. The issues/reasons which lead me to introduce them: - Ambiguous parameter/argument names E.g. project, tenant, job, to name a few. Sometimes a name, sometimes an object, sometimes an Ansible Job, sometimes a zuul job or build. Makes it hard to navigate in the code. Since some of those names are too generic it becomes a challenge when working on the heart of zuul. Probably not so much when dealing with small changes. Just to be clear I am not talking about cases here where one may intentionally want to pass an "project" argument which may be both, a project name and a project object, and the code can deal with both. Those occasions barely exist here. Usually it is one or the other. - NPE, missing property Setters called right after instantiating an object. Making the argument basically optional. Since scale-out-scheduler touches a lot of code I encountered more than once that something which supposed not to be none was after a bit of refactoring. Just by navigating in the code was not always easy to see where a certain argument is set. E.g. try to find usages of attribute job, project, etc and where those are set for certain object. You will get every class containing that attribute - basically, reducing it to a text search result. This (https://review.opendev.org/c/zuul/zuul/+/757149/14/zuul/scheduler.py) is an example where one can see a simplification of code and reducing the risk of runtime error where typings were involved. Tests are great but the theoretical case that someone may set some attribute to None (a new change or config case) may not be covered (I guess zuul tests do not cover 100% branches/complexity). While it is not bulletproof typings do help here. - Code Navigation IDE support, usage search, etc. - API changes and refactoring Was, and I believe still is, very much needed especially in big tasks like scale-out-scheduler. - Code documentation This goes a bit against the verbosity argument. When the type is actually documented in the comment section making it in addition prone to inconsistency. Type systems additionally help me to prevent some errors and make me thing time to time if my concepts, implementation is correct. I did make myself as the first guinea pig to test the efforts vs. benefit argument in zuul and it did serve me the purpose. After more colleagues were involved in that topic, we had the same discussion and came eventually to the same result. I understand there is an effort needed, but I also believe that the benefits outweigh the cost dramatically, especially in long-term and when considering a need-to usage. That said, I never thought about a 100% typing through the code base. For example, some classes, e.g. zuul.cmd.Scheduler may not benefit from typings as much, classes in zuul.model definitely would, due to their reuse thought-out the code. For the juniority argument I strongly believe that also experienced developers may find this beneficial especially in cases described above (one-time, short-term involvement). Unless there is, of course, an argument against such contributions (I am not aware of). Since, typings are now part of python and evolving over the past few versions, it does not seem to go away any time soon. From what I can see more and more pyhon code uses them. I do not agree that using typings is a sign of choosing the wrong language here. There may be use-cases where not using them may be more beneficial (fast prototyping, etc), and some use-cases where using them will benefit the project. Also using typings does not prevent loose/dynamic typing (e.g. Union or just not using them everywhere if it makes sense). It would be pity not to use the chance of a big change like scale-out-scheduler/zuul 4.x is to improve the code just to find ourselves few years down the road regretting it. The trend suggests that having the option of type hinting (while keeping dynamic types) is the way to go since it kind of offers the benefits of both worlds - other languages e.g. TypeScript, Kotlin JS, to name a few. Since, I felt this will be a controversial change (changes always are), I tried to keep the typings in a separate change in gerrit were ever I could - so you can easily remove them by abandoning those changes. They served me well and should I get another task in zuul I would like to find them again also in other parts of the code. Cheers Jan _______________________________________________ Zuul-discuss mailing list Zuul-discuss(a)lists.zuul-ci.org<mailto:Zuul-discuss@lists.zuul-ci.org> http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-discuss

1 0

Making gerrit zuul result comments more readable
by Sorin Sbarnea 02 Feb '21

02 Feb '21

At this moment the comments added by zuul on gerrit can be very hard to read, partially due to gerrit limitations I have projects where I cannot even fit them in a single screen! -- each job spans **3** rows instead of a single one. https://sbarnea.com/ss/Screen-Shot-2021-02-02-09-04-45.40.png While my hopes on adding markdown support on gerrit comments <https://bugs.chromium.org/p/gerrit/issues/detail?id=9299> are low, as nobody from google even did bother to get involved. (please comment/star ticket if you want it) I know that we plan to integrate the results in the UI and avoid comments, but that is long temp plan which does not fix the issue for email notifications at all. Here are some measures that we could do to lower the amount of wasted space: a) Put an emoji stating the result as the first item in the list, which should make it easy to spot the failed ones ❌ foo ... ✅ bar ... b) sort jobs alphabetically c) remove seconds from duration, they are useless even for the fastest jobs d) use shortened build-id for URLs, 5 chars should provide more than enough entropy If we do all of these we should make it fit on a single line, even for longer job names. Still, every single one would add visible benefits. What do you think? I also considered other alternatives like listing failed at top and non-voting at bottom, so the order will be failed, passed, non-voting, but I can see how this may struggle to be accepted. The root cause of this formatting issue is this style: .message { --gr-formatted-text-prose-max-width: 80ch; } Sadly due to the use of shadow-dom, I can no longer use css styling to override the stylesheet. -- /sorin

2 1

[Diversity] Diversity &Inclusion WG Meeting reminder Amy Marrich <amy@demarco.com> Mon, Jan 11, 7:49 AM to foundation, openstack-discuss, kata-dev, starlingx-discuss, airship-discuss, zuul-discuss The Diversity & Inclusion WG invites members of all OSF projects to our meeting today, at 17:00 UTC in the #openstack-diversity channel. The agenda can be found at https://etherpad.openstack.org/p/diversity-wg-agenda. Please feel free to add any other topics you wish to discuss at the meeting. Thanks, Amy (spotz)
by Amy Marrich 31 Jan '21

31 Jan '21

The Diversity & Inclusion WG invites members of all OSF projects to our meeting tomorrow, at 17:00 UTC in the #openstack-diversity channel. The agenda can be found at https://etherpad.openstack.org/p/diversity-wg-agenda. Please feel free to add any other topics you wish to discuss at the meeting. Thanks, Amy (spotz)

1 0

Re: swift-proxy repo creation
by okozachenko 28 Jan '21

28 Jan '21

I suggest that we can agree on the repository name first. I think 2 options for now - `zuul-storage-proxy` which includes any kinds of storage backend which can be configurable by envvar or conf file - `zuul-swift-proxy` which includes only swift proxy. And for other kinds of storage, we will create a new repo. I hope we decide the repo name soon. It also specifies the repo scope. Thank you.

5 7

[openinfra] 2020 Open Infra Foundation Annual Report
by Jimmy McArthur 28 Jan '21

28 Jan '21

Despite being a very different year than most, the Open Infrastructure community, which has over 110,000 community members, made it a productive and successful year. We could not have done this without each and every one of you. One of the biggest milestones that happened in the global community last year was that OpenStack, one of the top three most active open source projects with 15 million cores in production, marked its 10th anniversary in 2020. OpenStack is undoubtedly a success. Millions of CPU cores of computing infrastructure are provided through OpenStack. More than 70 regions of public clouds powered by OpenStack are available for users worldwide. Last year saw our 22nd (on-time) release, with more than 20,000 changes merged in only 5 months. A level of activity that placed OpenStack among the three most active open source development communities in the world. Now, we are seeing users who have been around for years talk about their growing OpenStack production deployments like Workday, LINE, and CERN, as well as new users talk about how they’re running other open source projects in production like Ant Group using Kata Containers, Verizon running StarlingX, and Volvo running Zuul for their autonomous driving software. The Open Infrastructure Foundation would like to extend a huge thanks to the global community for all of the work that went into 2020 and is continuing in 2021 to help people build and operate open infrastructure. Check out the OpenStack community’s achievements in 2020 from the OpenInfra Foundation Annual Report and join us (https://openinfra.dev/join/) [1] to build the next decade of open infrastructure! The 2020 OpenInfra Foundation Annual Report [2] Cheers, Jimmy [1] https://openinfra.dev/join [2] https://www.openstack.org/annual-reports/2020-openstack-foundation-annual-r… Let’s cheer on the new year and to the successful growth of the seeds planted over these past years.

1 0

newbie: wrong comment url posted
by Åsmund Østvold 27 Jan '21

27 Jan '21

hi Again, I am sorry to have to ask but. I have spent the day trying to find the solution. Zuul posts wrong url to logs in Gerrit comments. We get http://zuul-ci.resoptima.net:8000/65/3365/1/..... But we should get: http://zuul-ci.resoptima.local:8000/65/3365/1/.... The host that is running the docker-compose env had originally .net ending. Now it is renamed and I have to resolve the url issue. I have searched all of my setup folders. It has only minor changes to the docker-compose example provided. And I have not explisetly specified zuul-ci.resoptima.net anyware now. In an attempt to fix the issue I have added web.root and web.status_url in etc_zuul/zuul.conf.j2 without it taking effect. I have verified that the new config is present in the docker containers. [web] root=http://zuul-ci.resoptima.local status_url=http://zuul-ci.resoptima.local listen_address=0.0.0.0 Again any help would be appreciated! Even if we are having some small issues like this one I still enjoy Zuul very much! Regards, Åsmund

1 1

Gerrit: tags on changes not merged do not trigger builds
by Åsmund Østvold 27 Jan '21

27 Jan '21

hi, I assume the below request is a bit strange so I will give a short background: Our dev team produces some libraries that are used in different versions in our products. The distribution is done by publishing to https://myget.org/. To enable testing of pre-merge commits our previous build system did get triggered on tags pointing to unmerged gerrit changes. The issue: The tag pipeline defined like this: - pipeline: name: tag description: This pipeline runs jobs in response to any tag event. manager: independent precedence: high post-review: True trigger: gerrit: - event: ref-updated ref: ^refs/tags/.*$ success: mysql: failure: mysql: Our Zuul instans do not start builds when tags are pointing to change that is not merged. Builds do start when tags point to merged commits. I believe the relevant log is: 2020-12-18 10:12:03,656 INFO zuul.Pipeline.resoptima.tag: [e: 4435a9ca5b92495086f3083cf41e62fd] Adding change <Tag 0x7f3174086070 resoptima/irma.databaseextensions creates refs/tags/1.3.0-beta.1 on 8d0bdb76b5122402cc66d9f2b53bdf65d3409e7b> to queue <ChangeQueue tag: resoptima/irma.databaseextensions> in <Pipeline tag> 2020-12-18 10:12:03,657 INFO zuul.Pipeline.resoptima.tag: Reporting enqueue, action [] item <QueueItem 0x7f31542667f0 for <Tag 0x7f3174086070 resoptima/irma.databaseextensions creates refs/tags/1.3.0-beta.1 on 8d0bdb76b5122402cc66d9f2b53bdf65d3409e7b> in tag> 2020-12-18 10:12:03,657 WARNING zuul.GerritConnection: Unable to get change for <GerritTriggerEvent ref-replication-scheduled review.resoptima.net/resoptima/irma.databaseextensions> 2020-12-18 10:12:03,851 INFO zuul.MergeClient: [e: 4435a9ca5b92495086f3083cf41e62fd] Merge <gear.Job 0x7f31542fd8b0 handle: b'H:172.18.0.8:765' name: merger:refstate unique: 54a22a170b1a439c82acbb9948c8a109> complete, merged: False, updated: True, commit: None, branches: [''] 2020-12-18 10:12:22,555 WARNING zuul.GerritConnection: Unable to get change for <GerritTriggerEvent ref-replicated review.resoptima.net/resoptima/irma.databaseextensions> 2020-12-18 10:12:22,555 WARNING zuul.GerritConnection: Unable to get change for <GerritTriggerEvent ref-replication-done review.resoptima.net/resoptima/irma.databaseextensions> Is there something I can do to get this to work? Cheers, Åsmund

4 5

behavior change for pull-from-intermediate-registry
by Paul Belanger 26 Jan '21

26 Jan '21

Greetings, I believe we've encountered a new use case for pull-from-intermediate-registry and would like some feedback. TL;DR: pull-from-intermediate-registry will blindly overwrite images in the buildset-registry, I'd like to expose a way to toggle that feature. Today in zuul.a.c (ansible) we are in the process of migrating a lot of our testing to containers. As such, we have a new project called: https://github.com/ansible/network-ee/ This is the main repo used for building containers, and a job called: network-ee-build-container-image https://github.com/ansible/network-ee/blob/main/.zuul.d/jobs.yaml#L42 Basically, this job run generic enough to run in multiple projects, creating a new container image called: quay.io/ansible/network-ee:latest (there are more images, but we'll use this for the example) In our case, the new use case happens when using depends-on headers with network-ee-build-container-image jobs. We also have an image test job, which create: quay.io/ansible/network-ee-sanity-tests:latest using quay.io/ansible/network-ee:latest above. So for network-ee-sanity-tests job to run, first pause network-ee-build-container-image, then the sanity test job will run. The network-ee-sanity-tests job, parents to our: ansible-buildset-registry-consumer job Which, is basically the same as opendev's version. In this job, is where we run the pull-from-intermediate-registry role, to interact with the previous PR and copy images. The issue, in our case, is because our network-ee-build-container-image job runs in multiple places, it is possible for PR B, which depends on PR A, to have PR Bs quay.io/ansible/network-ee:latest images overwritten by PR A version. Here is an example: In https://dashboard.zuul.ansible.com/t/ansible/buildset/952c22f2df6c4a7181ce5… we have a failing job, network-ee-sanity-test. Looking at the logs, we have used: sha256:803564426f582436b6078d7ae203e2c109f4a47d04e48b78a2498c3755067aa2 https://ce377c2f657a3f4ccc50-902109811c92da02411ac82ade9b6da4.ssl.cf2.rackc… which you can see, is copied in from the intermediate registry: https://ce377c2f657a3f4ccc50-902109811c92da02411ac82ade9b6da4.ssl.cf2.rackc… If you dig more, sha256:8035644 is not produced from: https://dashboard.zuul.ansible.com/t/ansible/build/6bfc5ce56b2241f9bd709e14… our image build job in the buildset, but the parent job located at: https://cabd361c53760bec77a3-269ce6cee969dd0936b97c4252700a78.ssl.cf2.rackc… It took me a while, to fully understand what is happening, so I wanted to share here. In our case, I can think of 2 potential solutions. 1). update our ansible-buildset-registry-consumer job, to expose a flag to toggle the usage of pull-from-intermediate-registry. Given that our test jobs really don't need images from another PR, and we always rebuild, I think this is the safest option. 2). Expose a new flag in pull-from-intermediate-registry to first check if an image exists in the buildset registry, if not, don't copy over it. As it _should_ be a newer image, that was build in the current change, which already includes the changes of the previous depends-on change. Hopefully I explained clearly what is happening, if not, I can share more information. Paul

2 4

swift-proxy repo creation
by okozachenko 26 Jan '21

26 Jan '21

Hi Zuul team. Hope all are safe. I am trying to create a new repo under zuul namespace named swift-proxy. The new repo would include this code. https://review.opendev.org/c/zuul/zuul/+/708233 Let me explain a little about this PS. It is a wsgi proxy to proxy the requests like this; ``` ZUUL_HOST_ADDRESS/{CONTAINER_PREFIX}-{TENANT_NAME}/path -> SWIFT_SERVICE_ENDPOINT/{CONTAINER_PREFIX}-{TENANT_NAME}/path ``` This proxy requires clouds.yaml file and CLOUD_NAME envvar for the swift. Please give the feedback. Thank you. Open10k8s

5 5

Zuul-executor error
by sitakanta hotta 22 Jan '21

22 Jan '21

Here is my configuration. Am I missing host and port? [executor] manage_ansible=false log_config=/etc/zuul/executor-logging.conf job_dir=/var/lib/zuul/builds variables=/opt/project-config/zuul/site-variables.yaml private_key_file=/var/lib/zuul/ssh/nodepool_id_rsa trusted_ro_paths=/etc/openafs:/etc/ssl/certs:/var/lib/zuul/ssh trusted_rw_paths=/afs untrusted_ro_paths=/etc/ssl/certs disk_limit_per_job=5000 I get the following error Traceback (most recent call last): File "/usr/local/bin/zuul-executor", line 8, in <module> sys.exit(main()) File "/usr/local/lib/python3.8/site-packages/zuul/cmd/executor.py", line 113, in main Executor().main() File "/usr/local/lib/python3.8/site-packages/zuul/cmd/__init__.py", line 219, in main self.run() File "/usr/local/lib/python3.8/site-packages/zuul/cmd/executor.py", line 97, in run self.start_log_streamer() File "/usr/local/lib/python3.8/site-packages/zuul/cmd/executor.py", line 57, in start_log_streamer streamer = zuul.lib.log_streamer.LogStreamer( File "/usr/local/lib/python3.8/site-packages/zuul/lib/log_streamer.py", line 166, in __init__ self.server = LogStreamerServer((host, port), File "/usr/local/lib/python3.8/site-packages/zuul/lib/log_streamer.py", line 155, in __init__ super(LogStreamerServer, self).__init__(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/zuul/lib/streamer_utils.py", line 96, in __init__ socketserver.ThreadingTCPServer.__init__(self, *args, **kwargs) File "/usr/local/lib/python3.8/socketserver.py", line 448, in __init__ self.socket = socket.socket(self.address_family, File "/usr/local/lib/python3.8/socket.py", line 231, in __init__ _socket.socket.__init__(self, family, type, proto, fileno) OSError: [Errno 97] Address family not supported by protocol

2 2

Adopting Discourse for mailing list?
by Kenny Ho 21 Jan '21

21 Jan '21

Hi, Just curious, has anyone thought of adopting Discourse ( https://www.discourse.org/) for either Zuul or OpenDev (or both) for hosting mailing lists and forums? I am just wondering since Discourse seems to have more functionalities and it is open source. Some examples of open source organizations using Discourse are Fedora ( https://discussion.fedoraproject.org/) and Mozilla ( https://discourse.mozilla.org/) Regards, Kenny Ho

8 13

teaming up with patternfly team on building a generic ANSI/terminal component
by Sorin Sbarnea 16 Jan '21

16 Jan '21

As we had lots of discussions recently around ANSI processing in zuul, with multiple approaches for enabling ANSI in zuul I realized that maybe we should aim to resolve this at patternfly level, so any application using it may benefit from it. They have very good UX designers, something that we kinda miss around here. Few days ago I raised https://github.com/patternfly/patternfly-design/issues/955 and they want some extra feedback on this. I would like to invite those interested to watch and reply on this ticket, maybe we can end-up with an awesome solution. Cheers, -- /sorin

1 1

[yikes - marketing] Mini-Announcements/Social
by Jimmy McArthur 15 Jan '21

15 Jan '21

Hi everyone - We'd love to get some announcements out to help bring some notice and attention to the hard work the Zuul Community is doing. Generally we do this around big releases, but there's currently a bit of time until the Zuul 4 release. In the meantime, we'd really like to get some ideas from YOU - the community - on what we can help promote. We're not experts on CI/CD, so some context around what you would like to see promoted would be greatly helpful as well. For example, we'd love to push out some notices on: Nodepool 3.14.0 release New Elasticsearch feature Growth of the community/commercial support with the new Acme Gating and existing Vexxhost self-service The more context we can get the better. If there's some other new/in-the-works feature, please let us know. Cheers, Jimmy

4 5

My new role at Acme Gating
by jim＠acmegating.com 15 Jan '21

15 Jan '21

Hi, I'm pleased to announce that I have started a company to provide support, custom development, and consulting for Zuul. It's called Acme Gating. This isn't a forum for advertisement, so I'll keep this brief, though I do think it's important for folks to know what we all bring to the table as members of the community. My hope is that by growing the company, I can help support new Zuul users and thereby grow the Zuul community, all while continuing to support the Zuul project. There's a website[1] and twitter[2]. Please get in touch if there's anything you'd like to discuss. I look forward to continuing to be a part of this great community we've built. -Jim [1] https://acmegating.com/ [2] https://twitter.com/acmegating

3 3

migrating zuul-jobs towards an ansible collection
by Sorin Sbarnea 14 Jan '21

14 Jan '21

Does anyone have any reasons to oppose a gradual transition towards making zuul-jobs repository a real Ansible collection? If we do, we make it much easier to deploy and re-use these roles in order areas. We already followed most of the guidelines, the only real issue is around role names, which are not allowed to use use dashes, only underlines are supported. While collections are available only with ansible 2.9+ (2.8 support was only experimental), it does not mean that we cannot use the same code with older versions of Ansible. IMHO, we should not wait until the minimal version of Ansible we support would become 2.9 and just start to make changes towards building it as a collection. Regarding role re-names, probably we could make use of symlinks for keeping the old names as aliases and avoid breaking consumers. -- /sorin

2 1

authentication in the web UI: an update
by Matthieu Huin 14 Jan '21

14 Jan '21

Hello Zuul Community, TL;DR: "git review 769943 && cd doc/source/examples && docker-compose up", "firefox http://localhost:9000" and give feedback please! Long version: The changes implementing support for authentication-related features in the web UI should be ready for review. Since several parties voiced their interest in this, I'd appreciate community feedback so it can move forward to integration sooner than later. You can find the patch chain at this URL: https://review.opendev.org/q/topic:%22fffaff%22+(status:open%20OR%20status:… (do not mind the random topic name) The proposed features are: - Tenant scoped login ( https://review.opendev.org/c/zuul/zuul/+/734082 and https://review.opendev.org/c/zuul/zuul/+/735586 ) ie it is possible to associate one specific identity provider per tenant. - Once logged in, a User modal can be toggled on/off from the top navigation bar. The modal holds basic info on the user such as her name, username (with a pretty wizard hat next to it if she is allowed to perform admin actions as per the tenant's access rules). More importantly, you can also see there information on the user's token such as its expiry time, and the full token should the user want to use it in zuul-client. I've also thrown in a clipboard button that lets you copy a boilerplate command to use zuul-client with this token, and the right API endpoint (sometimes difficult to figure out for end users of tenant deployments). - Dequeue a live change from the status page ( https://review.opendev.org/c/zuul/zuul/+/734850 ) - Re-enqueue a buildset from the buildset's page ( https://review.opendev.org/c/zuul/zuul/+/736772 ) - Create a autohold request from a build's page ( https://review.opendev.org/c/zuul/zuul/+/768115 ) - Autoholds page, similar to the builds or buildsets pages ( https://review.opendev.org/c/zuul/zuul/+/768199 ) In order to simplify testing, I have documented how to configure a zuul instance to use Google OAuth or Keycloak as an Identity Provider in https://review.opendev.org/c/zuul/zuul/+/734082 . If you are familiar with other OpenID Connect Providers, it'd be great if you could ensure they play nice with zuul as well. If you'd rather not bother with setting up another service to test things, I have also modified the Docker compose used by zuul-quick-start, in https://review.opendev.org/c/zuul/zuul/+/769943 . It adds a keycloak container to the mix, and configures gerrit and zuul to use it as their identity provider. It also comes with predefined access rules and users as examples. This is much more straightforward, but requires a few tweaks before using it: * you need to map the 'keycloak' alias to localhost in /etc/hosts. This could be solved if there was a way to forward traffic inbound to http://localhost:8082 to the keycloak container, within other containers. * In doc/source/examples/docker-compose.yaml, you can comment out line 92 and uncomment ll.94-97 to build the zuul-web container based on the patch chain. This is recommended, but also increases significantly the duration of the build phase when doing "docker-compose up". * If you don't edit docker-compose.yaml, you will have to run a dev server of the UI: "cd web && REACT_APP_ZUUL_API='http://localhost:9000/api/' yarn react-scripts start". This will however trigger CORS-related errors with the admin API in your browser; with chrome you can disable CORS checks (do that only for testing!) by running chrome with the --disable-web-security CLI option. Note that the CORS related errors are fixed in the patch chain; they don't appear if building the zuul-web container as discussed above. Apologies for the big wall of text, but I put in the effort so that you don't have to. Do not hesitate to ping me on IRC (mhu) if you need help with this. I'm looking forward to community feedback, and hope we can get these handy features merged soon! Cheers, -- Matthieu Huin Senior Software Developer Red Hat <https://www.redhat.com> <https://red.ht/sig>

2 1

Only a few hours left: 2021 Board elections and Bylaws amendments
by Allison Price 13 Jan '21

13 Jan '21

Hi everyone, The 2021 Individual Director elections are currently open. All members who are eligible to vote should have received an email ballot earlier this week. You can review the candidates here: https://www.openstack.org/election/2021-individual-director-election/Candid… <https://www.openstack.org/election/2021-individual-director-election/Candid…>. If you are not currently a member of the Open Infrastructure Foundation, you can join here to vote in next year’s elections: https://openinfra.dev/join <https://openinfra.dev/join> This year, the election includes a set of amendments to the Foundation’s Bylaws. The amendments were approved unanimously by the Foundation Board in 2020. As a member class, the Individual Members must also approve these amendments. You can find more information in this mailing list post by Jonathan Bryce: http://lists.openstack.org/pipermail/foundation/2021-January/002939.html <http://lists.openstack.org/pipermail/foundation/2021-January/002939.html> The deadline to vote is this Friday, January 15 at 1800 UTC / 12pm CST. Please let me know if you have any questions. Allison Allison Price Open Infrastructure Foundation allison(a)openstack.org <mailto:allison@openstack.org>

1 0

[Diversity] Diversity &Inclusion WG Meeting reminder
by Amy Marrich 11 Jan '21

11 Jan '21

The Diversity & Inclusion WG invites members of all OSF projects to our meeting today, at 17:00 UTC in the #openstack-diversity channel. The agenda can be found at https://etherpad.openstack.org/p/diversity-wg-agenda. Please feel free to add any other topics you wish to discuss at the meeting. Thanks, Amy (spotz)

1 0

Project lead election
by jim＠acmegating.com 05 Jan '21

05 Jan '21

Hi, The position of Zuul Project Lead is an elected position described in our governance document[1]. I currently hold the position, and would be happy to continue in that position for another year. If elected, I plan to continue my current hands-off approach. I believe that all of the Zuul Maintainers are empowered to collectively decide on and execute our vision for the project. My purpose is to help articulate that vision when necessary, drive the process of attaining consensus, and be a point of contact and representative for the project in circumstances where a single person is needed (but in those cases, I promise to share as much of that work as possible with the rest of the team). I have uploaded a change[2] to update the expiration date of my term. If any other Zuul Maintainers would like to nominate themselves for the position, please upload a similar change and send a reply to this message to this list. The nomination period will close on January 14, 2021. If there are other candidates, we will hold a Condorcet election at that time. -Jim [1] https://zuul-ci.org/docs/zuul/reference/governance.html#zuul-project-lead [2] https://review.opendev.org/769420

1 0

Re: Gerrit: tags on changes not merged do not trigger builds
by Åsmund Østvold 18 Dec '20

18 Dec '20

Very sorry, this should have been sent to zuul-discuss(a)lists.zuul-ci.org. Will post to that list. On Fri, Dec 18, 2020, 13:25 Åsmund Østvold <asmundo(a)gmail.com> wrote: > hi, > > I assume the below request is a bit strange so I will give a short > background: > My company produces some libraries that are used in different versions in > our products. The distribution is done by publishing to https://myget.org/. > To enable testing of pre-merge commits our previous build system did > get triggered even on tags pointing to unmerged gerrit changes. > > > The issue: > The tag pipeline defined like this: > > - pipeline: > name: tag > description: This pipeline runs jobs in response to any tag event. > manager: independent > precedence: high > post-review: True > trigger: > gerrit: > - event: ref-updated > ref: ^refs/tags/.*$ > success: > mysql: > failure: > mysql: > > Our Zuul instans do not start builds when tags pointing to change that is > not merged. Builds do start when tags point to merged commits. I believe > the relevant log is: > > 2020-12-18 10:12:03,656 INFO zuul.Pipeline.resoptima.tag: [e: > 4435a9ca5b92495086f3083cf41e62fd] Adding change <Tag 0x7f3174086070 > resoptima/irma.databaseextensions creates refs/tags/1.3.0-beta.1 on > 8d0bdb76b5122402cc66d9f2b53bdf65d3409e7b> to queue <ChangeQueue tag: > resoptima/irma.databaseextensions> in <Pipeline tag> > 2020-12-18 10:12:03,657 INFO zuul.Pipeline.resoptima.tag: Reporting > enqueue, action [] item <QueueItem 0x7f31542667f0 for <Tag 0x7f3174086070 > resoptima/irma.databaseextensions creates refs/tags/1.3.0-beta.1 on > 8d0bdb76b5122402cc66d9f2b53bdf65d3409e7b> in tag> > 2020-12-18 10:12:03,657 WARNING zuul.GerritConnection: Unable to get > change for <GerritTriggerEvent ref-replication-scheduled > review.resoptima.net/resoptima/irma.databaseextensions> > 2020-12-18 10:12:03,851 INFO zuul.MergeClient: [e: > 4435a9ca5b92495086f3083cf41e62fd] Merge <gear.Job 0x7f31542fd8b0 handle: > b'H:172.18.0.8:765' name: merger:refstate unique: > 54a22a170b1a439c82acbb9948c8a109> complete, merged: False, updated: True, > commit: None, branches: [''] > 2020-12-18 10:12:22,555 WARNING zuul.GerritConnection: Unable to get > change for <GerritTriggerEvent ref-replicated > review.resoptima.net/resoptima/irma.databaseextensions> > 2020-12-18 10:12:22,555 WARNING zuul.GerritConnection: Unable to get > change for <GerritTriggerEvent ref-replication-done > review.resoptima.net/resoptima/irma.databaseextensions> > > > Is there something I can do to get this to work? > > Cheers, > Åsmund > >

1 0

what am I doing wrong
by Åsmund Østvold 18 Dec '20

18 Dec '20

hi Again, I like Zuul but it is a bit hard to get it correct. Sorry for the amount of text. A project has zuul.d/job.yaml: --- - job: name: build-and-integration-test parent: base run: - playbooks/build_project.yaml - zuul.d/run_testing.yaml required-projects: - zuul/irma-common-jobs and zuul.d/run_resting.yaml - hosts: all tasks: - name: asdf shell: cmd: | echo testing set -ex cd {{ zuul.project.src_dir }} ./integrationtests.sh feedback form Zuul is: Zuul encountered a syntax error while parsing its configuration in the repo ######/????.BARE master. The error was: Configuration item has more than one key. Each zuul.yaml configuration file must be a list of dictionaries with a single key, for example: - job: name: foo - project: name: bar Ensure that every item in the list is a dictionary with only one key (in this example, 'job' and 'project'). This error may be caused by insufficient indentation of the keys under the configuration item ('name' in this example). The incorrect values are around: hosts: all tasks: - name: asdf shell: Acording to https://zuul-ci.org/docs/zuul/reference/job_def#attr-job.run 'run' alow a list. I am sure I am making a newbie mistake but I can not spot it. Could you please help? cheers, Åsmund

4 5

new projects without restart
by Åsmund Østvold 18 Dec '20

18 Dec '20

hi, Number of projects wanting to use my Zuul setup is increasing. I am looking for a way to tell the Zuul setup that the number of 'untrusted_projects' have increased without having interrupted running builds. Can not find a solution in the documentation. I have tried the zuul German client and was not able to find a solution. Is this possible? Cheers, Åsmund

2 2

[all][SDK][zuul][DIB] CentOS-8
by Artem Goncharov 14 Dec '20

14 Dec '20

Hi everybody, Over around a week we have patches in SDK being blocked by functional test of the nodepool (https://123b844dd0bc940cd071-d2975a8f065f970ca29e743c021b8a36.ssl.cf1.rackc… <https://123b844dd0bc940cd071-d2975a8f065f970ca29e743c021b8a36.ssl.cf1.rackc…>) trying to build CentOS-8 image blocked by https://review.opendev.org/c/openstack/diskimage-builder/+/765963 <https://review.opendev.org/c/openstack/diskimage-builder/+/765963> caused by CentOS rename their packages (the DIB change helps to fix the issue - just verified locally). Normally I would say OK, let’s merge DIB fix and everything is fine, but due to a recent announcement of RedHat (IBM) to discontinue CentOS (non stream) I wonder, whether the DIB change as such is useful at all. At least for the moment I see a perhaps more appropriate fix in nodepool to build some other image (or Fedora or at least to use CentOS stream). DIB should be fixed as well, that’s clear, but maybe differently. Any way we need to come up with solution quickly, since our CI is blocked affecting quite some projects. P.S. Actually that issue is not only affecting our tests, but every Zuul installation trying to build CentOS-8 images. Any opinions? Artem

2 1

[zuul-ci.org website] Adding info re: managed hosting
by Jimmy McArthur 10 Dec '20

10 Dec '20

Hi - I've just requested a review to add Vexxhost Managed hosting, as well as some additional "Get Started" info. This would bypass the current redirect and go to a new section /get-started. Ideally, /start would redirect to /get-started once we all agree on language. https://review.opendev.org/c/zuul/zuul-website/+/766520 I just cobbled this together from a few sources, so please don't consider any verbiage finale. Please leave comments and thoughts in review. Thanks :) Jimmy

1 0

SQL reporters (how many is too many?)
by corvus＠inaugust.com 09 Dec '20

09 Dec '20

As you may recall, the v4/v5 plan is, briefly: * Make SQL connections required and default in every pipeline * release v4.0.0 * Make explicit SQL use in pipelines an error * release v4.1.0 * Do more HA stuff * release v5.0.0 See [1] for the plan in more detail. While reviewing the patch for the first step[2], I wondered if we want to think about changing the config file syntax for SQL connections. Currently the plan is to keep the existing style: [connection mydatabase] driver=sql dburi=... I'm wondering if we should change that to something more like: [database] dburi=... Essentially the question at hand is: is there a case for supporting more than one SQL connection? This will be true no matter which method we choose: * Per-pipeline sql connections are out. At the very least, a given tenant needs exactly one SQL connection in order for the web and other newer features to work. * Seamless upgrades will be supported through the v4.0/4.1 release process (there will be at least a release where both syntaxes are supported). A brief case for "[database]": * It is simpler to have a single database section in the config file. * Zuul already supports complete tenant isolation in its API and db schema, so there should be no need to separate data between different tenants via different connections. Truly paranoid users can run a separate Zuul for different tenants. * SQL databases scale really well, with a little bit of attention to our indexes and queries, we shouldn't need multiple connections for scalability purposes. A brief case for "[connection mydatabase]": * It's not difficult for us to assign a different SQL connection to a tenant. If we find a need to shard data for scalability reasons, this would be the best place to do it. Does anyone else have thoughts about this? Other reasons to prefer one over the other? Thanks, Jim [1] http://lists.zuul-ci.org/pipermail/zuul-discuss/2020-March/001158.html [2] https://review.opendev.org/630472

4 3

[Diversity] Diversity and Inclusion WG Meeting Change for January
by Amy Marrich 07 Dec '20

07 Dec '20

Due to the end of year holidays and the fact some people may still be out we will be moving our January meeting. As our backup meeting date is on Martin Luther King Day, the WG has selected January 11 to meet in IRC. As always I'll send out a reminder in the days leading up to the meeting. Thanks, Amy (spotz)

1 0

Steps to install Zuul ci using helm charts
by sitakanta hotta 04 Dec '20

04 Dec '20

Could you please refer to me if there are any zuul ci install steps using helm charts for kubernetes? Thanks Hotta

2 2

[Diversity] Diversity & Inclusion Meeting 12/7
by Amy Marrich 04 Dec '20

04 Dec '20

The Diversity & Inclusion WG invites members of all OSF projects to our next meeting Monday, December 7th, at 17:00 UTC in the #openstack-diversity channel. The agenda can be found at https://etherpad.openstack.org/p/diversity -wg-agenda. Please feel free to add any other topics you wish to discuss at the meeting. Thanks, Amy (spotz)

1 0