- Zuul-discuss - lists.opendev.org

Zuul scheduler Errors: 'NoneType' object has no attribute 'gauge'
by Sankar Palanivel 12 May '21

12 May '21

Hi Team, We are setting up zuul services on docker containers with openstack backed nodepool and external gerrit instance. When we try to push a change/comment in gerrit, we are getting below errors in scheduler container. It would be helpful if you can suggest a way to fix these errors. 2021-05-11 16:36:59,079 INFO zuul.GerritConnection: [e: 5bb47c4c919748158fd737fbc771f0ff] Updating <Change 0x7f3510395ca0 None 8658,2> 2021-05-11 16:36:59,162 INFO zuul.Pipeline.sample.check: [e: 5bb47c4c919748158fd737fbc771f0ff] Adding change <Change 0x7f3510395ca0 sandbox 8658,2> to queue <ChangeQueue check: sandbox> in <Pipeline check> 2021-05-11 16:36:59,162 INFO zuul.Pipeline.sample.check: Reporting enqueue, action [] item <QueueItem 99080e4bac5f4b36bb83feee1dec557c for <Change 0x7f3510395ca0 sandbox 8658,2> in check> 2021-05-11 16:36:59,569 INFO zuul.MergeClient: [e: 5bb47c4c919748158fd737fbc771f0ff] Merge <gear.Job 0x7f3510326a60 handle: b'H:172.19.0.7:13' name: merger:merge unique: 659107f2f6624a7680be7ac9c4f11956> complete, merged: True, updated: False, commit: d525dc035578c078849cf2a3b324d653d1aef03f, branches: [] 2021-05-11 16:36:59,590 INFO zuul.Pipeline.sample.check: [e: 5bb47c4c919748158fd737fbc771f0ff] Scheduling global repo state for item <QueueItem 99080e4bac5f4b36bb83feee1dec557c for <Change 0x7f3510395ca0 sandbox 8658,2> in check> 2021-05-11 16:36:59,753 INFO zuul.MergeClient: [e: 5bb47c4c919748158fd737fbc771f0ff] Merge <gear.Job 0x7f3510311fd0 handle: b'H:172.19.0.7:14' name: merger:refstate unique: 86f384a2cb3f4181ab6769e6857bc27c> complete, merged: False, updated: True, commit: None, branches: False 2021-05-11 16:36:59,775 INFO zuul.nodepool: [e: 5bb47c4c919748158fd737fbc771f0ff] Submitted node request <NodeRequest 200-0000000006 <NodeSet [<Node None ('infra-build-ubuntu1804-base',):infra-build-ubuntu1804-base>]>> 2021-05-11 16:37:07,288 INFO zuul.nodepool: [e: 5bb47c4c919748158fd737fbc771f0ff] Node request <NodeRequest 200-0000000006 <NodeSet [<Node 0000000008 ('infra-build-ubuntu1804-base',):infra-build-ubuntu1804-base>]>> fulfilled 2021-05-11 16:37:07,298 INFO zuul.nodepool: [e: 5bb47c4c919748158fd737fbc771f0ff] Accepting node request <NodeRequest 200-0000000006 <NodeSet [<Node 0000000008 ('infra-build-ubuntu1804-base',):infra-build-ubuntu1804-base>]>> 2021-05-11 16:37:07,307 INFO zuul.Pipeline.sample.check: [e: 5bb47c4c919748158fd737fbc771f0ff] Completed node request <NodeRequest 200-0000000006 <NodeSet [<Node 0000000008 ('infra-build-ubuntu1804-base',):infra-build-ubuntu1804-base>]>> for job testjob of item <QueueItem 99080e4bac5f4b36bb83feee1dec557c for <Change 0x7f3510395ca0 sandbox 8658,2> in check> with nodes <NodeSet [<Node 0000000008 ('infra-build-ubuntu1804-base',):infra-build-ubuntu1804-base>]> 2021-05-11 16:37:07,318 INFO zuul.nodepool: Setting nodeset <NodeSet [<Node 0000000008 ('infra-build-ubuntu1804-base',):infra-build-ubuntu1804-base>]> in use 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: [e: 5bb47c4c919748158fd737fbc771f0ff] Exception while executing job testjob for change <Change 0x7f3510395ca0 sandbox 8658,2>: 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: Traceback (most recent call last): 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: File "/usr/local/lib/python3.8/site-packages/zuul/manager/__init__.py", line 639, in _executeJobs 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: self.sched.nodepool.useNodeSet( 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: File "/usr/local/lib/python3.8/site-packages/zuul/nodepool.py", line 253, in useNodeSet 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: self.emitStatsResources() 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: File "/usr/local/lib/python3.8/site-packages/zuul/nodepool.py", line 90, in emitStatsResources 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: self.statsd.gauge(key, value, tenant=tenant, resource=resource) 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: AttributeError: 'NoneType' object has no attribute 'gauge' Thanks, Sankar Palanivel

4 5

Zuul on openshift.tv
by James E. Blair 10 May '21

10 May '21

Hi, Just a quick note to let folks know I will be talking about Zuul live on http://openshift.tv this Thursday at 17:00 UTC. -Jim

1 0

Zuul deployment options
by Sankar Palanivel 07 May '21

07 May '21

Hi Team, We are planning to use zuul for some of our projects. Could you please help me out on the following queries? 1. Best way to deploy zuul in production is using QuickStart (using docker-compose) option or install from scratch option? 2. Based on the selection from question 1, how zuul can be scaled for future requirements? 3. I dont find resource requirements (memory/cpu/disk) for any of the zuul services. Please confirm which of the zuul services are (executor, scheduler, nodepool, zookeeper, web, logs, DB) resource intensive? 4. Is it okayto have zookeeper standalone instance for nodepool launcher with openstack provider? or it needs to have a zookeeper in cluster mode? Thanks, Sankar Palanivel

2 5

[Diversity] Diversity & Inclusion WG Meeting reminder
by Amy Marrich 30 Apr '21

30 Apr '21

The Diversity & Inclusion WG invites members of all OSF projects to our meeting Monday May 3rd, at 17:00 UTC in the #openinfra-diversity channel. The agenda can be found at https://etherpad.openstack.org/p/diversity -wg-agenda. Please feel free to add any topics you wish to discuss at the meeting. Thanks, Amy (spotz)

1 0

help to getting speculative merged testing fixed
by Åsmund Østvold 30 Apr '21

30 Apr '21

hi Probably a newbie mistake. I have: - job: name: build-integration-tests parent: base run: playbooks/build_and_integrationtest.yaml required-projects: - resoptima/irma - name: resoptima/tools override-checkout: main - zuul/irma-common-jobs - resoptima/irma.fileimporter In the gate pipeline I get parallel unrelated merged executions, see attached image. I am sure I did something wrong. Any help appreciated! Cheers, Åsmund [image: image.png]

2 2

questions about locally running zuul repo tests
by mikeyang＠andestech.com 27 Apr '21

27 Apr '21

Hi, As part of my eval, I'm looking at the dev process. And we're interested in a Gitea driver (it looks like there was a WIP change<726391:%20more%20work%20towards%20a%20working%20gitea%20driver%20|%20https:…> last year). When I try to run the tests locally with "tox", following the instructions in TESTING.rst, it looks like the Docker Zookeeper instance does not start up properly and the majority of tests fail with a timeout. I tried modifying "tox.ini" in order to open/map port 2181 and I can see that "tox" starts up the zookeeper container, but the tests still timeout. Looking at the code, I see a couple of issues: 1. hostname ... how does tox know the hostname/ip-address of the container (with default bridge networking) 2. certificates ... are these required and how is tox configuring this? Or am I on the wrong path? I see that tools/test-setup-docker.sh seems to have a different way of running the tests. Thanks, - mike. CONFIDENTIALITY NOTICE: This e-mail (and its attachments) may contain confidential and legally privileged information or information protected from disclosure. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein is strictly prohibited. In this case, please immediately notify the sender by return e-mail, delete the message (and any accompanying documents) and destroy all printed hard copies. Thank you for your cooperation. Copyright ANDES TECHNOLOGY CORPORATION - All Rights Reserved.

2 3

Newbie question(s) for eval
by mikeyang＠andestech.com 20 Apr '21

20 Apr '21

Hi, I’m doing an eval of Zuul-CI for my team. I’ve previously used Jenkins and internally/purpose-built CI systems. So Ansible is new to me. I’m using the quick-start tutorial so this is all in the Docker containers and images from the tutorial. I’m running this on a Ubuntu bare-metal machine. And please let me know if there is a more appropriate forum to ask this question or get eval support. So what I tried to do was use a simple job from zuul-jobs. I thought `markdownlint` would be straight-forward. Here’s the rabbit hole that I went down. 1. The quick-start tutorial is using `Ubuntu Focal` images for the `node` 2. I added a second `node` using `Ubuntu Groovy` 3. I can run `testjob` and a custom job (playbook with `shell`s) on the Groovy node 4. I tried to run the `markdownlint` Zuul job on the Groovy node but it fails on the `ensure-nodejs` role * It tries to add `https://deb.nodesource.com/node_6.x groovy main` as a repo * This fails because `dists/groovy` only exists in the 10/12/14/15.x repos * I tried many different ways to assign `node_version` to 15, but the job always fails trying to add `node_6.x` From the Ansible documentation, I see that there are many places in precedence to overwrite a role’s default value, but I have not gotten any of these to work. And I’m unclear about how the precedence extends into Zuul’s `job.vars` and `jobs.extra-vars` (are these equivalent to passing on Ansible’s command line?). Thanks, - mike. CONFIDENTIALITY NOTICE: This e-mail (and its attachments) may contain confidential and legally privileged information or information protected from disclosure. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein is strictly prohibited. In this case, please immediately notify the sender by return e-mail, delete the message (and any accompanying documents) and destroy all printed hard copies. Thank you for your cooperation. Copyright ANDES TECHNOLOGY CORPORATION - All Rights Reserved.

3 5

Nested repo structures
by James E. Blair 20 Apr '21

20 Apr '21

Hi, We were recently presented with a case of a user who uses Gerrit's support for arbitrary repo hierarchies in an unexpected way. Essentially, they have both of the following projects defined in gerrit: * project * project/subproject That is to say, project is not just a pathname, it is a full-blown Gerrit repo. Further, this has nothing to do with git submodules or anything. There are several potential issues in Zuul related to nested repos: * Stats * Secret key storage * Time database * Internal merger repos * Workspace checkouts Stats are good. We already flatten project names there (eg, translate / to _). The secret key storage we are about to move to ZK. We can handle that right now. The time database will be eliminated as part of the v5 work as well, and replaced with SQL. Not a problem. That leaves the git repos. The main issue is that git doesn't deal well with git repos that are cloned into other git repos. We need to keep these separate. The internal merger repos (that is, the ones that users never see) will need a flattened namespace. That should be relatively straightforward since we can take care of that without a user-visible impact. Finally, that leaves the subject of this mail: the user-visible workspace checkouts. Once we get Zuul to a point where it doesn't error-out on a nested repo structure, we still have to decide how to lay it down on disk in the executor workspace. I can see three options: 1) Attempt to lay the repos out in the current fashion, with project/subproject cloned on top of project. I do not think we should do this. While we might be able to do this in the limited context of a workspace checkout (as compared to the internal merger repos), it would still be fraught with problems if, for example, anyone ever did add a "subproject" directory to "project". 2) Allow users to map projects to arbitrary repo checkouts. For this we might add a new mapping to a job like this: - job: checkouts: project: project project/subproject: subproject This would allow users to write out the repos wherever it suits them, whether that's a flat structure as above, or something more complex. They could also move just a single troublesome repo. In fact, you'd really only need the second line of the map above, the first is redundant. 3) Allow users to indicate they want a flat checkout. Given that this is the first use-case for this which we have seen, it's instructive to learn how the users who prompted this discussion are actually using their repos. They want a flat checkout. So they essentially only want the last component of the path to appear in their workspace. This option is a specialized version of the above that's less flexible, but probably simpler to use for this case: - job: flat-checkout: true That would map each project to its last path component. We would need to decide what to do if *that* caused a collision. Presumably we would error out. In this case, the user would have no recourse with Zuul (they would need to change a repo name or stop using a repo in that job). Options 2 and 3 are not mutually exclusive. We could do both so that the typical use would be #3 with #2 as a backup (or for someone with really creative ideas about how to lay down repos). We could also implement one and leave the other as an idea for future expansion. I suspect the internal implementation will be about the same for both. I think I'm inclined to start with #3 just because it concisely enables a specific pattern that may be useful in other contexts. Does anyone have feedback or a preference? -Jim P.S. I suspect we're going to find cases in zuul-jobs where we have assumed project.src_dir == 'src/'+project.canonical_name, but we can and should fix those. We made that a variable for a reason.

4 4

Proposed behavior change on reconfiguration
by James E. Blair 18 Apr '21

18 Apr '21

Hi, We recently merged some changes which improve the consistency of the git repos in different jobs within the same buildset. While it's always the case that every job in a buildset has the same view of the projects involved in its shared queue, projects outside of that queue (including projects involved in providing Zuul roles, playbooks and job required-projects) have not been synchronized, meaning that if a change merged in one of those projects between two jobs in a buildset starting, the earlier job would not see the change and the later job would. It seems more natural for Zuul to fix the state of all of the repos involved in all of the jobs of a buildset, and so the change that we recently merged did this. While this seems like a straightforward change, it had an unanticipated effect on reconfiguration. When Zuul performs a reconfiguration, any jobs that are added to a project are added to currently running buildsets and started; likewise, jobs may be removed from buildsets. Any jobs which are modified but have already run will retain their previous results, but jobs which have not yet run will do so with the new configuration. It turns out that the idea of fixing all of the repos involved in a buildset and also being able to dynamically add, remove, and alter jobs are fundamentally at odds. The simplest example of this is a reconfiguration that adds a playbook to the zuul-jobs repo, while a job on an unrelated repo is running If Zuul has frozen the state of all the repos involved (including zuul-jobs) then the updated version of the job will reference a playbook which will not appear in the previously frozen repo state. In other words, we can freeze both the configuration and the repos, or neither. Earlier in Zuul's development, we felt that the ability to add, remove, or alter jobs while they were running was fairly important. However, Zuul and its usage has evolved considerably since then. I'd like to propose that we continue down the path of freezing both the repo state and the job configuration within a buildset. This means than in the future, if a job is added, removed, or altered during a reconfiguration, it will not take effect for any currently running buildsets, but will take effect for any buildsets started after the reconfiguration (that includes new buildsets created on gate resets). Given the potentially significant change to a long-standing Zuul behavior, I ask that anyone relying on this or who feels that they may be affected by this to please let us know. Thanks, Jim

2 4

[Diversity] D&I WG Office Hours at the PTG
by Amy Marrich 16 Apr '21

16 Apr '21

The Diversity and Inclusion WG has grabbed our usual PTG slot Monday at 16:00UTC and we will be holding office hours to help any project with Inclusive Naming or other D&I related topics. If you'd like us to attend any of your sessions please let me know. If you've been interested in learning more about the WG please come join us! Thanks, Amy (spotz)

1 0

The error was: 'zuul_ssh_key_algorithm' is undefined
by Åsmund Østvold 31 Mar '21

31 Mar '21

Hi, Today a job that has not been changed for a long time( 1000+ executions) failed in a way I can not explain. The below snippet is from the "cleanup" after normal test execution that succeeded. Everything before and after looks to me to not have any problems. [test script output end] 2021-03-29 16:50:00.729468 | ubuntu-bionic | ok: Runtime: 0:02:02.256368 2021-03-29 16:50:00.751939 | 2021-03-29 16:50:00.752085 | PLAY RECAP 2021-03-29 16:50:00.752153 | ubuntu-bionic | ok: 2 changed: 2 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0 2021-03-29 16:50:00.752192 | 2021-03-29 16:50:00.992482 | RUN END RESULT_NORMAL: [trusted : review.resoptima.net/zuul/irma-common-jobs/playbooks/build_and_integrationt…] 2021-03-29 16:50:00.993174 | POST-RUN START: [trusted : review.resoptima.net/zuul/irma_config/playbooks/base/post-ssh.yaml@master] 2021-03-29 16:50:02.365604 | 2021-03-29 16:50:02.365802 | PLAY [all] 2021-03-29 16:50:02.400284 | 2021-03-29 16:50:02.400472 | TASK [remove-build-sshkey : Remove the build SSH key from all nodes] 2021-03-29 16:50:02.591665 | ubuntu-bionic | ERROR 2021-03-29 16:50:02.592037 | ubuntu-bionic | { 2021-03-29 16:50:02.592094 | ubuntu-bionic | "msg": "The task includes an option with an undefined variable. The error was: 'zuul_ssh_key_algorithm' is undefined\n\nThe error appears to be in '/var/lib/zuul/builds/e01e1c4faa834c079a62b6514ed2106f/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/remove-build-sshkey/tasks/main.yaml': line 1, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Remove the build SSH key from all nodes\n ^ here\n" 2021-03-29 16:50:02.592133 | ubuntu-bionic | } 2021-03-29 16:50:02.626484 | 2021-03-29 16:50:02.626654 | PLAY RECAP 2021-03-29 16:50:02.626737 | ubuntu-bionic | ok: 0 changed: 0 unreachable: 0 failed: 1 skipped: 0 rescued: 0 ignored: 0 [normal termination of job] Is this a known issue? As it is < 1/1000 probability of this happening it is not an important issue for me but wanted to check with you anyway. Regards, Åsmund

2 2

[marketing] Zuul 4 & 5 Press Release
by Jimmy McArthur 22 Mar '21

22 Mar '21

Hi Zuul Team - We're gearing up to start promoting the Zuul 4 and 5 release. I know we don't have a definitive timeline yet for the 5 release, but it seems clear that the Zuul community knows what features will land. Is there anyone in the community that could volunteer to put together a blogpost (location tbd - maybe openinfra.dev/blog or superuser). From that, we can extract features for promo and obviously push the blog out to our various marketing channels. Cheers, Jimmy

2 2

Getting a #403 Resource not accessible by integration"
by Cyril Lavier 19 Mar '21

19 Mar '21

Hi. I'm trying to setup a Zuul POC with Github, with 2 projects (one containing the jobs config/playbook and one being an actual project with a simple pipeline with a jsonlint job, to test). My Zuul is connected to Github, it can grab the repositories and receive calls from Github when creating a PR, but Zuul sends this message to the PR : ------------------------------------ Starting check jobs. Warning: Failed to update check run vanilla/check: 403 Resource not accessible by integration ------------------------------------ "vanilla" being the tenant name. In the Zuul WEB UI, the job is linked to the repository but it's still in "queued". I've checked the Github permissions and I don't find anything different than in this howto (https://zuul-ci.org/docs/zuul/howtos/github_setup.html) Also, in the meantime, no nodes are displayed in the "Nodes" section of the webUI, but I declared a static one (with the prerequisites from the doc) in the nodepool config. So I have 2 questions : - Why do I get this 403 error ?, or maybe a way to help me debug (debug logs don't help me a lot) - Why is there no nodes in the "Nodes" section of the webUI ? I assume the job being queued indefinitely may come from either the node not appearing or the 403 error. Thanks.

3 5

Type annotations
by jim＠acmegating.com 17 Mar '21

17 Mar '21

Hi, I'd like to propose that we remove type annotations and checking from our code. I don't think they are providing much value, but they come at a significant cost. They have very rarely (I think perhaps once or twice) caught errors. Meanwhile, we aren't going to relax our test coverage requirements, which should catch most things anyway. On the other hand, they cause the code to be much less readable and I believe that will ultimately lead to more errors. Developer and reviewer fatigue are important things to keep in mind. Some examples where I think annotations are particularly unhelpful: self.on_connect_listeners: List[Callable[[], None]] = [] Or (note the duplication): def __init__(self, ... info: Optional[zuul.model.WebInfo] = None): self.info: Optional[zuul.model.WebInfo] = info We've also had some occurrences where we ended up fighting the annotation system. I've mentioned to several of you that I do very much enjoy the benefits of statically typed languages (I write C code for fun), but I don't think that the type hints in Python are very well executed. Many of us chose to use Python instead of Java specifically for the dynamic typing and as a reaction against the extensive type requirements. The type hinting that is being added to Zuul seems like it carries the costs of Java without the benefits. We started this as a small experiment. We said that we would run the type checker in a mode where it would operate on implicit type information, thereby minimizing the additional cognitive load on authors and reviewers. We then started adding some minimal type information in strategic places to help it out. However, I'm now reviewing changes that are literally 3x larger than they otherwise would be because of all the additional repetitive type information going into them. I'm glad to try experiments like these, but I think this one has grown to the point where we need to decide if we want to go all-in on type annotations or remove them. I feel fairly strongly that the cost to authors and reviewers far outweighs the benefits, and we should make everyone's lives easier by removing them. -Jim

10 18

Dropping gerrit replication trigger event
by Tristan Cacqueray 16 Mar '21

16 Mar '21

Hello folks, After upgrading to Zuul version 4 and Gerrit version 3.2, our scheduler logs are flooded with ref-replication events (about 1GB of debug per hour). It seems like the gerrit driver could be improved to ignore unused events, and the queue processor could be less verbose to reduce the amount of noise. On the other hand, the replication event doesn't seems to be that useful and here is a change to ignore them: https://review.opendev.org/c/zuul/zuul/+/780809 If you do rely on replication event, please raise your voice in the review above. Thanks, -Tristan

2 1

vscode/codium editor for zuul config files
by Sorin Sbarnea 01 Mar '21

01 Mar '21

In case you happen to use vscode or codium, you can now use an extension that verified the syntax of the configuration files and provide auto-complete. Ansible Language extension v0.2.0 adds support for Zuul config files. https://marketplace.visualstudio.com/items?itemName=zbr.vscode-ansible&ssr=… https://open-vsx.org/extension/zbr/vscode-ansible If you like it useful, feel tree to leave a review. PS. Nope, I it does not work with vim, emacs, pycharm, ... Cheers Sorin

1 0

[all] April 2021 PTG Dates & Registration
by Kendall Nelson 01 Mar '21

01 Mar '21

Hello Everyone! I'm sure you all have been anxiously awaiting the announcement of the dates for the next virtual PTG[1]! The PTG will take place April 19-23, 2021! PTG registration is now open[2]. Like last time, it is free, but we will again be using it to communicate details about the event (schedules, passwords, etc), so please take two minutes to register! Early next week we will send out info to mailing lists about signing up teams. Also, the same as last time, we will have an ethercalc signup and a survey to gather some other data about your team. Can't wait to see you all there! -the Kendalls (diablo_rojo & wendallkaters) [1] https://www.openstack.org/ptg/ [2] https://april2021-ptg.eventbrite.com

1 0

Last call for 4.0 deprecations
by jim＠acmegating.com 15 Feb '21

15 Feb '21

Hi, We've merged the SQL database requirement, and as I write this, the ZooKeeper connection requirement patches are working their way through the gate. Those are the last big items blocking the 4.0 release. Before we issue the release, I'd like to ask folks to take a look for anything we've said that we would like to deprecate or remove in a major version bump to make sure we don't miss the opportunity. At this point, I don't think we should entertain last-minute new ideas, but rather, if we put something in the documentation quite some time ago saying that it would be removed in the future, let's make sure we take the opportunity to consider that now. Also, if anyone can think of any other blockers for 4.0, speak now please. :) Otherwise, I think we can cut the release very soon. -Jim

1 1

Scale out scheduler
by jim＠acmegating.com 13 Feb '21

13 Feb '21

Hi, The main blocker for releasing 4.0 at this point is merging the changes that implement the ZooKeeper connection requirement from every Zuul component. Currently the changes that implement those also include some extra changes that refactor the ZooKeeper connection classes. I think that's a good idea, but I'd like to separate concerns and focus on getting the requirements for the 4.0 release taken care of first. On Friday I discussed with some folks an approach to that, and today I finished up a rebase + rework of the series. I uploaded three new versions of these patches: https://review.opendev.org/775459 Mandatory Zookeeper connection for ZuulWeb https://review.opendev.org/775460 Connect executor to Zookeeper https://review.opendev.org/775461 Connect merger to Zookeeper I think they represent the minimum we need to get into 4.0. There is some boilerplate related to ZK connections in each service, but it's easy to reason about for now. I do think we should refactor the ZK connection class, but let's do that on top of these patches. The results will be clear and easy to understand. The original patches also increased the unit test job timeout because the jobs took approximately 25% longer to run. I think perhaps reviewers were under the impression that was necessary due to the new ZooKeeper connections in unit tests. I investigated the issue and found that was not the case. The actual cause was due to extra import statements that were added to support the addition of type annotations. After removing those type annotations and associated import statements, the test runtime remains the same as it currently is. You can read more details about it in the comments of patchset 2 of the last change. It's also worth keeping in mind that extraneous import statements can cause not only unanticipated problems like this, but can often exacerbate the problem of import loops, something we have seen occasionally in Zuul. Keeping additional imports for type annotations to a minimum will be beneficial. I propose the following next steps: 1) Re-propose 775459-775460 as new patchsets on their original changes in order to preserve the review history there. 2) Merge those changes and continue work on releasing 4.0. 3) Meanwhile: refactor the ZooKeeper connection class. 4) Continue work on the remaining scale-out scheduler changes. -Jim

1 0

Nodepool image building
by jim＠acmegating.com 10 Feb '21

10 Feb '21

Hi, I'd like to propose some new features related to Nodepool and image building. The short version is that right now image management is tied fairly closely to diskimage-builder within the nodepool-builder process. This is problematic for folks who want Nodepool image management but can't use DIB (ie, Windows images). It's even problematic for those who do want to use DIB, in that it's difficult to debug image builds. It's worth noting the original Zuulv3 spec had room for more than one nodepool-builder implementation (in an offhand comment, it mentioned that a snapshot-based nodepool-builder process could be created). However, that has not yet come to pass. Over the years, we've speculated about the idea of using Zuul to build images, and I think it's time to see if we can put that into practice. It makes a lot of sense in that we have a ready-made execution environment with Zuul -- build logs and history are well defined and accessible to anyone. It's flexible and can accommodate any image building system (DIB or otherwise). The main downside is the inception aspect, but I'm not worried about that (more later). I think the key requirements are: * Able to create images using any tooling. * Visibility into build history. * Nodepool-launcher should automatically use the "latest" build. Part of this solution is obvious, but I think there are open questions for other parts: * Should the Zuul job be responsible for uploading the image to the cloud? * Should Nodepool be responsible for deleting images? About that downside: clearly there's an issue when bootstrapping a new Zuul system -- how can you run jobs without images? Most new Zuul installations start by using cloud images from the provider. Any Zuul user who wants custom images likely already has some kind of job running on some kind of image anyway. I think it's a natural progression to create a Zuul job to build a custom image and then begin using that custom image in "real" Zuul jobs. Long story short: bootstrap the system with cloud-images. As an OpenDev admin, I think I would be comfortable with a Zuul job that built our DIB images using cloud provider images -- I don't think the differences between those images are so great as to make a DIB job difficult to write. However, if we did use this system, I'm not sure we would ever have to. We have enough clouds with our own images in them already, we could probably use our images to make new ones. In this case: a chicken with no egg. :) Let's work through how this might look. First the easy part, then I'm going to propose some alternate solutions. 1) Build the image: A Zuul job ("build-nodepool-image-ubuntu-focal") is triggered by a timer. It has a mutex to avoid duplicate image builds. It runs DIB in this case but could just as easily run packer or even snapshot the system it's running on (we can add roles in zuul-jobs for common image building systems). 2) Get the image into the cloud(s): a) Our first option is to do this work in Zuul. Have the job (or a dependent job) upload the image to the cloud(s) (if this was a snapshot build, that's a no-op -- it's already there). Advantage: the upload log is visible along with the build log. Disadvantage: if the upload fails, we won't retry like Nodepool does. b) An alternative would be to have Nodepool do this. Nodepool could even query the Zuul API to find the latest build of build-nodepool-image-ubuntu-focal, check the artifacts list, and download the image file from there. After that, it looks a lot like the current nodepool-builder process right after DIB runs (it has a file on disk and it starts to upload it). Advantage: we use all of the Nodepool image lifecycle management. Disadvantage: upload logs are hidden from users (like today). to integrate snapshot builds, we would have to download the snapshot and re-upload it. 3) Get nodepool-launcher to use the new image: a) If we use 2.a, then we have an image in the cloud without a record in Zookeeper. i) We could have nodepool-builder (or something like it) scan the cloud for images matching a certain pattern and create ZK records for them. ii) Or we could just have nodepool-launcher look for that pattern and use the latest image. b) If we use 2b, then launcher will work as it does today with nodepool-builder; no changes are needed. 4) Delete the image when it's no longer needed We should use the n-2 rule we have today so there's always at least 2 images in the cloud for easy rollback. a) If we used 2.a + 3.a.iii then we don't have ZK records. It would probably make sense to handle image deletes in Zuul jobs with this setup. Presumably the jobs would query the cloud API, list images, and delete them according to the age rules. The disadvantage here is we'd be re-implementing a good chuck of nodepool-builder in Ansible modules. It's doable, but may be a bit clunky. b) If we use 2.a + 3.a.i or 2.b + 3.b then we have records in ZK, so nodepool-builder (or something like it) can delete the images more or less how it does today. I think those are most of the choices. Now that I've typed all of that out, I think I really like the system with "b" for all the choices. I'm not too worried about the upload logs being hidden. And the snapshot system is workable, if not ideal. If we really wanted to support snapshots better, then 2.a + 3.a.i + 4.b would probably be best, but at the cost of moving the upload code into zuul-jobs roles. Or we could do the same thing we did last time and implement the "b" solution ("nodepool-zuul-builder") and if someone really wants to do snapshots, then we implement 2.a[snapshot] + 3.a.i + 4.b as ("nodepool-snapshot-builder"). That would probably be fairly efficient as far as code goes. Finally, a word about DIB and the current nodepool-builder: none of this requires that we change anything about the current nodepool-builder process. If we like it, we can keep it exactly as it is. Indeed, in OpenDev's case, we rely very heavily on the cache on the build host, so it's not at all clear that doing builds on throwaway nodes is ideal. Of course, we could do builds on static Zuul nodes and get the benefit of the cache. I think the way to approach this would be to implement the new system, then try using it with DIB, and if we love it and find no significant drawbacks, consider deprecating the current nodepool-builder. If we don't love it, we'll keep it. I'd appreciate feedback on this. I know this is a bit free-form; I'll take the feedback I get and synthesize it into a proper spec. -Jim PS: there's a fairly well-known hack for making non-DIB image builds by replacing DIB with a script that accepts the same arguments. The "b" plan could actually be implemented today with no changes to Nodepool using that same hack. A final implementation probably wouldn't look much different structurally.

5 6

job.provides/requires for periodic pipeline
by Vitaliy Lotorev 10 Feb '21

10 Feb '21

Hi, I have a jobA for project A (job builds images) and jobB for project B (job runs tests for those images). I set both jobs into a nightly periodic pipeline. periodic pipeline has jitter ±5 minutes. jobA and jobB run in pipeline asynchronously. jobB doesn't wait till jobA finishes and fetches latest successful build from jobA which appears to be built 24 hours ago (i.e. jobB tests in images built 24 hours ago). Questions: * Would setting jobA.provides=images and jobB.requires=images order them during one run? I.e both jobA and jobB would enqueue and jobB would wait while jobA finishes. Or should I use job.dependencies? * What are other assumptions for using requires/provides: ** Should be jobs in the same pipeline? ** if job has 'requires' how the job can be debugged separately without running jobs that define 'provides'? -- Regards, Vitaliy

2 3

Type annotations
by Jan.Kubovy＠bmw.de 09 Feb '21

09 Feb '21

…. > I'm not [yet] an expert on type annotations in Python, so I may have to > rely on others who know more details on what we can and should do in the > future, but here are some guidelines that might be a good starting > point: > > * Let's not bend over backwards to make mypy happy. That's not our > goal, so if something is tripping it up but is otherwise not ambiguous > to humans or IDEs, do the minimum work to ignore it. > > * Corollary: do we actually need/want to keep running mypy > automatically in our Zuul jobs? > > * Let's focus on improving interfaces, but not exhaustively typing every > variable. > > * If there are improvements coming in the future (like 3.9) let's do > less now and wait for them. > > And based on those, what do you think about these specific guidelines: > > * Do type method signatures. This is always a good idea without it most of the checking will not work. > * Don't recurse ("List[Callable]", not "List[Callable[[], None]]"). I would not make that a hard rule, more a recommendation. If there Is a benefit in certain cases I would allow it. One can also use type aliases and type vars to make it better usable and look less awkward. > * Avoid typing variable assignment inside of methods. (They should > mostly be inferred, and they do little to improve code comprehension.) If this works reliably, for everyone using it (not sure if this will not break for example usage search). Otherwise I don’t see as much drawback having a list of typed assignments in the init(). But I guess this Is a matter of taste. Jan _______________________________________________ Zuul-discuss mailing list Zuul-discuss(a)lists.zuul-ci.org<mailto:Zuul-discuss@lists.zuul-ci.org> http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-discuss

2 1

Type annotations
by Jan.Kubovy＠bmw.de 08 Feb '21

08 Feb '21

Hi all, my 0.02$ to the discussion: Although I am currently not an active contributor nor a reviewer a year ago I started the work on scale-out-scheduler as a part of my day job. This lead me to introduce the typings in parts of old code and use it in new code. As zuul is a community project I do believe that it relies not only on core developers, who are involved over long period of time but should also be code-friendly to contributors, who want to or are tasked to one-time, short-period jobs. While I do like dynamic languages and understand their advantages, I worked with e.g. with Ruby a decade and half ago and I work with python in other projects too, I do like more an as-complex-as-necessary and as-simple- and as-readable-as-possible code base. IMHO it always depends on a context: A small POC would probably never use type annotation, a huge code base like zuul should IMHO consider it. I do agree that typings add a certain degree verbosity (I was an early adopter of Kotlin - one of the reasons was reduced verbosity, optional type defs) but they also come with certain benefits - better code navigation, support when refactoring, IDE support, etc. Also, I do not understand use of type annotations as an attempt to make python statically typed. The issues/reasons which lead me to introduce them: - Ambiguous parameter/argument names E.g. project, tenant, job, to name a few. Sometimes a name, sometimes an object, sometimes an Ansible Job, sometimes a zuul job or build. Makes it hard to navigate in the code. Since some of those names are too generic it becomes a challenge when working on the heart of zuul. Probably not so much when dealing with small changes. Just to be clear I am not talking about cases here where one may intentionally want to pass an "project" argument which may be both, a project name and a project object, and the code can deal with both. Those occasions barely exist here. Usually it is one or the other. - NPE, missing property Setters called right after instantiating an object. Making the argument basically optional. Since scale-out-scheduler touches a lot of code I encountered more than once that something which supposed not to be none was after a bit of refactoring. Just by navigating in the code was not always easy to see where a certain argument is set. E.g. try to find usages of attribute job, project, etc and where those are set for certain object. You will get every class containing that attribute - basically, reducing it to a text search result. This (https://review.opendev.org/c/zuul/zuul/+/757149/14/zuul/scheduler.py) is an example where one can see a simplification of code and reducing the risk of runtime error where typings were involved. Tests are great but the theoretical case that someone may set some attribute to None (a new change or config case) may not be covered (I guess zuul tests do not cover 100% branches/complexity). While it is not bulletproof typings do help here. - Code Navigation IDE support, usage search, etc. - API changes and refactoring Was, and I believe still is, very much needed especially in big tasks like scale-out-scheduler. - Code documentation This goes a bit against the verbosity argument. When the type is actually documented in the comment section making it in addition prone to inconsistency. Type systems additionally help me to prevent some errors and make me thing time to time if my concepts, implementation is correct. I did make myself as the first guinea pig to test the efforts vs. benefit argument in zuul and it did serve me the purpose. After more colleagues were involved in that topic, we had the same discussion and came eventually to the same result. I understand there is an effort needed, but I also believe that the benefits outweigh the cost dramatically, especially in long-term and when considering a need-to usage. That said, I never thought about a 100% typing through the code base. For example, some classes, e.g. zuul.cmd.Scheduler may not benefit from typings as much, classes in zuul.model definitely would, due to their reuse thought-out the code. For the juniority argument I strongly believe that also experienced developers may find this beneficial especially in cases described above (one-time, short-term involvement). Unless there is, of course, an argument against such contributions (I am not aware of). Since, typings are now part of python and evolving over the past few versions, it does not seem to go away any time soon. From what I can see more and more pyhon code uses them. I do not agree that using typings is a sign of choosing the wrong language here. There may be use-cases where not using them may be more beneficial (fast prototyping, etc), and some use-cases where using them will benefit the project. Also using typings does not prevent loose/dynamic typing (e.g. Union or just not using them everywhere if it makes sense). It would be pity not to use the chance of a big change like scale-out-scheduler/zuul 4.x is to improve the code just to find ourselves few years down the road regretting it. The trend suggests that having the option of type hinting (while keeping dynamic types) is the way to go since it kind of offers the benefits of both worlds - other languages e.g. TypeScript, Kotlin JS, to name a few. Since, I felt this will be a controversial change (changes always are), I tried to keep the typings in a separate change in gerrit were ever I could - so you can easily remove them by abandoning those changes. They served me well and should I get another task in zuul I would like to find them again also in other parts of the code. Cheers Jan _______________________________________________ Zuul-discuss mailing list Zuul-discuss(a)lists.zuul-ci.org<mailto:Zuul-discuss@lists.zuul-ci.org> http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-discuss

1 0

Making gerrit zuul result comments more readable
by Sorin Sbarnea 02 Feb '21

02 Feb '21

At this moment the comments added by zuul on gerrit can be very hard to read, partially due to gerrit limitations I have projects where I cannot even fit them in a single screen! -- each job spans **3** rows instead of a single one. https://sbarnea.com/ss/Screen-Shot-2021-02-02-09-04-45.40.png While my hopes on adding markdown support on gerrit comments <https://bugs.chromium.org/p/gerrit/issues/detail?id=9299> are low, as nobody from google even did bother to get involved. (please comment/star ticket if you want it) I know that we plan to integrate the results in the UI and avoid comments, but that is long temp plan which does not fix the issue for email notifications at all. Here are some measures that we could do to lower the amount of wasted space: a) Put an emoji stating the result as the first item in the list, which should make it easy to spot the failed ones ❌ foo ... ✅ bar ... b) sort jobs alphabetically c) remove seconds from duration, they are useless even for the fastest jobs d) use shortened build-id for URLs, 5 chars should provide more than enough entropy If we do all of these we should make it fit on a single line, even for longer job names. Still, every single one would add visible benefits. What do you think? I also considered other alternatives like listing failed at top and non-voting at bottom, so the order will be failed, passed, non-voting, but I can see how this may struggle to be accepted. The root cause of this formatting issue is this style: .message { --gr-formatted-text-prose-max-width: 80ch; } Sadly due to the use of shadow-dom, I can no longer use css styling to override the stylesheet. -- /sorin

2 1

[Diversity] Diversity &Inclusion WG Meeting reminder Amy Marrich <amy@demarco.com> Mon, Jan 11, 7:49 AM to foundation, openstack-discuss, kata-dev, starlingx-discuss, airship-discuss, zuul-discuss The Diversity & Inclusion WG invites members of all OSF projects to our meeting today, at 17:00 UTC in the #openstack-diversity channel. The agenda can be found at https://etherpad.openstack.org/p/diversity-wg-agenda. Please feel free to add any other topics you wish to discuss at the meeting. Thanks, Amy (spotz)
by Amy Marrich 31 Jan '21

31 Jan '21

The Diversity & Inclusion WG invites members of all OSF projects to our meeting tomorrow, at 17:00 UTC in the #openstack-diversity channel. The agenda can be found at https://etherpad.openstack.org/p/diversity-wg-agenda. Please feel free to add any other topics you wish to discuss at the meeting. Thanks, Amy (spotz)

1 0