- Zuul-discuss - lists.opendev.org

Zuul scheduler Errors: 'NoneType' object has no attribute 'gauge'
by Sankar Palanivel 12 May '21

12 May '21

Hi Team, We are setting up zuul services on docker containers with openstack backed nodepool and external gerrit instance. When we try to push a change/comment in gerrit, we are getting below errors in scheduler container. It would be helpful if you can suggest a way to fix these errors. 2021-05-11 16:36:59,079 INFO zuul.GerritConnection: [e: 5bb47c4c919748158fd737fbc771f0ff] Updating <Change 0x7f3510395ca0 None 8658,2> 2021-05-11 16:36:59,162 INFO zuul.Pipeline.sample.check: [e: 5bb47c4c919748158fd737fbc771f0ff] Adding change <Change 0x7f3510395ca0 sandbox 8658,2> to queue <ChangeQueue check: sandbox> in <Pipeline check> 2021-05-11 16:36:59,162 INFO zuul.Pipeline.sample.check: Reporting enqueue, action [] item <QueueItem 99080e4bac5f4b36bb83feee1dec557c for <Change 0x7f3510395ca0 sandbox 8658,2> in check> 2021-05-11 16:36:59,569 INFO zuul.MergeClient: [e: 5bb47c4c919748158fd737fbc771f0ff] Merge <gear.Job 0x7f3510326a60 handle: b'H:172.19.0.7:13' name: merger:merge unique: 659107f2f6624a7680be7ac9c4f11956> complete, merged: True, updated: False, commit: d525dc035578c078849cf2a3b324d653d1aef03f, branches: [] 2021-05-11 16:36:59,590 INFO zuul.Pipeline.sample.check: [e: 5bb47c4c919748158fd737fbc771f0ff] Scheduling global repo state for item <QueueItem 99080e4bac5f4b36bb83feee1dec557c for <Change 0x7f3510395ca0 sandbox 8658,2> in check> 2021-05-11 16:36:59,753 INFO zuul.MergeClient: [e: 5bb47c4c919748158fd737fbc771f0ff] Merge <gear.Job 0x7f3510311fd0 handle: b'H:172.19.0.7:14' name: merger:refstate unique: 86f384a2cb3f4181ab6769e6857bc27c> complete, merged: False, updated: True, commit: None, branches: False 2021-05-11 16:36:59,775 INFO zuul.nodepool: [e: 5bb47c4c919748158fd737fbc771f0ff] Submitted node request <NodeRequest 200-0000000006 <NodeSet [<Node None ('infra-build-ubuntu1804-base',):infra-build-ubuntu1804-base>]>> 2021-05-11 16:37:07,288 INFO zuul.nodepool: [e: 5bb47c4c919748158fd737fbc771f0ff] Node request <NodeRequest 200-0000000006 <NodeSet [<Node 0000000008 ('infra-build-ubuntu1804-base',):infra-build-ubuntu1804-base>]>> fulfilled 2021-05-11 16:37:07,298 INFO zuul.nodepool: [e: 5bb47c4c919748158fd737fbc771f0ff] Accepting node request <NodeRequest 200-0000000006 <NodeSet [<Node 0000000008 ('infra-build-ubuntu1804-base',):infra-build-ubuntu1804-base>]>> 2021-05-11 16:37:07,307 INFO zuul.Pipeline.sample.check: [e: 5bb47c4c919748158fd737fbc771f0ff] Completed node request <NodeRequest 200-0000000006 <NodeSet [<Node 0000000008 ('infra-build-ubuntu1804-base',):infra-build-ubuntu1804-base>]>> for job testjob of item <QueueItem 99080e4bac5f4b36bb83feee1dec557c for <Change 0x7f3510395ca0 sandbox 8658,2> in check> with nodes <NodeSet [<Node 0000000008 ('infra-build-ubuntu1804-base',):infra-build-ubuntu1804-base>]> 2021-05-11 16:37:07,318 INFO zuul.nodepool: Setting nodeset <NodeSet [<Node 0000000008 ('infra-build-ubuntu1804-base',):infra-build-ubuntu1804-base>]> in use 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: [e: 5bb47c4c919748158fd737fbc771f0ff] Exception while executing job testjob for change <Change 0x7f3510395ca0 sandbox 8658,2>: 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: Traceback (most recent call last): 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: File "/usr/local/lib/python3.8/site-packages/zuul/manager/__init__.py", line 639, in _executeJobs 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: self.sched.nodepool.useNodeSet( 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: File "/usr/local/lib/python3.8/site-packages/zuul/nodepool.py", line 253, in useNodeSet 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: self.emitStatsResources() 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: File "/usr/local/lib/python3.8/site-packages/zuul/nodepool.py", line 90, in emitStatsResources 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: self.statsd.gauge(key, value, tenant=tenant, resource=resource) 2021-05-11 16:37:07,322 ERROR zuul.Pipeline.sample.check: AttributeError: 'NoneType' object has no attribute 'gauge' Thanks, Sankar Palanivel

4 5

Zuul on openshift.tv
by James E. Blair 10 May '21

10 May '21

Hi, Just a quick note to let folks know I will be talking about Zuul live on http://openshift.tv this Thursday at 17:00 UTC. -Jim

1 0

Zuul deployment options
by Sankar Palanivel 07 May '21

07 May '21

Hi Team, We are planning to use zuul for some of our projects. Could you please help me out on the following queries? 1. Best way to deploy zuul in production is using QuickStart (using docker-compose) option or install from scratch option? 2. Based on the selection from question 1, how zuul can be scaled for future requirements? 3. I dont find resource requirements (memory/cpu/disk) for any of the zuul services. Please confirm which of the zuul services are (executor, scheduler, nodepool, zookeeper, web, logs, DB) resource intensive? 4. Is it okayto have zookeeper standalone instance for nodepool launcher with openstack provider? or it needs to have a zookeeper in cluster mode? Thanks, Sankar Palanivel

2 5

[Diversity] Diversity & Inclusion WG Meeting reminder
by Amy Marrich 30 Apr '21

30 Apr '21

The Diversity & Inclusion WG invites members of all OSF projects to our meeting Monday May 3rd, at 17:00 UTC in the #openinfra-diversity channel. The agenda can be found at https://etherpad.openstack.org/p/diversity -wg-agenda. Please feel free to add any topics you wish to discuss at the meeting. Thanks, Amy (spotz)

1 0

help to getting speculative merged testing fixed
by Åsmund Østvold 30 Apr '21

30 Apr '21

hi Probably a newbie mistake. I have: - job: name: build-integration-tests parent: base run: playbooks/build_and_integrationtest.yaml required-projects: - resoptima/irma - name: resoptima/tools override-checkout: main - zuul/irma-common-jobs - resoptima/irma.fileimporter In the gate pipeline I get parallel unrelated merged executions, see attached image. I am sure I did something wrong. Any help appreciated! Cheers, Åsmund [image: image.png]

2 2

questions about locally running zuul repo tests
by mikeyang＠andestech.com 27 Apr '21

27 Apr '21

Hi, As part of my eval, I'm looking at the dev process. And we're interested in a Gitea driver (it looks like there was a WIP change<726391:%20more%20work%20towards%20a%20working%20gitea%20driver%20|%20https:…> last year). When I try to run the tests locally with "tox", following the instructions in TESTING.rst, it looks like the Docker Zookeeper instance does not start up properly and the majority of tests fail with a timeout. I tried modifying "tox.ini" in order to open/map port 2181 and I can see that "tox" starts up the zookeeper container, but the tests still timeout. Looking at the code, I see a couple of issues: 1. hostname ... how does tox know the hostname/ip-address of the container (with default bridge networking) 2. certificates ... are these required and how is tox configuring this? Or am I on the wrong path? I see that tools/test-setup-docker.sh seems to have a different way of running the tests. Thanks, - mike. CONFIDENTIALITY NOTICE: This e-mail (and its attachments) may contain confidential and legally privileged information or information protected from disclosure. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein is strictly prohibited. In this case, please immediately notify the sender by return e-mail, delete the message (and any accompanying documents) and destroy all printed hard copies. Thank you for your cooperation. Copyright ANDES TECHNOLOGY CORPORATION - All Rights Reserved.

2 3

Newbie question(s) for eval
by mikeyang＠andestech.com 20 Apr '21

20 Apr '21

Hi, I’m doing an eval of Zuul-CI for my team. I’ve previously used Jenkins and internally/purpose-built CI systems. So Ansible is new to me. I’m using the quick-start tutorial so this is all in the Docker containers and images from the tutorial. I’m running this on a Ubuntu bare-metal machine. And please let me know if there is a more appropriate forum to ask this question or get eval support. So what I tried to do was use a simple job from zuul-jobs. I thought `markdownlint` would be straight-forward. Here’s the rabbit hole that I went down. 1. The quick-start tutorial is using `Ubuntu Focal` images for the `node` 2. I added a second `node` using `Ubuntu Groovy` 3. I can run `testjob` and a custom job (playbook with `shell`s) on the Groovy node 4. I tried to run the `markdownlint` Zuul job on the Groovy node but it fails on the `ensure-nodejs` role * It tries to add `https://deb.nodesource.com/node_6.x groovy main` as a repo * This fails because `dists/groovy` only exists in the 10/12/14/15.x repos * I tried many different ways to assign `node_version` to 15, but the job always fails trying to add `node_6.x` From the Ansible documentation, I see that there are many places in precedence to overwrite a role’s default value, but I have not gotten any of these to work. And I’m unclear about how the precedence extends into Zuul’s `job.vars` and `jobs.extra-vars` (are these equivalent to passing on Ansible’s command line?). Thanks, - mike. CONFIDENTIALITY NOTICE: This e-mail (and its attachments) may contain confidential and legally privileged information or information protected from disclosure. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein is strictly prohibited. In this case, please immediately notify the sender by return e-mail, delete the message (and any accompanying documents) and destroy all printed hard copies. Thank you for your cooperation. Copyright ANDES TECHNOLOGY CORPORATION - All Rights Reserved.

3 5

Nested repo structures
by James E. Blair 20 Apr '21

20 Apr '21

Hi, We were recently presented with a case of a user who uses Gerrit's support for arbitrary repo hierarchies in an unexpected way. Essentially, they have both of the following projects defined in gerrit: * project * project/subproject That is to say, project is not just a pathname, it is a full-blown Gerrit repo. Further, this has nothing to do with git submodules or anything. There are several potential issues in Zuul related to nested repos: * Stats * Secret key storage * Time database * Internal merger repos * Workspace checkouts Stats are good. We already flatten project names there (eg, translate / to _). The secret key storage we are about to move to ZK. We can handle that right now. The time database will be eliminated as part of the v5 work as well, and replaced with SQL. Not a problem. That leaves the git repos. The main issue is that git doesn't deal well with git repos that are cloned into other git repos. We need to keep these separate. The internal merger repos (that is, the ones that users never see) will need a flattened namespace. That should be relatively straightforward since we can take care of that without a user-visible impact. Finally, that leaves the subject of this mail: the user-visible workspace checkouts. Once we get Zuul to a point where it doesn't error-out on a nested repo structure, we still have to decide how to lay it down on disk in the executor workspace. I can see three options: 1) Attempt to lay the repos out in the current fashion, with project/subproject cloned on top of project. I do not think we should do this. While we might be able to do this in the limited context of a workspace checkout (as compared to the internal merger repos), it would still be fraught with problems if, for example, anyone ever did add a "subproject" directory to "project". 2) Allow users to map projects to arbitrary repo checkouts. For this we might add a new mapping to a job like this: - job: checkouts: project: project project/subproject: subproject This would allow users to write out the repos wherever it suits them, whether that's a flat structure as above, or something more complex. They could also move just a single troublesome repo. In fact, you'd really only need the second line of the map above, the first is redundant. 3) Allow users to indicate they want a flat checkout. Given that this is the first use-case for this which we have seen, it's instructive to learn how the users who prompted this discussion are actually using their repos. They want a flat checkout. So they essentially only want the last component of the path to appear in their workspace. This option is a specialized version of the above that's less flexible, but probably simpler to use for this case: - job: flat-checkout: true That would map each project to its last path component. We would need to decide what to do if *that* caused a collision. Presumably we would error out. In this case, the user would have no recourse with Zuul (they would need to change a repo name or stop using a repo in that job). Options 2 and 3 are not mutually exclusive. We could do both so that the typical use would be #3 with #2 as a backup (or for someone with really creative ideas about how to lay down repos). We could also implement one and leave the other as an idea for future expansion. I suspect the internal implementation will be about the same for both. I think I'm inclined to start with #3 just because it concisely enables a specific pattern that may be useful in other contexts. Does anyone have feedback or a preference? -Jim P.S. I suspect we're going to find cases in zuul-jobs where we have assumed project.src_dir == 'src/'+project.canonical_name, but we can and should fix those. We made that a variable for a reason.

4 4

Proposed behavior change on reconfiguration
by James E. Blair 18 Apr '21

18 Apr '21

Hi, We recently merged some changes which improve the consistency of the git repos in different jobs within the same buildset. While it's always the case that every job in a buildset has the same view of the projects involved in its shared queue, projects outside of that queue (including projects involved in providing Zuul roles, playbooks and job required-projects) have not been synchronized, meaning that if a change merged in one of those projects between two jobs in a buildset starting, the earlier job would not see the change and the later job would. It seems more natural for Zuul to fix the state of all of the repos involved in all of the jobs of a buildset, and so the change that we recently merged did this. While this seems like a straightforward change, it had an unanticipated effect on reconfiguration. When Zuul performs a reconfiguration, any jobs that are added to a project are added to currently running buildsets and started; likewise, jobs may be removed from buildsets. Any jobs which are modified but have already run will retain their previous results, but jobs which have not yet run will do so with the new configuration. It turns out that the idea of fixing all of the repos involved in a buildset and also being able to dynamically add, remove, and alter jobs are fundamentally at odds. The simplest example of this is a reconfiguration that adds a playbook to the zuul-jobs repo, while a job on an unrelated repo is running If Zuul has frozen the state of all the repos involved (including zuul-jobs) then the updated version of the job will reference a playbook which will not appear in the previously frozen repo state. In other words, we can freeze both the configuration and the repos, or neither. Earlier in Zuul's development, we felt that the ability to add, remove, or alter jobs while they were running was fairly important. However, Zuul and its usage has evolved considerably since then. I'd like to propose that we continue down the path of freezing both the repo state and the job configuration within a buildset. This means than in the future, if a job is added, removed, or altered during a reconfiguration, it will not take effect for any currently running buildsets, but will take effect for any buildsets started after the reconfiguration (that includes new buildsets created on gate resets). Given the potentially significant change to a long-standing Zuul behavior, I ask that anyone relying on this or who feels that they may be affected by this to please let us know. Thanks, Jim

2 4

[Diversity] D&I WG Office Hours at the PTG
by Amy Marrich 16 Apr '21

16 Apr '21

The Diversity and Inclusion WG has grabbed our usual PTG slot Monday at 16:00UTC and we will be holding office hours to help any project with Inclusive Naming or other D&I related topics. If you'd like us to attend any of your sessions please let me know. If you've been interested in learning more about the WG please come join us! Thanks, Amy (spotz)

1 0