- Zuul-discuss - lists.opendev.org

The error was: 'zuul_ssh_key_algorithm' is undefined
by Åsmund Østvold 31 Mar '21

31 Mar '21

Hi, Today a job that has not been changed for a long time( 1000+ executions) failed in a way I can not explain. The below snippet is from the "cleanup" after normal test execution that succeeded. Everything before and after looks to me to not have any problems. [test script output end] 2021-03-29 16:50:00.729468 | ubuntu-bionic | ok: Runtime: 0:02:02.256368 2021-03-29 16:50:00.751939 | 2021-03-29 16:50:00.752085 | PLAY RECAP 2021-03-29 16:50:00.752153 | ubuntu-bionic | ok: 2 changed: 2 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0 2021-03-29 16:50:00.752192 | 2021-03-29 16:50:00.992482 | RUN END RESULT_NORMAL: [trusted : review.resoptima.net/zuul/irma-common-jobs/playbooks/build_and_integrationt…] 2021-03-29 16:50:00.993174 | POST-RUN START: [trusted : review.resoptima.net/zuul/irma_config/playbooks/base/post-ssh.yaml@master] 2021-03-29 16:50:02.365604 | 2021-03-29 16:50:02.365802 | PLAY [all] 2021-03-29 16:50:02.400284 | 2021-03-29 16:50:02.400472 | TASK [remove-build-sshkey : Remove the build SSH key from all nodes] 2021-03-29 16:50:02.591665 | ubuntu-bionic | ERROR 2021-03-29 16:50:02.592037 | ubuntu-bionic | { 2021-03-29 16:50:02.592094 | ubuntu-bionic | "msg": "The task includes an option with an undefined variable. The error was: 'zuul_ssh_key_algorithm' is undefined\n\nThe error appears to be in '/var/lib/zuul/builds/e01e1c4faa834c079a62b6514ed2106f/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/remove-build-sshkey/tasks/main.yaml': line 1, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Remove the build SSH key from all nodes\n ^ here\n" 2021-03-29 16:50:02.592133 | ubuntu-bionic | } 2021-03-29 16:50:02.626484 | 2021-03-29 16:50:02.626654 | PLAY RECAP 2021-03-29 16:50:02.626737 | ubuntu-bionic | ok: 0 changed: 0 unreachable: 0 failed: 1 skipped: 0 rescued: 0 ignored: 0 [normal termination of job] Is this a known issue? As it is < 1/1000 probability of this happening it is not an important issue for me but wanted to check with you anyway. Regards, Åsmund

2 2

[marketing] Zuul 4 & 5 Press Release
by Jimmy McArthur 22 Mar '21

22 Mar '21

Hi Zuul Team - We're gearing up to start promoting the Zuul 4 and 5 release. I know we don't have a definitive timeline yet for the 5 release, but it seems clear that the Zuul community knows what features will land. Is there anyone in the community that could volunteer to put together a blogpost (location tbd - maybe openinfra.dev/blog or superuser). From that, we can extract features for promo and obviously push the blog out to our various marketing channels. Cheers, Jimmy

2 2

Getting a #403 Resource not accessible by integration"
by Cyril Lavier 19 Mar '21

19 Mar '21

Hi. I'm trying to setup a Zuul POC with Github, with 2 projects (one containing the jobs config/playbook and one being an actual project with a simple pipeline with a jsonlint job, to test). My Zuul is connected to Github, it can grab the repositories and receive calls from Github when creating a PR, but Zuul sends this message to the PR : ------------------------------------ Starting check jobs. Warning: Failed to update check run vanilla/check: 403 Resource not accessible by integration ------------------------------------ "vanilla" being the tenant name. In the Zuul WEB UI, the job is linked to the repository but it's still in "queued". I've checked the Github permissions and I don't find anything different than in this howto (https://zuul-ci.org/docs/zuul/howtos/github_setup.html) Also, in the meantime, no nodes are displayed in the "Nodes" section of the webUI, but I declared a static one (with the prerequisites from the doc) in the nodepool config. So I have 2 questions : - Why do I get this 403 error ?, or maybe a way to help me debug (debug logs don't help me a lot) - Why is there no nodes in the "Nodes" section of the webUI ? I assume the job being queued indefinitely may come from either the node not appearing or the 403 error. Thanks.

3 5

Type annotations
by jim＠acmegating.com 17 Mar '21

17 Mar '21

Hi, I'd like to propose that we remove type annotations and checking from our code. I don't think they are providing much value, but they come at a significant cost. They have very rarely (I think perhaps once or twice) caught errors. Meanwhile, we aren't going to relax our test coverage requirements, which should catch most things anyway. On the other hand, they cause the code to be much less readable and I believe that will ultimately lead to more errors. Developer and reviewer fatigue are important things to keep in mind. Some examples where I think annotations are particularly unhelpful: self.on_connect_listeners: List[Callable[[], None]] = [] Or (note the duplication): def __init__(self, ... info: Optional[zuul.model.WebInfo] = None): self.info: Optional[zuul.model.WebInfo] = info We've also had some occurrences where we ended up fighting the annotation system. I've mentioned to several of you that I do very much enjoy the benefits of statically typed languages (I write C code for fun), but I don't think that the type hints in Python are very well executed. Many of us chose to use Python instead of Java specifically for the dynamic typing and as a reaction against the extensive type requirements. The type hinting that is being added to Zuul seems like it carries the costs of Java without the benefits. We started this as a small experiment. We said that we would run the type checker in a mode where it would operate on implicit type information, thereby minimizing the additional cognitive load on authors and reviewers. We then started adding some minimal type information in strategic places to help it out. However, I'm now reviewing changes that are literally 3x larger than they otherwise would be because of all the additional repetitive type information going into them. I'm glad to try experiments like these, but I think this one has grown to the point where we need to decide if we want to go all-in on type annotations or remove them. I feel fairly strongly that the cost to authors and reviewers far outweighs the benefits, and we should make everyone's lives easier by removing them. -Jim

10 18

Dropping gerrit replication trigger event
by Tristan Cacqueray 16 Mar '21

16 Mar '21

Hello folks, After upgrading to Zuul version 4 and Gerrit version 3.2, our scheduler logs are flooded with ref-replication events (about 1GB of debug per hour). It seems like the gerrit driver could be improved to ignore unused events, and the queue processor could be less verbose to reduce the amount of noise. On the other hand, the replication event doesn't seems to be that useful and here is a change to ignore them: https://review.opendev.org/c/zuul/zuul/+/780809 If you do rely on replication event, please raise your voice in the review above. Thanks, -Tristan

2 1

vscode/codium editor for zuul config files
by Sorin Sbarnea 01 Mar '21

01 Mar '21

In case you happen to use vscode or codium, you can now use an extension that verified the syntax of the configuration files and provide auto-complete. Ansible Language extension v0.2.0 adds support for Zuul config files. https://marketplace.visualstudio.com/items?itemName=zbr.vscode-ansible&ssr=… https://open-vsx.org/extension/zbr/vscode-ansible If you like it useful, feel tree to leave a review. PS. Nope, I it does not work with vim, emacs, pycharm, ... Cheers Sorin

1 0

[all] April 2021 PTG Dates & Registration
by Kendall Nelson 01 Mar '21

01 Mar '21

Hello Everyone! I'm sure you all have been anxiously awaiting the announcement of the dates for the next virtual PTG[1]! The PTG will take place April 19-23, 2021! PTG registration is now open[2]. Like last time, it is free, but we will again be using it to communicate details about the event (schedules, passwords, etc), so please take two minutes to register! Early next week we will send out info to mailing lists about signing up teams. Also, the same as last time, we will have an ethercalc signup and a survey to gather some other data about your team. Can't wait to see you all there! -the Kendalls (diablo_rojo & wendallkaters) [1] https://www.openstack.org/ptg/ [2] https://april2021-ptg.eventbrite.com

1 0

Last call for 4.0 deprecations
by jim＠acmegating.com 15 Feb '21

15 Feb '21

Hi, We've merged the SQL database requirement, and as I write this, the ZooKeeper connection requirement patches are working their way through the gate. Those are the last big items blocking the 4.0 release. Before we issue the release, I'd like to ask folks to take a look for anything we've said that we would like to deprecate or remove in a major version bump to make sure we don't miss the opportunity. At this point, I don't think we should entertain last-minute new ideas, but rather, if we put something in the documentation quite some time ago saying that it would be removed in the future, let's make sure we take the opportunity to consider that now. Also, if anyone can think of any other blockers for 4.0, speak now please. :) Otherwise, I think we can cut the release very soon. -Jim

1 1

Scale out scheduler
by jim＠acmegating.com 13 Feb '21

13 Feb '21

Hi, The main blocker for releasing 4.0 at this point is merging the changes that implement the ZooKeeper connection requirement from every Zuul component. Currently the changes that implement those also include some extra changes that refactor the ZooKeeper connection classes. I think that's a good idea, but I'd like to separate concerns and focus on getting the requirements for the 4.0 release taken care of first. On Friday I discussed with some folks an approach to that, and today I finished up a rebase + rework of the series. I uploaded three new versions of these patches: https://review.opendev.org/775459 Mandatory Zookeeper connection for ZuulWeb https://review.opendev.org/775460 Connect executor to Zookeeper https://review.opendev.org/775461 Connect merger to Zookeeper I think they represent the minimum we need to get into 4.0. There is some boilerplate related to ZK connections in each service, but it's easy to reason about for now. I do think we should refactor the ZK connection class, but let's do that on top of these patches. The results will be clear and easy to understand. The original patches also increased the unit test job timeout because the jobs took approximately 25% longer to run. I think perhaps reviewers were under the impression that was necessary due to the new ZooKeeper connections in unit tests. I investigated the issue and found that was not the case. The actual cause was due to extra import statements that were added to support the addition of type annotations. After removing those type annotations and associated import statements, the test runtime remains the same as it currently is. You can read more details about it in the comments of patchset 2 of the last change. It's also worth keeping in mind that extraneous import statements can cause not only unanticipated problems like this, but can often exacerbate the problem of import loops, something we have seen occasionally in Zuul. Keeping additional imports for type annotations to a minimum will be beneficial. I propose the following next steps: 1) Re-propose 775459-775460 as new patchsets on their original changes in order to preserve the review history there. 2) Merge those changes and continue work on releasing 4.0. 3) Meanwhile: refactor the ZooKeeper connection class. 4) Continue work on the remaining scale-out scheduler changes. -Jim

1 0

Nodepool image building
by jim＠acmegating.com 10 Feb '21

10 Feb '21

Hi, I'd like to propose some new features related to Nodepool and image building. The short version is that right now image management is tied fairly closely to diskimage-builder within the nodepool-builder process. This is problematic for folks who want Nodepool image management but can't use DIB (ie, Windows images). It's even problematic for those who do want to use DIB, in that it's difficult to debug image builds. It's worth noting the original Zuulv3 spec had room for more than one nodepool-builder implementation (in an offhand comment, it mentioned that a snapshot-based nodepool-builder process could be created). However, that has not yet come to pass. Over the years, we've speculated about the idea of using Zuul to build images, and I think it's time to see if we can put that into practice. It makes a lot of sense in that we have a ready-made execution environment with Zuul -- build logs and history are well defined and accessible to anyone. It's flexible and can accommodate any image building system (DIB or otherwise). The main downside is the inception aspect, but I'm not worried about that (more later). I think the key requirements are: * Able to create images using any tooling. * Visibility into build history. * Nodepool-launcher should automatically use the "latest" build. Part of this solution is obvious, but I think there are open questions for other parts: * Should the Zuul job be responsible for uploading the image to the cloud? * Should Nodepool be responsible for deleting images? About that downside: clearly there's an issue when bootstrapping a new Zuul system -- how can you run jobs without images? Most new Zuul installations start by using cloud images from the provider. Any Zuul user who wants custom images likely already has some kind of job running on some kind of image anyway. I think it's a natural progression to create a Zuul job to build a custom image and then begin using that custom image in "real" Zuul jobs. Long story short: bootstrap the system with cloud-images. As an OpenDev admin, I think I would be comfortable with a Zuul job that built our DIB images using cloud provider images -- I don't think the differences between those images are so great as to make a DIB job difficult to write. However, if we did use this system, I'm not sure we would ever have to. We have enough clouds with our own images in them already, we could probably use our images to make new ones. In this case: a chicken with no egg. :) Let's work through how this might look. First the easy part, then I'm going to propose some alternate solutions. 1) Build the image: A Zuul job ("build-nodepool-image-ubuntu-focal") is triggered by a timer. It has a mutex to avoid duplicate image builds. It runs DIB in this case but could just as easily run packer or even snapshot the system it's running on (we can add roles in zuul-jobs for common image building systems). 2) Get the image into the cloud(s): a) Our first option is to do this work in Zuul. Have the job (or a dependent job) upload the image to the cloud(s) (if this was a snapshot build, that's a no-op -- it's already there). Advantage: the upload log is visible along with the build log. Disadvantage: if the upload fails, we won't retry like Nodepool does. b) An alternative would be to have Nodepool do this. Nodepool could even query the Zuul API to find the latest build of build-nodepool-image-ubuntu-focal, check the artifacts list, and download the image file from there. After that, it looks a lot like the current nodepool-builder process right after DIB runs (it has a file on disk and it starts to upload it). Advantage: we use all of the Nodepool image lifecycle management. Disadvantage: upload logs are hidden from users (like today). to integrate snapshot builds, we would have to download the snapshot and re-upload it. 3) Get nodepool-launcher to use the new image: a) If we use 2.a, then we have an image in the cloud without a record in Zookeeper. i) We could have nodepool-builder (or something like it) scan the cloud for images matching a certain pattern and create ZK records for them. ii) Or we could just have nodepool-launcher look for that pattern and use the latest image. b) If we use 2b, then launcher will work as it does today with nodepool-builder; no changes are needed. 4) Delete the image when it's no longer needed We should use the n-2 rule we have today so there's always at least 2 images in the cloud for easy rollback. a) If we used 2.a + 3.a.iii then we don't have ZK records. It would probably make sense to handle image deletes in Zuul jobs with this setup. Presumably the jobs would query the cloud API, list images, and delete them according to the age rules. The disadvantage here is we'd be re-implementing a good chuck of nodepool-builder in Ansible modules. It's doable, but may be a bit clunky. b) If we use 2.a + 3.a.i or 2.b + 3.b then we have records in ZK, so nodepool-builder (or something like it) can delete the images more or less how it does today. I think those are most of the choices. Now that I've typed all of that out, I think I really like the system with "b" for all the choices. I'm not too worried about the upload logs being hidden. And the snapshot system is workable, if not ideal. If we really wanted to support snapshots better, then 2.a + 3.a.i + 4.b would probably be best, but at the cost of moving the upload code into zuul-jobs roles. Or we could do the same thing we did last time and implement the "b" solution ("nodepool-zuul-builder") and if someone really wants to do snapshots, then we implement 2.a[snapshot] + 3.a.i + 4.b as ("nodepool-snapshot-builder"). That would probably be fairly efficient as far as code goes. Finally, a word about DIB and the current nodepool-builder: none of this requires that we change anything about the current nodepool-builder process. If we like it, we can keep it exactly as it is. Indeed, in OpenDev's case, we rely very heavily on the cache on the build host, so it's not at all clear that doing builds on throwaway nodes is ideal. Of course, we could do builds on static Zuul nodes and get the benefit of the cache. I think the way to approach this would be to implement the new system, then try using it with DIB, and if we love it and find no significant drawbacks, consider deprecating the current nodepool-builder. If we don't love it, we'll keep it. I'd appreciate feedback on this. I know this is a bit free-form; I'll take the feedback I get and synthesize it into a proper spec. -Jim PS: there's a fairly well-known hack for making non-DIB image builds by replacing DIB with a script that accepts the same arguments. The "b" plan could actually be implemented today with no changes to Nodepool using that same hack. A final implementation probably wouldn't look much different structurally.

5 6