- service-discuss - lists.opendev.org

Parallel production jobs changes
by Ian Wienand 07 Dec '21

07 Dec '21

Hi, To recap: currently production deployment jobs run sequentially. Zuul starts the job on an executor, which is setup to log into the bastion host. The job sets up the system-config playbooks on the bastion host and Ansible is run from there against the production server. To run in parallel, each job needs to not assume it owns the system-config playbooks on the bastion host. Each Zuul *buildset* can use the same system-config playbook checkout though. To achieve this we need to rework the dependencies; each production job needs to depend on a common source-setup job. Once the source is setup on the bastion host, the actual production jobs can run in parallel. To the changes... Firstly, I believe we're doing the setup steps for the executor to log into bridge twice: https://review.opendev.org/c/opendev/system-config/+/818190 removes this duplication, and should be safe to merge. As pointed out in prior reviews when running in the periodic or hourly pipelines each job overrides that bastion host checkout to master. https://review.opendev.org/c/opendev/base-jobs/+/818189 moves this step into base-jobs, in preparation for only being done once by the separate source-setup job. I believe this will be safe to merge; system-config will just do it again in an idempotent way, until: https://review.opendev.org/c/opendev/system-config/+/818191 merges, which drops this step from system-config. We can then merge the system-config job dependency updates in https://review.opendev.org/c/opendev/system-config/+/807672 This should mean that all jobs not only rely on the correct base jobs, but jobs that need certificates, etc. will be relying on the letsencrypt job, etc. This should be safe to merge as nothing should actually change, we just have stricter dependencies. After this, I think we are ready to refactor the base jobs into the two separate steps -- firstly setup the keys on the executor to log into the bastion host, then setup the source to use on the bastion host: https://review.opendev.org/c/opendev/base-jobs/+/807807 This initial refactor should be safe to merge as it creates two new jobs, but the existing base job keeps running both steps as-is. Then we are ready for the penultimate change: https://review.opendev.org/c/opendev/system-config/+/807808 This updates the system-config jobs to all depend on "infra-prod-setup-src" which will be the canonical job that sets up the source repository on bridge.o.o. All other jobs in the buildset will depend on this job, ensuring consistency for a run. This should also be safe, as it again doesn't actually change ordering. Once all this is in, we need the final change to enable parallel running (and think about correct semaphores between periodic/hourly and regular runs). That is yet to be written, but we have enough to get to that point! -i

1 1

Team Meeting Agenda for December 7, 2021
by Clark Boylan 07 Dec '21

07 Dec '21

Hello, We will meet with this agenda on December 7, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20211207) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies *** We ended up in a weird spot after relying on the per buildset update source job: **** We weren't syncing updated repo content to bridge but to a single use test node **** We weren't hard depending on the job so if it didn't run we would run the other jobs without updated sources **** We weren't adding the update source job to hourly and daily periodic buildsets *** Next steps happening in https://review.opendev.org/c/opendev/system-config/+/820320 ** User management on our systems (clarkb 20211207) *** Still struggling with matrix-gerritbot. Now openssh on it complains that uid 11000 doesn't exist. *** Eventually convert mariadb container's from uid 999 to something that makes more sense on the system. ** Adding a lists.openinfra.dev mailman site (fungi 20211207) *** Ansible newlist invocations still struggling *** https://review.opendev.org/c/opendev/system-config/+/820397 and parent are a stack to investigate and fix this. ** keycloak.opendev.org exists now (clarkb 20211207) *** This can be used to gain more familiarity with the tool to implement our authentication spec. *** Are we treating this as production ready? Might be good to avoid that for now while we investigate how to apply this to our previous spec? ** Gerrit User Summit (clarkb 20211207) *** Gerrit 3.2 is EOL *** We can integrate with the new Checks UI stuff if we implement a Checks plugin for Zuul. **** This plugin would run within Gerrit and poll an API on the Zuul side for CI results and render them in a consistent manner. *** Composable Submit Requirements **** This is a new system for expressing what is required to merge a change **** Relies on boolean query language using Gerrit's existing query predicates. **** Mostly we should be careful to avoid accidentally allowing too much access when this shows up. *** ChronicleMap libmodule plugin persistent caches **** This replaces H2 caches for a number of things and gives the caches persistence. Can apparently have a big impact on performance **** One suggestion was that Gerrit incorporate this by default. ** Spring cleaning our Nodepool images (clarkb 20211207) *** CentOS 8, Fedora 34, Gentoo, OpenSUSE 15, and OpenSUSE tumbleweed are either EOL, going EOL, or lacking care necessary to keep building up to date images. *** https://etherpad.opendev.org/p/xvu2oKUQVLiHIHsvGRUt draft communications for proposed cleanups. * Open discussion

1 0

UbuntuOne/Launchpad two-factor authentication
by Jeremy Stanley 02 Dec '21

02 Dec '21

For a little over a year Ian, Clark and I have been using the multi-factor authentication feature of UbuntuOne SSO (i.e. Launchpad) in order to more strongly secure the accounts we rely on for OpenID logins to the Web interfaces of our services like Gerrit and StoryBoard. It's gone smoothly, I think, and so we're probably overdue on our plan to offer this capability to other OpenDev users. Support for this is not enabled by default, your SSO account needs to be a member of a group which is granted the feature. We have one such group authorized for this purpose, which can be found here: https://launchpad.net/~opendev-2fa Please see the information and important caveats documented in the group description. I expect the process would be something like using the LP group members page to request membership for your account, and then one of the group administrators would approve the request, after which you would be able to proceed with configuration of your token or other HOTP/TOTP authenticator. I'm bringing it up here first for discussion, in order to see if anyone has any concerns or related suggestions, but barring none I'd like to move forward with a "soft" (quiet) call for wider testing the first week of December. -- Jeremy Stanley

3 5

Team Meeting Agenda for November 30, 2021
by Clark Boylan 30 Nov '21

30 Nov '21

We will meet with this agenda on November 30, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Gerrit User Summit happening December 2&3 virtually. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20211130) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies *** https://review.opendev.org/c/opendev/system-config/+/807808 Update system-config once per buildset. *** https://review.opendev.org/c/opendev/base-jobs/+/818297/ Reduce actions needed to be taken in base-jobs. *** Next up is thinking about semaphores and actually trying to run jobs in parallel. Dependencies should be correct. ** Zuul multi scheduler setup (clarkb 20211130) *** Possible for zuul updates to break rolling system updates. Not sure if we've got signal on when that happens yet. ** User management on our systems (clarkb 20211130) *** Ran into problems with matrix-gerritbot. There is an updated image we can pull and use to try again. *** Eventually convert mariadb container's from uid 999 to something that makes more sense on the system. ** UbuntuOne/Launchpad two-factor OpenID authentication availability (fungi 20211130) *** http://lists.opendev.org/pipermail/service-discuss/2021-November/000298.html ** Adding a lists.openinfra.dev mailman site (fungi 20211130) *** https://review.opendev.org/818826 ** Proxying and caching Ansible Galaxy in our providers (fungi 20211130) *** https://review.opendev.org/818787 ** Gerrit Account cleanups (clarkb 20211130) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml * Open discussion

1 0

Infra Team Meeting Agenda for November 16, 2021
by Clark Boylan 16 Nov '21

16 Nov '21

We will meet with this agenda on November 16, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Gerrit User Summit happening December 2&3 virtually. ** clarkb out next week. Should we skip the meeting on November 23? * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20211116) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211116) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Zuul multi scheduler setup (clarkb 20211116) *** Zuul is currently running with two schedulers (zuul01.o.o and zuul02.o.o with zuul02.o.o being "primary") *** Did first rolling restart of schedulers over the weekend. *** Zuul-web should return consistent results now as it talk to ZooKeeper directly. ** User management on our systems (clarkb 20211116) *** Give gerritbot and matrix-gerritbot a shared user: https://review.opendev.org/c/opendev/system-config/+/816769/ *** Eventually convert mariadb container's from uid 999 to something that makes more sense on the system. ** Caching openstack/openstack on our DIB images (clarkb 20211116) *** There are semi frequent errors when updating the DIB cache for openstack/openstack *** Seems related to verifying or updating submodule content. *** Should we simply stop caching this repo entirely? It isn't really used for much. * Open discussion

1 0

Team Meeting Agenda for November 9, 2021
by Clark Boylan 09 Nov '21

09 Nov '21

Hello, we will meet on November 9, 2021 at 19:00UTC in #opendev-meeting with this agenda (note the DST change in many parts of the world): == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20211109) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211109) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Zuul multi scheduler setup (clarkb 20211109) *** Zuul is currently running with two schedulers (zuul01.o.o and zuul02.o.o with zuul02.o.o being "primary") *** We have tracked down a number of bugs Monday with corvus fixing many of them. *** Overall seems stable enough. *** Note the "flapping" status page can be weird. ** User management on our systems (clarkb 20211109) *** Be explicit about uid/gid ranges: https://review.opendev.org/c/opendev/system-config/+/816869/ **** 0-999 system, 1000-1999 unallocated, 2000-2999 for infra-root users, 3000-9999 host level users, 10k - 64k container users that need uids on the host as well for bind mounts. *** Clean up unused bootstrapping users: https://review.opendev.org/c/opendev/system-config/+/816771 *** Give gerritbot and matrix-gerritbot a shared user: https://review.opendev.org/c/opendev/system-config/+/816769/ *** Eventually convert mariadb container's from uid 999 to something that makes more sense on the system. * Open discussion

1 0

Infra Team Meeting Agenda for November 2, 2021
by Clark Boylan 02 Nov '21

02 Nov '21

We will meet on November 2, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Gerrit User Summit details arriving soon. I've been told they would be interested to hear from us on how we do automated testing and management of Gerrit. * Actions from last meeting * Specs Review ** Mailman 3 spec https://review.opendev.org/810990 * Topics ** Improving OpenDev's CD throughput (clarkb 20211102) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211102) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Fedora 34 test node booting problems (clarkb 20211102) *** Changes to Fedora's kernel packaging broke Xen *** Not yet sure if that may have also somehow broken OVH and iweb. ** Zuul multi scheduler setup (clarkb 20211102) *** Zuul ran with two schedulers for the first time over the last weekend. *** First jobs started by one scheduler and completed by another ran. *** Had to revert due to bugs in caching. *** Expect Zuul restarts as the scale out scheduler work in Zuul progresses. ** FIPS testing in our CI system (clarkb 20211102) *** There is interest in testing various pieces of software against FIPS enabled systems in our Zuul. *** We are not building special FIPS images instead a Zuul role exists to update supported platforms and boot them into FIPS mode. *** Some tests have problems with ephemeral state being lost if the FIPS role runs too late. Then tests fail for unexpected reasons. Worth checking on reboot ordering relative to other test setup if there are problems. * Open discussion

1 0

Team Meeting Agenda for October 26, 2021
by Clark Boylan 26 Oct '21

26 Oct '21

We will meet on October 26, 2021 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review ** Mailman 3 spec https://review.opendev.org/810990 * Topics ** Improving OpenDev's CD throughput (clarkb 20211026) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211026) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Fedora 34 test node booting problems (clarkb 20211026) *** Changes to Fedora's kernel packaging broke Xen *** Not yet sure if that may have also somehow broken OVH and iweb. ** Begin planning for Gerrit 3.4 upgrade (clarkb 20211026) *** Read through release notes and identify areas of concern. **** https://www.gerritcodereview.com/3.4.html *** Do we need to modify system-config-run-review-3.4 or system-config-upgrade-review jobs to test any specific behaviors? *** Probably worth holding a 3.4 test node and doing a skim for any unexpected behaviors. * Open discussion

1 0

Team Meeting Agenda for October 19, 2021
by Clark Boylan 19 Oct '21

19 Oct '21

We will meet October 19, 2021 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** PTG this week. *** OpenDev session Wednesday October 20, 2021 at 14:00 - 16:00 UTC in https://meetpad.opendev.org/oct2021-ptg-opendev *** Zuul session Thursday October 21, 2021 at 14:00 UTC in https://meetpad.opendev.org/zuul-2021-10-21 * Actions from last meeting * Specs Review ** Mailman 3 spec https://review.opendev.org/810990 * Topics ** Improving OpenDev's CD throughput (clarkb 20211019) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211019) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Gerrit project renames (clarkb 20211019) *** Overall went well. *** Possible trouble renaming secrets and doing ZK secrets backups? *** We accidentally updated all Gitea projects. Should we just do that by default: https://review.opendev.org/c/opendev/system-config/+/814443 ? ** Improve zuul restarts (frickler 20211014) *** Docs at https://docs.opendev.org/opendev/system-config/latest/zuul.html#restarting-… need updating **** What to restart (scheduler,web,fingergw?) and how (docker restart vs. docker-compose) **** When to run the re-enqueue **** Collecting debug information *** Don't reenqueue periodic jobs * Open discussion Note the PTG is happening, if there are conflicts we can probably focus on the most important items (zuul restart docs and post project rename stuff) and keep the meeting short.

1 0

Gerrit 3.3 upgrade: dashboards and attention sets
by Ian Wienand 13 Oct '21

13 Oct '21

Hello You may have noticed the recent upgrade of Gerrit to version 3.3 on review.opendev.org. We note two things users should be aware of - Dashboard links If your dashboard uses https://review.opendev.org/#/dashboard/... it may no longer work, despite being the documented way to create dashboards. We are currently unclear if this is a bug or a feature [1] but in the mean time, removing the "/#/" (i.e. just use https://review.opendev.org/dashboard/...) makes things work. - Attention sets You have hopefully seen a new section on the main page "Your Turn". This is the Gerrit "Attention Set" feature [2]. This is an optional feature, but we feel it is worth spending some time with it to see how this can fit into our workflows. One thing to be aware of is that adding a vote by default adds the uploader in the attention set. While this makes sense in some models (particuarly non-gating where you are expected to merge yourself), in our environment usually adding a +1 or +2 with no comments is not something that needs the attention of the uploader -- it's either waiting for more review, or Zuul will merge it after gating tests. At the bottom of the modal box where you are inputting your votes you can easily modify the attention set. Look out for it next time you're reviewing! Consider removing others if your comments do not require new attention. If we pollute other people's attention lists with changes that don't actually need their attention the feature will not be useful. So it is worth us all trying to work with the system and be mindful of who and when we add people (we have raised an issue to discuss our workflow with this feature, but it will not be an instant fix [3]). Thanks! OpenDev Admins [1] https://groups.google.com/g/repo-discuss [2] https://gerrit-review.googlesource.com/Documentation/user-attention-set.html [3] https://bugs.chromium.org/p/gerrit/issues/detail?id=15154

1 0