- service-discuss - lists.opendev.org

Mailman 3 service upgrades
by Jeremy Stanley 26 Aug '23

26 Aug '23

A few minutes ago we upgraded the production Mailman 3 server (currently hosting this lists.opendev.org domain and also lists.zuul-ci.org) to the latest releases of the various Mailman component applications. If you notice any problems you think might be related to the upgrade, please do let us know here on the service-discuss mailing list or in the #opendev channel of the OFTC IRC network. -- Jeremy Stanley on behalf of the OpenDev Collaboratory sysadmins

1 0

Clarkb's Service Coordinator Self Nomination
by Clark Boylan 23 Aug '23

23 Aug '23

We have just about reached the end of the OpenDev Service Coordination nomination period, and I haven't seen any nominations. I brought this up in our weekly team meeting just now, and it sounds like people are happy for me to continue in the role. Given that feedback I nominate myself to serve another six months as OpenDev Service Coordinator. Expect more of the same direction from me. I'm hoping we'll continue upgrading older servers, move our container images to Bookworm, and maybe eventually get some stuff running on podman so that we can revive quay.io hosting of our images. If you'd like something different your window of opportunity is quickly fading as we said August 22 would be the end of the nomination period. To be clear, I'd be happy to support someone else in the role if they want to give it a go. Thank you for your consideration, Clark

1 0

Team Meeting Agenda for August 22, 2023
by Clark Boylan 22 Aug '23

22 Aug '23

We will meet with this agenda on August 22, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Service Coordinator Election (clarkb 20230822) *** Nomination period ends today (the day of the meeting) ** Bastion host (ianw 20230627) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230627) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit updates (clarkb 20230718) *** Replication task file leaks. **** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 **** https://review.opendev.org/c/opendev/system-config/+/884779 Revert bind mounts for Gerrit plugin data ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Next up mirrors and meetpad. *** Clarkb needs to look into cleaning up the old CI registry node. ** Rax IAD image upload struggles (clarkb 20230822) *** The Rax IAD region has struggled to upload images within the image upload timeout. This seems to get worse the more images you upload **** Almost like images are processed serially and we create a queue that gets backed up. *** No issue has been filed with rax yet, but we're testing incremental unpausing of images to get more data. *** We also have hundreds of leaked images in the other two regions (there were well over a thousand in IAD but those have been cleaned up manually now). ** Fedora cleanup (clarkb 20230627) *** Need changes to remove the label and nodeset from Zuul/Nodepool ** Gitea 1.20 (clarkb 20230627) *** https://review.opendev.org/c/opendev/system-config/+/886993 **** Ready to go if we are happy with the resulting behaviors. *** We disable oauth2 but need to configure multiple oauth2 jwt settings in order to avoid startup errors. **** Need to update prod secrets with the oauth2 jwt settings before merging. *** https://github.com/go-gitea/gitea/issues/25377 ** Zuul changes and updates (clarkb 20230822) *** Ansible 8 now availalble. We should convert the OpenDev tenant. Notify openstack tenant that we will switch them after the upcoming release *** Stdout and stderr to be uncombined in Zuul Ansible. Currently this is a job setting for testing purposes. We'll need to start looking at how this impacts us. *** Early failure detection regexes should arrive soon. This is potentially useful for our users? ** Python container updates (tonyb 20230718) *** Some moderate progress in the first round *** https://review.opendev.org/q/topic:bookworm-python and https://review.opendev.org/q/topic:force-base-image-build *** I noticed that https://opendev.org/openstack/python-openstackclient/src/branch/master/Dock… is using a 3.9 tag which is buster **** We should stick to the explicit debian release + explicit python release versions for things. We can delete the old tags to force errors that indicate people should update the tags they use. *** We should write a tool to find and clean leaked tags on docker hub (and quay.io?) * Open discussion

1 0

Team meeting Agenda for August 15, 2023
by Clark Boylan 15 Aug '23

15 Aug '23

We will meet on August 15, 2023 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20230627) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230627) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit updates (clarkb 20230718) *** Replication task file leaks. **** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 **** https://review.opendev.org/c/opendev/system-config/+/884779 Revert bind mounts for Gerrit plugin data ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Next up mirrors and meetpad. *** Clarkb needs to look into cleaning up the old CI registry node. ** Fedora cleanup (clarkb 20230627) *** Can we accelerate this removal now that upstream has killed the package mirrors? We should be able to simply remove the nodes now? ** Gitea 1.20 (clarkb 20230627) *** https://review.opendev.org/c/opendev/system-config/+/886993 **** Updating log collection so that we can double check changes to the access logs that are listed in the breaking changes list. *** We disable oauth2 but need to configure multiple oauth2 jwt settings in order to avoid startup errors. *** https://github.com/go-gitea/gitea/issues/25377 ** Etherpad 1.9.1 (clarkb 20230627) *** https://review.opendev.org/c/opendev/system-config/+/887006 *** The held node running the latest version of this change appears to work for clarkb. Others should test and review the change appropriately. ** Python container updates (tonyb 20230718) *** Some moderate progress in the first round *** https://review.opendev.org/q/topic:bookworm-python and https://review.opendev.org/q/topic:force-base-image-build *** I noticed that https://opendev.org/openstack/python-openstackclient/src/branch/master/Dock… is using a 3.9 tag which is buster **** We should stick to the explicit debian release + explicit python release versions for things. We can delete the old tags to force errors that indicate people should update the tags they use. *** We should write a tool to find and clean leaked tags on docker hub (and quay.io?) * Open discussion

1 0

Team Meeting Agenda for August 1, 2023
by Clark Boylan 01 Aug '23

01 Aug '23

We will meet with this agenda on August 1, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20230627) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230627) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit updates (clarkb 20230718) *** Replication task file leaks. **** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 **** https://review.opendev.org/c/opendev/system-config/+/884779 Revert bind mounts for Gerrit plugin data ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Next up mirrors and meetpad. *** Clarkb needs to look into cleaning up the old CI registry node. ** Fedora cleanup (clarkb 20230627) *** Proposed the removal of Fedora mirrors and/or images https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… **** Tonyb has been looking at moving existing fedora test nodes off of our mirrors. Not straightforward as it requires zuul-jobs role updates. ** Quo vadis storyboard (frickler 20230627) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG ** Gitea 1.20 (clarkb 20230627) *** https://review.opendev.org/c/opendev/system-config/+/886993 **** Updating log collection so that we can double check changes to the access logs that are listed in the breaking changes list. *** We disable oauth2 but need to configure multiple oauth2 jwt settings in order to avoid startup errors. *** https://github.com/go-gitea/gitea/issues/25377 ** Etherpad 1.9.1 (clarkb 20230627) *** https://review.opendev.org/c/opendev/system-config/+/887006 *** The held node running the latest version of this change appears to work for clarkb. Others should test and review the change appropriately. ** Python container updates (tonyb 20230718) *** Some moderate progress in the first round *** https://review.opendev.org/q/topic:bookworm-python and https://review.opendev.org/q/topic:force-base-image-build *** I noticed that https://opendev.org/openstack/python-openstackclient/src/branch/master/Dock… is using a 3.9 tag which is buster **** We should stick to the explicit debian release + explicit python release versions for things. We can delete the old tags to force errors that indicate people should update the tags they use. *** We should write a tool to find and clean leaked tags on docker hub (and quay.io?) ** Meetpad LE cert refresh (clarkb 20230725) *** LE refreshed the cert for meetpad but the handler that copies the file and restarts services does not appear to have run. *** Ideally we would manually update acme to trigger a rerun and debug from there. * Open discussion

1 0

Team Meeting Agenda for July 25, 2023
by Clark Boylan 25 Jul '23

25 Jul '23

We will meet with this agenda on July 25, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20230627) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230627) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit updates (clarkb 20230718) *** Update our Gerrit images for new releases (3.7.4 and 3.8.1) **** https://review.opendev.org/c/opendev/system-config/+/885317 *** Replication task file leaks. **** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 **** https://review.opendev.org/c/opendev/system-config/+/884779 Revert bind mounts for Gerrit plugin data *** Restart for the update_bugs hook script revert we merged (and whatever else) ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Next up mirrors and meetpad. *** Clarkb needs to look into cleaning up the old CI registry node. ** Fedora cleanup (clarkb 20230627) *** Proposed the removal of Fedora mirrors and/or images https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… **** Tonyb has been looking at moving existing fedora test nodes off of our mirrors. Not straightforward as it requires zuul-jobs role updates. ** Quo vadis storyboard (frickler 20230627) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG ** Gitea 1.20 (clarkb 20230627) *** https://review.opendev.org/c/opendev/system-config/+/886993 **** Updating log collection so that we can double check changes to the access logs that are listed in the breaking changes list. *** We disable oauth2 but need to configure multiple oauth2 jwt settings in order to avoid startup errors. *** https://github.com/go-gitea/gitea/issues/25377 ** Etherpad 1.9.1 (clarkb 20230627) *** https://review.opendev.org/c/opendev/system-config/+/887006 *** The held node running the latest version of this change appears to work for clarkb. Others should test and review the change appropriately. ** Python container updates (tonyb 20230718) *** Some moderate progress in the first round *** https://review.opendev.org/q/topic:bookworm-python and https://review.opendev.org/q/topic:force-base-image-build *** I noticed that https://opendev.org/openstack/python-openstackclient/src/branch/master/Dock… is using a 3.9 tag which is buster **** We should stick to the explicit debian release + explicit python release versions for things. We can delete the old tags to force errors that indicate people should update the tags they use. *** Should we remove the old tags similare to change_789650_3.9 ? **** We should write a small tool that can go in the zuul-jobs role content for the container/docker jobs to scan for and clean up these tags if changes are merged. ** Meetpad LE cert refresh (clarkb 20230725) *** LE refreshed the cert for meetpad but the handler that copies the file and restarts services does not appear to have run. *** Ideally we would manually update acme to trigger a rerun and debug from there. * Open discussion

1 0

Team Meeting Agenda for July 18, 2023
by Clark Boylan 17 Jul '23

17 Jul '23

We will meet at 19:00 UTC on July 18, 2023 in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20230627) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230627) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit updates (clarkb 20230718) *** Update our Gerrit images for new releases (3.7.4 and 3.8.1) **** https://review.opendev.org/c/opendev/system-config/+/885317 *** Replication task file leaks. **** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 **** https://review.opendev.org/c/opendev/system-config/+/884779 Revert bind mounts for Gerrit plugin data *** Gerrit reject implict merge changes **** https://review.opendev.org/c/opendev/system-config/+/885318 Merge this change once All-Projects is updated to reject implicit merges. ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Next up mirrors and meetpad. *** Zuul executors have been updated to jammy. *** Clarkb needs to look into cleaning up the old CI registry node. ** Fedora cleanup (clarkb 20230627) *** Proposed the removal of Fedora mirrors and/or images https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… **** Tonyb has been looking at moving existing fedora test nodes off of our mirrors. Not straightforward as it requires zuul-jobs role updates. ** Quo vadis storyboard (frickler 20230627) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG ** Gitea 1.20 (clarkb 20230627) *** https://review.opendev.org/c/opendev/system-config/+/886993 *** We disable oauth2 but need to configure multiple oauth2 jwt settings in order to avoid startup errors. *** https://github.com/go-gitea/gitea/issues/25377 ** Etherpad 1.9.1 (clarkb 20230627) *** https://review.opendev.org/c/opendev/system-config/+/887006 *** There were issues with user name and color settings. Resyncing our config to use null values instead of false boolean values seems to address this. But further testing on a held node is required. ** Python container updates (tonyb 20230718) *** Some moderate progress in the first round *** https://review.opendev.org/q/topic:bookworm-python and https://review.opendev.org/q/topic:force-base-image-build *** I noticed that https://opendev.org/openstack/python-openstackclient/src/branch/master/Dock… is using a 3.9 tag which is buster *** Should we update those tags? *** Should we remove the old tags similare to change_789650_3.9 ? * Open discussion

1 0

Stories not closing on gerrit WF
by ronald.stone＠windriver.com 13 Jul '23

13 Jul '23

I'm not sure if this is the correct group. Please refer me elsewhere if this is inappropriate. StX docs gerrits are no longer triggering referenced stories to close when they workflow. How can we get this working again?

2 3

Team Meeting Agenda for July 11, 2023
by Jeremy Stanley 11 Jul '23

11 Jul '23

We will meet with this Agenda on June 27, 2023 at 19:00 UTC in #opendev-meeting: = Weekly Project Infrastructure team meeting = The OpenDev Team holds public weekly meetings in <code><nowiki>#opendev-meeting</nowiki></code> on OFTC, Tuesdays at 1900 UTC. Everyone interested in infrastructure and process surrounding automated testing and deployment is encouraged to attend. Please feel free to add agenda items (and your IRC nick in parenthesis). == Agenda for next meeting == * Announcements ** Clarkb unable to attend he team meeting on July 4. * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20230627) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230627) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit replication leaked tasks (clarkb 20230627) *** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 **** https://review.opendev.org/c/opendev/system-config/+/884779 Revert bind mounts for Gerrit plugin data ** Gerrit reject implict merge changes (clarkb 20230627) *** We merged a revert of a change on branch foo to branch bar which implicitly merged foo into bar *** Gerrit project config receive.rejectImplicitMerges can be set to true to prevent this changes from being pushed *** Should we go ahead and set that on All-Projects? The default value is to inherit from the parent config. ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** CI registry is done. (Cleanup of old host still needs to be done). *** Next up mirrors, meetpad, and ci registry. *** Please grab a node or two. *** Zuul installation being updated as Jammy enables use of Podman ** Fedora cleanup (clarkb 20230627) *** Proposed the removal of Fedora mirrors and/or images https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… **** Tonyb has been looking at moving existing fedora test nodes off of our mirrors. Not straightforward as it requires zuul-jobs role updates. ** Quo vadis storyboard (frickler 20230627) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG ** Gitea 1.20 (clarkb 20230627) *** https://review.opendev.org/c/opendev/system-config/+/886993?usp=dashboard *** https://github.com/go-gitea/gitea/issues/25377 ** Etherpad 1.9.1 (clarkb 20230627) *** https://review.opendev.org/c/opendev/system-config/+/887006?usp=dashboard *** The commit tagged 1.9.1 doesn't actually build due to lock errors. These were fixed in followup commits. Hopefully we get a 1.9.2. * Open discussion -- Jeremy Stanley

1 0

Team Meeting Agenda for June 27, 2023
by Clark Boylan 27 Jun '23

27 Jun '23

We will meet with this Agenda on June 27, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Clarkb unable to attend he team meeting on July 4. * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20230627) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230627) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit replication leaked tasks (clarkb 20230627) *** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 **** https://review.opendev.org/c/opendev/system-config/+/884779 Revert bind mounts for Gerrit plugin data ** Gerrit reject implict merge changes (clarkb 20230627) *** We merged a revert of a change on branch foo to branch bar which implicitly merged foo into bar *** Gerrit project config receive.rejectImplicitMerges can be set to true to prevent this changes from being pushed *** Should we go ahead and set that on All-Projects? The default value is to inherit from the parent config. ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** CI registry is done. (Cleanup of old host still needs to be done). *** Next up mirrors, meetpad, and ci registry. *** Please grab a node or two. *** Zuul installation being updated as Jammy enables use of Podman ** Fedora cleanup (clarkb 20230627) *** Proposed the removal of Fedora mirrors and/or images https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… **** Tonyb has been looking at moving existing fedora test nodes off of our mirrors. Not straightforward as it requires zuul-jobs role updates. ** Quo vadis storyboard (frickler 20230627) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG ** Gitea 1.20 (clarkb 20230627) *** https://review.opendev.org/c/opendev/system-config/+/886993?usp=dashboard *** https://github.com/go-gitea/gitea/issues/25377 ** Etherpad 1.9.1 (clarkb 20230627) *** https://review.opendev.org/c/opendev/system-config/+/887006?usp=dashboard *** The commit tagged 1.9.1 doesn't actually build due to lock errors. These were fixed in followup commits. Hopefully we get a 1.9.2. * Open discussion

1 0