August 2023 - service-discuss - lists.opendev.org

Second 2023 Election for OpenDev Service Coordinator
by Clark Boylan 30 Aug '23

30 Aug '23

It is time to elect our OpenDev service coordinator for the next six months. As mentioned during the last weekly team meeting we will open the nomination period from August 8, 2023 to August 22, 2023 using UTC as the timezone. If we have more than one nominee in that time frame we can run an election between the candidates from August 23, 2023 to August 30, 2023 (again using UTC as the timezone). If you are interested in running, feel free to reach out to me for more information or with any questions you may have. The OpenDev governance documentation [0] also has information about the role and its responsibilities. [0] https://docs.opendev.org/opendev/system-config/latest/project.html#governan… Clark

2 2

Team meeting agenda for August 29, 2023
by Clark Boylan 29 Aug '23

29 Aug '23

We will meet with this agenda on August 29, 2023 at 19:00UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Service Coordinator Election (clarkb 20230822) *** clarkb was the only nominee. ** Bastion host (ianw 20230627) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230627) *** https://etherpad.opendev.org/p/mm3migration *** All outstanding changes to fix vhosting, upgrade mailman, and clean up config management have landed. *** Time to schedule some list domain migrations. ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Next up mirrors and meetpad. *** Clarkb needs to look into cleaning up the old CI registry node. ** Rax IAD image upload struggles (clarkb 20230822) *** The Rax IAD region has struggled to upload images within the image upload timeout. This seems to get worse the more images you upload **** Almost like images are processed serially and we create a queue that gets backed up. *** No issue has been filed with rax yet, but we're testing incremental unpausing of images to get more data. *** We also have hundreds of leaked images in the other two regions (there were well over a thousand in IAD but those have been cleaned up manually now). ** Fedora cleanup (clarkb 20230627) *** https://review.opendev.org/c/opendev/base-jobs/+/892380 Remove the fedora-latest nodeset ** Zuul Ansible 8 default *** Email sent to announce the common default updates https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… *** Every tenant but OpenStack is running Ansible 8 by default now. *** Plan to do OpenStack on September 4 (this date happens to be a US holiday but I can probably be around to shepherd things through). ** Python container updates (tonyb 20230718) *** https://review.opendev.org/q/hashtag:bookworm+status:open Next round of image rebuilds onto bookworm. *** Containers like gitea aren't based on our base image but should be updated too. * Open discussion

1 0

Mailman 3 service upgrades
by Jeremy Stanley 25 Aug '23

25 Aug '23

A few minutes ago we upgraded the production Mailman 3 server (currently hosting this lists.opendev.org domain and also lists.zuul-ci.org) to the latest releases of the various Mailman component applications. If you notice any problems you think might be related to the upgrade, please do let us know here on the service-discuss mailing list or in the #opendev channel of the OFTC IRC network. -- Jeremy Stanley on behalf of the OpenDev Collaboratory sysadmins

1 0

Clarkb's Service Coordinator Self Nomination
by Clark Boylan 23 Aug '23

23 Aug '23

We have just about reached the end of the OpenDev Service Coordination nomination period, and I haven't seen any nominations. I brought this up in our weekly team meeting just now, and it sounds like people are happy for me to continue in the role. Given that feedback I nominate myself to serve another six months as OpenDev Service Coordinator. Expect more of the same direction from me. I'm hoping we'll continue upgrading older servers, move our container images to Bookworm, and maybe eventually get some stuff running on podman so that we can revive quay.io hosting of our images. If you'd like something different your window of opportunity is quickly fading as we said August 22 would be the end of the nomination period. To be clear, I'd be happy to support someone else in the role if they want to give it a go. Thank you for your consideration, Clark

1 0

Team Meeting Agenda for August 22, 2023
by Clark Boylan 22 Aug '23

22 Aug '23

We will meet with this agenda on August 22, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Service Coordinator Election (clarkb 20230822) *** Nomination period ends today (the day of the meeting) ** Bastion host (ianw 20230627) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230627) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit updates (clarkb 20230718) *** Replication task file leaks. **** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 **** https://review.opendev.org/c/opendev/system-config/+/884779 Revert bind mounts for Gerrit plugin data ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Next up mirrors and meetpad. *** Clarkb needs to look into cleaning up the old CI registry node. ** Rax IAD image upload struggles (clarkb 20230822) *** The Rax IAD region has struggled to upload images within the image upload timeout. This seems to get worse the more images you upload **** Almost like images are processed serially and we create a queue that gets backed up. *** No issue has been filed with rax yet, but we're testing incremental unpausing of images to get more data. *** We also have hundreds of leaked images in the other two regions (there were well over a thousand in IAD but those have been cleaned up manually now). ** Fedora cleanup (clarkb 20230627) *** Need changes to remove the label and nodeset from Zuul/Nodepool ** Gitea 1.20 (clarkb 20230627) *** https://review.opendev.org/c/opendev/system-config/+/886993 **** Ready to go if we are happy with the resulting behaviors. *** We disable oauth2 but need to configure multiple oauth2 jwt settings in order to avoid startup errors. **** Need to update prod secrets with the oauth2 jwt settings before merging. *** https://github.com/go-gitea/gitea/issues/25377 ** Zuul changes and updates (clarkb 20230822) *** Ansible 8 now availalble. We should convert the OpenDev tenant. Notify openstack tenant that we will switch them after the upcoming release *** Stdout and stderr to be uncombined in Zuul Ansible. Currently this is a job setting for testing purposes. We'll need to start looking at how this impacts us. *** Early failure detection regexes should arrive soon. This is potentially useful for our users? ** Python container updates (tonyb 20230718) *** Some moderate progress in the first round *** https://review.opendev.org/q/topic:bookworm-python and https://review.opendev.org/q/topic:force-base-image-build *** I noticed that https://opendev.org/openstack/python-openstackclient/src/branch/master/Dock… is using a 3.9 tag which is buster **** We should stick to the explicit debian release + explicit python release versions for things. We can delete the old tags to force errors that indicate people should update the tags they use. *** We should write a tool to find and clean leaked tags on docker hub (and quay.io?) * Open discussion

1 0

Team meeting Agenda for August 15, 2023
by Clark Boylan 15 Aug '23

15 Aug '23

We will meet on August 15, 2023 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20230627) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230627) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit updates (clarkb 20230718) *** Replication task file leaks. **** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 **** https://review.opendev.org/c/opendev/system-config/+/884779 Revert bind mounts for Gerrit plugin data ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Next up mirrors and meetpad. *** Clarkb needs to look into cleaning up the old CI registry node. ** Fedora cleanup (clarkb 20230627) *** Can we accelerate this removal now that upstream has killed the package mirrors? We should be able to simply remove the nodes now? ** Gitea 1.20 (clarkb 20230627) *** https://review.opendev.org/c/opendev/system-config/+/886993 **** Updating log collection so that we can double check changes to the access logs that are listed in the breaking changes list. *** We disable oauth2 but need to configure multiple oauth2 jwt settings in order to avoid startup errors. *** https://github.com/go-gitea/gitea/issues/25377 ** Etherpad 1.9.1 (clarkb 20230627) *** https://review.opendev.org/c/opendev/system-config/+/887006 *** The held node running the latest version of this change appears to work for clarkb. Others should test and review the change appropriately. ** Python container updates (tonyb 20230718) *** Some moderate progress in the first round *** https://review.opendev.org/q/topic:bookworm-python and https://review.opendev.org/q/topic:force-base-image-build *** I noticed that https://opendev.org/openstack/python-openstackclient/src/branch/master/Dock… is using a 3.9 tag which is buster **** We should stick to the explicit debian release + explicit python release versions for things. We can delete the old tags to force errors that indicate people should update the tags they use. *** We should write a tool to find and clean leaked tags on docker hub (and quay.io?) * Open discussion

1 0

Team Meeting Agenda for August 1, 2023
by Clark Boylan 01 Aug '23

01 Aug '23

We will meet with this agenda on August 1, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20230627) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230627) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit updates (clarkb 20230718) *** Replication task file leaks. **** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 **** https://review.opendev.org/c/opendev/system-config/+/884779 Revert bind mounts for Gerrit plugin data ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Next up mirrors and meetpad. *** Clarkb needs to look into cleaning up the old CI registry node. ** Fedora cleanup (clarkb 20230627) *** Proposed the removal of Fedora mirrors and/or images https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… **** Tonyb has been looking at moving existing fedora test nodes off of our mirrors. Not straightforward as it requires zuul-jobs role updates. ** Quo vadis storyboard (frickler 20230627) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG ** Gitea 1.20 (clarkb 20230627) *** https://review.opendev.org/c/opendev/system-config/+/886993 **** Updating log collection so that we can double check changes to the access logs that are listed in the breaking changes list. *** We disable oauth2 but need to configure multiple oauth2 jwt settings in order to avoid startup errors. *** https://github.com/go-gitea/gitea/issues/25377 ** Etherpad 1.9.1 (clarkb 20230627) *** https://review.opendev.org/c/opendev/system-config/+/887006 *** The held node running the latest version of this change appears to work for clarkb. Others should test and review the change appropriately. ** Python container updates (tonyb 20230718) *** Some moderate progress in the first round *** https://review.opendev.org/q/topic:bookworm-python and https://review.opendev.org/q/topic:force-base-image-build *** I noticed that https://opendev.org/openstack/python-openstackclient/src/branch/master/Dock… is using a 3.9 tag which is buster **** We should stick to the explicit debian release + explicit python release versions for things. We can delete the old tags to force errors that indicate people should update the tags they use. *** We should write a tool to find and clean leaked tags on docker hub (and quay.io?) ** Meetpad LE cert refresh (clarkb 20230725) *** LE refreshed the cert for meetpad but the handler that copies the file and restarts services does not appear to have run. *** Ideally we would manually update acme to trigger a rerun and debug from there. * Open discussion

1 0