- service-discuss - lists.opendev.org

Team Meeting Agenda for April 23, 2024
by Clark Boylan 23 Apr '24

23 Apr '24

We will meet with this agenda on April 23, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Etherpad and mailman are done. Gerrit and Gitea are all that remain. ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial is next but currently busy with PTG, Release, and other tasks. *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Building Ubuntu Noble Nodes (clarkb 20240416) *** https://review.opendev.org/c/opendev/glean/+/915907 Glean updates for python3.12 support *** Appears we will need a debootstrap from debian unstable to build noble ** Etherpad 2.0.x Upgrade planning (clarkb 20240416) *** https://review.opendev.org/c/opendev/system-config/+/914119 *** Etherpad 2.0.2 removes the old APIKEY API auth method and now uses Oauth2.0 client_credentials instead. This new system is not yet released, but we are working to determine how to use it pre release. ** Add reference to the project-team-guide in the fail ci msg (fungi 20240423) *** https://review.opendev.org/914189 ** Gerrit 3.9 Upgrade Planning (clarkb 20240423) *** https://www.gerritcodereview.com/3.9.html Release Notes *** Downgrade is possible with an offline reindex using the old war. *** Need to compile an upgrade checklist (go over breaking changes and also test the downgrade). *** Server could stand to be upgraded as well but I think it is probably best to decouple these concerns. ** Wiki SSL Cert Renewal (clarkb 20240423) *** We have just under 4 weeks to renew the cert. This is one of the last (maybe the only) manually managed ssl cert. *** Will probably delay until we have less than a week left before processing it. * Open discussion

1 0

Team Meeting Agenda for April 16, 2024
by Clark Boylan 16 Apr '24

16 Apr '24

== Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Etherpad, Gitea, Gerrit, and Mailman could use upgrades. *** https://review.opendev.org/c/opendev/system-config/+/911000 Upgrade etherpad mariadb to 10.11 *** https://review.opendev.org/c/opendev/system-config/+/915183 Upgrade mailman3 mariadb to 10.11 ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial is next but currently busy with PTG, Release, and other tasks. *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Building Ubuntu Noble Nodes (clarkb 20240416) *** https://review.opendev.org/c/opendev/glean/+/915907 Glean updates for python3.12 support *** Appears we will need a debootstrap from debian unstable to build noble ** Review02 had an oops last night (clarkb 20240326) *** Found the server was shutdown. After giving it a few minutes to potentially resolve itself (mostly worried about cloud action) clarkb proceeded to manually start the instance then start the containers. *** mnaser reports it may have been an OOM event on the hosting side. ** Project Renames (clarkb 20240227) *** https://review.opendev.org/c/opendev/system-config/+/911622 Move gerrit replication queue aside during project renames. *** Need to set a hard time for the 19th and send an announcement. ** Etherpad 2.0.x Upgrade planning (clarkb 20240416) *** https://review.opendev.org/c/opendev/system-config/+/914119 *** Etherpad 2.0.2 removes the old APIKEY API auth method and now uses Oauth2.0 client_credentials instead. This new system is not yet released, but we are working to determine how to use it pre release. * Open discussion

1 1

Cancelling Tomorrow's (April 9, 2024) Team Meeting
by Clark Boylan 09 Apr '24

09 Apr '24

At a previous meeting several people indicated a preference to cancel this week's meeting due to the PTG. Let's make that official and go ahead and say the meeting is cancelled tomorrow April 9, 2024. You can always bring up discussion on this mailing list or on IRC in #opendev for items that need to be discussed broadly before next week. Clark

1 0

Team Meeting Agenda for March 26, 2024
by Clark Boylan 27 Mar '24

27 Mar '24

We will meet with this agenda on March 26, 2024 (today) at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack Release next week then PTG the week after ** Put your PTG agenda items on the etherpad: https://etherpad.opendev.org/p/apr2024-ptg-opendev * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Etherpad, Gitea, Gerrit, and Mailman could use upgrades. *** https://review.opendev.org/c/opendev/system-config/+/911000 Upgrade etherpad mariadb to 10.11 ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial is next but currently busy with PTG, Release, and other tasks. *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Rebuilding Gerrit Images (clarkb 20240312) *** Gerrit 3.9.2 has been released finally. *** https://review.opendev.org/c/opendev/system-config/+/912470 Update our 3.9 image to 3.9.2 **** This will also rebuild our 3.8.4 image so we should try and restart prod gerrit on the new 3.8.4 image when available. *** Sounds like there are a number of bugfixes that a rebuild will get us. May be worth doing this just after the openstack release completes? ** Review02 had an oops last night (clarkb 20240326) *** Found the server was shutdown. After giving it a few minutes to potentially resolve itself (mostly worried about cloud action) clarkb proceeded to manually start the instance then start the containers. *** mnaser reports it may have been an OOM event on the hosting side. ** Rackspace MFA Requirement (clarkb 20240312) *** MFA is enabled. Enforcement day is today. Please lookout for any issues. ** Project Renames (clarkb 20240227) *** https://review.opendev.org/c/opendev/system-config/+/911622 Move gerrit replication queue aside during project renames. *** Penciled in April 19, 2024 submit your rename requests now. ** Nodepool image delete after upload (clarkb 20240319) *** Nodepool now has the ability to delete on disk files for images after they are uploaded. We could potentially keep only small qcow2s using this functionality to save disk space. * Open discussion

1 0

Team Meeting Agenda for March 19, 2024
by Clark Boylan 19 Mar '24

19 Mar '24

We will meet with this agenda on March 19, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Put your PTG agenda items on the etherpad: https://etherpad.opendev.org/p/apr2024-ptg-opendev * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Etherpad, Gitea, Gerrit, Refstack, and Mailman could use upgrades. *** https://review.opendev.org/c/opendev/system-config/+/911000 Upgrade etherpad mariadb to 10.11 *** https://review.opendev.org/c/opendev/system-config/+/910999 Upgrade refstack mariadb to 10.11 ** AFS Mirror cleanups (clarkb 20240220) *** CentOS 7 is now gone. Next up is Ubuntu Xenial *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Rebuilding Gerrit Images (clarkb 20240312) *** Gerrit 3.9.2 has been released finally. *** https://review.opendev.org/c/opendev/system-config/+/912470 Update our 3.9 image to 3.9.2 **** This will also rebuild our 3.8.4 image so we should try and restart prod gerrit on the new 3.8.4 image when available. ** Rackspace MFA Requirement (clarkb 20240312) *** We intend on manually enabling MFA on March 20, 2024 before the March 26, 2024 deadline. ** Project Renames (clarkb 20240227) *** https://review.opendev.org/c/opendev/system-config/+/911622 Move gerrit replication queue aside during project renames. *** Penciled in April 19, 2024 submit your rename requests now. ** Linaro cloud cert renewal (clarkb 20240319) *** The cert for the cloud APIs expires on April 15 and will need to be manually rotated before then. ** Nodepool image delete after upload (clarkb 20240319) *** Nodepool now has the ability to delete on disk files for images after they are uploaded. We could potentially keep only small qcow2s using this functionality to save disk space. * Open discussion

1 0

Team Meeting Agenda for March 12, 2024
by Clark Boylan 12 Mar '24

12 Mar '24

We will meet with this agenda on March 12, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** DST changes are starting to occur for some of us. Double check your timezone hasn't moved relative to the 19:00 UTC meeting time. * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Etherpad, Gitea, Gerrit, Refstack, and Mailman could use upgrades. *** https://review.opendev.org/c/opendev/system-config/+/911000 Upgrade etherpad mariadb to 10.11 *** https://review.opendev.org/c/opendev/system-config/+/910999 Upgrade refstack mariadb to 10.11 ** AFS Mirror cleanups (clarkb 20240220) *** CentOS 7 removal is in progress: topic: drop-centos-7 **** Announced that we would finalize CentOS 7 removal on March 15 to give OpenStack time to clean stuff up and avoid unexpected release problems *** When removing Debian mirror content afs01.dfw.openstack.org lost a volume backing /dev/vicepa and had a big sad. Not sure if related or coincidence. *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Rebuilding Gerrit Images (clarkb 20240312) *** Gerrit 3.9.2 has been released finally. *** https://review.opendev.org/c/opendev/system-config/+/912470 Update our 3.9 image to 3.9.2 **** This will also rebuild our 3.8.4 image so we should try and restart prod gerrit on the new 3.8.4 image when available. ** Rackspace MFA Requirement (clarkb 20240312) *** Rackspace is requiring all accounts switch to multi factor auth March 26, 2024 *** We moved our clouds.yaml auth over to `rackspaceauth` using an `api_key` as normal auth through the API will no longer work after MFA is enabled *** Still an open question if swift log uploads for zuul jobs need changing, but indications are they already use dedicated credentials which are unlikely to be affected. *** Should we go ahead and opt into MFA now ahead of the deadline so that we can control when our API interactions will be impacted? ** Project Renames (clarkb 20240227) *** https://review.opendev.org/c/opendev/system-config/+/911622 Move gerrit replication queue aside during project renames. *** Penciled in April 19, 2024 submit your rename requests now. ** PTG Planning (clarkb 20240305) *** https://ptg.opendev.org/ptg.html *** Times have been selected. Feel free to add agenda items here: https://etherpad.opendev.org/p/apr2024-ptg-opendev * Open discussion

1 0

Team Meeting Agenda for March 5, 2024
by Clark Boylan 05 Mar '24

05 Mar '24

We will meet with this Agenda on March 5, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Etherpad, Gitea, Gerrit, Refstack, and Mailman could use upgrades. *** https://review.opendev.org/c/opendev/system-config/+/911000 Upgrade etherpad mariadb to 10.11 *** https://review.opendev.org/c/opendev/system-config/+/910999 Upgrade refstack mariadb to 10.11 ** AFS Mirror cleanups (clarkb 20240220) *** OpenSUSE Leap and Debian Buster are gone now *** Have started with CentOS 7 removal: topic: drop-centos-7 **** Announced that we would finalize CentOS 7 removal on March 15 to give OpenStack time to clean stuff up and avoid unexpected release problems *** When removing Debian mirror content afs01.dfw.openstack.org lost a volume backing /dev/vicepa and had a big sad. Not sure if related or coincidence. *** Can followup with webserver log processing to determine which other mirrors may be dead. ** OpenDev email hosting (clarkb 20240220) *** Would be good to come to a consensus on whether or not we want ot host this ourselves or just pay someone to do it. ** Project Renames (clarkb 20240227) *** Do we need to update the tooling to move the replication waiting queue aside during the rename process? *** Penciled in April 19, 2024 submit your rename requests now. ** PTG Planning (clarkb 20240305) *** https://ptg.opendev.org/ptg.html *** Very little schedule content is in place at this point. Clarkb proposes we do something like Wednesday 0400-0600 and Thursday 1400-1600UTC. Gives enough time between blocks to catch up on sleep. * Open discussion

1 0

Team Meeting Agenda for February 27, 2024
by Clark Boylan 27 Feb '24

27 Feb '24

We will meet with this agenda on February 27, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Service Coordinator Election Results (clarkb 20240227) *** Single nominee was clarkb *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Paste, Etherpad, Gitea, Gerrit, Refstack, and Mailman could use upgrades. Starting with Paste due to its simplicity. *** https://review.opendev.org/c/opendev/system-config/+/909471 Upgrade paste.o.o mariadb to 10.11 ** AFS Mirror cleanups (clarkb 20240220) *** OpenSUSE Leap has been removed **** topic:drop-opensuse *** Debian Buster is in progress **** topic:drop-buster *** CentOS 7 and Ubuntu Xenial still to do. *** Can followup with webserver log processing to determine which other mirrors may be dead. ** OpenDev email hosting (clarkb 20240220) *** Would be good to come to a consensus on whether or not we want ot host this ourselves or just pay someone to do it. ** Gitea 1.21.7 Upgrade (clarkb 20240227) *** https://review.opendev.org/c/opendev/system-config/+/909941 ** git-review vendoring the commit message hook (clarkb 20240227) *** https://review.opendev.org/c/opendev/git-review/+/910275 ** Project Renames (clarkb 20240227) *** Do we need to update the tooling to move the replication waiting queue aside during the rename process? * Open discussion

1 0

Clarkb's Service Coordinator Nomination for February 2024
by Clark Boylan 21 Feb '24

21 Feb '24

As previously mentioned on this list [0] it is Service Coordinator nomination time. In fact the nomination period ends about two hours after I send this email out. There have been no other nominees, and I am willing to continue operating in this capacity. Consider this my self nomination for the Service coordinator position. I continue to think it would be great to have others in the role, and I will happily support whoever that may be in the future should we have additional volunteers. Until that time, you can expect I'll continue to try and push for the sustainability of our services. We have been able to lean on Zuul, Ansible, Containers, and CI testing to reliably keep our services up to date, and we should continue to do this or similar. The consistent deployment practices reduces mental overhead understanding how things operate, and our CI testing gives us confidence when we make changes. [0] https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… Thank you for your consideration, Clark

1 0

Team Meeting Agenda for February 20, 2024
by Clark Boylan 20 Feb '24

20 Feb '24

We will meet with this agenda on February 20, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Service coordinator nominations open February 6, 2024 - February 20, 2024 *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… *** No nominees as of the time the agenda was put together. Nominations stay open 4 hours beyond the end of our meeting. * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** https://review.opendev.org/c/opendev/system-config/+/908512 Ensure new servers have sufficient CPU version levels. *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 *** Hound looks to be a fairly simple service with indexed/ached data on disk. It's currently running Focal with "bookworm" containers Any reason to stay there? *** I think there was a suggestion we could implement prometheus and once were happy with that drop cacti, Is that correct? ** PrePTG Summary (clarkb 20240220) *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… *** Code review practices are worth reviewing for accuracy *** Happy to followup on other topics or have others write up summaries from their perspective. ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Paste, Etherpad, Gitea, Gerrit, Refstack, and Mailman could use upgrades. Starting with Paste due to its simplicity. ** AFS Mirror cleanups (clarkb 20240220) *** We will remove OpenSUSE Leap, Debian Buster, CentOS 7, and Ubuntu Xenial *** Can followup with webserver log processing to determine which other mirrors may be dead. ** OpenDev email hosting (clarkb 20240220) *** Would be good to come to a consensus on whether or not we want ot host this ourselves or just pay someone to do it. * Open discussion

1 0