December 2025 - service-discuss

Cancelling our Team Meetings December 23 and December 30, 2025
by Clark Boylan 20 Dec '25

20 Dec '25

Hello, Things seem to have slowed down quite a bit with the end of the year holidays. I'm going to go ahead and cancel our next meeting on December 23. We had previously decided to cancel the December 30 meeting as well. We will be back January 6, 2026. See you then. If anything comes up between now and then feel free to start a thread on this mailing list or jump onto IRC and bring it up there. Clark

1 0

Team Meeting Agenda for December 16, 2025
by Clark Boylan 16 Dec '25

16 Dec '25

We will meet with this agenda on December 16, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** December 30 meeting will be cancelled. Other meetings may be cancelled depending on attendance * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Cleanup Tasks (clarkb 20250401) *** Gerrit 3.11 supports Java 21 and Gerrit 3.12 requires Java 21 **** The next big Gerrit todo item is to update our container image to Trixie and Java 21. This will enable us to build and test Gerrit 3.12 as well **** Other containers have updated to Trixie and seem to be working well. **** https://review.opendev.org/c/opendev/system-config/+/970160 Switches Gerrit to Java 21 on Debian Trixie *** https://review.opendev.org/q/hashtag:%22upgrade-gerrit-3.11%22+status:open *** Additionally, testing of gerritlib and jeepyb is updated to improve coverage of those tools with newer Gerrit. Changes are in the list linked above. *** Hit an unexpected exception that appears to mostly just be noise: **** https://issues.gerritcodereview.com/issues/467103128 ** Rename project app-kubernetes-module-manager to app-kernel-module-management (clarkb 20251125) *** Rename happened December 12 at approximately 2000 UTC *** Discovered that gerrit2 doesn't own its own homedir impacting some Ansible tasks in the rename playbook **** https://review.opendev.org/c/opendev/system-config/+/970919 *** Also decided we should have manage-projects record logs the same way as other infra-prod jobs **** https://review.opendev.org/c/opendev/system-config/+/970920 ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** https://docs.opendev.org/opendev/infra-specs/latest/specs/matrix_for_opende… *** Created a new @status account for statusbot. **** Gave it a valid email address using a plus address suffix and account creation succeeded but recorded an error setting the email address *** Private vars should be updated and we can land the change to update statusbot configs whenever we like **** https://review.opendev.org/c/opendev/system-config/+/969328 ** zuul-registry Image Pruning of OCI Images (clarkb 20251216) *** We (re)discovered that speculative images in the intermediate registry seem to get pruned too soon *** Images older than 180 are expected to be pruned but these were ~2 days old and removed *** Investigation shows that the pruning routine handles docker image types properly but not oci image types. *** https://review.opendev.org/c/zuul/zuul-registry/+/970480/ This change fixes that and has a few parents/children addressing general CI concerns. ** Adding Bad Crawler Honeypots to our Sites (clarkb 20251216) *** As the web crawlers go crazy on the Internet more and more people notice the disruption they cause *** We can continue to make our services faster and bigger to keep up, but pushing against some of the demand is also a good idea *** One idea from fungi is that we set up a honeypot of invalid Disallowed data (via robots.txt) and anyone that fetches it gets blocked *** https://review.opendev.org/c/opendev/system-config/+/970674 Implement a PoC honeypot against docs.opendev.org * Open discussion

1 0

Team Meeting Agenda for December 9, 2025
by Clark Boylan 09 Dec '25

09 Dec '25

We will meet with this agenda on December 9, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** December 30 meeting will be cancelled. Other meetings may be cancelled depending on attendance * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade (clarkb 20250401) *** Upgrade happened on December 7 *** We are running Gerrit 3.11 on top of Debian Bookworm with Java 17 *** Gerrit 3.11 supports Java 21 and Gerrit 3.12 requires Java 21 **** The next big Gerrit todo item is to update our container image to Trixie and Java 21. This will enable us to build and test Gerrit 3.12 as well *** https://review.opendev.org/q/hashtag:%22upgrade-gerrit-3.11%22+status:open *** Additionally, testing of gerritlib and jeepyb is updated to improve coverage of those tools with newer Gerrit. Changes are in the list linked above. *** Hit an unexpected exception that appears to mostly just be noise: **** https://issues.gerritcodereview.com/issues/467103128 ** Rename project app-kubernetes-module-manager to app-kernel-module-management (clarkb 20251125) *** Fungi is looking at renaming this project on December 12 at 2100 UTC ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** https://docs.opendev.org/opendev/infra-specs/latest/specs/matrix_for_opende… *** Current blocker is adding a new bot account for the status bot on matrix *** EMS requires email for all newly created accounts. We can use a beta dashboard to do so without validating the email address *** Another option would be to use plus addressing to give bots working email addresses *** It isn't clear if we can use a single email address with multiple accounts ** Zuul Launcher Updates (clarkb 20251125) *** We found a corrupted image upload in raxflex sjc3 again on November 21. Previous occurrence was discovered on October 6 *** https://review.opendev.org/c/zuul/zuul/+/968090 Will validate the glance checksum against our checksum and reject mismatches **** It isn't clear to clarkb if this is safe as glance may change the checksum value if it does a format conversion on the backend? *** Floating IPs attached to held nodes were deleted in raxflex. Likely due to our leaked fip cleanup routine. Debugging has been added to the launcher to track this down. *** Clarkb will send email to Rackspace today to follow up on this incase it got lost during the holiday week at the end of November. * Open discussion

1 0

Team Meeting Agenda for December 2, 2025
by Clark Boylan 02 Dec '25

02 Dec '25

We will meet with this agenda on December 2, 2025 at 19:00 UTC in #opendev-meeting == Agenda for next meeting == * Announcements ** December 30 meeting will be cancelled. Other meetings may be cancelled depending on attendance * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… *** Upgrade is happening Sunday December 7, 2025 *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Held nodes here: **** 3.10: https://217.182.142.186 **** 3.11: server lost its floating ip again and needs a new one. *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out. ** Rename project app-kubernetes-module-manager to app-kernel-module-management (clarkb 20251125) *** Fungi indicates that he can try to drive the renaming week of December 8-12 after the Gerrit upgrade ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** https://docs.opendev.org/opendev/infra-specs/latest/specs/matrix_for_opende… *** The new room exists and gerritbot has joined it *** Statusbot needs to be configured for Matrix and we need to schedule a cut over day for eavesdrop and our usage of the room. *** https://review.opendev.org/q/hashtag:%22opendev-matrix%22+status:open ** Zuul Launcher Updates (clarkb 20251125) *** We found a corrupted image upload in raxflex sjc3 again on November 21. Previous occurrence was discovered on October 6 *** https://review.opendev.org/c/zuul/zuul/+/968090 Will validate the glance checksum against our checksum and reject mismatches **** It isn't clear to clarkb if this is safe as glance may change the checksum value if it does a format conversion on the backend? *** Floating IPs attached to held nodes were deleted in raxflex. Likely due to our leaked fip cleanup routine. Debugging has been added to the launcher to track this down. * Open discussion

1 0