April 2026 - service-discuss - lists.opendev.org

Team Meeting Agenda for April 14, 2026
by Clark Boylan 13 Apr '26

13 Apr '26

We will meet with this agenda on April 14, 2026 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Clarkb will be out on Friday * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** mnasiadka has been working to replace some of the older mirror nodes. Please look out for changes related to this effort. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Dealing with web crawlers (clarkb 20251216) *** We have deployed anubis to lists.opendev.org and it seems to be working well enough there *** We have also deployed anubis on the gitea servers. This was after spending much of last week fighting the crawler flood **** Would be good to look at getting PROXY protocol support working with gitea to make debugging easier *** Should we be looking at adding anubis to other services like static? ** Deploying a Prometheus for Server Metrics (clarkb 20260331) *** https://review.opendev.org/c/opendev/system-config/+/980840 *** This change and its child deploy prometheus with node exporter to collect server metrics *** Napkin math says that a 1TB volume should get us about 60 days of metrics. mnasiadka also indicates that Prometheus doesn't handle longer term metrics super well *** Ideally we would collect at least a years' worth of data. Can we make that happen with Prometheus? *** Do we need to look at Prometheus adjacent tools like Mimir or Thanos? **** Both of these solutions seem to tie into Prometheus using Prometheus as the data collection system. Then they store the data in a different system which can handle long term storage more nimbly. Then for queries they speak promql and prometheus apis allowing you to point tools like grafana at them as if they were prometheus. ** Upgrade Ansible to v9 (clarkb 20260310) *** https://docs.ansible.com/projects/ansible/latest/reference_appendices/relea… *** https://review.opendev.org/c/opendev/system-config/+/976282 *** Based on Ansible's python support Matrix Ansible 9 gives us a good deal of flexibility for bridge and remote nodes *** Ansible 9 also fixes problems with the use of pkg_resources in the Ansible ip module *** Any concerns with proceeding with the upgrade since tests look good? ** Gerrit Account Cleanups (clarkb 20260317) *** Since the upgrade to Gerrit notedb we've had account inconsistencies that prevent us from push to the external ids ref/table directly. *** clarkb did a bunch of work to get the number down from hundreds to about 33 consistency errors before stalling out. *** The tail was the most difficult as it wasn't clear what the more appropriate fix for each account would be *** Since then it has been years and those accounts are likely inactive and unused. We can rerun the Gerrit consistency check, feed the info back through our audit script then decide if we need to be careful with any of these accounts *** Chances are we can simply disable them all and remove the conflicting external ids. *** If we take good notes we can reconstruct the accounts as appropriate after the fact without Gerrit downtime should one of these users show up and wonder what happened. ** Gerrit 3.12 Upgrade Followup (clarkb 20260310) *** https://review.opendev.org/q/hashtag:upgrade-gerrit-3.12+status:open Followup changes for Gerrit 3.12 upgrade here. *** Things seem to have gone well. In addition to the changes above there are a few things we should plan to do **** Monitor H2 v2 cache file sizes and get a sense for whether or not they grow like H2 v1 cache files did **** Clean up files that were created during the upgrade process. In particular the old H2 v1 caches. ** Gerrit 3.13 Upgrade Planning (clarkb 20260414) *** Due to the PTG next week and travel the week after I am targetting the 3.13 upgrade to end of May/early June *** Gerrit 3.13 adds an unconfigurable AI Code Review helper button to changes **** https://groups.google.com/g/repo-discuss/c/duY8pKj3qBg discusses this a bit more *** Gerrit 3.13 removes support for Robot comments so Zuul will start making normal inline comments *** This also means that the Zuul restarts performed as part of the upgrade process are actually required when we upgrade to 3.13 to get Zuul's Gerrit version detection sorted out. ** Ubuntu Resolute Test Nodes (clarkb 20260331) *** https://review.opendev.org/c/openstack/diskimage-builder/+/982231 Add Resolute testing to dib *** https://review.opendev.org/c/opendev/zuul-providers/+/982182 Add Resolute images to Zuul *** Ubuntu Bionic mirror content has been removed. We can probably start the process of mirroring Resolute packages. ** OpenInfra PTG Prep (clarkb 20260331) *** The next PTG is happening April 20-24 which is next week. *** We will want to put meetpad and jvm nodes in the emergency file prior to the event to prevent unwanted upgrade disruptions. *** Is there any other prep work that we think should be done ahead of the event? ** Noble Docker Not Talking to Podman Socket for all Operations (clarkb 20260414) *** During the Gerrit 3.12 upgrade we noticed that `docker image ls` doesn't work on Noble nodes anymore due to API version support mismatches between Docker and Podman *** `podman image ls` does work just fine and is what we used *** The problem appears to be due to noble-updates upgrading the docker.io package compared to noble proper *** The problem does not appear to affect all docker subcommands `docker ps -a` works just fine. *** Please keep an eye out for problems in configuration management caused by this. * Open discussion

1 0

Opendev data erasure request - from Robert Stofan - request: M6OC4FQV
by bobbystofan454＠gmail.com 10 Apr '26

10 Apr '26

Hi Opendev (opendev.org) My name is Robert Stofan, I've used your service in the past. However, I'm now making the conscious decision to reduce my digital footprint and as a result I ask you to please delete any personal data of mine you have stored on your systems. I have initiated this request myself and it was sent from my own personal inbox (see From address of this email). Would appreciate your cooperation and an email confirming when the deletion has been completed. My personal details are: - Name: Robert Stofan - Email: bobbystofan454(a)gmail.com Context: I received an email from your company on *2024 April 04* which indicates your systems hold my personal data. Thanks, Robert Stofan Sent using McAfee® Online Account Cleanup Request ID: M6OC4FQV

1 0

Team Meeting Agenda for April 7, 2026
by Clark Boylan 06 Apr '26

06 Apr '26

We will meet with this agenda on April 7, 2026 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** mnasiadka has been working to replace some of the older mirror nodes. Please look out for changes related to this effort. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Dealing with web crawlers (clarkb 20251216) *** We have deployed anubis to lists.opendev.org and it seems to be working well enough there *** https://review.opendev.org/c/opendev/system-config/+/983061 Anubis deployment for Gitea **** This is a bit more complicated than lists due to how Gitea was configured with apache. There is a refactoring change prior to the anubis change that makes it more similar to other services. Specifically Gitea will no longer terminal ssl itself and all http traffic will go through Apache. ** Deploying a Prometheus for Server Metrics (clarkb 20260331) *** https://review.opendev.org/c/opendev/system-config/+/980840 *** This change and its child deploy prometheus with node exporter to collect server metrics *** Napkin math says that a 1TB volume should get us about 60 days of metrics. mnasiadka also indicates that Prometheus doesn't handle longer term metrics super well *** Ideally we would collect at least a years' worth of data. Can we make that happen with Prometheus? *** Do we need to look at Prometheus adjacent tools like Mimir or Thanos? **** Both of these solutions seem to tie into Prometheus using Prometheus as the data collection system. Then they store the data in a different system which can handle long term storage more nimbly. Then for queries they speak promql and prometheus apis allowing you to point tools like grafana at them as if they were prometheus. ** Upgrade Ansible to v9 (clarkb 20260310) *** https://docs.ansible.com/projects/ansible/latest/reference_appendices/relea… *** https://review.opendev.org/c/opendev/system-config/+/976282 *** Based on Ansible's python support Matrix Ansible 9 gives us a good deal of flexibility for bridge and remote nodes *** Ansible 9 also fixes problems with the use of pkg_resources in the Ansible ip module *** Any concerns with proceeding with the upgrade since tests look good? ** Gerrit Account Cleanups (clarkb 20260317) *** Since the upgrade to Gerrit notedb we've had account inconsistencies that prevent us from push to the external ids ref/table directly. *** clarkb did a bunch of work to get the number down from hundreds to about 33 consistency errors before stalling out. *** The tail was the most difficult as it wasn't clear what the more appropriate fix for each account would be *** Since then it has been years and those accounts are likely inactive and unused. We can rerun the Gerrit consistency check, feed the info back through our audit script then decide if we need to be careful with any of these accounts *** Chances are we can simply disable them all and remove the conflicting external ids. *** If we take good notes we can reconstruct the accounts as appropriate after the fact without Gerrit downtime should one of these users show up and wonder what happened. ** Gerrit 3.12 and 3.13 Upgrade Planning (clarkb 20260310) *** Gerrit 3.12 upgrade will happen April 12 at 20:00 UTC. This is a Sunday. *** https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… *** https://etherpad.opendev.org/p/gerrit-upgrade-3.12 3.12 Upgrade planning document *** Prep changes for Gerrit have landed and Gerrit was restarted on them. Work is ongoing to finish preparations and testing before Sunday. *** Will attempt Gerrit 3.13 upgrade late May/early June. ** Ubuntu Resolute Test Nodes (clarkb 20260331) *** https://review.opendev.org/c/openstack/diskimage-builder/+/982231 Add Resolute testing to dib *** https://review.opendev.org/c/opendev/zuul-providers/+/982182 Add Resolute images to Zuul *** Drop Ubuntu Bionic mirror data to make room for Resolute ** OpenInfra PTG Prep (clarkb 20260331) *** The next PTG is happening April 20-24. *** We will want to put meetpad and jvm nodes in the emergency file prior to the event to prevent unwanted upgrade disruptions. *** Is there any other prep work that we think should be done ahead of the event? * Open discussion

1 0