- service-discuss - lists.opendev.org

Team Meeting Agenda for July 7, 2026
by Clark Boylan 06 Jul '26

06 Jul '26

We will meet with this agenda on July 7, 2026 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents *** Next on the list are graphite and backup servers *** backup03.ca-ymq-1.vexxhost.opendev.org has been launched and is being backed up too **** https://review.opendev.org/c/opendev/system-config/+/995420 Starting backup02 removal here *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Deploying a Prometheus for Server Metrics (clarkb 20260331) *** https://review.opendev.org/c/opendev/system-config/+/980840 *** This change and its child deploy prometheus with node exporter to collect server metrics *** These two changes simplify the setup and testing of prometheus and node exporter **** https://review.opendev.org/c/zuul/zuul-jobs/+/994564 manage /etc/hosts with public IPs **** https://review.opendev.org/c/opendev/system-config/+/994565 Use public IPs in system-config-run jobs ** Dealing with alien zuul config errors in the openstack tenant (frickler 20260617) *** clarkb reached out to starlingx and airship about this **** Someone from airship is in the opendev matrix room now and their zuul tenant config errors appear to be gone **** https://lists.starlingx.io/archives/list/starlingx-discuss@lists.starlingx.… ** Gitea 1.26.4 Upgrade (clarkb 20260622) *** https://review.opendev.org/c/opendev/system-config/+/994326 Upgrade Gitea to 1.26.4 *** Its time to upgrade to the next Gitea bugfix release ** Bump Anubis difficult to 5 (clarkb 20260630) *** There is some evidence that bots are regularly solving the Anubis challenge *** The challenges are slowing them down enough that services continue to be mostly responsive *** Should we increase the difficulty one level to slow them down even futher? *** This will impact regular users too which is likely the primary consideration we should make. *** https://review.opendev.org/c/opendev/system-config/+/995096 ** Planning Gerrit Project Renames (clarkb 20260622) *** We have a request to rename x/cursive to openstack/cursive *** Any concern with project ownership doing that? The current group membership includes people from Johns Hopkins University and OpenStack Barbican *** Aiming for July 9 at ~2100 UTC **** Clarkb is unlikely to make this time *** https://etherpad.opendev.org/p/opendev-project-renames-20260709 ** Matrix Meetbot Porting (clarkb 20260707) *** Anil has started a port of the IRC meetbot to Matrix *** https://review.opendev.org/c/opendev/system-config/+/995755 stack starts here ** OpenMetal Cloud Offline (clarkb 20260707) *** Zuul had errors talking to this cloud so the cloud was disabled *** `openstack server list` reports HTTP 500 errors * Open discussion

1 0

Team Meeting Agenda for June 30, 2026
by Clark Boylan 29 Jun '26

29 Jun '26

We will meet with this agenda on June 30, 2026 at 1900 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents *** Next on the list are graphite and backup servers *** backup03.ca-ymq-1.vexxhost.opendev.org has been launched and is being backed up to **** https://review.opendev.org/c/opendev/system-config/+/995420 Starting backup02 removal here *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Deploying a Prometheus for Server Metrics (clarkb 20260331) *** https://review.opendev.org/c/opendev/system-config/+/980840 *** This change and its child deploy prometheus with node exporter to collect server metrics *** These two changes simplify the setup and testing of prometheus and node exporter **** https://review.opendev.org/c/zuul/zuul-jobs/+/994564 manage /etc/hosts with public IPs **** https://review.opendev.org/c/opendev/system-config/+/994565 Use public IPs in system-config-run jobs ** Larger VM sizes for tests (corvus 20260618) *** corvus has been testing python 3.14 with zuul; zuul unit tests now use slightly more than 8GB under 3.14 *** We have 16gb nodes, but in two clouds, rax-classic and vexxhost, they have fewer vcpus than their 8gb counterparts, so we need to use 32gb nodes to compensate *** Are we okay with this? Alternatives? ** Dealing with alien zuul config errors in the openstack tenant (frickler 20260617) *** Currently there are still 185 zuul config errors in the openstack tenant, despite my year-long struggle to get rid of them. *** Most of these are from "alien" repos (74 airship, 29 starlingx) that I have no motivation to fix myself with my OpenStack hats on *** Efforts to motivate these projects to clean up their errors themselves have mostly failed *** I still believe that cleaning these up and being able to easily identify fresh errors is important for the healthyness of the CI setup as a whole *** One pretty strong action would be to move these repos into their own tenant(s) or a different shared one like opendev *** I acknowledge that without further work this would break their CI setup, but I'm questioning now whether that impact would be worse than the impact the current situation has on my ability to maintain the OpenStack CI *** Other ideas or opinions are welcome *** clarkb reached out to starlingx and airship about this **** Airship indicated they would like to avoid the extra work involved in setting up a separate tenant **** clarkb pointed out to them that they would need to fix their zuul config errors and be reachable via email or matrix at a bare minumum if we want to make that work. **** https://lists.starlingx.io/archives/list/starlingx-discuss@lists.starlingx.… ** Gitea 1.26.4 Upgrade (clarkb 20260622) *** https://review.opendev.org/c/opendev/system-config/+/994326 Upgrade Gitea to 1.26.4 *** Its time to upgrade to the next Gitea bugfix release ** Bump Anubis difficult to 5 (clarkb 20260630) *** There is some evidence that bots are regularly solving the Anubis challenge *** The challenges are slowing them down enough that services continue to be mostly responsive *** Should we increase the difficulty one level to slow them down even further? *** This will impact regular users too which is likely the primary consideration we should make. *** https://review.opendev.org/c/opendev/system-config/+/995096 ** Planning Gerrit Project Renames (clarkb 20260622) *** We have a request to rename x/cursive to openstack/cursive *** Any concern with project ownership doing that? The current group membership includes people from Johns Hopkins University and OpenStack Barbican *** Aiming for July 9 at ~2100 UTC * Open discussion

1 0

Team Meeting Agenda for June 23, 2026
by Clark Boylan 22 Jun '26

22 Jun '26

We will meet with this agenda on June 23, 2026 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents *** Next on the list are graphite and backup servers *** backup03.ca-ymq-1.vexxhost.opendev.org has been launched and is being backed up too **** clarkb checked paste's backups to the new server and they appear to be working there **** fungi moved wiki over to backup to this server from the old server **** I think we can probably start the tear down of the old server now *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Deploying a Prometheus for Server Metrics (clarkb 20260331) *** https://review.opendev.org/c/opendev/system-config/+/980840 *** This change and its child deploy prometheus with node exporter to collect server metrics *** mnasiadka has proposed we use GrepTimeDB in conjunction with Prometheus to collect metrics over a longer period of time ** Gerrit Account Cleanups (clarkb 20260317) *** Since the upgrade to Gerrit notedb we've had account inconsistencies that prevent us from push to the external ids ref/table directly. *** clarkb did a bunch of work to get the number down from hundreds to about 33 consistency errors before stalling out. *** The tail was the most difficult as it wasn't clear what the more appropriate fix for each account would be *** Since then it has been years and those accounts are likely inactive and unused. We can rerun the Gerrit consistency check, feed the info back through our audit script then decide if we need to be careful with any of these accounts *** Chances are we can simply disable them all and remove the conflicting external ids. *** If we take good notes we can reconstruct the accounts as appropriate after the fact without Gerrit downtime should one of these users show up and wonder what happened. ** Zuul reporting empty public_v6 addresses for test nodes (clarkb 20260519) *** Zuul is reporting public_v6 values of '' for test nodes that do have working ipv6 in clouds like rax classic and ovh *** This may be an openstack api bug, an openstacksdk bug, or a zuul-launcher bug. *** Be aware this may impact the behavior of some test jobs. *** We will need to dig into why this is happening to understand it better. ** Larger VM sizes for tests (corvus 20260618) *** corvus has been testing python 3.14 with zuul; zuul unit tests now use slightly more than 8GB under 3.14 *** We have 16gb nodes, but in two clouds, rax-classic and vexxhost, they have fewer vcpus than their 8gb counterparts, so we need to use 32gb nodes to compensate *** Are we okay with this? Alternatives? ** Dealing with alien zuul config errors in the openstack tenant (frickler 20260617) *** Currently there are still 185 zuul config errors in the openstack tenant, despite my year-long struggle to get rid of them. *** Most of these are from "alien" repos (74 airship, 29 starlingx) that I have no motivation to fix myself with my OpenStack hats on *** Efforts to motivate these projects to clean up their errors themselves have mostly failed *** I still believe that cleaning these up and being able to easily identify fresh errors is important for the healthyness of the CI setup as a whole *** One pretty strong action would be to move these repos into their own tenant(s) or a different shared one like opendev *** I acknowledge that without further work this would break their CI setup, but I'm questioning now whether that impact would be worse than the impact the current situation has on my ability to maintain the OpenStack CI *** Other ideas or opinions are welcome ** Etherpad 3.3.2 Upgrade (clarkb 20260622) *** https://review.opendev.org/c/opendev/system-config/+/994325 Upgrade Etherpad to 3.3.2 *** Held node at 50.56.159.16 with a test clarkb-test pad. ** Gitea 1.26.4 Upgrade (clarkb 20260622) *** https://review.opendev.org/c/opendev/system-config/+/994326 Upgrade Gitea to 1.26.4 *** Its time to upgrade to the next Gitea bugfix release ** Working with LF IT to get some help in OpenDev (clarkb 20260622) *** clarkb and fungi are meeting internally later this week to see if we can point others at the LF to work that OpenDev would like to get done *** Don't be surprised if you see new faces contributing and please welcome and help them ** Planning Gerrit Project Renames (clarkb 20260622) *** We have a request to rename x/cursive to openstack/cursive *** When we do want to try and do that? *** Any concern with project ownership doing that? The current group membership includes people from Johns Hopkins University and OpenStack Barbican * Open discussion

1 0

Team Meeting Agenda for June 16, 2026
by Jeremy Stanley 15 Jun '26

15 Jun '26

We will meet with this agenda on June 16, 2026 at 19:00 UTC in #opendev-meeting: * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents *** Next on the list are graphite and backup servers *** backup03.ca-ymq-1.vexxhost.opendev.org has been launched and is being backed up too **** Preliminary checks all looked good. Should double check that all expected hosts are backing up to this server and confirm that a backup is mountable and looks accurate. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Deploying a Prometheus for Server Metrics (clarkb 20260331) *** https://review.opendev.org/c/opendev/system-config/+/980840 *** This change and its child deploy prometheus with node exporter to collect server metrics *** mnasiadka has proposed we use GrepTimeDB in conjunction with Prometheus to collect metrics over a longer period of time ** Gerrit Account Cleanups (clarkb 20260317) *** Since the upgrade to Gerrit notedb we've had account inconsistencies that prevent us from push to the external ids ref/table directly. *** clarkb did a bunch of work to get the number down from hundreds to about 33 consistency errors before stalling out. *** The tail was the most difficult as it wasn't clear what the more appropriate fix for each account would be *** Since then it has been years and those accounts are likely inactive and unused. We can rerun the Gerrit consistency check, feed the info back through our audit script then decide if we need to be careful with any of these accounts *** Chances are we can simply disable them all and remove the conflicting external ids. *** If we take good notes we can reconstruct the accounts as appropriate after the fact without Gerrit downtime should one of these users show up and wonder what happened. ** Gerrit 3.13 Upgrade Followup (clarkb 20260414) *** We upgraded Gerrit last Friday to 3.13.6 from 3.12.7 *** https://etherpad.opendev.org/p/gerrit-upgrade-3.13 Upgrade plan and process that was used *** Seems to be working, but a few things have been noticed **** The lack of robot comment apis has created some questions. This was anticipated and we expect hard refreshes of browser clients to fix this **** The robot comment api removal also broke gertty **** corvus has suggested that Zuul stop using robot comments early as these comments effectively disappear post upgrade **** Custom "Your -> Foo" links in the web ui can be set to open in new tabs. This is configurable through your user preferences in Gerrit **** Some changes show up as conflicting with each other despite having a direct git tree relationship *** https://review.opendev.org/c/opendev/system-config/+/991945/ and children will drop Gerrit 3.12 images, add 3.14 images, and test a 3.13 to 3.14 upgrade just as soon as we're confident we won't rollback to 3.12. ** Upgrading Grafana (clarkb 20260609) *** https://review.opendev.org/c/opendev/system-config/+/991155 This change will upgrade grafana to the latest version *** https://23.253.20.165/ is a held node for testing purposes. Things look good to clarkb and we can probably proceed. ** Zuul reporting empty public_v6 addresses for test nodes (clarkb 20260519) *** Zuul is reporting public_v6 values of '' for test nodes that do have working ipv6 in clouds like rax classic and ovh *** This may be an openstack api bug, an openstacksdk bug, or a zuul-launcher bug. *** Be aware this may impact the behavior of some test jobs. *** We will need to dig into why this is happening to understand it better. * Open discussion -- Jeremy Stanley

1 0

Team Meeting Agenda for June 9, 2026
by Clark Boylan 08 Jun '26

08 Jun '26

We will meet with this agenda on June 9, 2026 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** clarkb will be out June 15 and 16. Fungi has volunteered to chair the team meeting on June 16 * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents *** Next on the list are graphite and backup servers *** backup03.ca-ymq-1.vexxhost.opendev.org has been launched and is being backed up too **** Preliminary checks all looked good. Should double check that all expected hosts are backing up to this server and confirm that a backup is mountable and looks accurate. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Deploying a Prometheus for Server Metrics (clarkb 20260331) *** https://review.opendev.org/c/opendev/system-config/+/980840 *** This change and its child deploy prometheus with node exporter to collect server metrics *** mnasiadka has proposed we use GrepTimeDB in conjunction with Prometheus to collect metrics over a longer period of time ** Gerrit Account Cleanups (clarkb 20260317) *** Since the upgrade to Gerrit notedb we've had account inconsistencies that prevent us from push to the external ids ref/table directly. *** clarkb did a bunch of work to get the number down from hundreds to about 33 consistency errors before stalling out. *** The tail was the most difficult as it wasn't clear what the more appropriate fix for each account would be *** Since then it has been years and those accounts are likely inactive and unused. We can rerun the Gerrit consistency check, feed the info back through our audit script then decide if we need to be careful with any of these accounts *** Chances are we can simply disable them all and remove the conflicting external ids. *** If we take good notes we can reconstruct the accounts as appropriate after the fact without Gerrit downtime should one of these users show up and wonder what happened. ** Gerrit 3.13 Upgrade Followup (clarkb 20260414) *** We upgraded Gerrit last Friday to 3.13.6 from 3.12.7 *** https://etherpad.opendev.org/p/gerrit-upgrade-3.13 Upgrade plan and process that was used *** Seems to be working, but a few things have been noticed **** The lack of robot comment apis has created some questions. This was anticipated and we expect hard refreshes of browser clients to fix this **** The robot comment api removal also broke gertty **** corvus has suggested that Zuul stop using robot comments early as these comments effectively disappear post upgrade **** Custom "Your -> Foo" links in the web ui can be set to open in new tabs. This is configurable through your user preferences in Gerrit **** Some changes show up as conflicting with each other despite having a direct git tree relationship *** https://review.opendev.org/c/opendev/system-config/+/991945/ and children will drop Gerrit 3.12 images, add 3.14 images, and test a 3.13 to 3.14 upgrade just as soon as we're confident we won't rollback to 3.12. ** Upgrading Grafana (clarkb 20260609) *** https://review.opendev.org/c/opendev/system-config/+/991155 This change will upgrade grafana to the latest version *** https://23.253.20.165/ is a held node for testing purposes. Things look good to clarkb and we can probably proceed. ** Zuul reporting empty public_v6 addresses for test nodes (clarkb 20260519) *** Zuul is reporting public_v6 values of '' for test nodes that do have working ipv6 in clouds like rax classic and ovh *** This may be an openstack api bug, an openstacksdk bug, or a zuul-launcher bug. *** Be aware this may impact the behavior of some test jobs. *** We will need to dig into why this is happening to understand it better. * Open discussion

1 0

Team Meeting agenda for June 2, 2026
by Clark Boylan 01 Jun '26

01 Jun '26

We will meet with this agenda on June 2, 2026 at 1900 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents *** Next on the list are graphite and backup servers *** backup03.ca-ymq-1.vexxhost.opendev.org has been launched and is being backed up too **** Preliminary checks all looked good. Should double check that all expected hosts are backing up to this server and confirm that a backup is mountable and looks accurate. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Deploying a Prometheus for Server Metrics (clarkb 20260331) *** https://review.opendev.org/c/opendev/system-config/+/980840 *** This change and its child deploy prometheus with node exporter to collect server metrics *** mnasiadka has proposed we use GrepTimeDB in conjunction with Prometheus to collect metrics over a longer period of time ** Gerrit Account Cleanups (clarkb 20260317) *** Since the upgrade to Gerrit notedb we've had account inconsistencies that prevent us from push to the external ids ref/table directly. *** clarkb did a bunch of work to get the number down from hundreds to about 33 consistency errors before stalling out. *** The tail was the most difficult as it wasn't clear what the more appropriate fix for each account would be *** Since then it has been years and those accounts are likely inactive and unused. We can rerun the Gerrit consistency check, feed the info back through our audit script then decide if we need to be careful with any of these accounts *** Chances are we can simply disable them all and remove the conflicting external ids. *** If we take good notes we can reconstruct the accounts as appropriate after the fact without Gerrit downtime should one of these users show up and wonder what happened. ** Gerrit 3.13 Upgrade Planning (clarkb 20260414) *** Upgrade is happening this Friday June 5, at 1600 UTC *** https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… *** Gerrit 3.13 removes support for Robot comments so Zuul will start making normal inline comments *** This also means that the Zuul restarts performed as part of the upgrade process are actually required when we upgrade to 3.13 to get Zuul's Gerrit version detection sorted out. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.13 Beginnings of an upgrade plan document ** Etherpad 3.2.0 Upgrade (clarkb 20260519) *** 3.0.0, 3.1.0, and 3.2.0 have been released *** 2.7.3 introduced session cleanups in the database. But this created OOMs. We have disabled this feature. It should be fixed in 3.2.0. *** https://github.com/ether/etherpad/blob/v3.2.0/CHANGELOG.md Big change appears to be the ability for etherpad to self update. We will not opt into this functionality and already have it disabled *** https://review.opendev.org/c/opendev/system-config/+/990531 ** Zuul reporting empty public_v6 addresses for test nodes (clarkb 20260519) *** Zuul is reporting public_v6 values of '' for test nodes that do have working ipv6 in clouds like rax classic and ovh *** This may be an openstack api bug, an openstacksdk bug, or a zuul-launcher bug. *** Be aware this may impact the behavior of some test jobs. *** We will need to dig into why this is happening to understand it better. * Open discussion

1 0

Team Meeting Agenda for May 26, 2026
by Clark Boylan 25 May '26

25 May '26

We will meet with this agenda on May 26, 2026 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents *** Next on the list are graphite and backup servers *** Backup server replacement is in progress. Should proceed soon. We'll then need to double check backups function as expected against the new server. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Deploying a Prometheus for Server Metrics (clarkb 20260331) *** https://review.opendev.org/c/opendev/system-config/+/980840 *** This change and its child deploy prometheus with node exporter to collect server metrics *** mnasiadka has proposed we use GrepTimeDB in conjunction with Prometheus to collect metrics over a longer period of time ** Upgrade Ansible to v9 (clarkb 20260310) *** https://docs.ansible.com/projects/ansible/latest/reference_appendices/relea… *** This has mostly gone well. *** https://review.opendev.org/c/opendev/system-config/+/989022 and https://review.opendev.org/c/opendev/system-config/+/988698/ should address the last known issue with the update. ** Gerrit Account Cleanups (clarkb 20260317) *** Since the upgrade to Gerrit notedb we've had account inconsistencies that prevent us from push to the external ids ref/table directly. *** clarkb did a bunch of work to get the number down from hundreds to about 33 consistency errors before stalling out. *** The tail was the most difficult as it wasn't clear what the more appropriate fix for each account would be *** Since then it has been years and those accounts are likely inactive and unused. We can rerun the Gerrit consistency check, feed the info back through our audit script then decide if we need to be careful with any of these accounts *** Chances are we can simply disable them all and remove the conflicting external ids. *** If we take good notes we can reconstruct the accounts as appropriate after the fact without Gerrit downtime should one of these users show up and wonder what happened. ** Gerrit 3.13 Upgrade Planning (clarkb 20260414) *** Clarkb would like to target a 3.13 upgrade for the end of May/early June. How does Friday June 5 look for others? *** Gerrit 3.13 removes support for Robot comments so Zuul will start making normal inline comments *** This also means that the Zuul restarts performed as part of the upgrade process are actually required when we upgrade to 3.13 to get Zuul's Gerrit version detection sorted out. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.13 Beginnings of an upgrade plan document *** 3.12.7 to 3.13.6 upgrades looked good. The next steps are announcing the upgrade and working through the minor TODOs in the etherpad plan document. ** Etherpad 3.1.0 Upgrade (clarkb 20260519) *** 2.7.3 introduced session cleanups in the database. But this created OOMs. We have disabled this feature. It should be fixed in the release after 3.1.0. *** 3.0.0 and 3.1.0 have been released *** https://github.com/ether/etherpad/blob/v3.1.0/CHANGELOG.md Big change appears to be the ability for etherpad to self update. I assume we would disable this and control etherpad via container images. ** Gitea 1.26.2 Upgrade (clarkb 20260526) *** https://review.opendev.org/c/opendev/system-config/+/989448 ** Zuul reporting empty public_v6 addresses for test nodes (clarkb 20260519) *** Zuul is reporting public_v6 values of '' for test nodes that do have working ipv6 in clouds like rax classic and ovh *** This may be an openstack api bug, an openstacksdk bug, or a zuul-launcher bug. *** Be aware this may impact the behavior of some test jobs. *** We will need to dig into why this is happening to understand it better. * Open discussion

1 0

Team Meeting Agenda for May 19, 2026
by Clark Boylan 18 May '26

18 May '26

We will meet with this agenda on May 19, 2026 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Dealing with web crawlers (clarkb 20251216) *** We have seen ghcr.io hosted anubis images return errors during some deployment jobs. If this becomes consistent we may need to mirror the image *** static02 and static04 are being cleaned up. *** Anything else to monitor or can we close this item up for now? ** Deploying a Prometheus for Server Metrics (clarkb 20260331) *** https://review.opendev.org/c/opendev/system-config/+/980840 *** This change and its child deploy prometheus with node exporter to collect server metrics *** mnasiadka has proposed we use GrepTimeDB in conjunction with Prometheus to collect metrics over a longer period of time ** Upgrade Ansible to v9 (clarkb 20260310) *** https://docs.ansible.com/projects/ansible/latest/reference_appendices/relea… *** This has mostly gone well. *** We had to force some hosts to use python2 due to their python3 being too old *** https://review.opendev.org/c/opendev/ansible-role-puppet/+/989028 ansible-role-puppet needs to stop using deprecated include tasks. ** Gerrit Account Cleanups (clarkb 20260317) *** Since the upgrade to Gerrit notedb we've had account inconsistencies that prevent us from push to the external ids ref/table directly. *** clarkb did a bunch of work to get the number down from hundreds to about 33 consistency errors before stalling out. *** The tail was the most difficult as it wasn't clear what the more appropriate fix for each account would be *** Since then it has been years and those accounts are likely inactive and unused. We can rerun the Gerrit consistency check, feed the info back through our audit script then decide if we need to be careful with any of these accounts *** Chances are we can simply disable them all and remove the conflicting external ids. *** If we take good notes we can reconstruct the accounts as appropriate after the fact without Gerrit downtime should one of these users show up and wonder what happened. ** Gerrit 3.13 Upgrade Planning (clarkb 20260414) *** Clarkb would like to target a 3.13 upgrade for the end of May/early June. How does Friday June 5 look for others? *** Gerrit 3.13 removes support for Robot comments so Zuul will start making normal inline comments *** This also means that the Zuul restarts performed as part of the upgrade process are actually required when we upgrade to 3.13 to get Zuul's Gerrit version detection sorted out. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.13 Beginnings of an upgrade plan document *** Clarkb will be retesting the upgrade process now that 3.12.7 and 3.13.6 images are available. ** Etherpad 3.1.0 Upgrade (clarkb 20260519) *** Upgrade to 2.7.3 seems to have worked well enough *** 3.0.0 and 3.1.0 have been released *** https://github.com/ether/etherpad/blob/v3.1.0/CHANGELOG.md Big change appears to be the ability for etherpad to self update. I assume we would disable this and control etherpad via container images. ** Zuul reporting empty public_v6 addresses for test nodes (clarkb 20260519) *** Zuul is reporting public_v6 values of '' for test nodes that do have working ipv6 in clouds like rax classic and ovh *** This may be an openstack api bug, an openstacksdk bug, or a zuul-launcher bug. *** Be aware this may impact the behavior of some test jobs. *** We will need to dig into why this is happening to understand it better. * Open discussion

1 0

Team Meeting Agenda for May 12, 2026
by Clark Boylan 11 May '26

11 May '26

We will meet with this agenda on May 12, 2026 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** mnasiadka has replaced our older mirror nodes and is now looking at other low hanging upgrade fruit. Please help review these changes to migrate servers. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Dealing with web crawlers (clarkb 20251216) *** We have deployed anubis to lists.opendev.org and it seems to be working well enough there *** We have also deployed anubis on the gitea servers. This was after spending much of last week fighting the crawler flood **** Would be good to look at getting PROXY protocol support working with gitea to make debugging easier *** Should we be looking at adding anubis to other services like static? ** Deploying a Prometheus for Server Metrics (clarkb 20260331) *** https://review.opendev.org/c/opendev/system-config/+/980840 *** This change and its child deploy prometheus with node exporter to collect server metrics *** Napkin math says that a 1TB volume should get us about 60 days of metrics. mnasiadka also indicates that Prometheus doesn't handle longer term metrics super well *** Ideally we would collect at least a years' worth of data. Can we make that happen with Prometheus? *** Do we need to look at Prometheus adjacent tools like Mimir or Thanos? **** Both of these solutions seem to tie into Prometheus using Prometheus as the data collection system. Then they store the data in a different system which can handle long term storage more nimbly. Then for queries they speak promql and prometheus apis allowing you to point tools like grafana at them as if they were prometheus. ** Upgrade Ansible to v9 (clarkb 20260310) *** https://docs.ansible.com/projects/ansible/latest/reference_appendices/relea… *** https://review.opendev.org/c/opendev/system-config/+/976282 *** Based on Ansible's python support Matrix Ansible 9 gives us a good deal of flexibility for bridge and remote nodes *** Ansible 9 also fixes problems with the use of pkg_resources in the Ansible ip module *** Any concerns with proceeding with the upgrade since tests look good? ** Gerrit Account Cleanups (clarkb 20260317) *** Since the upgrade to Gerrit notedb we've had account inconsistencies that prevent us from push to the external ids ref/table directly. *** clarkb did a bunch of work to get the number down from hundreds to about 33 consistency errors before stalling out. *** The tail was the most difficult as it wasn't clear what the more appropriate fix for each account would be *** Since then it has been years and those accounts are likely inactive and unused. We can rerun the Gerrit consistency check, feed the info back through our audit script then decide if we need to be careful with any of these accounts *** Chances are we can simply disable them all and remove the conflicting external ids. *** If we take good notes we can reconstruct the accounts as appropriate after the fact without Gerrit downtime should one of these users show up and wonder what happened. ** Gerrit 3.13 Upgrade Planning (clarkb 20260414) *** The 3.12 upgrade seems to have gone well **** H2 v2 caches appear to be no worse than H2 v1 caches **** Old H2 v1 caches have been removed from review03 at this point *** Clarkb would like to target a 3.13 upgrade for the end of May/early June *** https://review.opendev.org/c/opendev/system-config/+/987375 Update 3.12 and 3.13 images to the latest versions **** This includes a fix for the previously unconfigurable AI review button *** Gerrit 3.13 removes support for Robot comments so Zuul will start making normal inline comments *** This also means that the Zuul restarts performed as part of the upgrade process are actually required when we upgrade to 3.13 to get Zuul's Gerrit version detection sorted out. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.13 Beginnings of an upgrade plan document ** Etherpad 2.7.3 Upgrade *** https://review.opendev.org/c/opendev/system-config/+/985843 Etherpad 2.7.3 is out and this change will upgrade us to it *** Held node looks good to clarkb **** Default text on new pads inherits pad creator color **** Pad creator gets a pad deletion token that is shown when they first create the pad ** Ubuntu Resolute Test Nodes (clarkb 20260331) *** Images, labels, nodesets, etc have been added for Ubuntu Resolute *** Image uploads ran into problems with the new intermediate swift container which should be resolved now but uploads will take some time to catch up *** Still need to figure out our plan for mirroring the Resolute packages ** Noble Docker Not Talking to Podman Socket for all Operations (clarkb 20260414) *** During the Gerrit 3.12 upgrade we noticed that `docker image ls` doesn't work on Noble nodes anymore due to API version support mismatches between Docker and Podman *** `podman image ls` does work just fine and is what we used *** The problem appears to be due to noble-updates upgrading the docker.io package compared to noble proper *** The problem does not appear to affect all docker subcommands `docker ps -a` works just fine. *** Please keep an eye out for problems in configuration management caused by this. * Open discussion

1 0

Team Meeting Agenda for May 5, 2026
by Clark Boylan 04 May '26

04 May '26

After a couple of weeks away we are back for the OpenDev team meeting this week. We will meet on May 5, 2026 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** mnasiadka has replaced our older mirror nodes and is now looking at other low hanging upgrade fruit. Please help review these changes to migrate servers. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Dealing with web crawlers (clarkb 20251216) *** We have deployed anubis to lists.opendev.org and it seems to be working well enough there *** We have also deployed anubis on the gitea servers. This was after spending much of last week fighting the crawler flood **** Would be good to look at getting PROXY protocol support working with gitea to make debugging easier *** Should we be looking at adding anubis to other services like static? ** Deploying a Prometheus for Server Metrics (clarkb 20260331) *** https://review.opendev.org/c/opendev/system-config/+/980840 *** This change and its child deploy prometheus with node exporter to collect server metrics *** Napkin math says that a 1TB volume should get us about 60 days of metrics. mnasiadka also indicates that Prometheus doesn't handle longer term metrics super well *** Ideally we would collect at least a years' worth of data. Can we make that happen with Prometheus? *** Do we need to look at Prometheus adjacent tools like Mimir or Thanos? **** Both of these solutions seem to tie into Prometheus using Prometheus as the data collection system. Then they store the data in a different system which can handle long term storage more nimbly. Then for queries they speak promql and prometheus apis allowing you to point tools like grafana at them as if they were prometheus. ** Upgrade Ansible to v9 (clarkb 20260310) *** https://docs.ansible.com/projects/ansible/latest/reference_appendices/relea… *** https://review.opendev.org/c/opendev/system-config/+/976282 *** Based on Ansible's python support Matrix Ansible 9 gives us a good deal of flexibility for bridge and remote nodes *** Ansible 9 also fixes problems with the use of pkg_resources in the Ansible ip module *** Any concerns with proceeding with the upgrade since tests look good? ** Gerrit Account Cleanups (clarkb 20260317) *** Since the upgrade to Gerrit notedb we've had account inconsistencies that prevent us from push to the external ids ref/table directly. *** clarkb did a bunch of work to get the number down from hundreds to about 33 consistency errors before stalling out. *** The tail was the most difficult as it wasn't clear what the more appropriate fix for each account would be *** Since then it has been years and those accounts are likely inactive and unused. We can rerun the Gerrit consistency check, feed the info back through our audit script then decide if we need to be careful with any of these accounts *** Chances are we can simply disable them all and remove the conflicting external ids. *** If we take good notes we can reconstruct the accounts as appropriate after the fact without Gerrit downtime should one of these users show up and wonder what happened. ** Gerrit 3.13 Upgrade Planning (clarkb 20260414) *** The 3.12 upgrade seems to have gone well **** H2 v2 caches appear to be no worse than H2 v1 caches **** Old H2 v1 caches have been removed from review03 at this point *** Clarkb would like to target a 3.13 upgrade for the end of May/early June *** Gerrit 3.13 made the AI review button configurable but I don't think that is in a release yet *** Gerrit 3.13 removes support for Robot comments so Zuul will start making normal inline comments *** This also means that the Zuul restarts performed as part of the upgrade process are actually required when we upgrade to 3.13 to get Zuul's Gerrit version detection sorted out. *** Clarkb will start on an upgrade document soon ** Gitea 1.26 Upgrade *** https://review.opendev.org/c/opendev/system-config/+/985834 Gitea 1.26 is out and this change will upgrade us to it *** Should hold a node and do a quick sanity check before upgrading ** Etherpad 2.7.2 Upgrade *** https://review.opendev.org/c/opendev/system-config/+/985843 Etherpad 2.7.2 is out and this change will upgrade us to it *** Should probably hold a node and test things. The changelog refers to fixes to chat as well as the time slider undo functionality ** Ubuntu Resolute Test Nodes (clarkb 20260331) *** https://review.opendev.org/c/opendev/zuul-providers/+/982182 Add Resolute images to Zuul *** Ubuntu Bionic mirror content has been removed. We can probably start the process of mirroring Resolute packages. ** Noble Docker Not Talking to Podman Socket for all Operations (clarkb 20260414) *** During the Gerrit 3.12 upgrade we noticed that `docker image ls` doesn't work on Noble nodes anymore due to API version support mismatches between Docker and Podman *** `podman image ls` does work just fine and is what we used *** The problem appears to be due to noble-updates upgrading the docker.io package compared to noble proper *** The problem does not appear to affect all docker subcommands `docker ps -a` works just fine. *** Please keep an eye out for problems in configuration management caused by this. * Open discussion

1 0