We will meet with this agenda on June 9, 2026 at 19:00 UTC in #opendev-meeting:
== Agenda for next meeting ==
* Announcements
** clarkb will be out June 15 and 16. Fungi has volunteered to chair the team meeting on June 16
* Actions from last meeting
* Specs Review
* Topics
** Upgrading Old Servers (clarkb 20230627)
*** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents
*** Next on the list are graphite and backup servers
*** backup03.ca-ymq-1.vexxhost.opendev.org has been launched and is being backed up too
**** Preliminary checks all looked good. Should double check that all expected hosts are backing up to this server and confirm that a backup is mountable and looks accurate.
*** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic
** Deploying a Prometheus for Server Metrics (clarkb 20260331)
*** https://review.opendev.org/c/opendev/system-config/+/980840
*** This change and its child deploy prometheus with node exporter to collect server metrics
*** mnasiadka has proposed we use GrepTimeDB in conjunction with Prometheus to collect metrics over a longer period of time
** Gerrit Account Cleanups (clarkb 20260317)
*** Since the upgrade to Gerrit notedb we've had account inconsistencies that prevent us from push to the external ids ref/table directly.
*** clarkb did a bunch of work to get the number down from hundreds to about 33 consistency errors before stalling out.
*** The tail was the most difficult as it wasn't clear what the more appropriate fix for each account would be
*** Since then it has been years and those accounts are likely inactive and unused. We can rerun the Gerrit consistency check, feed the info back through our audit script then decide if we need to be careful with any of these accounts
*** Chances are we can simply disable them all and remove the conflicting external ids.
*** If we take good notes we can reconstruct the accounts as appropriate after the fact without Gerrit downtime should one of these users show up and wonder what happened.
** Gerrit 3.13 Upgrade Followup (clarkb 20260414)
*** We upgraded Gerrit last Friday to 3.13.6 from 3.12.7
*** https://etherpad.opendev.org/p/gerrit-upgrade-3.13 Upgrade plan and process that was used
*** Seems to be working, but a few things have been noticed
**** The lack of robot comment apis has created some questions. This was anticipated and we expect hard refreshes of browser clients to fix this
**** The robot comment api removal also broke gertty
**** corvus has suggested that Zuul stop using robot comments early as these comments effectively disappear post upgrade
**** Custom "Your -> Foo" links in the web ui can be set to open in new tabs. This is configurable through your user preferences in Gerrit
**** Some changes show up as conflicting with each other despite having a direct git tree relationship
*** https://review.opendev.org/c/opendev/system-config/+/991945/ and children will drop Gerrit 3.12 images, add 3.14 images, and test a 3.13 to 3.14 upgrade just as soon as we're confident we won't rollback to 3.12.
** Upgrading Grafana (clarkb 20260609)
*** https://review.opendev.org/c/opendev/system-config/+/991155 This change will upgrade grafana to the latest version
*** https://23.253.20.165/ is a held node for testing purposes. Things look good to clarkb and we can probably proceed.
** Zuul reporting empty public_v6 addresses for test nodes (clarkb 20260519)
*** Zuul is reporting public_v6 values of '' for test nodes that do have working ipv6 in clouds like rax classic and ovh
*** This may be an openstack api bug, an openstacksdk bug, or a zuul-launcher bug.
*** Be aware this may impact the behavior of some test jobs.
*** We will need to dig into why this is happening to understand it better.
* Open discussion
We will meet with this agenda on June 2, 2026 at 1900 UTC in #opendev-meeting:
== Agenda for next meeting ==
* Announcements
* Actions from last meeting
* Specs Review
* Topics
** Upgrading Old Servers (clarkb 20230627)
*** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents
*** Next on the list are graphite and backup servers
*** backup03.ca-ymq-1.vexxhost.opendev.org has been launched and is being backed up too
**** Preliminary checks all looked good. Should double check that all expected hosts are backing up to this server and confirm that a backup is mountable and looks accurate.
*** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic
** Deploying a Prometheus for Server Metrics (clarkb 20260331)
*** https://review.opendev.org/c/opendev/system-config/+/980840
*** This change and its child deploy prometheus with node exporter to collect server metrics
*** mnasiadka has proposed we use GrepTimeDB in conjunction with Prometheus to collect metrics over a longer period of time
** Gerrit Account Cleanups (clarkb 20260317)
*** Since the upgrade to Gerrit notedb we've had account inconsistencies that prevent us from push to the external ids ref/table directly.
*** clarkb did a bunch of work to get the number down from hundreds to about 33 consistency errors before stalling out.
*** The tail was the most difficult as it wasn't clear what the more appropriate fix for each account would be
*** Since then it has been years and those accounts are likely inactive and unused. We can rerun the Gerrit consistency check, feed the info back through our audit script then decide if we need to be careful with any of these accounts
*** Chances are we can simply disable them all and remove the conflicting external ids.
*** If we take good notes we can reconstruct the accounts as appropriate after the fact without Gerrit downtime should one of these users show up and wonder what happened.
** Gerrit 3.13 Upgrade Planning (clarkb 20260414)
*** Upgrade is happening this Friday June 5, at 1600 UTC
*** https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/…
*** Gerrit 3.13 removes support for Robot comments so Zuul will start making normal inline comments
*** This also means that the Zuul restarts performed as part of the upgrade process are actually required when we upgrade to 3.13 to get Zuul's Gerrit version detection sorted out.
*** https://etherpad.opendev.org/p/gerrit-upgrade-3.13 Beginnings of an upgrade plan document
** Etherpad 3.2.0 Upgrade (clarkb 20260519)
*** 3.0.0, 3.1.0, and 3.2.0 have been released
*** 2.7.3 introduced session cleanups in the database. But this created OOMs. We have disabled this feature. It should be fixed in 3.2.0.
*** https://github.com/ether/etherpad/blob/v3.2.0/CHANGELOG.md Big change appears to be the ability for etherpad to self update. We will not opt into this functionality and already have it disabled
*** https://review.opendev.org/c/opendev/system-config/+/990531
** Zuul reporting empty public_v6 addresses for test nodes (clarkb 20260519)
*** Zuul is reporting public_v6 values of '' for test nodes that do have working ipv6 in clouds like rax classic and ovh
*** This may be an openstack api bug, an openstacksdk bug, or a zuul-launcher bug.
*** Be aware this may impact the behavior of some test jobs.
*** We will need to dig into why this is happening to understand it better.
* Open discussion