November 2025 - service-discuss

Team Meeting Agenda for November 25, 2025
by Clark Boylan 24 Nov '25

24 Nov '25

We will meet on November 25, 2025 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Holiday weekend for many of us this weekend. * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** Gerrit 3.13 has been released so we should really try to upgrade to 3.11 and then 3.12 soon. *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Held nodes here: **** 3.10: https://217.182.142.186 **** 3.11: https://50.56.159.76/ *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out. *** Proposing December 7 at 2100 UTC for the upgrade itself ** Rename project app-kubernetes-module-manager to app-kernel-module-management (clarkb 20251125) *** Biggest question is when do we want to try and do this? Before or after the 3.11 Upgrade? Also where does it fit with holidays etc. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** https://docs.opendev.org/opendev/infra-specs/latest/specs/matrix_for_opende… *** The new room exists (thank you tonyb) and updates to statusbot have begun (thank you corvus) *** https://review.opendev.org/q/hashtag:%22opendev-matrix%22+status:open ** Zuul Launcher Updates (clarkb 20251125) *** We found a corrupted image upload in raxflex sjc3 again on November 21. Previous occurrence was discovered on October 6 *** https://review.opendev.org/c/zuul/zuul/+/968090 Will validate the glance checksum against our checksum and reject mismatches **** It isn't clear to clarkb if this is safe as glance may change the checksum value if it does a format conversion on the backend? *** Floating IPs attached to held nodes were deleted in raxflex. Likely due to our leaked fip cleanup routine. Debugging has been added to the launcher to track this down. ** Gitea 1.25.2 Upgrade (clarkb 20251125) *** https://review.opendev.org/c/opendev/system-config/+/968245 Upgrade gitea to 1.25.2 ** December Meeting Planning (clarkb 20251125) *** Holidays are approaching for many of us. Do we want to proactively cancel meetings? *** December 23, 30, and January 6 are the dates most likely to be impacted *** clarkb can likely host a meeting on each of these days if we wish to have them. * Open discussion

1 0

Re: Issue Adding OpenStack-Helm Repository via Helm
by Clark Boylan 21 Nov '25

21 Nov '25

Note this email was originally sent to the incident list, but I'm following up here on the public list as this is not a security concern. On 11/21/25 8:05 AM, Francesco La Porta wrote: > I am experiencing an issue when trying to add the OpenStack-Helm > repository using Helm. Running the following command: helm repo add > openstack-helm https://tarballs.opendev.org/openstack/openstack-helm > <https://tarballs.opendev.org/openstack/openstack-helm> > > returns the following error: > > Error: looks like "https://tarballs.opendev.org/openstack/openstack-helm > <https://tarballs.opendev.org/openstack/openstack-helm>" is not a valid > chart repository or cannot be reached: context deadline exceeded > (Client.Timeout or context cancellation while reading body) > > > This occurs consistently and prevents me from using the OpenStack-Helm > charts via Helm. > > Could you please confirm if there is an ongoing issue with the > repository or advise on any alternative methods to access the charts? This is probably not an intended behavior, but is unfortunately an expected one. The issue here is an underlying artifact of using Apache2 mod_autoindex to build index.html content on the fly from data backed by OpenAFS in a directory with thousands of entries. Opening this URL with wget and my browser takes about 3 minutes and 12 seconds. OpenAFS performs poorly with many thousands of entries and to make things worse Apache is stat'ing the directory entries to include modification times and file sizes in the generated index which takes time. The good news is I think you can workaround this today using the --timeout option to `helm repo add` and the OpenStack Helm team can change how they manage their Helm repository to improve its performance. `helm repo add` has a default timeout of 2 minutes. Given my testing says fetching this data takes around 3 minutes 12 seconds I think if you set the timeout to say 5 minutes using --timeout you should be able to fetch the data (if slowly). Separately, the OpenStack Helm team needs to stop writing all of this data to a single directory. Currently there are just under 15k entries in the directory. In addition to slowness stat'ing thousands of files OpenAFS directories have a hard limit on total entries between 15k and 60k based on the lengths of the filenames. We estimate the current limit for that directory is about 32k. Instead what they should be doing instead is sharding the data into subdirectories. For example the current file at openstack/openstack-helm/nova-2025.2.0+f6be959ce.tgz could instead be stored at openstack/openstack-helm/nova/2025.2.0/nova-2025.2.0+f6be959ce.tgz then openstack/openstack-helm/index.yaml should be updated to match. Doing this will drastically cut down on the total number of entries in each directory which will improve performance and avoid the hard limits. Now I recognize that you aren't going to fix that for them, and I can't either. I think a good next step would be for you to engage with the OpenStack Helm team directly and point out that this is both problematic for you and that there is something they can do to fix it. That team lists communication options here: https://docs.openstack.org/openstack-helm/latest/readme.html#communication and you should feel free to refer to this email thread or quote it when you reach out to them. An archived version of this email should show up in https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/ shortly after you receive it. Clark

1 0

Team Meeting Agenda for November 18, 2025
by Clark Boylan 17 Nov '25

17 Nov '25

We will meet with this agenda on November 18, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** Gerrit 3.13 has been released so we should really try to upgrade to 3.11 and then 3.12 soon. *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Held nodes here: **** 3.10: https://217.182.142.186 **** 3.11: https://174.143.59.58 *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** https://docs.opendev.org/opendev/infra-specs/latest/specs/matrix_for_opende… *** The new room exists (thank you tonyb) and updates to statusbot have begun (thank you corvus) *** https://review.opendev.org/q/hashtag:%22opendev-matrix%22+status:open ** Gitea Performance (clarkb 20251028) *** All traffic should be through the load balancer now *** Memcached size limits have been increased *** We should monitor and determine if these changes have improved performance * Open discussion

1 0

Team Meeting Agenda for November 11, 2025
by Clark Boylan 11 Nov '25

11 Nov '25

We will meet with this agenda on November 11, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** Gerrit 3.13 has been released so we should really try to upgrade to 3.11 and then 3.12 soon. *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** https://review.opendev.org/c/opendev/system-config/+/966083 Fix container bind mounts for Gerrit *** https://review.opendev.org/c/opendev/system-config/+/966084 Update to Gerrit 3.10.9 and 3.11.7 *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** The spec (954826) has merged. *** clarkb will start working on this as soon as time permits. ** Upgrade Zuul Zookeeper Cluster to 3.9 (clarkb 20251111) *** https://review.opendev.org/c/opendev/system-config/+/966612 *** This will add some new features Zuul can take advantage of ** Gitea 1.25.1 Upgrade (clarkb 20251028) *** https://review.opendev.org/c/opendev/system-config/+/965960 Upgrade Gitea to 1.25.1 ** Gitea Performance (clarkb 20251028) *** All traffic should be through the load balancer now *** Memcached size limits have been increased *** We should monitor and determine if these changes have improved performance ** Raxflex DFW3 Disabled (clarkb 20251104) *** We discovered the raxflex dfw3 mirror had afs cache errors. fs flushall appeared to get stuck and the kernel oopsed. After forcing a shutdown via openstack the server refuses to boot *** We've asked dan_with what our next steps should be via IRC *** Worst case we rebuild the mirror with a new cinder volume and replace the whole thing. * Open discussion

1 0

Team Meeting Agenda for November 4, 2025
by Clark Boylan 03 Nov '25

03 Nov '25

We will meet with this agenda on November 4, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Clarkb out early Friday and all of Monday. Expect a meeting Tuesday with a possibly delayed agenda. * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** New held nodes are now available. *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Gerrit Spontaneous Shutdown During Summit (clarkb 20251028) *** The Gerrit server was discovered to be in a shutdown state with no recorded errors *** The Nova team indicates this could happen if the hypervisor is unable to allocate memory. The host would be shutdown and Nova would notice but not record an error *** Its possible the underlying cause is something else; either way we need to work with Vexxhost to identify the source of the problem and mitigate it *** Is someone willing to figure out how to file a ticket with Vexxhost? This was their suggested approach last time we had problems. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** AFS mirror content updates (clarkb 20250916) *** https://review.opendev.org/c/opendev/system-config/+/965334 Mirror trixie packages *** Rocky Linux didn't hit these mirror issues because we don't mirror any rocky linux releases. Debian expects mirror content for all releases. ** Zuul Launcher Updates (clarkb 20250923) *** Nodes can be from mixed providers again due to a bug when reassigning unassigned nodes. *** https://review.opendev.org/c/zuul/zuul/+/965954 Fix assignment of unassigned nodes. ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** The spec (954826) has merged. *** clarkb will start working on this as soon as time permits. ** Etherpad 2.5.2 Upgrade (clarkb 20250805) *** https://github.com/ether/etherpad-lite/blob/v2.5.2/CHANGELOG.md *** https://review.opendev.org/c/opendev/system-config/+/956593 *** This fixes our css issues and clarkb believes we are ready to upgrade. ** Gitea 1.25.0 Upgrade (clarkb 20251028) *** https://review.opendev.org/c/opendev/system-config/+/965960 Upgrade Gitea to 1.25.0 ** Gitea Performance (clarkb 20251028) *** https://review.opendev.org/c/opendev/system-config/+/964728 Don't allow direct backend access **** Some performance issues seem related to crawlers hitting backends directly so they aren't balanced properly **** Most recent issue was via traffic to the load balancer frontend so this won't fix everything **** Means we'll have to take extra steps to test backends directly (ssh port forwards) *** https://review.opendev.org/c/opendev/system-config/+/965420 Increase memcached cache size to mitigate effect of crawlers poisoning the cache ** Raxflex DFW3 Disabled (clarkb 20251104) *** We discovered the raxflex dfw3 mirror had afs cache errors. fs flushall appeared to get stuck and the kernel oopsed. After forcing a shutdown via openstack the server refuses to boot *** We've asked dan_with what our next steps should be via IRC *** Worst case we rebuild the mirror with a new cinder volume and replace the whole thing. * Open discussion

1 0