We will meet with this agenda on October 6, 2026 at 19:00 UTC in #opendev-meeting:opendev.org on Matrix:
== Agenda for next meeting ==
* Announcements
** OpenInfra PTG happening October 12-16.
*** We will cancel our normal weekly meeting on October 13, but clarkb will keep the office hour on the 13th
* Actions from last meeting
* Specs Review
* Topics
** Upgrading Old Servers (clarkb 20230627)
*** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents
*** Next on the list are graphite and the rax backup server
*** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic
*** Would be good to start thinking about replacing Jammy servers.
** Deploying a Prometheus for Server Metrics (clarkb 20260331)
*** We have mostly working graphs in grafana now
*** Next steps:
*** Fix the disk metrics collection for prometheus01
**** https://review.opendev.org/c/opendev/system-config/+/1008848
*** Clean up the existing dashboard (remove unused/unneeded panels/graphs)
**** https://review.opendev.org/c/openstack/project-config/+/1008493
*** Add more servers to the node exporter group to begin collecting metrics for them
** IPTables to NFTables Conversion for control plane nodes (clarkb 20260811)
*** We are starting to see distros (like Debian) drop support for legacy IPtables tooling
*** There are options for preserving iptables rule compatibiltiy but it is probably time for us to start thinking in terms of nftables instead of iptables
*** This should make us more future proof and should hopefully make rules easier to read
*** Plan is to update the two different proposed approaches to merge them together. Taking the testing from one set of changes and the nftables native ruleset from the other and combine them into the thing we will work on deploying.
** Potential new cloud donor (clarkb 20260825)
*** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t…
*** Clark met with this cloud at the recent Summit. Sounds like they may have a small demo setup available for us soon.
*** They have emailed and indicated that we should create an account in their systems and let them know what the account is so they can give us access to the test/demo environment.
** Struggles Building Test Node Images (clarkb 20260915)
*** Zuul launchers ran out of disk space. This has since been resolved and builds look happier.
*** Uploads to rax classic are failing on 60 second read timeouts making requests to glance to start the image import tasks using the glance tasks api.
*** One theory is that this may be related to the leaked contents in the temporary swift container used to shuffle images from our end to rax glance.
*** A good next step is trying to clean up those leaked objects in those container(s).
** OpenAFS Disk Consumption (clarkb 20261006)
*** We are hovering around 95% total disk use in our OpenAFS cluster
*** clarkb cleaned up unneeded wheel mirror volumes for old distro releases which freed up ~60GB of disk
**** https://review.opendev.org/c/openstack/project-config/+/1008533 cleanup afs grafana graphs for removed volumes
*** We have a number of old volumes that can be cleared out with a little bit of work (though this won't free up significant disk)
**** https://review.opendev.org/c/opendev/system-config/+/1008520
*** Dropping reprepro contents file production will speed up reprepro runtime
**** https://review.opendev.org/c/opendev/system-config/+/1008055
*** We don't need source packages in Debian
**** https://review.opendev.org/c/opendev/system-config/+/1008547
*** Then we're left with some hard decisions that will impact our users
**** clarkb proposes that we stop mirroring arm64 packages as they represent ~50% of the total disk consumption for each distro but are a tiny fraction of our job workloads
**** https://review.opendev.org/c/opendev/system-config/+/1008548 stack starts here with centos then debian then ubuntu
** Gerrit 3.14 Upgrade Planning (clarkb 20261006)
*** https://etherpad.opendev.org/p/gerrit-upgrade-3.14 Planning will go in this document
*** Gerrit has announced that Gerrit 3.15 is targeted to release on November 13
*** We try to stick to a one release behind upstream upgrade cadence which means we should plan to upgrade to 3.14 late November/early December
*** Clarkb has done the last few upgrades. Is anyone else interested in driving this particular upgrade?
* Open discussion