- service-discuss - lists.opendev.org

Team Meeting Agenda for September 30, 2025
by Clark Boylan 29 Sep '25

29 Sep '25

We will meet with this agenda on September 30, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack release happening this week on October 1 * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** New held nodes for Gerrit 3.10.8 and 3.11.5 here: **** https://zuul.opendev.org/t/openstack/build/54f6629a3041466ca2b1cc6bf17886c4 **** https://zuul.opendev.org/t/openstack/build/c9051c435bf7414b986c37256f71538e *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** AFS mirror content cleanup (clarkb 20250916) *** Can we automate cleanups for reprepro content? In other words are the steps cheap enough during the noop case to run with our regular reprepro update script? *** Additional volumes to look into cleanups for include ceph, docker, and puppetlabs package mirrors as well as our python wheel caches *** CentOS 9 Stream grows at a rate that requires regular quota bumps. Is there anything we can do about that? *** With these cleanups and server upgrades completed we can probably start to consider adding new mirrors again. **** Trixie, CentOS 10 Stream, Rocky Linux ? ** Zuul Launcher Updates (clarkb 20250923) *** Networking for rax flex should be sorted now *** Still have leaked ERROR nodes in rax-ord waiting on rax to clear them out as we cannot *** Also haven't heard anything about rax-dfw API problems. We can probably try re-enabling that region and see if the issue persists. ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** The spec (954826) has merged. *** clarkb will start working on this as soon as time permits. ** Pre PTG Planning (clarkb 20250624) *** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document *** Clarkb has added a number of topic Ideas. Please add more. *** Times: Tuesday October 7 1800-2000 UTC, Wednesday October 8 1500-1700 UTC, Thursday October 9 1500-1700 (next week) *** The Tuesday time block will effectively replace our weekly IRC meeting next week. ** Etherpad 2.5.0 Upgrade (clarkb 20250805) *** https://github.com/ether/etherpad-lite/blob/v2.5.0/CHANGELOG.md *** This release is reported to have fixed our css problems. *** https://review.opendev.org/c/opendev/system-config/+/956593/ *** 104.130.127.119 is a held node for testing. You need to edit /etc/hosts to point etherpad.opendev.org at that IP. * Open discussion

1 0

Team Meeting Agenda for September 23, 2025
by Clark Boylan 22 Sep '25

22 Sep '25

We will meet with this agenda on September 23, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** clarkb will be out Thursday the 25th. * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** New held nodes for Gerrit 3.10.8 and 3.11.5 here: **** https://zuul.opendev.org/t/openstack/build/54f6629a3041466ca2b1cc6bf17886c4 **** https://zuul.opendev.org/t/openstack/build/c9051c435bf7414b986c37256f71538e *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** AFS mirror content cleanup (clarkb 20250916) *** Debian stretch, bullseye backports, and openeuler content has been cleaned up. *** Can we automate cleanups for reprepro content? In other words are the steps cheap enough during the noop case to run with our regular reprepro update script? *** Additional volumes to look into cleanups for include ceph, docker, and puppetlabs package mirrors as well as our python wheel caches *** CentOS 9 Stream grows at a rate that requires regular quota bumps. Is there anything we can do about that? *** With these cleanups and server upgrades completed we can probably start to consider adding new mirrors again. ** Lists Server Slowness (clarkb 20250909) *** Mailman's data is now backed by a dedicated volume capable of many more IOPs than the root disk. Let us know if you see new occurrences of unexpected slowness. ** Deleting ze11 (clarkb 20250923) *** The ze11 zuul executor has been shutdown for some time due to network slowness cloning the nova repository. *** We have a 10 minute timeout for that clone, most nodes do it in ~4 minutes, but ze11 does it in about 12-13 minutes. *** This slowness may also impact job runtimes as well *** At this point 11 or even 10 executors is probably sufficient for our needs the simplest solution here may be to delete ze11 entirely (and ze12 if we find its existence confusing) ** Zuul Launcher Updates (clarkb 20250923) *** We've run into weirdness with rax flex clouds booting instances with two network interfaces if networks are configured via zuul-providers config. This does not occur when networks are configured via clouds.yaml instead. This will need debugging *** All existing nodepool nodes have been deleted out of all cloud regions except for rax-ord. There are a number of nodepool instances there in an ERROR state that we cannot delete ourselves. *** I have sent email to Rackspace asking them to clean those nodes up as I believe they do count against our quota. *** We have also disabled rax-dfw as it started having API errors similar to what we saw in IAD and ORD previously. My email included questions about this situation too. ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** The spec (954826) has merged. *** clarkb will start working on this as soon as time permits. ** Pre PTG Planning (clarkb 20250624) *** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document *** Clarkb has added a number of topic Ideas. Please add more. *** Times: Tuesday October 7 1800-2000 UTC, Wednesday October 8 1500-1700 UTC, Thursday October 9 1500-1700 *** The Tuesday time block would effectively replace our weekly IRC meeting. ** Etherpad 2.5.0 Upgrade (clarkb 20250805) *** https://github.com/ether/etherpad-lite/blob/v2.5.0/CHANGELOG.md *** This release is reported to have fixed our css problems. *** https://review.opendev.org/c/opendev/system-config/+/956593/ *** 104.130.127.119 is a held node for testing. You need to edit /etc/hosts to point etherpad.opendev.org at that IP. * Open discussion

1 0

Team Meeting Agenda for September 16, 2025
by Clark Boylan 15 Sep '25

15 Sep '25

We will meet with this agenda on September 16, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Moving OpenDev's python-base/python-builder/uwsgi-base Images to Quay (clarkb 20250812) *** The images have moved and we're no longer publishing to docker hub *** https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… *** I believe that all images within OpenDev that consume these base images have been updated to pull from the new location (including Gerrit) *** If you find images that need updates please push a change and let us know. ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** New held nodes for Gerrit 3.10.8 and 3.11.5 here: **** https://zuul.opendev.org/t/openstack/build/54f6629a3041466ca2b1cc6bf17886c4 **** https://zuul.opendev.org/t/openstack/build/c9051c435bf7414b986c37256f71538e *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** The entire openafs and kerberos system has been upgraded to Noble except for afs02.dfw.openstack.org. Waiting on RW volume moves to complete before performing this last upgrade **** Will need to remember to remove all of the hosts from the emergency file when done. *** Next on the list are graphite and backup servers *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** AFS mirror content cleanup (clarkb 20250916) *** Fungi identified a number of AFS mirror cleanups that can be performed once server upgrades are complete *** OpenEuler mirror content is no longer needed as we don't have any test images *** Debian stretch and bullseye backports are no longer needed as they are both old enough to have aged out *** For the reprepro content we should also look into automatically clearing content if the cost to do so is actually a noop when we haven't removed releases from the mirror list. ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** The spec (954826) has merged. *** clarkb will start working on this as soon as time permits. ** Pre PTG Planning (clarkb 20250624) *** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document *** Clarkb has added a number of topic Ideas. Please add more. *** Times: Tuesday October 7 1800-2000 UTC, Wednesday October 8 1500-1700 UTC, Thursday October 9 1500-1700 *** The Tuesday time block would effectively replace our weekly IRC meeting. ** Etherpad 2.5.0 Upgrade (clarkb 20250805) *** https://github.com/ether/etherpad-lite/blob/v2.5.0/CHANGELOG.md *** This release is reported to have fixed our css problems. *** https://review.opendev.org/c/opendev/system-config/+/956593/ *** 104.130.127.119 is a held node for testing. You need to edit /etc/hosts to point etherpad.opendev.org at that IP. ** Lists Server Slowness (clarkb 20250909) *** We think we have tracked this down to the different disk_io_index values between Performance and Standard flavors on rax classic *** Specifically the value is 2 on Standard (lists) and 40 on mirror and fio measures about a 20x differences in iops during randread testing to match *** We will need to migrate the mailman3 installation off of the server disk and onto a cinder volume to improve the number of iops. * Open discussion

1 0

Team Meeting Agenda for September 9, 2025
by Clark Boylan 08 Sep '25

08 Sep '25

We will meet with this agenda on September 9, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Held nodes for these two builds: **** https://zuul.opendev.org/t/openstack/build/f1ca0d1f2e054829a4506ececb58bed3 **** https://zuul.opendev.org/t/openstack/build/588723b923e94901af3065143d9df818 *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Kerberos and OpenAFS servers all updated to Jammy. afs01.dfw.openstack.org was subsequently updated to Noble and then began to unping **** Seems likely that Noble is simply not honoring the old /etc/network/interfaces scripts and needs netplan or similar. We should be able to rescue the node and set that directly or set a password for login via the console to do it interactively. **** I've also discovered that all but afs02.dfw.openstack.org and kdc04.opendev.org are running on an older xen version. In CI we've seen Noble boot with a single vcpu on the older hypervisors. We may also need to request these hosts get migrated to new hypervisors before updating. We can see what afs01.dfw.openstack.org looks like when networking is happy again. *** Next on the list are graphite and backup servers *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** https://review.opendev.org/c/opendev/infra-specs/+/954826 Spec outlining the motivation and plan for Matrix trialing ** Pre PTG Planning (clarkb 20250624) *** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document *** Clarkb has added a number of topic Ideas. Please add more. *** Times: Tuesday October 7 1800-2000 UTC, Wednesday October 8 1500-1700 UTC, Thursday October 9 1500-1700 *** The Tuesday time block would effectively replace our weekly IRC meeting. ** Loss of upstream Debian bullseye-backports mirror (fungi 20250805) *** https://review.opendev.org/956497 temporarily ignore undefinedtarget *** Zuul-jobs will no longer enable debian backports by default on September 9 (day of this meeting) **** https://lists.zuul-ci.org/archives/list/zuul-announce@lists.zuul-ci.org/thr… *** Once zuul-jobs is updated we can delete the content from our mirrors and drop the workaround ** Etherpad 2.5.0 Upgrade (clarkb 20250805) *** https://github.com/ether/etherpad-lite/blob/v2.5.0/CHANGELOG.md *** This release is reported to have fixed our css problems. *** https://review.opendev.org/c/opendev/system-config/+/956593/ *** 104.130.127.119 is a held node for testing. You need to edit /etc/hosts to point etherpad.opendev.org at that IP. ** Moving OpenDev's python-base/python-builder/uwsgi-base Images to Quay (clarkb 20250812) *** All of our containers that are build from these base images are now running on Noble nodes with podman *** A zuul user reported problems with the move to always using buildx to build images. This has been corrected by setting network=host on the build itself (not just the custom builder) *** If there are no other complaints by the time of this meeting I think we are in good shape to actually proceed with the container image move on our side. ** Adding Debian Trixie Base Python Container Images (clarkb 20250826) *** https://review.opendev.org/c/opendev/system-config/+/958480 *** Trixie is here. Let's bootstrap some container images so that we can migrate over to the new release in our containers. ** Dropping Ubuntu Bionic Test Nodes (clarkb 20250826) *** Zuul will be dropping Ansible 9 support soon. When that happens Ubuntu Bionic test nodes will stop working in Zuul as Python on them is too old for Ansible 11 *** Decided that since Bionic is well past its EOL date we don't need to make a big announcement. At least not yet. But cleanup is a good thing. *** I wrote a number of changes to OpenDev tools like glean, bindep, and git-review to remove Bionic based test jobs *** Please call out others that you notice or push up changes for them too. *** It should still be possible to run python2.7 jobs if we need them on Ubuntu Jammy. Note that tox 4 is not compatible with python2.7 so we need to pin tox in the jobs to <4 ** Lists Server Slowness (clarkb 20250909) *** Like every other service mailman is getting bombarded by web crawlers *** User agent lists have been updated to try and cut down on some of the impacts *** Service is extremely slow at times. When this occurs top reports significant iowait *** Thoughts on further improvements that could be made? * Open discussion

1 0

Team Meeting Agenda for September 2, 2025
by Clark Boylan 02 Sep '25

02 Sep '25

We will meet on September 2, 2025 (today) at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Held nodes for these two builds: **** https://zuul.opendev.org/t/openstack/build/f1ca0d1f2e054829a4506ececb58bed3 **** https://zuul.opendev.org/t/openstack/build/588723b923e94901af3065143d9df818 *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Kerberos and OpenAFS servers are in the middle of getting upgraded. Most are running Jammy now. Then after everything is running Jammy the next round of updates will get us on Noble. *** Next on the list are graphite and backup servers *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** https://review.opendev.org/c/opendev/infra-specs/+/954826 Spec outlining the motivation and plan for Matrix trialing ** Pre PTG Planning (clarkb 20250624) *** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document *** Clarkb has added a number of topic Ideas. Please add more. *** Times: Tuesday October 7 1800-2000 UTC, Wednesday October 8 1500-1700 UTC, Thursday October 9 1500-1700 *** The Tuesday time block would effectively replace our weekly IRC meeting. ** Loss of upstream Debian bullseye-backports mirror (fungi 20250805) *** https://review.opendev.org/956497 temporarily ignore undefinedtarget *** Zuul-jobs will no longer enable debian backports by default on September 9 **** https://lists.zuul-ci.org/archives/list/zuul-announce@lists.zuul-ci.org/thr… *** Once zuul-jobs is updated we can delete the content from our mirrors and drop the workaround ** Etherpad 2.5.0 Upgrade (clarkb 20250805) *** https://github.com/ether/etherpad-lite/blob/v2.5.0/CHANGELOG.md *** This release is reported to have fixed our css problems. *** https://review.opendev.org/c/opendev/system-config/+/956593/ *** 104.130.127.119 is a held node for testing. You need to edit /etc/hosts to point etherpad.opendev.org at that IP. ** Moving OpenDev's python-base/python-builder/uwsgi-base Images to Quay (clarkb 20250812) *** All of our containers that are build from these base images are now running on Noble nodes with podman *** Turns out to use speculative images via the buildset registry we always need to build with a custom buildx builder **** https://review.opendev.org/c/zuul/zuul-jobs/+/958783 *** Once that lands we can switch the default builder back to docker in system-config then use a consistent builder for all container image builds ** Adding Debian Trixie Base Python Container Images (clarkb 20250826) *** https://review.opendev.org/c/opendev/system-config/+/958480 *** Trixie is here. Let's bootstrap some container images so that we can migrate over to the new release in our containers. ** Dropping Ubuntu Bionic Test Nodes (clarkb 20250826) *** Zuul will be dropping Ansible 9 support soon. When that happens Ubuntu Bionic test nodes will stop working in Zuul as Python on them is too old for Ansible 11 *** Decided that since Bionic is well past its EOL date we don't need to make a big announcement. At least not yet. But cleanup is a good thing. *** https://review.opendev.org/q/hashtag:%22drop-bionic%22+status:open *** This cleanup will also enable us to drop additional content from the afs ubuntu mirrors which will make afs server upgrades slightly easier. ** Temporary Shutdown of raxflex sjc3 for provider maintenance window (clarkb 20250902) *** https://review.opendev.org/c/opendev/zuul-providers/+/959200 *** The cinder volume backing our mirror cache is going to be unavailable tomorrow for maintenance. Clarkb will disable the region in Zuul and shutdown the mirror today in preparation. * Open discussion

1 0

Team Meeting Agenda for August 26, 2025
by Clark Boylan 25 Aug '25

25 Aug '25

We will meet with this agenda on August 26, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Held nodes for these two builds: **** https://zuul.opendev.org/t/openstack/build/f1ca0d1f2e054829a4506ececb58bed3 **** https://zuul.opendev.org/t/openstack/build/588723b923e94901af3065143d9df818 *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Kerberos and OpenAFS servers are in the middle of getting upgraded. Most are running Jammy now. Then after everything is running Jammy the next round of updates will get us on Noble. *** Next on the list are graphite and backup servers *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** https://review.opendev.org/c/opendev/infra-specs/+/954826 Spec outlining the motivation and plan for Matrix trialing ** Pre PTG Planning (clarkb 20250624) *** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document *** Clarkb has added a number of topic Ideas. Please add more. *** Times: Tuesday October 7 1800-2000 UTC, Wednesday October 8 1500-1700 UTC, Thursday October 9 1500-1700 *** The Tuesday time block would effectively replace our weekly IRC meeting. ** Service Coordinator Election Planning (clarkb 20250729) *** Nomination period is over. Clarkb was the only nominee. Clarkb is it again by default. If there are any questions or concerns let's air them now then we can consider this official and done. ** Loss of upstream Debian bullseye-backports mirror (fungi 20250805) *** https://review.opendev.org/956497 temporarily ignore undefinedtarget *** the workaround isn't too bad, but we need to make a longer-term decision about our mirror contents and the configure-mirrors role ** Etherpad 2.5.0 Upgrade (clarkb 20250805) *** https://github.com/ether/etherpad-lite/blob/v2.5.0/CHANGELOG.md *** This release is reported to have fixed our css problems. *** https://review.opendev.org/c/opendev/system-config/+/956593/ *** 104.130.127.119 is a held node for testing. You need to edit /etc/hosts to point etherpad.opendev.org at that IP. ** Moving OpenDev's python-base/python-builder/uwsgi-base Images to Quay (clarkb 20250812) *** All of our containers that are build from these base images are now running on Noble nodes with podman *** It should be safe for us to move the base images then rebuild everything using quay as the source. *** https://review.opendev.org/c/opendev/system-config/+/957277 ** Adding Debian Trixie Base Python Container Images (clarkb 20250826) *** https://review.opendev.org/c/opendev/system-config/+/958480 *** Trixie is here. Let's bootstrap some container images so that we can migrate over to the new release in our containers. ** Dropping Ubuntu Bionic Test Nodes (clarkb 20250826) *** Zuul will be dropping Ansible 9 support soon. When that happens Ubuntu Bionic test nodes will stop working in Zuul as Python on them is too old for Ansible 11 *** Should we announce that we're ripping these nodes out now in an effort to begin the cleanup process so that by the time the Zuul changes happen we're well on our way to a happy state? *** This cleanup will also enable us to drop additional content from the afs ubuntu mirrors which will make afs server upgrades slightly easier. * Open discussion

1 0

Team Meeting Agenda for August 19, 2025
by Clark Boylan 18 Aug '25

18 Aug '25

We will meet with this agenda on August 19, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Held nodes for these two builds: **** https://zuul.opendev.org/t/openstack/build/f1ca0d1f2e054829a4506ececb58bed3 **** https://zuul.opendev.org/t/openstack/build/588723b923e94901af3065143d9df818 *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are kerberos, openafs, graphite, and backup servers *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** https://review.opendev.org/c/opendev/infra-specs/+/954826 Spec outlining the motivation and plan for Matrix trialing ** Pre PTG Planning (clarkb 20250624) *** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document *** Clarkb has added a number of topic Ideas. Please add more. *** Proposed times: Tuesday October 7 1800-2000 UTC, Wednesday October 8 1500-1700 UTC, Thursday October 9 1500-1700 *** The Tuesday time block would effectively replace our weekly IRC meeting. ** Service Coordinator Election Planning (clarkb 20250729) *** Nomination Period open from August 5, 2025 to August 19, 2025. If necessary we will hold an election from August 20, 2025 to August 27, 2025. All date ranges and times will be in UTC. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Loss of upstream Debian bullseye-backports mirror (fungi 20250805) *** https://review.opendev.org/956497 temporarily ignore undefinedtarget *** the workaround isn't too bad, but we need to make a longer-term decision about our mirror contents and the configure-mirrors role ** Etherpad 2.4.2 Upgrade (clarkb 20250805) *** https://github.com/ether/etherpad-lite/blob/v2.4.2/CHANGELOG.md *** Theming updates have broken the no-skin and custom skin themes. We use no-skin. https://github.com/ether/etherpad-lite/issues/7065 *** We could consider switching to Colibris skin which does work. This skin presents etherpads in a google docs piece of paper manner. ** Moving OpenDev's python-base/python-builder/uwsgi-base Images to Quay (clarkb 20250812) *** All of our containers that are build from these base images are now running on Noble nodes with podman *** It should be safe for us to move the base images then rebuild everything using quay as the source. *** https://review.opendev.org/c/opendev/system-config/+/957277 ** Bootstrapping rax-flex iad3 (clarkb 20250819) *** We should be able to use rax-flex iad3 now for additional capacity and redundancy *** https://review.opendev.org/c/opendev/system-config/+/957810 Initial bootstrapping steps *** Will need to boot a mirror and configure the region in opendev/zuul-providers. * Open discussion

1 0

Clarkb's OpenDev Service Coordinator Nomination
by Clark Boylan 18 Aug '25

18 Aug '25

Per the plan sent in this announcement [0] the OpenDev Service Coordinator nomination period ends in just over 24 hours. I haven't seen any indication that anyone else is interested in running. I am willing to continue in the role. This email makes it official. I am nominating myself for the role of OpenDev Service Coordinator. My goals for the next six months are to wrap up the server upgrades that have been in progress for some time. We've made great progress here; enough to have all python based containers running on Ubuntu Noble. Eventually all of our container images should be hosted on quay.io as well. Gerrit is due for an upgrade. We'll run a pre PTG event to ensure everyone has a chance to sync back up on where our priorities lie and what outstanding efforts could use some help. Finally, I suspect we'll work with Zuul to get zuul-launcher officially released as part of the Zuul suite of tools. Hopefully this sounds good. If not feel free to run as well. I'm more than happy to have someone else step into the role and trade off duties for a bit. [0] https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… Clark

1 0

Help w/ storyboard issue -- cannot delete label
by Khalil, Ghada 14 Aug '25

14 Aug '25

Hi all, I hope someone here can point me to the right contact to help with this issue. I get the following error when trying to delete the "stx.11.0" label from this storyboard: https://storyboard.openstack.org/#!/story/2011361 500: DELETE /api/v1/stories/2011361/tags: DELETE statement on table 'story_storytags' expected to delete 1 row(s); Only 2 were matched. Any ideas on how to fix this issue? From the storyboard history, it appears that the tag was added twice somehow. Thanks, Ghada Khalil StarlingX Release Prime

2 1

Team Meeting Agenda for August 12, 2025
by Clark Boylan 11 Aug '25

11 Aug '25

We will meet with this agenda on August 12, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Held nodes for these two builds: **** https://zuul.opendev.org/t/openstack/build/f1ca0d1f2e054829a4506ececb58bed3 **** https://zuul.opendev.org/t/openstack/build/588723b923e94901af3065143d9df818 *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Fungi announced refstack removal and cleanup: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… *** Need to cleanup the eavesdrop01 server and its DNS records. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-running-todo-list ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** https://review.opendev.org/c/opendev/infra-specs/+/954826 Spec outlining the motivation and plan for Matrix trialing ** Pre PTG Planning (clarkb 20250624) *** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document *** Clarkb has added a number of topic Ideas. Please add more. *** Proposed times: Tuesday October 7 1800-2000 UTC, Wednesday October 8 1500-1700 UTC, Thursday October 9 1500-1700 *** The Tuesday time block would effectively replace our weekly IRC meeting. ** Service Coordinator Election Planning (clarkb 20250729) *** Nomination Period open from August 5, 2025 to August 19, 2025. If necessary we will hold an election from August 20, 2025 to August 27, 2025. All date ranges and times will be in UTC. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… *** Clarkb is more than happy to support someone new in the role. Please reach out if there are questions or interest. *** One week left in the nomination period. ** Loss of upstream Debian bullseye-backports mirror (fungi 20250805) *** https://review.opendev.org/956497 temporarily ignore undefinedtarget *** the workaround isn't too bad, but we need to make a longer-term decision about our mirror contents and the configure-mirrors role ** Adding Ansible 11 to Zuul (corvus 20250805) *** Zuul is growing support for Ansible 11 and dropping Ansible 8 *** What is our rollout plan for OpenDev's Zuul deployment and its tenants? ** Etherpad 2.4.2 Upgrade (clarkb 20250805) *** https://github.com/ether/etherpad-lite/blob/v2.4.2/CHANGELOG.md *** Theming updates have broken the no-skin and custom skin themes. We use no-skin. https://github.com/ether/etherpad-lite/issues/7065 *** We could consider switching to Colibris skin which does work. This skin presents etherpads in a google docs piece of paper manner. ** Cleaning Up the Old voip.ms Account (clarkb 20250812) *** The foundation noticed a small charge on our voip.ms account recently. We haven't used it since asterisk went away. *** Fungi and I would like to clean that up. If we configure meetpad to interface with POTS we can reconfigure SIP and set up a new account or resurrect the old one. *** Any concerns with this plan? ** Moving OpenDev's python-base/python-builder/uwsig-base Images to Quay (clarkb 20250812) *** All of our containers that are build from these base images are now running on Noble nodes with podman *** It should be safe for us to move the base images then rebuild everything using quay as the source. *** Concerns **** Projects outside of OpenDev use these base images (zuul and vexxhost for example). Do we need to coordinate with them or just do our best to get everything updated in a short span of time? **** Image builds that are built using docker as the underlying build system may still fail to speculatively fetch images from Quay. I believe that using buildx/buildkit which is now docker's default fixes this; however, I am unsure if we need to make updates to our build jobs to make that work. * Open discussion

1 0