service-discuss

Download

service-discuss@lists.opendev.org

September 2025

1 participants
2 discussions

Team Meeting Agenda for September 9, 2025
by Clark Boylan 08 Sep '25

08 Sep '25

We will meet with this agenda on September 9, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Held nodes for these two builds: **** https://zuul.opendev.org/t/openstack/build/f1ca0d1f2e054829a4506ececb58bed3 **** https://zuul.opendev.org/t/openstack/build/588723b923e94901af3065143d9df818 *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Kerberos and OpenAFS servers all updated to Jammy. afs01.dfw.openstack.org was subsequently updated to Noble and then began to unping **** Seems likely that Noble is simply not honoring the old /etc/network/interfaces scripts and needs netplan or similar. We should be able to rescue the node and set that directly or set a password for login via the console to do it interactively. **** I've also discovered that all but afs02.dfw.openstack.org and kdc04.opendev.org are running on an older xen version. In CI we've seen Noble boot with a single vcpu on the older hypervisors. We may also need to request these hosts get migrated to new hypervisors before updating. We can see what afs01.dfw.openstack.org looks like when networking is happy again. *** Next on the list are graphite and backup servers *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** https://review.opendev.org/c/opendev/infra-specs/+/954826 Spec outlining the motivation and plan for Matrix trialing ** Pre PTG Planning (clarkb 20250624) *** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document *** Clarkb has added a number of topic Ideas. Please add more. *** Times: Tuesday October 7 1800-2000 UTC, Wednesday October 8 1500-1700 UTC, Thursday October 9 1500-1700 *** The Tuesday time block would effectively replace our weekly IRC meeting. ** Loss of upstream Debian bullseye-backports mirror (fungi 20250805) *** https://review.opendev.org/956497 temporarily ignore undefinedtarget *** Zuul-jobs will no longer enable debian backports by default on September 9 (day of this meeting) **** https://lists.zuul-ci.org/archives/list/zuul-announce@lists.zuul-ci.org/thr… *** Once zuul-jobs is updated we can delete the content from our mirrors and drop the workaround ** Etherpad 2.5.0 Upgrade (clarkb 20250805) *** https://github.com/ether/etherpad-lite/blob/v2.5.0/CHANGELOG.md *** This release is reported to have fixed our css problems. *** https://review.opendev.org/c/opendev/system-config/+/956593/ *** 104.130.127.119 is a held node for testing. You need to edit /etc/hosts to point etherpad.opendev.org at that IP. ** Moving OpenDev's python-base/python-builder/uwsgi-base Images to Quay (clarkb 20250812) *** All of our containers that are build from these base images are now running on Noble nodes with podman *** A zuul user reported problems with the move to always using buildx to build images. This has been corrected by setting network=host on the build itself (not just the custom builder) *** If there are no other complaints by the time of this meeting I think we are in good shape to actually proceed with the container image move on our side. ** Adding Debian Trixie Base Python Container Images (clarkb 20250826) *** https://review.opendev.org/c/opendev/system-config/+/958480 *** Trixie is here. Let's bootstrap some container images so that we can migrate over to the new release in our containers. ** Dropping Ubuntu Bionic Test Nodes (clarkb 20250826) *** Zuul will be dropping Ansible 9 support soon. When that happens Ubuntu Bionic test nodes will stop working in Zuul as Python on them is too old for Ansible 11 *** Decided that since Bionic is well past its EOL date we don't need to make a big announcement. At least not yet. But cleanup is a good thing. *** I wrote a number of changes to OpenDev tools like glean, bindep, and git-review to remove Bionic based test jobs *** Please call out others that you notice or push up changes for them too. *** It should still be possible to run python2.7 jobs if we need them on Ubuntu Jammy. Note that tox 4 is not compatible with python2.7 so we need to pin tox in the jobs to <4 ** Lists Server Slowness (clarkb 20250909) *** Like every other service mailman is getting bombarded by web crawlers *** User agent lists have been updated to try and cut down on some of the impacts *** Service is extremely slow at times. When this occurs top reports significant iowait *** Thoughts on further improvements that could be made? * Open discussion

1 0

Team Meeting Agenda for September 2, 2025
by Clark Boylan 02 Sep '25

02 Sep '25

We will meet on September 2, 2025 (today) at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Held nodes for these two builds: **** https://zuul.opendev.org/t/openstack/build/f1ca0d1f2e054829a4506ececb58bed3 **** https://zuul.opendev.org/t/openstack/build/588723b923e94901af3065143d9df818 *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Kerberos and OpenAFS servers are in the middle of getting upgraded. Most are running Jammy now. Then after everything is running Jammy the next round of updates will get us on Noble. *** Next on the list are graphite and backup servers *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** https://review.opendev.org/c/opendev/infra-specs/+/954826 Spec outlining the motivation and plan for Matrix trialing ** Pre PTG Planning (clarkb 20250624) *** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document *** Clarkb has added a number of topic Ideas. Please add more. *** Times: Tuesday October 7 1800-2000 UTC, Wednesday October 8 1500-1700 UTC, Thursday October 9 1500-1700 *** The Tuesday time block would effectively replace our weekly IRC meeting. ** Loss of upstream Debian bullseye-backports mirror (fungi 20250805) *** https://review.opendev.org/956497 temporarily ignore undefinedtarget *** Zuul-jobs will no longer enable debian backports by default on September 9 **** https://lists.zuul-ci.org/archives/list/zuul-announce@lists.zuul-ci.org/thr… *** Once zuul-jobs is updated we can delete the content from our mirrors and drop the workaround ** Etherpad 2.5.0 Upgrade (clarkb 20250805) *** https://github.com/ether/etherpad-lite/blob/v2.5.0/CHANGELOG.md *** This release is reported to have fixed our css problems. *** https://review.opendev.org/c/opendev/system-config/+/956593/ *** 104.130.127.119 is a held node for testing. You need to edit /etc/hosts to point etherpad.opendev.org at that IP. ** Moving OpenDev's python-base/python-builder/uwsgi-base Images to Quay (clarkb 20250812) *** All of our containers that are build from these base images are now running on Noble nodes with podman *** Turns out to use speculative images via the buildset registry we always need to build with a custom buildx builder **** https://review.opendev.org/c/zuul/zuul-jobs/+/958783 *** Once that lands we can switch the default builder back to docker in system-config then use a consistent builder for all container image builds ** Adding Debian Trixie Base Python Container Images (clarkb 20250826) *** https://review.opendev.org/c/opendev/system-config/+/958480 *** Trixie is here. Let's bootstrap some container images so that we can migrate over to the new release in our containers. ** Dropping Ubuntu Bionic Test Nodes (clarkb 20250826) *** Zuul will be dropping Ansible 9 support soon. When that happens Ubuntu Bionic test nodes will stop working in Zuul as Python on them is too old for Ansible 11 *** Decided that since Bionic is well past its EOL date we don't need to make a big announcement. At least not yet. But cleanup is a good thing. *** https://review.opendev.org/q/hashtag:%22drop-bionic%22+status:open *** This cleanup will also enable us to drop additional content from the afs ubuntu mirrors which will make afs server upgrades slightly easier. ** Temporary Shutdown of raxflex sjc3 for provider maintenance window (clarkb 20250902) *** https://review.opendev.org/c/opendev/zuul-providers/+/959200 *** The cinder volume backing our mirror cache is going to be unavailable tomorrow for maintenance. Clarkb will disable the region in Zuul and shutdown the mirror today in preparation. * Open discussion

1 0