- service-discuss - lists.opendev.org

Team Meeting Agenda for November 11, 2025
by Clark Boylan 11 Nov '25

11 Nov '25

We will meet with this agenda on November 11, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** Gerrit 3.13 has been released so we should really try to upgrade to 3.11 and then 3.12 soon. *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** https://review.opendev.org/c/opendev/system-config/+/966083 Fix container bind mounts for Gerrit *** https://review.opendev.org/c/opendev/system-config/+/966084 Update to Gerrit 3.10.9 and 3.11.7 *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** The spec (954826) has merged. *** clarkb will start working on this as soon as time permits. ** Upgrade Zuul Zookeeper Cluster to 3.9 (clarkb 20251111) *** https://review.opendev.org/c/opendev/system-config/+/966612 *** This will add some new features Zuul can take advantage of ** Gitea 1.25.1 Upgrade (clarkb 20251028) *** https://review.opendev.org/c/opendev/system-config/+/965960 Upgrade Gitea to 1.25.1 ** Gitea Performance (clarkb 20251028) *** All traffic should be through the load balancer now *** Memcached size limits have been increased *** We should monitor and determine if these changes have improved performance ** Raxflex DFW3 Disabled (clarkb 20251104) *** We discovered the raxflex dfw3 mirror had afs cache errors. fs flushall appeared to get stuck and the kernel oopsed. After forcing a shutdown via openstack the server refuses to boot *** We've asked dan_with what our next steps should be via IRC *** Worst case we rebuild the mirror with a new cinder volume and replace the whole thing. * Open discussion

1 0

Team Meeting Agenda for November 4, 2025
by Clark Boylan 04 Nov '25

04 Nov '25

We will meet with this agenda on November 4, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Clarkb out early Friday and all of Monday. Expect a meeting Tuesday with a possibly delayed agenda. * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** New held nodes are now available. *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Gerrit Spontaneous Shutdown During Summit (clarkb 20251028) *** The Gerrit server was discovered to be in a shutdown state with no recorded errors *** The Nova team indicates this could happen if the hypervisor is unable to allocate memory. The host would be shutdown and Nova would notice but not record an error *** Its possible the underlying cause is something else; either way we need to work with Vexxhost to identify the source of the problem and mitigate it *** Is someone willing to figure out how to file a ticket with Vexxhost? This was their suggested approach last time we had problems. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** AFS mirror content updates (clarkb 20250916) *** https://review.opendev.org/c/opendev/system-config/+/965334 Mirror trixie packages *** Rocky Linux didn't hit these mirror issues because we don't mirror any rocky linux releases. Debian expects mirror content for all releases. ** Zuul Launcher Updates (clarkb 20250923) *** Nodes can be from mixed providers again due to a bug when reassigning unassigned nodes. *** https://review.opendev.org/c/zuul/zuul/+/965954 Fix assignment of unassigned nodes. ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** The spec (954826) has merged. *** clarkb will start working on this as soon as time permits. ** Etherpad 2.5.2 Upgrade (clarkb 20250805) *** https://github.com/ether/etherpad-lite/blob/v2.5.2/CHANGELOG.md *** https://review.opendev.org/c/opendev/system-config/+/956593 *** This fixes our css issues and clarkb believes we are ready to upgrade. ** Gitea 1.25.0 Upgrade (clarkb 20251028) *** https://review.opendev.org/c/opendev/system-config/+/965960 Upgrade Gitea to 1.25.0 ** Gitea Performance (clarkb 20251028) *** https://review.opendev.org/c/opendev/system-config/+/964728 Don't allow direct backend access **** Some performance issues seem related to crawlers hitting backends directly so they aren't balanced properly **** Most recent issue was via traffic to the load balancer frontend so this won't fix everything **** Means we'll have to take extra steps to test backends directly (ssh port forwards) *** https://review.opendev.org/c/opendev/system-config/+/965420 Increase memcached cache size to mitigate effect of crawlers poisoning the cache ** Raxflex DFW3 Disabled (clarkb 20251104) *** We discovered the raxflex dfw3 mirror had afs cache errors. fs flushall appeared to get stuck and the kernel oopsed. After forcing a shutdown via openstack the server refuses to boot *** We've asked dan_with what our next steps should be via IRC *** Worst case we rebuild the mirror with a new cinder volume and replace the whole thing. * Open discussion

1 0

Team Meeting Agenda for October 28, 2025
by Clark Boylan 28 Oct '25

28 Oct '25

I know the PTG is happening this week which might make this a bit tough, but it has been a while since we had a meeting so I want to ensure we get a chance to cover what is happening in OpenDev. We will meet with this agenda on October 28, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Need to hold new test nodes after some Zuul launcher updates cleared out the old ones. *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Gerrit Spontaneous Shutdown During Summit (clarkb 20251028) *** The Gerrit server was discovered to be in a shutdown state with no recorded errors *** The Nova team indicates this could happen if the hypervisor is unable to allocate memory. The host would be shutdown and Nova would notice but not record an error *** Its possible the underlying cause is something else; either way we need to work with Vexxhost to identify the source of the problem and mitigate it ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** AFS mirror content cleanup (clarkb 20250916) *** Can we automate cleanups for reprepro content? In other words are the steps cheap enough during the noop case to run with our regular reprepro update script? *** Additional volumes to look into cleanups for include ceph, docker, and puppetlabs package mirrors as well as our python wheel caches *** CentOS 9 Stream grows at a rate that requires regular quota bumps. Is there anything we can do about that? *** With these cleanups and server upgrades completed we can probably start to consider adding new mirrors again. **** Trixie, CentOS 10 Stream, Rocky Linux ? ** Zuul Launcher Updates (clarkb 20250923) *** NODE_FAILURES created by a bug in zuul-launcher **** https://review.opendev.org/c/zuul/zuul/+/964893 is the fix which we should roll out as soon as it has merged ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** The spec (954826) has merged. *** clarkb will start working on this as soon as time permits. ** Etherpad 2.5.1 Upgrade (clarkb 20250805) *** https://github.com/ether/etherpad-lite/blob/v2.5.1/CHANGELOG.md *** There is an even newer 2.5.1 release. We'll need to recycle the test nodes and check if CSS has improved *** Note we shouldn't upgrade until after the PTG as the event relies on etherpad. ** Gitea 1.24.7 Upgrade (clarkb 20251028) *** https://review.opendev.org/c/opendev/system-config/+/964899/ Upgrade Gitea to 1.24.7 ** Gitea Performance (clarkb 20251028) *** https://review.opendev.org/c/opendev/system-config/+/964728 Don't allow direct backend access **** Some performance issues seem related to crawlers hitting backends directly so they aren't balanced properly **** Most recent issue was via traffic to the load balancer frontend so this won't fix everything **** Means we'll have to take extra steps to test backends directly (ssh port forwards) ** PBR Updates to Handle Setuptools Deprecations (clarkb 20251028) *** Heads up setuptools has an October 31 deadline for no longer relying on easy_install tooling *** https://review.opendev.org/c/openstack/pbr/+/964712/ and children aim to address this * Open discussion

1 0

Re:Re: git review issue
by 赵奇慧 27 Oct '25

27 Oct '25

Hi Tony and Sean, Thanks for your instructions. I've reset my original commit, and changed to a new txt file with link to the video inside. But new problem happens which needs your help. Now the issue has changed to following. Seem like some configuration isn't right. And for the required command and response: --------------------------------------- 赵奇慧中国移动通信研究院网络所手机：13810659120 邮箱：zhaoqihui(a)chinamobile.com ----邮件原文---- 发件人：Tony Breeds 收件人："赵奇慧" 抄　送: service-discuss 发送时间：2025-10-27 10:22:01 主题：Re: git review issue On Sat, 25 Oct 2025 at 01:03, 赵奇慧 wrote: > > Dear experts, > > I'm trying to upload a demo video under cfn/computing-native repo. The demo video is about 25MB. It seems like I've successfully done "git add" and "git commit", but unable to get feedback from server when doing "git review". The following figure shows the situation. May I ask for help or instructions on this? Hello Qihui, As Sean has replied, binary objects are not handled super well with gerrit. Having said that git review should definitely do more than your screenshots show. Can you please reply with the output from the following commands. * git review --version * git remote -v * git review -v That will help us diagnose the issue. Yours Tony. Subject：Re: git review issue On Sat, 25 Oct 2025 at 01:03, 赵奇慧 wrote: > > Dear experts, > > I'm trying to upload a demo video under cfn/computing-native repo. The demo video is about 25MB. It seems like I've successfully done "git add" and "git commit", but unable to get feedback from server when doing "git review". The following figure shows the situation. May I ask for help or instructions on this? Hello Qihui, As Sean has replied, binary objects are not handled super well with gerrit. Having said that git review should definitely do more than your screenshots show. Can you please reply with the output from the following commands. * git review --version * git remote -v * git review -v That will help us diagnose the issue. Yours Tony.

1 1

git review issue
by 赵奇慧 27 Oct '25

27 Oct '25

Dear experts, I'm trying to upload a demo video under cfn/computing-native repo. The demo video is about 25MB. It seems like I've successfully done "git add" and "git commit", but unable to get feedback from server when doing "git review". The following figure shows the situation. May I ask for help or instructions on this? Best, Qihui --------------------------------------- 赵奇慧中国移动通信研究院网络所手机：13810659120 邮箱：zhaoqihui(a)chinamobile.com

3 2

Cancelling October 14, 2025's Team Meeting
by Clark Boylan 14 Oct '25

14 Oct '25

Due to an expected lack of participants I'm going to go ahead and cancel tomorrow's team meeting. As always feel free to bring topics up for discussion on this mailing list or in #opendev on OFTC. Clark

1 0

OpenDev October 2025 Pre PTG
by Clark Boylan 06 Oct '25

06 Oct '25

This is a reminder that instead of holding our typical weekly meeting tomorrow on IRC we will be hosting our first block of the OpenDev Pre PTG event from 1800-2000 UTC on the meetpad that matches our agenda etherpad: https://etherpad.opendev.org/p/opendev-preptg-october-2025. We've found that during the PTG the OpenDev team tends to be busy with other PTG activities which makes it hard to focus on OpenDev specific topics. Hosting a Pre PTG gives us the ability to focus and acts as a quick test of the meetpad system before the larger event. Feel free to add agenda items to the etherpad and we'll see you there tomorrow. There are also additional blocks of time noted on the agenda for Wednesday and Thursday should we need them. Clark

1 0

Team Meeting Agenda for September 30, 2025
by Clark Boylan 30 Sep '25

30 Sep '25

We will meet with this agenda on September 30, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack release happening this week on October 1 * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** New held nodes for Gerrit 3.10.8 and 3.11.5 here: **** https://zuul.opendev.org/t/openstack/build/54f6629a3041466ca2b1cc6bf17886c4 **** https://zuul.opendev.org/t/openstack/build/c9051c435bf7414b986c37256f71538e *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** AFS mirror content cleanup (clarkb 20250916) *** Can we automate cleanups for reprepro content? In other words are the steps cheap enough during the noop case to run with our regular reprepro update script? *** Additional volumes to look into cleanups for include ceph, docker, and puppetlabs package mirrors as well as our python wheel caches *** CentOS 9 Stream grows at a rate that requires regular quota bumps. Is there anything we can do about that? *** With these cleanups and server upgrades completed we can probably start to consider adding new mirrors again. **** Trixie, CentOS 10 Stream, Rocky Linux ? ** Zuul Launcher Updates (clarkb 20250923) *** Networking for rax flex should be sorted now *** Still have leaked ERROR nodes in rax-ord waiting on rax to clear them out as we cannot *** Also haven't heard anything about rax-dfw API problems. We can probably try re-enabling that region and see if the issue persists. ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** The spec (954826) has merged. *** clarkb will start working on this as soon as time permits. ** Pre PTG Planning (clarkb 20250624) *** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document *** Clarkb has added a number of topic Ideas. Please add more. *** Times: Tuesday October 7 1800-2000 UTC, Wednesday October 8 1500-1700 UTC, Thursday October 9 1500-1700 (next week) *** The Tuesday time block will effectively replace our weekly IRC meeting next week. ** Etherpad 2.5.0 Upgrade (clarkb 20250805) *** https://github.com/ether/etherpad-lite/blob/v2.5.0/CHANGELOG.md *** This release is reported to have fixed our css problems. *** https://review.opendev.org/c/opendev/system-config/+/956593/ *** 104.130.127.119 is a held node for testing. You need to edit /etc/hosts to point etherpad.opendev.org at that IP. * Open discussion

1 0

Team Meeting Agenda for September 23, 2025
by Clark Boylan 23 Sep '25

23 Sep '25

We will meet with this agenda on September 23, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** clarkb will be out Thursday the 25th. * Actions from last meeting * Specs Review * Topics ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** New held nodes for Gerrit 3.10.8 and 3.11.5 here: **** https://zuul.opendev.org/t/openstack/build/54f6629a3041466ca2b1cc6bf17886c4 **** https://zuul.opendev.org/t/openstack/build/c9051c435bf7414b986c37256f71538e *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** Next on the list are graphite and backup servers *** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** AFS mirror content cleanup (clarkb 20250916) *** Debian stretch, bullseye backports, and openeuler content has been cleaned up. *** Can we automate cleanups for reprepro content? In other words are the steps cheap enough during the noop case to run with our regular reprepro update script? *** Additional volumes to look into cleanups for include ceph, docker, and puppetlabs package mirrors as well as our python wheel caches *** CentOS 9 Stream grows at a rate that requires regular quota bumps. Is there anything we can do about that? *** With these cleanups and server upgrades completed we can probably start to consider adding new mirrors again. ** Lists Server Slowness (clarkb 20250909) *** Mailman's data is now backed by a dedicated volume capable of many more IOPs than the root disk. Let us know if you see new occurrences of unexpected slowness. ** Deleting ze11 (clarkb 20250923) *** The ze11 zuul executor has been shutdown for some time due to network slowness cloning the nova repository. *** We have a 10 minute timeout for that clone, most nodes do it in ~4 minutes, but ze11 does it in about 12-13 minutes. *** This slowness may also impact job runtimes as well *** At this point 11 or even 10 executors is probably sufficient for our needs the simplest solution here may be to delete ze11 entirely (and ze12 if we find its existence confusing) ** Zuul Launcher Updates (clarkb 20250923) *** We've run into weirdness with rax flex clouds booting instances with two network interfaces if networks are configured via zuul-providers config. This does not occur when networks are configured via clouds.yaml instead. This will need debugging *** All existing nodepool nodes have been deleted out of all cloud regions except for rax-ord. There are a number of nodepool instances there in an ERROR state that we cannot delete ourselves. *** I have sent email to Rackspace asking them to clean those nodes up as I believe they do count against our quota. *** We have also disabled rax-dfw as it started having API errors similar to what we saw in IAD and ORD previously. My email included questions about this situation too. ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** The spec (954826) has merged. *** clarkb will start working on this as soon as time permits. ** Pre PTG Planning (clarkb 20250624) *** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document *** Clarkb has added a number of topic Ideas. Please add more. *** Times: Tuesday October 7 1800-2000 UTC, Wednesday October 8 1500-1700 UTC, Thursday October 9 1500-1700 *** The Tuesday time block would effectively replace our weekly IRC meeting. ** Etherpad 2.5.0 Upgrade (clarkb 20250805) *** https://github.com/ether/etherpad-lite/blob/v2.5.0/CHANGELOG.md *** This release is reported to have fixed our css problems. *** https://review.opendev.org/c/opendev/system-config/+/956593/ *** 104.130.127.119 is a held node for testing. You need to edit /etc/hosts to point etherpad.opendev.org at that IP. * Open discussion

1 0

Team Meeting Agenda for September 16, 2025
by Clark Boylan 16 Sep '25

16 Sep '25

We will meet with this agenda on September 16, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Moving OpenDev's python-base/python-builder/uwsgi-base Images to Quay (clarkb 20250812) *** The images have moved and we're no longer publishing to docker hub *** https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… *** I believe that all images within OpenDev that consume these base images have been updated to pull from the new location (including Gerrit) *** If you find images that need updates please push a change and let us know. ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** New held nodes for Gerrit 3.10.8 and 3.11.5 here: **** https://zuul.opendev.org/t/openstack/build/54f6629a3041466ca2b1cc6bf17886c4 **** https://zuul.opendev.org/t/openstack/build/c9051c435bf7414b986c37256f71538e *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. **** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** The entire openafs and kerberos system has been upgraded to Noble except for afs02.dfw.openstack.org. Waiting on RW volume moves to complete before performing this last upgrade **** Will need to remember to remove all of the hosts from the emergency file when done. *** Next on the list are graphite and backup servers *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** AFS mirror content cleanup (clarkb 20250916) *** Fungi identified a number of AFS mirror cleanups that can be performed once server upgrades are complete *** OpenEuler mirror content is no longer needed as we don't have any test images *** Debian stretch and bullseye backports are no longer needed as they are both old enough to have aged out *** For the reprepro content we should also look into automatically clearing content if the cost to do so is actually a noop when we haven't removed releases from the mirror list. ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** The spec (954826) has merged. *** clarkb will start working on this as soon as time permits. ** Pre PTG Planning (clarkb 20250624) *** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document *** Clarkb has added a number of topic Ideas. Please add more. *** Times: Tuesday October 7 1800-2000 UTC, Wednesday October 8 1500-1700 UTC, Thursday October 9 1500-1700 *** The Tuesday time block would effectively replace our weekly IRC meeting. ** Etherpad 2.5.0 Upgrade (clarkb 20250805) *** https://github.com/ether/etherpad-lite/blob/v2.5.0/CHANGELOG.md *** This release is reported to have fixed our css problems. *** https://review.opendev.org/c/opendev/system-config/+/956593/ *** 104.130.127.119 is a held node for testing. You need to edit /etc/hosts to point etherpad.opendev.org at that IP. ** Lists Server Slowness (clarkb 20250909) *** We think we have tracked this down to the different disk_io_index values between Performance and Standard flavors on rax classic *** Specifically the value is 2 on Standard (lists) and 40 on mirror and fio measures about a 20x differences in iops during randread testing to match *** We will need to migrate the mailman3 installation off of the server disk and onto a cinder volume to improve the number of iops. * Open discussion

1 0