- service-discuss - lists.opendev.org

Team Meeting Agenda for July 29, 2025
by Clark Boylan 29 Jul '25

29 Jul '25

We will meet with this agenda on July 29, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Clarkb out Monday August 4 * Actions from last meeting * Specs Review * Topics ** Zuul-launcher migration(corvus 20240910) *** Mixed provider nodesets should only be possible now if the only way to fulfill the requests is through different providers (think openstack vm label and k8s pod label in one nodeset). *** Replaced gitea-lb02 with gitea-lb03 as IPv6 issues on gitea-lb02 were impacting image build jobs *** Also updated load balancer health checks to hopefully get a more accurate picture of when gitea is running properly on each node. *** Nodepool is gone. ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Held nodes for these two builds: **** https://zuul.opendev.org/t/openstack/build/f1ca0d1f2e054829a4506ececb58bed3 **** https://zuul.opendev.org/t/openstack/build/588723b923e94901af3065143d9df818 *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack and eavesdrop servers remaining. *** Eavesdrop stores data on a cinder volume. We will need a downtime to stop bots on the old server long enough to copy/move data to the new server before starting services there. Is there a particularly good time for doing this? *** https://review.opendev.org/c/opendev/system-config/+/955544/ prep step for eavesdrop move to Noble *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Vexxhost backup server inaccessible (clarkb 20250729) *** The backup server running in vexxhost has become unreachable but Nova considers it to be up. We probably need to ask Nova to reboot it. *** Keep an eye out for similar issues going forward. Might be worth noting what tipped fungi off to this as well. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** https://review.opendev.org/c/opendev/infra-specs/+/954826 Spec outlining the motivation and plan for Matrix trialing ** Pre PTG Planning (clarkb 20250624) *** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document ** Service Coordinator Election Planning (clarkb 20250729) *** Proposal: Nomination Period open from August 5, 2025 to August 19, 2025. If necessary we will hold an election from August 20, 2025 to August 27, 2025. All date ranges and times will be in UTC. *** Clarkb is more than happy to support someone new in the role. Please reach out if there are questions or interest. * Open discussion

1 0

Team Meeting Agenda for July 22, 2025
by Clark Boylan 22 Jul '25

22 Jul '25

We will meet on July 22, 2025 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher migration(corvus 20240910) *** Mixed provider nodesets are impacting some jobs **** https://review.opendev.org/c/zuul/zuul/+/955545 is current proposal due to lack of verbose errors from openstack clouds *** Image uploads may be overwhelming rax-flex sjc3 swift proxies **** Python requests/urllib3 is returning a 499 error code which can happen when proxies timeout connections to the backend and pass that along to the client. **** May need to sync up with rackspace on this as there may need to be updates to their proxies **** May also consider a different swift location? *** Nodepool is shutdown and no longer providing any images or nodes to zuul. ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Held nodes for these two builds: **** https://zuul.opendev.org/t/openstack/build/f1ca0d1f2e054829a4506ececb58bed3 **** https://zuul.opendev.org/t/openstack/build/588723b923e94901af3065143d9df818 *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack and eavesdrop servers remaining. *** https://review.opendev.org/c/opendev/system-config/+/955544/ prep step for eavesdrop move to Noble *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** OFTC Matrix bridge no longer supporting new users (clarkb 20250520) *** https://review.opendev.org/c/opendev/infra-specs/+/954826 Spec outlining the motivation and plan for Matrix trialing ** Pre PTG Planning (clarkb 20250624) *** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document * Open discussion

1 0

Team Meeting Agenda for July 15, 2025
by Clark Boylan 15 Jul '25

15 Jul '25

I will be out tomorrow, but Jeremy has volunteered to chair the meeting. The meeting will be held at 19:00 UTC in #opendev-meeting, tomorrow, July 15, 2025 with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher migration(corvus 20240910) *** Mixed provider nodesets are impacting some jobs **** The bugfixes seem to have this under control. clarkb hasn't seen any recent complaints about this. *** Image uploads may be overwhelming rax-flex sjc3 swift proxies **** Python requests/urllib3 is returning a 499 error code which can happen when proxies timeout connections to the backend and pass that along to the client. **** May need to sync up with rackspace on this as there may need to be updates to their proxies **** May also consider a different swift location? *** New Images for Trixie, Centos 10 Stream, and Rocky Linux 10 *** Ubuntu Bionic Arm64 is the last label still provided by Nodepool **** Job cleanup branches off from https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/954761/ ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Omnibus update to Gerrit images planned for Friday or Monday to incorporate several updates (zuul status plugin, CLA removals, and quay.io image hosting) **** https://review.opendev.org/c/opendev/system-config/+/882900 Host Gerrit images on quay.io **** https://review.opendev.org/c/opendev/system-config/+/954376 CLA file removal **** Needs to be done after CLA usage is removed from project ACLs *** Need to hold new test nodes. Perhaps that is best for after the container images move? *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack and eavesdrop servers remaining. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** OFTC Matrix bridge no longer supporting new users (clarkb 20250520) *** https://review.opendev.org/c/opendev/infra-specs/+/954826 Spec outlining the motivation and plan for Matrix trialing ** Pre PTG Planning (clarkb 20250624) *** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document * Open discussion

1 0

OpenDev Team Meeting Agenda for July 8, 2025
by Clark Boylan 08 Jul '25

08 Jul '25

We will meet with this agenda on July 8, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher migration(corvus 20240910) *** Mixed provider nodesets are impacting some jobs **** corvus was able to run down a couple of bugs around that, but some jobs (periodic jobs from provided examples) still occasionally see mixed providers *** Image build gnutls errors continue to be a problem when image builds update git repos **** Current workaround is dib has been updated to retry those git repo updates **** Looking at haproxy logs clarkb sees initial updates come from ipv4 addresses, then ipv6 addresses, then the failure occurs and things move back to ipv4 before finally using ipv6 and completing. **** https://zuul.opendev.org/t/opendev/build/2f4200f24ae4492295d5c43307e93038/l… That behavior was seen with this build **** One theory is that nodes are not booting with working ipv6 so use ipv4, then get get an RA or similar that causes ipv6 to work. Then something breaks ipv6 causing a fallback to ipv4. ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Automated testing is currently testing this generally works for us. *** https://review.opendev.org/c/opendev/system-config/+/882900 Host Gerrit images on quay.io *** Need to hold new test nodes. Perhaps that is best for after the container images move? *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack and eavesdrop servers remaining. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** EL10 Test Nodes (clarkb 20250708) *** https://review.opendev.org/c/opendev/zuul-providers/+/953460 CentOS 10 Stream *** https://review.opendev.org/c/opendev/zuul-providers/+/954265 Rocky Linux 10 ** Adding Debian Trixie Test Nodes (clarkb 20250708) *** https://review.opendev.org/c/opendev/zuul-providers/+/951471 ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** OFTC Matrix bridge no longer supporting new users (clarkb 20250520) *** Clarkb to write a spec explaining the motivation for a move to matrix for opendev comms. *** Then idea is to do a bit of a trial and see if it does help users who primarily rely on matrix for IRC as well as new users. ** Pre PTG Planning (clarkb 20250624) *** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document * Open discussion

1 0

Team Meeting Agenda for July 1, 2025
by Clark Boylan 01 Jul '25

01 Jul '25

We will meet with this agenda on July 1, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher migration(corvus 20240910) *** Much of the job configs in OpenDev have converted to using labels provided by niz/zuul-launcher as of June 30. *** Be on the lookout for unexpected behaviors *** At some point we'll need to plan for nodepool cleanup. *** Image builds are plagued by gnutls connection issues here and there. One appraoch is to have git fetches retry in image builds. **** https://review.opendev.org/c/openstack/diskimage-builder/+/721581 ** Gerrit shutdown problems (clarkb 20250527) *** This was almost certainly related to our h2 compaction time increase change. *** We reverted that change and when restarting to pick up the revert I manually ran kill -HUP against Gerrit. It took ages again. I ran lsof and strace and we could see that Gerrit was reading, writing, and syncing to one of the h2 cache files. *** I expect SIGINT to work via docker compose and podman at this point without issue as h2 compaction should be quick. *** We'll just continue to delete the large h2 cache files during restarts. ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Automated testing is currently testing this generally works for us. *** 104.130.253.194 is a held Gerrit 3.11 node for testing purposes. *** https://review.opendev.org/c/opendev/system-config/+/882900 Host Gerrit images on quay.io *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack and eavesdrop servers remaining. *** clarkb has replaced the zookeeper cluster with a new triple (zk01-03) *** corvus replaced all zuul servers (reused existing names) *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** OFTC Matrix bridge no longer supporting new users (clarkb 20250520) *** Clarkb to write a spec explaining the motivation for a move to matrix for opendev comms. *** Then idea is to do a bit of a trial and see if it does help users who primarily rely on matrix for IRC as well as new users. ** Pre PTG Planning (clarkb 20250624) *** OpenStack release is October 1 *** OpenInfra Summit is October 17-19 *** PTG is October 27-31 *** Would 2-3 days the week of October 6-10 work for us to have some focused OpenDev Pre PTG time? * Open discussion

1 0

Request to Clear Previous Commits and Start Fresh
by 田玉桐 24 Jun '25

24 Jun '25

I am tianyutong, responsible for the project heterogeneous-distributed-training-framework.(cfn/heterogeneous-distributed-training-framework: Heterogeneous distributed training framework researches the training method and framework over heterogeneous hardware. - heterogeneous-distributed-training-framework - OpenDev: Free Software Needs Free Tools) I attempted to overwrite the previous commits on the following branches: master, release_1.0.0, and core_0.8.0. However, due to merge conflicts, the commit history became corrupted, and the format is now incorrect. I need assistance in clearing all the previous commits and starting fresh. I would like to push new commits without keeping any of the previous history. Could you please guide me on how to remove the old commits and push the new changes? Best regards, tianyutong

2 1

Re:Re:Re: Authorization and Branch Creation in OpenDev Repo
by 田玉桐 24 Jun '25

24 Jun '25

Dear OpenDev Community, I am Tianyutong from China Mobile, responsible for the project cfn/heterogeneous-distributed-training-framework. I encountered a permission issue while trying to create a new branch, and I need your assistance to resolve it. Issue Description When attempting to create a branch named "release_1.0.0" via the OpenDev web interface (following the official documentation), I received the following error: An error occurred Error 403 (Forbidden): not permitted: create on refs/heads/release_1.0.0 Endpoint: /projects/*/branches/* The error screenshot is attached for reference. Request Could you please grant our team permission to create new branches in cfn/heterogeneous-distributed-training-framework? We need this privilege to manage version branches for our project. Thank you for your support! Best regards, Tianyutongtianyutong(a)sd.chinamobile.com ----邮件原文---- 发件人："赵奇慧" <zhaoqihui(a)chinamobile.com> 收件人：fungi <fungi(a)yuggoth.org> 抄　送: "黄蕾" <huangleiyjy(a)chinamobile.com>,tianyutong <tianyutong(a)sd.chinamobile.com>,chenzhengwei <chenzhengwei(a)sd.chinamobile.com>,qihuiz93 <qihuiz93(a)163.com> 发送时间：2025-06-12 09:32:52 主题：Re:Re: Authorization and Branch Creation in OpenDev Repo Hi Jeremy, Thank you so much for your kind help and advice. We'll try with the link. Best, Qihui --------------------------------------- 赵奇慧中国移动通信研究院网络所手机：13810659120 邮箱：zhaoqihui(a)chinamobile.com ----邮件原文---- 发件人：fungi 收件人："赵奇慧" 抄　送: "黄蕾" ,tianyutong ,chenzhengwei ,qihuiz93 发送时间：2025-06-11 20:50:29 主题：Re: Authorization and Branch Creation in OpenDev Repo On 2025-06-11 17:18:48 +0800 (+0800), 赵奇慧 wrote: > This is Qihui Zhao from China Mobile and OIF CFN WG. We write this > email to ask about how to get management and branch creation > authorization on repo > cfn/heterogeneous-distributed-training-framework. [...] I believe the documentation you're looking for is here: https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#branches In the future, I recommend asking questions on the service-discuss(a)lists.opendev.org mailing list or in the #opendev channel on the OFTC IRC network, since there are many people in both places who could answer and I'm often not available due to other obligations. Hope that helps! -- Jeremy Stanley Subject：Re: Authorization and Branch Creation in OpenDev Repo On 2025-06-11 17:18:48 +0800 (+0800), 赵奇慧 wrote: > This is Qihui Zhao from China Mobile and OIF CFN WG. We write this > email to ask about how to get management and branch creation > authorization on repo > cfn/heterogeneous-distributed-training-framework. [...] I believe the documentation you're looking for is here: https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#branches In the future, I recommend asking questions on the service-discuss(a)lists.opendev.org mailing list or in the #opendev channel on the OFTC IRC network, since there are many people in both places who could answer and I'm often not available due to other obligations. Hope that helps! -- Jeremy Stanley

3 4

Team Meeting Agenda for June 24, 2025
by Clark Boylan 24 Jun '25

24 Jun '25

We will meet on June 24, 2025 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher migration(corvus 20240910) *** Auto hold functionality is rolling out *** Are there any other required features before all OpenDev tenants switch to use zuul launcher managed resources? ** Gerrit shutdown problems (clarkb 20250527) *** After successfully using SIGINT to shutdown gerrit a few weeks ago the recent restart last week failed to shutdown and timed out to SIGKILL *** https://review.opendev.org/c/opendev/system-config/+/950595 One theory is that h2 compaction time may be slowing down shutdown enough to time out *** Another theory is that SIGINT just doesn't work as expected and we need to use SIGHUP (which podman blocks by default) *** Proposal: When convenient time for a restart we land 950595. Since that won't apply until the next restart we manually issue a kill -HUP against the gerrit process to see if that works better. Then once h2 compaction cleanup is in place the restart after can try SIGINT again. ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Automated testing is currently testing this generally works for us. *** 104.130.253.194 is a held Gerrit 3.11 node for testing purposes. *** https://review.opendev.org/c/opendev/system-config/+/882900 Host Gerrit images on quay.io *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack, mirror-update, eavesdrop, zuul schedulers, and zookeeper servers remaining. *** clarkb is in the process of rolling out a new mirror-update with new zookeepers to follow *** corvus is replacing zuul servers over the next week or two *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** OFTC Matrix bridge no longer supporting new users (clarkb 20250520) *** Clarkb to write a spec explaining the motivation for a move to matrix for opendev comms. *** Then idea is to do a bit of a trial and see if it does help users who primarily rely on matrix for IRC as well as new users. ** Adding CentOS 10 Stream Support to Glean, DIB, and Nodepool (clarkb 20250527) *** https://review.opendev.org/c/openstack/diskimage-builder/+/949942 DIB functional testing without Nodepool **** This will allow us to have greater control of the devstack environment allowing us to configure cpu features that CentOS 10 Stream requires *** https://review.opendev.org/c/openstack/diskimage-builder/+/934045 DIB support for CentOS 10 Stream ** projects.yaml normalization *** a (possibly intentional) regression in ruamel.yaml has started adding spaces to the end of wrapped strings, akin to format=flowed E-mail wrapping **** https://sourceforge.net/p/ruamel-yaml/tickets/546/ *** this is impacting our normalization job output **** https://review.opendev.org/952006 ** Pre PTG Planning (clarkb 20250624) *** OpenStack release is October 1 *** OpenInfra Summit is October 17-19 *** PTG is October 27-31 *** Would 2-3 days the week of October 6-10 work for us to have some focused OpenDev Pre PTG time? * Open discussion

1 0

Team Meeting Agenda for June 17, 2025
by Jeremy Stanley 17 Jun '25

17 Jun '25

We will meet with this agenda on June 17, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) ** Gerrit shutdown problems (clarkb 20250527) *** After successfully using SIGINT to shutdown gerrit a few weeks ago the recent restart last week failed to shutdown and timed out to SIGKILL *** https://review.opendev.org/c/opendev/system-config/+/950595 One theory is that h2 compaction time may be slowing down shutdown enough to time out *** Another theory is that SIGINT just doesn't work as expected and we need to use SIGHUP (which podman blocks by default) *** Proposal: When convenient time for a restart we land 950595. Since that won't apply until the next restart we manually issue a kill -HUP against the gerrit process to see if that works better. Then once h2 compaction cleanup is in place the restart after can try SIGINT again. ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Automated testing is currently testing this generally works for us. *** 104.130.253.194 is a held Gerrit 3.11 node for testing purposes. *** https://review.opendev.org/c/opendev/system-config/+/882900 Host Gerrit images on quay.io *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack, mirror-update, eavesdrop, zuul schedulers, and zookeeper servers remaining. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** OFTC Matrix bridge no longer supporting new users (clarkb 20250520) *** https://github.com/matrix-org/matrix-appservice-irc/issues/1851 *** This means that matrix bridge users don't see messages from newer IRC nicks/users *** Options: **** Declare bankruptcy: Matrix needs a Matrix client and IRC needs an IRC client **** Pay EMS for a bridge. This is apparently billed per active IRC bridge user and could be a large increase on current EMS hosting cost **** Host our own bridge. https://postmarketos.org/blog/2025/03/31/matrix-bridge-migration/ has useful related info **** Embrace Matrix and move there. This represents a compromise between those who want IRC and those who want something like Slack/Discord. But for both sides the important functionality/needs seems to be preserved. (IRC users continue to get to use open source tooling with federated servers. Slack/Discord users get a system that runs over HTTPS using clients with modern bells and whistles). ** Enabling hashtags globally (corvus 20250523) *** Agreed to apply this the day of the meeting (June 3, 2025). *** Announced at https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… ** Adding CentOS 10 Stream Support to Glean, DIB, and Nodepool (clarkb 20250527) *** https://review.opendev.org/c/opendev/glean/+/941672 Glean NetworkManager Keyfile support *** https://review.opendev.org/c/openstack/diskimage-builder/+/934045 DIB support for CentOS 10 Stream *** There have been a few struggles **** CentOS 10 Stream requires x86-64-v3 which clouds other than rax classic can provide (or about only 40% of our capacity). **** CentOS 10 Stream upstream images label the / partition as /boot with a partition uuid. This seems like an upstream bug. centos-minimal element mitigates this. ** projects.yaml normalization *** a (possibly intentional) regression in ruamel.yaml has started adding spaces to the end of wrapped strings, akin to format=flowed E-mail wrapping **** https://sourceforge.net/p/ruamel-yaml/tickets/546/ *** this is impacting our normalization job output **** https://review.opendev.org/952006 * Open discussion -- Jeremy Stanley

1 0

Team Meeting Agenda for June 10, 2025
by Jeremy Stanley 10 Jun '25

10 Jun '25

We will meet with this agenda on June 10, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) ** Gerrit shutdown problems (clarkb 20250527) *** After successfully using SIGINT to shutdown gerrit a few weeks ago the recent restart last week failed to shutdown and timed out to SIGKILL *** https://review.opendev.org/c/opendev/system-config/+/950595 One theory is that h2 compaction time may be slowing down shutdown enough to time out *** Another theory is that SIGINT just doesn't work as expected and we need to use SIGHUP (which podman blocks by default) *** Proposal: When convenient time for a restart we land 950595. Since that won't apply until the next restart we manually issue a kill -HUP against the gerrit process to see if that works better. Then once h2 compaction cleanup is in place the restart after can try SIGINT again. ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Automated testing is currently testing this generally works for us. *** 104.130.253.194 is a held Gerrit 3.11 node for testing purposes. *** https://review.opendev.org/c/opendev/system-config/+/882900 Host Gerrit images on quay.io *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack, mirror-update, eavesdrop, zuul schedulers, and zookeeper servers remaining. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** OFTC Matrix bridge no longer supporting new users (clarkb 20250520) *** https://github.com/matrix-org/matrix-appservice-irc/issues/1851 *** This means that matrix bridge users don't see messages from newer IRC nicks/users *** Options: **** Declare bankruptcy: Matrix needs a Matrix client and IRC needs an IRC client **** Pay EMS for a bridge. This is apparently billed per active IRC bridge user and could be a large increase on current EMS hosting cost **** Host our own bridge. https://postmarketos.org/blog/2025/03/31/matrix-bridge-migration/ has useful related info **** Embrace Matrix and move there. This represents a compromise between those who want IRC and those who want something like Slack/Discord. But for both sides the important functionality/needs seems to be preserved. (IRC users continue to get to use open source tooling with federated servers. Slack/Discord users get a system that runs over HTTPS using clients with modern bells and whistles). ** Enabling hashtags globally (corvus 20250523) *** Agreed to apply this the day of the meeting (June 3, 2025). *** Announced at https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… ** Adding CentOS 10 Stream Support to Glean, DIB, and Nodepool (clarkb 20250527) *** https://review.opendev.org/c/opendev/glean/+/941672 Glean NetworkManager Keyfile support *** https://review.opendev.org/c/openstack/diskimage-builder/+/934045 DIB support for CentOS 10 Stream *** There have been a few struggles **** CentOS 10 Stream requires x86-64-v3 which clouds other than rax classic can provide (or about only 40% of our capacity). **** CentOS 10 Stream upstream images label the / partition as /boot with a partition uuid. This seems like an upstream bug. centos-minimal element mitigates this. * Open discussion -- Jeremy Stanley

1 0