July 2025 - service-discuss - lists.opendev.org

August 2025 OpenDev Service Coordinator Election
by Clark Boylan 30 Jul '25

30 Jul '25

It is that time of year again where we need to elect an OpenDev Service Coordinator [0] for the next six months. Nominations will be open from August 5, 2025 to August 19, 2025. If we have more than one candidate we will vote for our preferred candidate from August 20, 2025 to August 27, 2025. All dates are based on a UTC clock. Please send your self nominations to this mailing list during the nomination time frame. You are eligible to run and vote if you have contributed to OpenDev git repositories in the last 12 months (this period will be August 5, 2024 to August 4, 2025). I continue to encourage anyone who may be interested to reach out and ask questions. I am happy to support someone new in the role. [0] https://docs.opendev.org/opendev/system-config/latest/project.html#governan… Clark

1 0

Team Meeting Agenda for July 29, 2025
by Clark Boylan 28 Jul '25

28 Jul '25

We will meet with this agenda on July 29, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Clarkb out Monday August 4 * Actions from last meeting * Specs Review * Topics ** Zuul-launcher migration(corvus 20240910) *** Mixed provider nodesets should only be possible now if the only way to fulfill the requests is through different providers (think openstack vm label and k8s pod label in one nodeset). *** Replaced gitea-lb02 with gitea-lb03 as IPv6 issues on gitea-lb02 were impacting image build jobs *** Also updated load balancer health checks to hopefully get a more accurate picture of when gitea is running properly on each node. *** Nodepool is gone. ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Held nodes for these two builds: **** https://zuul.opendev.org/t/openstack/build/f1ca0d1f2e054829a4506ececb58bed3 **** https://zuul.opendev.org/t/openstack/build/588723b923e94901af3065143d9df818 *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack and eavesdrop servers remaining. *** Eavesdrop stores data on a cinder volume. We will need a downtime to stop bots on the old server long enough to copy/move data to the new server before starting services there. Is there a particularly good time for doing this? *** https://review.opendev.org/c/opendev/system-config/+/955544/ prep step for eavesdrop move to Noble *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Vexxhost backup server inaccessible (clarkb 20250729) *** The backup server running in vexxhost has become unreachable but Nova considers it to be up. We probably need to ask Nova to reboot it. *** Keep an eye out for similar issues going forward. Might be worth noting what tipped fungi off to this as well. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520) *** https://review.opendev.org/c/opendev/infra-specs/+/954826 Spec outlining the motivation and plan for Matrix trialing ** Pre PTG Planning (clarkb 20250624) *** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document ** Service Coordinator Election Planning (clarkb 20250729) *** Proposal: Nomination Period open from August 5, 2025 to August 19, 2025. If necessary we will hold an election from August 20, 2025 to August 27, 2025. All date ranges and times will be in UTC. *** Clarkb is more than happy to support someone new in the role. Please reach out if there are questions or interest. * Open discussion

1 0

Team Meeting Agenda for July 22, 2025
by Clark Boylan 21 Jul '25

21 Jul '25

We will meet on July 22, 2025 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher migration(corvus 20240910) *** Mixed provider nodesets are impacting some jobs **** https://review.opendev.org/c/zuul/zuul/+/955545 is current proposal due to lack of verbose errors from openstack clouds *** Image uploads may be overwhelming rax-flex sjc3 swift proxies **** Python requests/urllib3 is returning a 499 error code which can happen when proxies timeout connections to the backend and pass that along to the client. **** May need to sync up with rackspace on this as there may need to be updates to their proxies **** May also consider a different swift location? *** Nodepool is shutdown and no longer providing any images or nodes to zuul. ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Held nodes for these two builds: **** https://zuul.opendev.org/t/openstack/build/f1ca0d1f2e054829a4506ececb58bed3 **** https://zuul.opendev.org/t/openstack/build/588723b923e94901af3065143d9df818 *** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us. *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack and eavesdrop servers remaining. *** https://review.opendev.org/c/opendev/system-config/+/955544/ prep step for eavesdrop move to Noble *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** OFTC Matrix bridge no longer supporting new users (clarkb 20250520) *** https://review.opendev.org/c/opendev/infra-specs/+/954826 Spec outlining the motivation and plan for Matrix trialing ** Pre PTG Planning (clarkb 20250624) *** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document * Open discussion

1 0

Team Meeting Agenda for July 15, 2025
by Clark Boylan 14 Jul '25

14 Jul '25

I will be out tomorrow, but Jeremy has volunteered to chair the meeting. The meeting will be held at 19:00 UTC in #opendev-meeting, tomorrow, July 15, 2025 with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher migration(corvus 20240910) *** Mixed provider nodesets are impacting some jobs **** The bugfixes seem to have this under control. clarkb hasn't seen any recent complaints about this. *** Image uploads may be overwhelming rax-flex sjc3 swift proxies **** Python requests/urllib3 is returning a 499 error code which can happen when proxies timeout connections to the backend and pass that along to the client. **** May need to sync up with rackspace on this as there may need to be updates to their proxies **** May also consider a different swift location? *** New Images for Trixie, Centos 10 Stream, and Rocky Linux 10 *** Ubuntu Bionic Arm64 is the last label still provided by Nodepool **** Job cleanup branches off from https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/954761/ ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Omnibus update to Gerrit images planned for Friday or Monday to incorporate several updates (zuul status plugin, CLA removals, and quay.io image hosting) **** https://review.opendev.org/c/opendev/system-config/+/882900 Host Gerrit images on quay.io **** https://review.opendev.org/c/opendev/system-config/+/954376 CLA file removal **** Needs to be done after CLA usage is removed from project ACLs *** Need to hold new test nodes. Perhaps that is best for after the container images move? *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack and eavesdrop servers remaining. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** OFTC Matrix bridge no longer supporting new users (clarkb 20250520) *** https://review.opendev.org/c/opendev/infra-specs/+/954826 Spec outlining the motivation and plan for Matrix trialing ** Pre PTG Planning (clarkb 20250624) *** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document * Open discussion

1 0

OpenDev Team Meeting Agenda for July 8, 2025
by Clark Boylan 07 Jul '25

07 Jul '25

We will meet with this agenda on July 8, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher migration(corvus 20240910) *** Mixed provider nodesets are impacting some jobs **** corvus was able to run down a couple of bugs around that, but some jobs (periodic jobs from provided examples) still occasionally see mixed providers *** Image build gnutls errors continue to be a problem when image builds update git repos **** Current workaround is dib has been updated to retry those git repo updates **** Looking at haproxy logs clarkb sees initial updates come from ipv4 addresses, then ipv6 addresses, then the failure occurs and things move back to ipv4 before finally using ipv6 and completing. **** https://zuul.opendev.org/t/opendev/build/2f4200f24ae4492295d5c43307e93038/l… That behavior was seen with this build **** One theory is that nodes are not booting with working ipv6 so use ipv4, then get get an RA or similar that causes ipv6 to work. Then something breaks ipv6 causing a fallback to ipv4. ** Gerrit 3.11 Upgrade Planning (clarkb 20250401) *** https://www.gerritcodereview.com/3.11.html *** Please check this for any concerns with the way we use Gerrit. *** Automated testing is currently testing this generally works for us. *** https://review.opendev.org/c/opendev/system-config/+/882900 Host Gerrit images on quay.io *** Need to hold new test nodes. Perhaps that is best for after the container images move? *** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup. *** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade *** Will need to select a date once we know more about the upgrade itself. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack and eavesdrop servers remaining. *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic ** EL10 Test Nodes (clarkb 20250708) *** https://review.opendev.org/c/opendev/zuul-providers/+/953460 CentOS 10 Stream *** https://review.opendev.org/c/opendev/zuul-providers/+/954265 Rocky Linux 10 ** Adding Debian Trixie Test Nodes (clarkb 20250708) *** https://review.opendev.org/c/opendev/zuul-providers/+/951471 ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** OFTC Matrix bridge no longer supporting new users (clarkb 20250520) *** Clarkb to write a spec explaining the motivation for a move to matrix for opendev comms. *** Then idea is to do a bit of a trial and see if it does help users who primarily rely on matrix for IRC as well as new users. ** Pre PTG Planning (clarkb 20250624) *** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document * Open discussion

1 0