April 2025 - service-discuss - lists.opendev.org

Team Meeting Agenda for April 29, 2025
by Clark Boylan 28 Apr '25

28 Apr '25

We will meet with this agenda on April 29, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** The May 6th team meeting has been cancelled. * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** All current x86_64 providers are enrolled in the launcher now. Those with smaller and bigger flavors are setup to provide those labels too. *** We have at least one arm64 image built by niz now. Thank you mnasiadka! ** Container hygiene tasks (clarkb 20250318) *** https://review.opendev.org/q/topic:%22opendev-python3.12%22+status:open Update images to use python3.12 *** Only limnoria remains. Need to find a good non disruptive time for irc meetings to land this change. *** https://review.opendev.org/c/opendev/system-config/+/948247 Force docker hub access to happen over ipv4 for better rate limits. ** Switching Gerrit to run on Review03 (clarkb 20250401) *** https://etherpad.opendev.org/p/i_vt63v18c3RKX2VyCs3 Notes on the migration plan *** The move from review02 to review03 is largely complete *** The old review02 server is shutdown and removed from our inventory *** At least one new project has been created on review03 *** The last remaining task is to clean up the old review02 server and possibly its volumes. **** I'm guessing we want to keep the volumes. The boot volume should get a name assigned to it so we know what it is in the future **** Any other input before we delete the instance? *** Now it is time to start thinking about the Gerrit 3.11 upgrade. **** https://www.gerritcodereview.com/3.11.html ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack, mirror-update, eavesdrop, zuul schedulers, and zookeeper servers remaining. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** Rotating mailman 3 logs (clarkb 20250325) *** Some of the mailman3 log files have grown quite large and could stand some rotation *** Unfortunately fungi found evidence that the codebase may not handle rotation gracefully. *** Not sure if we want to just force the issue or try to fix it upstream or something else. ** Renewing wiki's cert (clarkb 20250422) *** Wiki's cert expires soon. Clarkb needs to replace it this week. Please say something if it isn't done by Friday. ** Occasional Log Upload Failures to OVH (clarkb 20250429) *** We've recently noticed very occasional POST_FAILURE results due to not being able to upload logs to OVH (both GRA and BHS) *** This doesn't happen frequently enough to remove the backends. By the time we notice it seems to have gone away. *** Keep an eye out for more cases and if it becomes persistent we may need more debugging. * Open discussion

1 0

Team Meeting Agenda for April 22, 2025
by Clark Boylan 21 Apr '25

21 Apr '25

We will meet with this agenda on April 22, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** All current x86_64 providers are enrolled in the launcher now. Those with smaller and bigger flavors are setup to provide those labels too. *** Differences discovered in handling of private_ip vars for ipv4 and ipv6. Nodepool would set the private_ip to the public_ip if there is no private_ip. NIZ does not do that currently. *** One idea is to leave the NIZ behavior in place for private_ip but then add a new var that can interpret the value for jobs. *** We have volunteers to port image builds from nodepool to niz. ** Container hygiene tasks (clarkb 20250318) *** https://review.opendev.org/q/topic:%22opendev-python3.12%22+status:open Update images to use python3.12 *** Rebuilding our base images then switching things over to those new base images seems like a good way to get updates. ** Switching Gerrit to run on Review03 (clarkb 20250401) *** https://etherpad.opendev.org/p/i_vt63v18c3RKX2VyCs3 Notes on the migration plan *** Servers switched on April 21, 2025 *** There is a bit of followup to do when we are satisfied we won't roll back: **** https://review.opendev.org/c/opendev/system-config/+/947758 Drop review02 from inventory **** https://review.opendev.org/c/opendev/system-config/+/947759 Drop ignored docker compose version specifier **** https://review.opendev.org/c/opendev/system-config/+/882900 Migrate gerrit images to quay *** The last change for the quay move may be exposing a problem with our assumptions around testing with quay that we should double check with lodgeit and hound. *** Now it is time to start thinking about the Gerrit 3.11 upgrade. **** https://www.gerritcodereview.com/3.11.html ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack, mirror-update, eavesdrop, zuul schedulers, and zookeeper servers remaining. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** Rotating mailman 3 logs (clarkb 20250325) *** Some of the mailman3 log files have grown quite large and could stand some rotation *** Unfortunately fungi found evidence that the codebase may not handle rotation gracefully. *** Not sure if we want to just force the issue or try to fix it upstream or something else. ** Moving hound image to quay.io (clarkb 20250415) *** Now that codesearch is a Noble node running with podman we can host the hound image on quay and still have speculative testing. *** https://review.opendev.org/c/opendev/system-config/+/947010 *** See note about in the Gerrit topic. We should be careful with this update to ensure the testing is doing what we expect. ** Renewing wiki's cert (clarkb 20250422) *** Wiki's cert expires soon. We will need to manually provision a new annual cert and apply it to the server. * Open discussion

1 0

Image availability in OpenDev
by James E. Blair 18 Apr '25

18 Apr '25

Hi, The OpenDev community needs volunteers to help with the maintenance of the following images, otherwise we may need to remove them from Zuul: * CentOS Stream (x86 and Arm) * Rocky Linux (x86 and Arm) * Ubuntu (Arm only) * Debian (Arm only) The preceding images are all active and in use; the following images do not currently build and without further action are already slated to be removed: * openEuler * Gentoo The Zuul project is nearing the end of an effort to update the way that it launches nodes when running tests. This promises to bring a number of benefits to OpenDev users, including better responsiveness, availability, scaling, and more end-user control over the configuration. In short, we are removing Nodepool as a standalone project and incorporating all of its functionality into Zuul itself. OpenDev currently builds the base images we use for test nodes using nodepool-builder, which is a standalone daemon running on a small number of dedicated hosts which runs diskimage-builder and uploads the results to the clouds we use. This approach ensures a consistent image across the many clouds that we employ, plus effective caching of content used in jobs. The configuration for these image builds is, as is standard for every part of OpenDev, public and changes to it are able to be proposed by any member of our community. Yet there is limited testing of those changes, and new image builds can only happen after configuration changes are merged and deployed. With the new system, image builds will happen in Zuul jobs, which makes it far easier to observe the builds and examine the results. Most importantly, any changes to the image configuration are entirely self-testing. Which leads us to the request for volunteers. We have already copied over and adapted the x86 Debian and Ubuntu images to the new system. But we need someone with knowledge of Arm, CentOS, and Rocky images to port the old configuration to the new system. It's not a lot of work, but from time to time these images do need updates (for new OS releases, for example), so it's important that we have a community of users who are able to assist in that maintenance. If no volunteers are willing to help maintain this community resource, we may need to remove support for these images. Starting around May 15, we would consider these images best-effort: we may not run them in all providers as we move more of our workload to zuul-launcher. Jobs that use them may start to fail with NODE_ERROR. Later, once we finish the move to zuul-launcher (tentatively July 1), any request for nodes without images would return NODE_ERROR. Along with myself, others in the OpenDev community are willing to assist anyone who is interested in working on this. It is an excellent opportunity to get involved in helping to support the community: it requires no previous knowledge of OpenDev systems or any special access. If you are willing to port the nodepool-builder configuration for these images to the new Zuul system, please let us know (by either responding to this thread on service-discuss, or contacting us in OFTC in #opendev). Here are some relevant links to get you started: The current nodepool-builder configuration: https://opendev.org/openstack/project-config/src/branch/master/nodepool/nod… The new zuul-launcher configuration: https://opendev.org/opendev/zuul-providers/src/branch/master/zuul.d/image-b… Thanks in advance, -Jim (on behalf of the OpenDev project)

3 2

Team Meeting Agenda for April 15, 2025
by Clark Boylan 14 Apr '25

14 Apr '25

Welcome back from our week "off" (not really as the PTG was demanding out time instead). We will have our weekly meeting this week on April 15, 2025 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** All current x86_64 providers are enrolled in the launcher now. Those with smaller and bigger flavors are setup to provide those labels too. *** VHD support has been added to image builds. Node leak cleanups are getting added. ** Container hygiene tasks (clarkb 20250318) *** https://review.opendev.org/q/topic:%22opendev-python3.12%22+status:open Update images to use python3.12 *** Rebuilding our base images then switching things over to those new base images seems like a good way to get updates. ** Switching Gerrit to run on Review03 (clarkb 20250401) *** A new BFV server of the same size as the old server has been booted in vexxhost ca-ymq-1 *** https://etherpad.opendev.org/p/i_vt63v18c3RKX2VyCs3 Notes on the migration plan *** We have been working through the process of spinning up and testing the new server *** https://review.opendev.org/c/opendev/system-config/+/947044 Prep work to ensure port 29418 ssh host keys match on old and new servers *** April 21, 2025 at 1600 UTC is the planned outage time to migrate. *** DNS updates are slightly tricky with Gerrit being the server that is updated ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack, mirror-update, eavesdrop, zuul schedulers, and zookeeper servers remaining. ** Running certcheck on bridge (clarkb 20250114) *** Suggestion is that we just run this as an infra-prod job instead as all state should be available there. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** Rotating mailman 3 logs (clarkb 20250325) *** Some of the mailman3 log files have grown quite large and could stand some rotation *** Unfortunately fungi found evidence that the codebase may not handle rotation gracefully. *** Not sure if we want to just force the issue or try to fix it upstream or something else. ** Moving hound image to quay.io (clarkb 20250415) *** Now that codesearch is a Noble node running with podman we can host the hound image on quay and still have speculative testing. *** https://review.opendev.org/c/opendev/system-config/+/947010 *** We can do similar with Gerrit after running on the new host and not rolling back * Open discussion

1 0

Team Meeting Agenda for April 1, 2025
by Clark Boylan 31 Mar '25

31 Mar '25

We will meet with this agenda on April 1, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack releases Epoxy 2025.1 Wednesday. Please be aware of this as you make changes ** The PTG is being hosted on Meetpad April 7-11. * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** All current x86_64 providers are enrolled in the launcher now. Those with smaller and bigger flavors are setup to provide those labels too. ** Container hygiene tasks (clarkb 20250318) *** https://review.opendev.org/q/topic:%22opendev-python3.12%22+status:open Update images to use python3.12 *** Rebuilding our base images then switching things over to those new base images seems like a good way to get updates. ** Booting a new Gerrit server (clarkb 20250401) *** Its about time to start preparing the new Gerrit server. We have a decent amount of experience with Noble now and it seems to work in several clouds *** Clarkb thinks we should stick to the same flavor size wherever we host. This likely means Vexxhost is still the best candidate location despite the IPv6 issues *** If we boot in Vexxhost we should use the equivalent v3 flavor so that we don't have to boot from volume (this simplifies server rescue should we need it). *** Maybe aim for swapping the new server into production the week after the PTG? ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** On the easy list we've got refstack, mirror-update, eavesdrop, zuul schedulers, and zookeeper servers remaining. ** Running certcheck on bridge (clarkb 20250114) *** Suggestion is that we just run this as an infra-prod job instead as all state should be available there. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** Rotating mailman 3 logs (clarkb 20250325) *** Some of the mailman3 log files have grown quite large and could stand some rotation *** Unfortunately fungi found evidence that the codebase may not handle rotation gracefully. *** Not sure if we want to just force the issue or try to fix it upstream or something else. * Open discussion

1 0