February 2025 - service-discuss

Help w/ storyboard issue
by Khalil, Ghada 26 Feb '25

26 Feb '25

Hi all, I hope someone here can point me to the right contact to help with this issue. I had previously created a storyboard: https://storyboard.openstack.org/#!/story/2011332 -- titled: Platform Application Maintenance - ptp-notification It was accessible. We were able to create tasks and link them to code reviews previously: - https://review.opendev.org/c/starlingx/ptp-notification-armada-app/+/938075 - https://review.opendev.org/c/starlingx/ptp-notification-armada-app/+/940343 However, for some reason, we now get an API error saying the story is not found. 404: GET /api/v1/stories/2011332: Story 2011332 not found Thanks, Ghada Khalil StarlingX Release Prime

2 1

Team Meeting Agenda for February 25, 2025
by Clark Boylan 25 Feb '25

25 Feb '25

We will meet with this agenda on February 25, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** We are building Jammy, Noble, and Bullseye images now *** Flavors are in place for 4GB, 8GB, and 16GB test nodes. *** https://review.opendev.org/c/zuul/zuul/+/940824 Somewhat successful dogfooding in this zuul change ** Fixing known_hosts generation on bridge (clarkb 20250225) *** https://review.opendev.org/c/opendev/system-config/+/942307 *** TL;DR is that we weren't updating system-config on bridge before reading the inventory file to generate known_hosts on bridge *** This means that we could fail infra-prod-base trying to ssh to newly added servers *** Sometimes everything just worked because the -lb jobs weren't depending on anything else and if they ran early enough system-config would update by chance. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Sprinting to Upgrade Servers to Focal (clarkb 20250204) *** Managed to upgrade two servers *** Ran into problems with AppArmor on Noble interferring with podman and rsyslogd *** Will try to continue upgrading things as time goes on. *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint ** Redeploying raxflex resources (clarkb 20250225) *** We have access to a new raxflex region. To align with the old region and to correct network MTUs we need to redeploy the old region on the new project/tenant. *** Trying to determine whether or not we are required to use floating IPs before booting new mirrors. ** Running certcheck on bridge (clarkb 20250114) *** Any concern running this service in the bastion? *** Would better match how testing works. *** We would need to disable it on the cacti server. ** Service Coordinator Election (clarkb 20250107) *** Nominations closed and only clarkb was nominated *** Unless someone was missed clarkb is it by default and we don't need an election. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Using more robust Gitea caches (clarkb 20250225) *** Currently gitea uses a default cache setup with a single Go hashmap *** It is theorized that this hashmap grows such that eventually Go GC creates noticeable pause the world behavior impacting service response time *** https://review.opendev.org/c/opendev/system-config/+/942650 Set up gitea with memcached for caching *** As an alternative we could use the twoqueue adapter in gitea but this is compiled into the binary so would still be potentially impacting the GC system. But is one less container image and service. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup * Open discussion

1 0

Clarkb's OpenDev Service Coordinator Nomination
by Clark Boylan 19 Feb '25

19 Feb '25

Per the plan sent in https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… the OpenDev service coordinator nomination period ends in less than an hour. I haven't seen any other nominations yet and after some discussion in our weekly meeting it seems that folks are happy for me to be it again. I am sending this email to formally nominate myself to make it official. My intention for the next six months is largely to update servers to Noble or newer enabling us to get onto podman runtimes for containers and eventually stop publishing images to docker hub. I'd also like to see us continue to update services like Gerrit and Gitea, adopt the new zuul-launcher system within our Zuul deployment, and continue to push our use of Zuul forward as an illustration of what is possible with this powerful tool. If this sounds good to you then I'm happy to help guide us in that direction for the next six months. Otherwise, I am still more than happy to help anyone else who may wish to step into the role. Perhaps in six months someone else will be willing to step up? Clark

1 0

Team Meeting Agenda for February 18, 2025
by Clark Boylan 18 Feb '25

18 Feb '25

We will meet with this agenda on February 18, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** We are building Jammy, Noble, and Bullseye images now *** Flavors are in place for 4GB, 8GB, and 16GB test nodes. *** https://review.opendev.org/c/opendev/system-config/+/942018 Increase working disk space for zuul-launcher ** Unpinning our Grafana deployment (clarkb 20250128) *** Grafana02 is deployed and running the latest Grafana 10.x release *** 10.x reports angular dashboards are deprecated *** https://review.opendev.org/c/opendev/system-config/+/940997 Update to Grafana 11 **** Screenshots seem to indicate that we don't have deprecation warnings anymore however json definition files don't differ significantly from 10.x. Do we want to proceed or hold a node and debug further? ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** Discovered that `podman kill -s HUP` and `docker-compose kill -s HUP` fail on noble with podman apparmor permission issues **** I think we can work around this by issuing `kill -s HUP` directly to the container process pid **** Issue is fixed upstream. No movement on downstream bug to backport. **** https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2040483 ** Sprinting to Upgrade Servers to Focal (clarkb 20250204) *** Managed to upgrade two servers *** Ran into problems with AppArmor on Noble interferring with podman and rsyslogd *** Will try to continue upgrading things as time goes on. *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint *** https://review.opendev.org/c/opendev/system-config/+/941997 Optimize when LE edits trigger jobs ** Running certcheck on bridge (clarkb 20250114) *** Any concern running this service in the bastion? *** Would better match how testing works. *** We would need to disable it on the cacti server. ** Service Coordinator Election (clarkb 20250107) *** Nominations Open From February 4, 2025 to February 18, 2025 (You have until EOD UTC time today) *** Voting February 19, 2025 to February 26, 2025 if necessary *** All times and dates will be UTC based. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup * Open discussion

1 0

Team Meeting Agenda for February 11, 2025
by Clark Boylan 11 Feb '25

11 Feb '25

We will meet with this agenda on February 11, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** We are building Jammy, Noble, and Bullseye images now *** Flavors are in place for 4GB, 8GB, and 16GB test nodes. *** Discovered a zuul bug in the launcher implementation that affected jobs with empty buildsets (including noop jobs). Should be fixed now. ** Unpinning our Grafana deployment (clarkb 20250128) *** Grafana02 is deployed and running the latest Grafana 10.x release *** 10.x reports angular dashboards are deprecated *** https://review.opendev.org/c/opendev/system-config/+/940997 Update to Grafana 11 **** Screenshots seem to indicate that we don't have deprecation warnings anymore however json definition files don't differ significantly from 10.x. Do we want to proceed or hold a node and debug further? ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** Discovered that `podman kill -s HUP` and `docker-compose kill -s HUP` fail on noble with podman apparmor permission issues **** I think we can work around this by issuing `kill -s HUP` directly to the container process pid **** https://github.com/containers/common/issues/1898 **** https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2040483 ** Sprinting to Upgrade Servers to Focal (clarkb 20250204) *** Getting new servers going is much faster if we are able to review and address problems quickly *** Happening this week *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** Will attempt to track details on this etherpad ** Running certcheck on bridge (clarkb 20250114) *** Any concern running this service in the bastion? *** Would better match how testing works. *** We would need to disable it on the cacti server. ** Service Coordinator Election (clarkb 20250107) *** Nominations Open From February 4, 2025 to February 18, 2025 (one week remaining!) *** Voting February 19, 2025 to February 26, 2025 *** All times and dates will be UTC based. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup * Open discussion

1 0

Team Meeting Agenda for February 4, 2025
by Clark Boylan 04 Feb '25

04 Feb '25

We will meet on February 4, 2025 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** zuul-launcher related configs (images, labels, providers, etc) have been refactored into https://opendev.org/opendev/zuul-providers *** Zuul itself is starting to explore dogfooding of the new launcher managed images in jobs zuul runs. ** Unpinning our Grafana deployment (clarkb 20250128) *** https://review.opendev.org/c/opendev/system-config/+/940073 *** https://review.opendev.org/c/opendev/system-config/+/940654 *** Plan is to deploy new grafana on a new noble node (allows us to keep old grafana around on focal for now as a fallback) *** grafana01 is in the emergency file. Landing the two changes above should deploy what we need on grafana02. Once happy we update dns to point grafana.o.o at grafana02.o.o. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Sprinting to Upgrade Servers to Focal (clarkb 20250204) *** Getting new servers going is much faster if we are able to review and address problems quickly *** Clarkb proposes we plan to sprint next week (can use meetpad too if we like) and try to upgrade as many servers as we possible can to Noble *** This will help build confidence that noble and podman work well for us before we schedule a Gerrit upgrade. ** Switch to quay.io/opendevmirror images where possible (clarkb 20250128) *** Mariadb is an easy switch but updating the image from one location to another (even if the hash is the same) causes a service restart **** https://review.opendev.org/q/topic:%22quay-mariadb%22+status:open *** Other changes to consider: Pulling our python images from quay.io. We lose speculative testing of the base image but that may be a reasonable compromise. ** Running certcheck on bridge (clarkb 20250114) *** Any concern running this service in the bastion? *** Would better match how testing works. *** We would need to disable it on the cacti server. ** Service Coordinator Election (clarkb 20250107) *** Nominations Open From February 4, 2025 to February 18, 2025 *** Voting February 19, 2025 to February 26, 2025 *** All times and dates will be UTC based. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup * Open discussion

1 0