- service-discuss - lists.opendev.org

Team Meeting Agenda for March 4, 2025
by Clark Boylan 04 Mar '25

04 Mar '25

We will meet with this agenda on March 4, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Redeploying raxflex resources (clarkb 20250225) *** We have access to a new raxflex region. To align with the old region and to correct network MTUs we need to redeploy the old region on the new project/tenant. *** https://review.opendev.org/q/hashtag:flex-dfw3 ** Zuul-launcher image builds (corvus 20240910) *** We are building Jammy, Noble, and Bullseye images now *** Flavors are in place for 4GB, 8GB, and 16GB test nodes. *** https://review.opendev.org/c/zuul/zuul/+/940824 Somewhat successful dogfooding in this zuul change ** Updating Flavors in OVH (clarkb 20250304) *** https://etherpad.opendev.org/p/ovh-flavors *** The zuul-launcher experiment is looking at expanding into OVH, but we only have 8GB flavors there. ** Running infra-prod Jobs in Parallel on Bridge (clarkb 20250304) *** https://review.opendev.org/c/opendev/system-config/+/942439 *** This change refactors how we run infra-prod-* jobs on bridge to enable running them in parallel *** The current iteration will continue to run them serially with an additional semaphore which can be expanded from a single lock to 2 or 5 etc when we're happy with it *** Need to pick a good day when we are able to observe the behavior of the system in order to catch issues as soon as possible. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Sprinting to Upgrade Servers to Focal (clarkb 20250204) *** Will try to continue upgrading things as time goes on. *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint ** Docker Hub rate limits (clarkb 20250304) *** Docker hub announced they would be changing rate limits from 100 pulls per 6 hours to 10 pulls per hour on March 1, 2025 *** This could be good for us if the shorter reset period works well with our zuul jobs as they tend to run for some time but not 6 hours *** Or it could be bad for us. *** Early data doesn't show any indicators for it being better or worse yet. *** Also keep in mind ipv6 addresses are rate limited per /64. We might consider forcing docker hub access through ipv4 via /etc/hosts entries. ** Running certcheck on bridge (clarkb 20250114) *** Any concern running this service in the bastion? *** Would better match how testing works. *** We would need to disable it on the cacti server. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup * Open discussion

1 0

Help w/ storyboard issue
by Khalil, Ghada 26 Feb '25

26 Feb '25

Hi all, I hope someone here can point me to the right contact to help with this issue. I had previously created a storyboard: https://storyboard.openstack.org/#!/story/2011332 -- titled: Platform Application Maintenance - ptp-notification It was accessible. We were able to create tasks and link them to code reviews previously: - https://review.opendev.org/c/starlingx/ptp-notification-armada-app/+/938075 - https://review.opendev.org/c/starlingx/ptp-notification-armada-app/+/940343 However, for some reason, we now get an API error saying the story is not found. 404: GET /api/v1/stories/2011332: Story 2011332 not found Thanks, Ghada Khalil StarlingX Release Prime

2 1

Team Meeting Agenda for February 25, 2025
by Clark Boylan 25 Feb '25

25 Feb '25

We will meet with this agenda on February 25, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** We are building Jammy, Noble, and Bullseye images now *** Flavors are in place for 4GB, 8GB, and 16GB test nodes. *** https://review.opendev.org/c/zuul/zuul/+/940824 Somewhat successful dogfooding in this zuul change ** Fixing known_hosts generation on bridge (clarkb 20250225) *** https://review.opendev.org/c/opendev/system-config/+/942307 *** TL;DR is that we weren't updating system-config on bridge before reading the inventory file to generate known_hosts on bridge *** This means that we could fail infra-prod-base trying to ssh to newly added servers *** Sometimes everything just worked because the -lb jobs weren't depending on anything else and if they ran early enough system-config would update by chance. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Sprinting to Upgrade Servers to Focal (clarkb 20250204) *** Managed to upgrade two servers *** Ran into problems with AppArmor on Noble interferring with podman and rsyslogd *** Will try to continue upgrading things as time goes on. *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint ** Redeploying raxflex resources (clarkb 20250225) *** We have access to a new raxflex region. To align with the old region and to correct network MTUs we need to redeploy the old region on the new project/tenant. *** Trying to determine whether or not we are required to use floating IPs before booting new mirrors. ** Running certcheck on bridge (clarkb 20250114) *** Any concern running this service in the bastion? *** Would better match how testing works. *** We would need to disable it on the cacti server. ** Service Coordinator Election (clarkb 20250107) *** Nominations closed and only clarkb was nominated *** Unless someone was missed clarkb is it by default and we don't need an election. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Using more robust Gitea caches (clarkb 20250225) *** Currently gitea uses a default cache setup with a single Go hashmap *** It is theorized that this hashmap grows such that eventually Go GC creates noticeable pause the world behavior impacting service response time *** https://review.opendev.org/c/opendev/system-config/+/942650 Set up gitea with memcached for caching *** As an alternative we could use the twoqueue adapter in gitea but this is compiled into the binary so would still be potentially impacting the GC system. But is one less container image and service. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup * Open discussion

1 0

Clarkb's OpenDev Service Coordinator Nomination
by Clark Boylan 18 Feb '25

18 Feb '25

Per the plan sent in https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… the OpenDev service coordinator nomination period ends in less than an hour. I haven't seen any other nominations yet and after some discussion in our weekly meeting it seems that folks are happy for me to be it again. I am sending this email to formally nominate myself to make it official. My intention for the next six months is largely to update servers to Noble or newer enabling us to get onto podman runtimes for containers and eventually stop publishing images to docker hub. I'd also like to see us continue to update services like Gerrit and Gitea, adopt the new zuul-launcher system within our Zuul deployment, and continue to push our use of Zuul forward as an illustration of what is possible with this powerful tool. If this sounds good to you then I'm happy to help guide us in that direction for the next six months. Otherwise, I am still more than happy to help anyone else who may wish to step into the role. Perhaps in six months someone else will be willing to step up? Clark

1 0

Team Meeting Agenda for February 18, 2025
by Clark Boylan 18 Feb '25

18 Feb '25

We will meet with this agenda on February 18, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** We are building Jammy, Noble, and Bullseye images now *** Flavors are in place for 4GB, 8GB, and 16GB test nodes. *** https://review.opendev.org/c/opendev/system-config/+/942018 Increase working disk space for zuul-launcher ** Unpinning our Grafana deployment (clarkb 20250128) *** Grafana02 is deployed and running the latest Grafana 10.x release *** 10.x reports angular dashboards are deprecated *** https://review.opendev.org/c/opendev/system-config/+/940997 Update to Grafana 11 **** Screenshots seem to indicate that we don't have deprecation warnings anymore however json definition files don't differ significantly from 10.x. Do we want to proceed or hold a node and debug further? ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** Discovered that `podman kill -s HUP` and `docker-compose kill -s HUP` fail on noble with podman apparmor permission issues **** I think we can work around this by issuing `kill -s HUP` directly to the container process pid **** Issue is fixed upstream. No movement on downstream bug to backport. **** https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2040483 ** Sprinting to Upgrade Servers to Focal (clarkb 20250204) *** Managed to upgrade two servers *** Ran into problems with AppArmor on Noble interferring with podman and rsyslogd *** Will try to continue upgrading things as time goes on. *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint *** https://review.opendev.org/c/opendev/system-config/+/941997 Optimize when LE edits trigger jobs ** Running certcheck on bridge (clarkb 20250114) *** Any concern running this service in the bastion? *** Would better match how testing works. *** We would need to disable it on the cacti server. ** Service Coordinator Election (clarkb 20250107) *** Nominations Open From February 4, 2025 to February 18, 2025 (You have until EOD UTC time today) *** Voting February 19, 2025 to February 26, 2025 if necessary *** All times and dates will be UTC based. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup * Open discussion

1 0

Team Meeting Agenda for February 11, 2025
by Clark Boylan 11 Feb '25

11 Feb '25

We will meet with this agenda on February 11, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** We are building Jammy, Noble, and Bullseye images now *** Flavors are in place for 4GB, 8GB, and 16GB test nodes. *** Discovered a zuul bug in the launcher implementation that affected jobs with empty buildsets (including noop jobs). Should be fixed now. ** Unpinning our Grafana deployment (clarkb 20250128) *** Grafana02 is deployed and running the latest Grafana 10.x release *** 10.x reports angular dashboards are deprecated *** https://review.opendev.org/c/opendev/system-config/+/940997 Update to Grafana 11 **** Screenshots seem to indicate that we don't have deprecation warnings anymore however json definition files don't differ significantly from 10.x. Do we want to proceed or hold a node and debug further? ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** Discovered that `podman kill -s HUP` and `docker-compose kill -s HUP` fail on noble with podman apparmor permission issues **** I think we can work around this by issuing `kill -s HUP` directly to the container process pid **** https://github.com/containers/common/issues/1898 **** https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2040483 ** Sprinting to Upgrade Servers to Focal (clarkb 20250204) *** Getting new servers going is much faster if we are able to review and address problems quickly *** Happening this week *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** Will attempt to track details on this etherpad ** Running certcheck on bridge (clarkb 20250114) *** Any concern running this service in the bastion? *** Would better match how testing works. *** We would need to disable it on the cacti server. ** Service Coordinator Election (clarkb 20250107) *** Nominations Open From February 4, 2025 to February 18, 2025 (one week remaining!) *** Voting February 19, 2025 to February 26, 2025 *** All times and dates will be UTC based. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup * Open discussion

1 0

Team Meeting Agenda for February 4, 2025
by Clark Boylan 04 Feb '25

04 Feb '25

We will meet on February 4, 2025 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** zuul-launcher related configs (images, labels, providers, etc) have been refactored into https://opendev.org/opendev/zuul-providers *** Zuul itself is starting to explore dogfooding of the new launcher managed images in jobs zuul runs. ** Unpinning our Grafana deployment (clarkb 20250128) *** https://review.opendev.org/c/opendev/system-config/+/940073 *** https://review.opendev.org/c/opendev/system-config/+/940654 *** Plan is to deploy new grafana on a new noble node (allows us to keep old grafana around on focal for now as a fallback) *** grafana01 is in the emergency file. Landing the two changes above should deploy what we need on grafana02. Once happy we update dns to point grafana.o.o at grafana02.o.o. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Sprinting to Upgrade Servers to Focal (clarkb 20250204) *** Getting new servers going is much faster if we are able to review and address problems quickly *** Clarkb proposes we plan to sprint next week (can use meetpad too if we like) and try to upgrade as many servers as we possible can to Noble *** This will help build confidence that noble and podman work well for us before we schedule a Gerrit upgrade. ** Switch to quay.io/opendevmirror images where possible (clarkb 20250128) *** Mariadb is an easy switch but updating the image from one location to another (even if the hash is the same) causes a service restart **** https://review.opendev.org/q/topic:%22quay-mariadb%22+status:open *** Other changes to consider: Pulling our python images from quay.io. We lose speculative testing of the base image but that may be a reasonable compromise. ** Running certcheck on bridge (clarkb 20250114) *** Any concern running this service in the bastion? *** Would better match how testing works. *** We would need to disable it on the cacti server. ** Service Coordinator Election (clarkb 20250107) *** Nominations Open From February 4, 2025 to February 18, 2025 *** Voting February 19, 2025 to February 26, 2025 *** All times and dates will be UTC based. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup * Open discussion

1 0

Asking permission to scrape OpenStack docs for external search
by Jay Faulkner 29 Jan '25

29 Jan '25

Hello OpenDev admins, Mehdi, one of our developer relations engineers here at GR-OSS, and I were talking about the state of OpenStack docs, and how a repeated complaint we hear is that search is not extremely useful. He noted that for https://consuldot.net GR-OSS uses a service called https://algolia.com to provide search (top right corner of consuldot.net <http://consuldot.net>). We were hoping to pilot this for OpenStack and see how well the search worked against our documentation, but when going through the process they specifically request you have permission from the website owners before they crawl ( https://docsearch.algolia.com/apply/ ) -- so I'm reaching out to you to try and get this permission. Right now our ONLY goal is to get a prototype working for this to demo to the larger OpenStack community -- honestly I just want to see if it's worth the value before we have the larger discussion around externally-hosted search. So basically, we're looking for permission to point their crawler at OpenStack documentation (we may start with just Ironic for simplicity). We'll create a larger discussion including the foundation, TC, and larger community if it appears to be a worthwhile pursuit. Let me know what you think! Thanks, Jay Faulkner

2 1

Team Meeting Agenda for January 28, 2025
by Clark Boylan 28 Jan '25

28 Jan '25

== Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** https://review.opendev.org/q/hashtag:niz+status:open is next set of work happening in Zuul ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** Thoughts on using Noble **** Rely on borg 1.2.8 backups to existing servers until we have new backup servers running borg 1.4.x **** Some docker compose setups may need to be edited to set a restart policy of always **** Podman package updates don't appear to require container restarts. But we should observe this behavior if possible. ** Switch to quay.io/opendevmirror images where possible (clarkb 20250128) *** Mariadb is an easy switch but updating the image from one location to another (even if the hash is the same) causes a service restart *** Other changes to consider: Pulling our python images from quay.io. We lose speculative testing of the base image but that may be a reasonable compromise. ** Unpinning our Grafana deployment (clarkb 20250128) *** https://review.opendev.org/c/opendev/system-config/+/940073 *** Testing shows this may still exhibit the problem we saw that caused us to pin *** Trying to use a held node to test directly. ** Increasing Exim's Acceptable Line Length (clarkb 20250128) *** https://review.opendev.org/c/opendev/system-config/+/940248 *** Bounces were being generated by our own Exim system on mailman due to line lengths being too long. This is configurable and we can avoid the problem entirely. ** Running certcheck on bridge (clarkb 20250114) *** Any concern running this service in the bastion? *** Would better match how testing works. *** We would need to disable it on the cacti server. ** Service Coordinator Election (clarkb 20250107) *** Nominations Open From February 4, 2025 to February 18, 2025 *** Voting February 19, 2025 to February 26, 2025 *** All times and dates will be UTC based. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Beginning of the Year (Virtual) Meetup Recap (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup *** Tried to condense the notes down into a TODO list for the next $TIME * Open discussion

1 0

February 2025 OpenDev Service Coordinator Election
by Clark Boylan 21 Jan '25

21 Jan '25

It is that time of year again where we need to elect an OpenDev Service Coordinator [0] for the next six months. Nominations will be open from February 4, 2025 to February 18, 2025. If we have more than one candidate we will vote for our preferred candidate from February 19, 2025 to February 26, 2025. All dates are based on a UTC clock. Please send your self nominations to this mailing list during the nomination time frame. You are eligible to run and vote if you have contributed to OpenDev git repositories in the last 12 months (this period will be February 4, 2024 to February 3, 2025). I continue to encourage anyone who may be interested to reach out and ask questions. I am happy to support someone new in the role. [0] https://docs.opendev.org/opendev/system-config/latest/project.html#governan… Clark

1 0