- service-discuss - lists.opendev.org

Zuul errors
by ronald.stone＠windriver.com 18 Mar '25

18 Mar '25

Apologies if this is the wrong group. Zuul appears to be broken. See https://zuul.opendev.org/t/openstack/build/179c40f7e382474cb9c676b812caa6fb

2 1

Team Meeting Agenda for March 18, 2025
by Clark Boylan 18 Mar '25

18 Mar '25

We will meet on March 18, 2025 at 19:00 UTC with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** We are building Jammy, Noble, and Bullseye images now *** Flavors are in place for 4GB, 8GB, and 16GB test nodes. *** https://review.opendev.org/c/zuul/zuul/+/940824 Somewhat successful dogfooding in this zuul change ** Updating Flavors in OVH (clarkb 20250304) *** https://etherpad.opendev.org/p/ovh-flavors *** The zuul-launcher experiment is looking at expanding into OVH, but we only have 8GB flavors there. *** OVH said March 17 (today) was not good for them and they would be in touch to reschedule later. ** Container hygiene tasks (clarkb 20250318) *** https://review.opendev.org/c/opendev/system-config/+/944799 drop python 3.10 image builds *** https://review.opendev.org/c/opendev/system-config/+/944789 rebuild python 3.11 and 3.12 base images *** https://review.opendev.org/q/topic:%22opendev-python3.12%22+status:open Update images to use python3.12 *** Rebuilding our base images then switching things over to those new base images seems like a good way to get updates. ** Dropping uWSGI (clarkb 20250318) *** uWSGI has never built reliable and we are hitting a new segfault during compilation *** uWSGI is also on life support and not getting new development just bugfixes here and there *** Thinking about switching to Granian which is an ASGI/RSGI/WSGI server written in Rust that publishes wheels for aarch64 and x86_64 *** In theory this means we should be able to get away with one server regardless of the protocol that web services we choose to run use. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Sprinting to Upgrade Servers to Focal (clarkb 20250204) *** Will try to continue upgrading things as time goes on. *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint *** nb05 and nb06 are in the process of getting deployed on Noble with podman and docker compose. ** Running certcheck on bridge (clarkb 20250114) *** Suggestion is that we just run this as an infra-prod job instead as all state should be available there. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** Packaging updates for bindep (fungi 20250318) *** https://review.opendev.org/938570 Drop requirements.txt *** https://review.opendev.org/940711 Drop auxiliary requirements files *** It would be nice to get consensus on whether or not to include these when replicating the new packaging methods to our other tools. * Open discussion

1 0

Team Meeting Agenda for March 11, 2025
by Clark Boylan 11 Mar '25

11 Mar '25

We will meet with this agenda on March 11, 2025 at 19:00 UTC (note that North America and maybe others have switched to DST) in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** We are building Jammy, Noble, and Bullseye images now *** Flavors are in place for 4GB, 8GB, and 16GB test nodes. *** https://review.opendev.org/c/zuul/zuul/+/940824 Somewhat successful dogfooding in this zuul change ** Updating Flavors in OVH (clarkb 20250304) *** https://etherpad.opendev.org/p/ovh-flavors *** The zuul-launcher experiment is looking at expanding into OVH, but we only have 8GB flavors there. *** Plan is to target March 17 for OVH flavor surgery ** Running infra-prod Jobs in Parallel on Bridge (clarkb 20250304) *** https://review.opendev.org/c/opendev/system-config/+/943488 Actually run infra-prod jobs in parallel *** The refactor to infra-prod jobs has gone in but with a semaphore limit of 1 which continues to serialize things *** There were some small issues. **** bootstrap-bridge needed extra required-projects to setup repos like project-config in periodic pipelines **** project-config and dns zone repos needed to be setup with bootstrap-bridge as the parent job to ensure repos are configured and locks are respected *** Seems like things work serially at this point. Time to proceed with parallel execution? ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Sprinting to Upgrade Servers to Focal (clarkb 20250204) *** Will try to continue upgrading things as time goes on. *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint ** Running certcheck on bridge (clarkb 20250114) *** Suggestion is that we just run this as an infra-prod job instead as all state should be available there. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup * Open discussion

1 0

Team Meeting Agenda for March 4, 2025
by Clark Boylan 04 Mar '25

04 Mar '25

We will meet with this agenda on March 4, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Redeploying raxflex resources (clarkb 20250225) *** We have access to a new raxflex region. To align with the old region and to correct network MTUs we need to redeploy the old region on the new project/tenant. *** https://review.opendev.org/q/hashtag:flex-dfw3 ** Zuul-launcher image builds (corvus 20240910) *** We are building Jammy, Noble, and Bullseye images now *** Flavors are in place for 4GB, 8GB, and 16GB test nodes. *** https://review.opendev.org/c/zuul/zuul/+/940824 Somewhat successful dogfooding in this zuul change ** Updating Flavors in OVH (clarkb 20250304) *** https://etherpad.opendev.org/p/ovh-flavors *** The zuul-launcher experiment is looking at expanding into OVH, but we only have 8GB flavors there. ** Running infra-prod Jobs in Parallel on Bridge (clarkb 20250304) *** https://review.opendev.org/c/opendev/system-config/+/942439 *** This change refactors how we run infra-prod-* jobs on bridge to enable running them in parallel *** The current iteration will continue to run them serially with an additional semaphore which can be expanded from a single lock to 2 or 5 etc when we're happy with it *** Need to pick a good day when we are able to observe the behavior of the system in order to catch issues as soon as possible. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Sprinting to Upgrade Servers to Focal (clarkb 20250204) *** Will try to continue upgrading things as time goes on. *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint ** Docker Hub rate limits (clarkb 20250304) *** Docker hub announced they would be changing rate limits from 100 pulls per 6 hours to 10 pulls per hour on March 1, 2025 *** This could be good for us if the shorter reset period works well with our zuul jobs as they tend to run for some time but not 6 hours *** Or it could be bad for us. *** Early data doesn't show any indicators for it being better or worse yet. *** Also keep in mind ipv6 addresses are rate limited per /64. We might consider forcing docker hub access through ipv4 via /etc/hosts entries. ** Running certcheck on bridge (clarkb 20250114) *** Any concern running this service in the bastion? *** Would better match how testing works. *** We would need to disable it on the cacti server. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup * Open discussion

1 0

Help w/ storyboard issue
by Khalil, Ghada 26 Feb '25

26 Feb '25

Hi all, I hope someone here can point me to the right contact to help with this issue. I had previously created a storyboard: https://storyboard.openstack.org/#!/story/2011332 -- titled: Platform Application Maintenance - ptp-notification It was accessible. We were able to create tasks and link them to code reviews previously: - https://review.opendev.org/c/starlingx/ptp-notification-armada-app/+/938075 - https://review.opendev.org/c/starlingx/ptp-notification-armada-app/+/940343 However, for some reason, we now get an API error saying the story is not found. 404: GET /api/v1/stories/2011332: Story 2011332 not found Thanks, Ghada Khalil StarlingX Release Prime

2 1

Team Meeting Agenda for February 25, 2025
by Clark Boylan 25 Feb '25

25 Feb '25

We will meet with this agenda on February 25, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** We are building Jammy, Noble, and Bullseye images now *** Flavors are in place for 4GB, 8GB, and 16GB test nodes. *** https://review.opendev.org/c/zuul/zuul/+/940824 Somewhat successful dogfooding in this zuul change ** Fixing known_hosts generation on bridge (clarkb 20250225) *** https://review.opendev.org/c/opendev/system-config/+/942307 *** TL;DR is that we weren't updating system-config on bridge before reading the inventory file to generate known_hosts on bridge *** This means that we could fail infra-prod-base trying to ssh to newly added servers *** Sometimes everything just worked because the -lb jobs weren't depending on anything else and if they ran early enough system-config would update by chance. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Sprinting to Upgrade Servers to Focal (clarkb 20250204) *** Managed to upgrade two servers *** Ran into problems with AppArmor on Noble interferring with podman and rsyslogd *** Will try to continue upgrading things as time goes on. *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint ** Redeploying raxflex resources (clarkb 20250225) *** We have access to a new raxflex region. To align with the old region and to correct network MTUs we need to redeploy the old region on the new project/tenant. *** Trying to determine whether or not we are required to use floating IPs before booting new mirrors. ** Running certcheck on bridge (clarkb 20250114) *** Any concern running this service in the bastion? *** Would better match how testing works. *** We would need to disable it on the cacti server. ** Service Coordinator Election (clarkb 20250107) *** Nominations closed and only clarkb was nominated *** Unless someone was missed clarkb is it by default and we don't need an election. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Using more robust Gitea caches (clarkb 20250225) *** Currently gitea uses a default cache setup with a single Go hashmap *** It is theorized that this hashmap grows such that eventually Go GC creates noticeable pause the world behavior impacting service response time *** https://review.opendev.org/c/opendev/system-config/+/942650 Set up gitea with memcached for caching *** As an alternative we could use the twoqueue adapter in gitea but this is compiled into the binary so would still be potentially impacting the GC system. But is one less container image and service. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup * Open discussion

1 0

Clarkb's OpenDev Service Coordinator Nomination
by Clark Boylan 19 Feb '25

19 Feb '25

Per the plan sent in https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… the OpenDev service coordinator nomination period ends in less than an hour. I haven't seen any other nominations yet and after some discussion in our weekly meeting it seems that folks are happy for me to be it again. I am sending this email to formally nominate myself to make it official. My intention for the next six months is largely to update servers to Noble or newer enabling us to get onto podman runtimes for containers and eventually stop publishing images to docker hub. I'd also like to see us continue to update services like Gerrit and Gitea, adopt the new zuul-launcher system within our Zuul deployment, and continue to push our use of Zuul forward as an illustration of what is possible with this powerful tool. If this sounds good to you then I'm happy to help guide us in that direction for the next six months. Otherwise, I am still more than happy to help anyone else who may wish to step into the role. Perhaps in six months someone else will be willing to step up? Clark

1 0

Team Meeting Agenda for February 18, 2025
by Clark Boylan 18 Feb '25

18 Feb '25

We will meet with this agenda on February 18, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** We are building Jammy, Noble, and Bullseye images now *** Flavors are in place for 4GB, 8GB, and 16GB test nodes. *** https://review.opendev.org/c/opendev/system-config/+/942018 Increase working disk space for zuul-launcher ** Unpinning our Grafana deployment (clarkb 20250128) *** Grafana02 is deployed and running the latest Grafana 10.x release *** 10.x reports angular dashboards are deprecated *** https://review.opendev.org/c/opendev/system-config/+/940997 Update to Grafana 11 **** Screenshots seem to indicate that we don't have deprecation warnings anymore however json definition files don't differ significantly from 10.x. Do we want to proceed or hold a node and debug further? ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** Discovered that `podman kill -s HUP` and `docker-compose kill -s HUP` fail on noble with podman apparmor permission issues **** I think we can work around this by issuing `kill -s HUP` directly to the container process pid **** Issue is fixed upstream. No movement on downstream bug to backport. **** https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2040483 ** Sprinting to Upgrade Servers to Focal (clarkb 20250204) *** Managed to upgrade two servers *** Ran into problems with AppArmor on Noble interferring with podman and rsyslogd *** Will try to continue upgrading things as time goes on. *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint *** https://review.opendev.org/c/opendev/system-config/+/941997 Optimize when LE edits trigger jobs ** Running certcheck on bridge (clarkb 20250114) *** Any concern running this service in the bastion? *** Would better match how testing works. *** We would need to disable it on the cacti server. ** Service Coordinator Election (clarkb 20250107) *** Nominations Open From February 4, 2025 to February 18, 2025 (You have until EOD UTC time today) *** Voting February 19, 2025 to February 26, 2025 if necessary *** All times and dates will be UTC based. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup * Open discussion

1 0

Team Meeting Agenda for February 11, 2025
by Clark Boylan 11 Feb '25

11 Feb '25

We will meet with this agenda on February 11, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** We are building Jammy, Noble, and Bullseye images now *** Flavors are in place for 4GB, 8GB, and 16GB test nodes. *** Discovered a zuul bug in the launcher implementation that affected jobs with empty buildsets (including noop jobs). Should be fixed now. ** Unpinning our Grafana deployment (clarkb 20250128) *** Grafana02 is deployed and running the latest Grafana 10.x release *** 10.x reports angular dashboards are deprecated *** https://review.opendev.org/c/opendev/system-config/+/940997 Update to Grafana 11 **** Screenshots seem to indicate that we don't have deprecation warnings anymore however json definition files don't differ significantly from 10.x. Do we want to proceed or hold a node and debug further? ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** Discovered that `podman kill -s HUP` and `docker-compose kill -s HUP` fail on noble with podman apparmor permission issues **** I think we can work around this by issuing `kill -s HUP` directly to the container process pid **** https://github.com/containers/common/issues/1898 **** https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2040483 ** Sprinting to Upgrade Servers to Focal (clarkb 20250204) *** Getting new servers going is much faster if we are able to review and address problems quickly *** Happening this week *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint **** Will attempt to track details on this etherpad ** Running certcheck on bridge (clarkb 20250114) *** Any concern running this service in the bastion? *** Would better match how testing works. *** We would need to disable it on the cacti server. ** Service Coordinator Election (clarkb 20250107) *** Nominations Open From February 4, 2025 to February 18, 2025 (one week remaining!) *** Voting February 19, 2025 to February 26, 2025 *** All times and dates will be UTC based. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup * Open discussion

1 0

Team Meeting Agenda for February 4, 2025
by Clark Boylan 04 Feb '25

04 Feb '25

We will meet on February 4, 2025 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** zuul-launcher related configs (images, labels, providers, etc) have been refactored into https://opendev.org/opendev/zuul-providers *** Zuul itself is starting to explore dogfooding of the new launcher managed images in jobs zuul runs. ** Unpinning our Grafana deployment (clarkb 20250128) *** https://review.opendev.org/c/opendev/system-config/+/940073 *** https://review.opendev.org/c/opendev/system-config/+/940654 *** Plan is to deploy new grafana on a new noble node (allows us to keep old grafana around on focal for now as a fallback) *** grafana01 is in the emergency file. Landing the two changes above should deploy what we need on grafana02. Once happy we update dns to point grafana.o.o at grafana02.o.o. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Sprinting to Upgrade Servers to Focal (clarkb 20250204) *** Getting new servers going is much faster if we are able to review and address problems quickly *** Clarkb proposes we plan to sprint next week (can use meetpad too if we like) and try to upgrade as many servers as we possible can to Noble *** This will help build confidence that noble and podman work well for us before we schedule a Gerrit upgrade. ** Switch to quay.io/opendevmirror images where possible (clarkb 20250128) *** Mariadb is an easy switch but updating the image from one location to another (even if the hash is the same) causes a service restart **** https://review.opendev.org/q/topic:%22quay-mariadb%22+status:open *** Other changes to consider: Pulling our python images from quay.io. We lose speculative testing of the base image but that may be a reasonable compromise. ** Running certcheck on bridge (clarkb 20250114) *** Any concern running this service in the bastion? *** Would better match how testing works. *** We would need to disable it on the cacti server. ** Service Coordinator Election (clarkb 20250107) *** Nominations Open From February 4, 2025 to February 18, 2025 *** Voting February 19, 2025 to February 26, 2025 *** All times and dates will be UTC based. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup * Open discussion

1 0