March 2025 - service-discuss - lists.opendev.org

Zuul errors
by ronald.stone＠windriver.com 18 Mar '25

18 Mar '25

Apologies if this is the wrong group. Zuul appears to be broken. See https://zuul.opendev.org/t/openstack/build/179c40f7e382474cb9c676b812caa6fb

2 1

Team Meeting Agenda for March 18, 2025
by Clark Boylan 18 Mar '25

18 Mar '25

We will meet on March 18, 2025 at 19:00 UTC with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** We are building Jammy, Noble, and Bullseye images now *** Flavors are in place for 4GB, 8GB, and 16GB test nodes. *** https://review.opendev.org/c/zuul/zuul/+/940824 Somewhat successful dogfooding in this zuul change ** Updating Flavors in OVH (clarkb 20250304) *** https://etherpad.opendev.org/p/ovh-flavors *** The zuul-launcher experiment is looking at expanding into OVH, but we only have 8GB flavors there. *** OVH said March 17 (today) was not good for them and they would be in touch to reschedule later. ** Container hygiene tasks (clarkb 20250318) *** https://review.opendev.org/c/opendev/system-config/+/944799 drop python 3.10 image builds *** https://review.opendev.org/c/opendev/system-config/+/944789 rebuild python 3.11 and 3.12 base images *** https://review.opendev.org/q/topic:%22opendev-python3.12%22+status:open Update images to use python3.12 *** Rebuilding our base images then switching things over to those new base images seems like a good way to get updates. ** Dropping uWSGI (clarkb 20250318) *** uWSGI has never built reliable and we are hitting a new segfault during compilation *** uWSGI is also on life support and not getting new development just bugfixes here and there *** Thinking about switching to Granian which is an ASGI/RSGI/WSGI server written in Rust that publishes wheels for aarch64 and x86_64 *** In theory this means we should be able to get away with one server regardless of the protocol that web services we choose to run use. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Sprinting to Upgrade Servers to Focal (clarkb 20250204) *** Will try to continue upgrading things as time goes on. *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint *** nb05 and nb06 are in the process of getting deployed on Noble with podman and docker compose. ** Running certcheck on bridge (clarkb 20250114) *** Suggestion is that we just run this as an infra-prod job instead as all state should be available there. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup ** Packaging updates for bindep (fungi 20250318) *** https://review.opendev.org/938570 Drop requirements.txt *** https://review.opendev.org/940711 Drop auxiliary requirements files *** It would be nice to get consensus on whether or not to include these when replicating the new packaging methods to our other tools. * Open discussion

1 0

Team Meeting Agenda for March 11, 2025
by Clark Boylan 11 Mar '25

11 Mar '25

We will meet with this agenda on March 11, 2025 at 19:00 UTC (note that North America and maybe others have switched to DST) in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** We are building Jammy, Noble, and Bullseye images now *** Flavors are in place for 4GB, 8GB, and 16GB test nodes. *** https://review.opendev.org/c/zuul/zuul/+/940824 Somewhat successful dogfooding in this zuul change ** Updating Flavors in OVH (clarkb 20250304) *** https://etherpad.opendev.org/p/ovh-flavors *** The zuul-launcher experiment is looking at expanding into OVH, but we only have 8GB flavors there. *** Plan is to target March 17 for OVH flavor surgery ** Running infra-prod Jobs in Parallel on Bridge (clarkb 20250304) *** https://review.opendev.org/c/opendev/system-config/+/943488 Actually run infra-prod jobs in parallel *** The refactor to infra-prod jobs has gone in but with a semaphore limit of 1 which continues to serialize things *** There were some small issues. **** bootstrap-bridge needed extra required-projects to setup repos like project-config in periodic pipelines **** project-config and dns zone repos needed to be setup with bootstrap-bridge as the parent job to ensure repos are configured and locks are respected *** Seems like things work serially at this point. Time to proceed with parallel execution? ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Sprinting to Upgrade Servers to Focal (clarkb 20250204) *** Will try to continue upgrading things as time goes on. *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint ** Running certcheck on bridge (clarkb 20250114) *** Suggestion is that we just run this as an infra-prod job instead as all state should be available there. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup * Open discussion

1 0

Team Meeting Agenda for March 4, 2025
by Clark Boylan 04 Mar '25

04 Mar '25

We will meet with this agenda on March 4, 2025 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Redeploying raxflex resources (clarkb 20250225) *** We have access to a new raxflex region. To align with the old region and to correct network MTUs we need to redeploy the old region on the new project/tenant. *** https://review.opendev.org/q/hashtag:flex-dfw3 ** Zuul-launcher image builds (corvus 20240910) *** We are building Jammy, Noble, and Bullseye images now *** Flavors are in place for 4GB, 8GB, and 16GB test nodes. *** https://review.opendev.org/c/zuul/zuul/+/940824 Somewhat successful dogfooding in this zuul change ** Updating Flavors in OVH (clarkb 20250304) *** https://etherpad.opendev.org/p/ovh-flavors *** The zuul-launcher experiment is looking at expanding into OVH, but we only have 8GB flavors there. ** Running infra-prod Jobs in Parallel on Bridge (clarkb 20250304) *** https://review.opendev.org/c/opendev/system-config/+/942439 *** This change refactors how we run infra-prod-* jobs on bridge to enable running them in parallel *** The current iteration will continue to run them serially with an additional semaphore which can be expanded from a single lock to 2 or 5 etc when we're happy with it *** Need to pick a good day when we are able to observe the behavior of the system in order to catch issues as soon as possible. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Sprinting to Upgrade Servers to Focal (clarkb 20250204) *** Will try to continue upgrading things as time goes on. *** https://etherpad.opendev.org/p/opendev-server-replacement-sprint ** Docker Hub rate limits (clarkb 20250304) *** Docker hub announced they would be changing rate limits from 100 pulls per 6 hours to 10 pulls per hour on March 1, 2025 *** This could be good for us if the shorter reset period works well with our zuul jobs as they tend to run for some time but not 6 hours *** Or it could be bad for us. *** Early data doesn't show any indicators for it being better or worse yet. *** Also keep in mind ipv6 addresses are rate limited per /64. We might consider forcing docker hub access through ipv4 via /etc/hosts entries. ** Running certcheck on bridge (clarkb 20250114) *** Any concern running this service in the bastion? *** Would better match how testing works. *** We would need to disable it on the cacti server. ** Working through our TODO list (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup * Open discussion

1 0