December 2024 - service-discuss

Team Meeting Agenda for December 17, 2024
by Clark Boylan 17 Dec '24

17 Dec '24

We will meet with this agenda on December 17, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** December 17, 2024 is the last meeting of the year. We will resume our weekly meeting on January 7,2025 * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** Raw image handling is being sped up through the use of zlib *** Can still use additional image build jobs for the other images we care about. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Docker Hub's New Rate Limits (clarkb 20241119) *** https://www.docker.com/blog/november-2024-updated-plans-announcement/ *** 10 pulls per IP per hour *** Disabling the use of the caching proxy server seems to have helped though we still hit rate limits. *** We discovered last week that IPv6 /64s are treated as a single IP for rate limit purposes. This is almost certainly making our situation worse. **** Do we want to shim in ipv4 records for docker hub hosts in /etc/hosts to force the use of ipv4 until we're sufficiently off of dockerhub? ** Docker compose plugin with podman service for servers (corvus 20241001) *** topic:podman-prep in system-config *** Modified our install-docker role to install docker-compose-v2 and podman from distro repos on Ubuntu Noble (older systems are unchanged) *** podman.socket is configured to listen on docker's socket pat and docker.service and docker.socket are disabled and docker.socket's path is changed to avoid conflict if reenabled. *** This makes docker and docker compose commands all work talking to podman on the backend. Podman commands also appear to work and see resources created by docker even though podman appears to use a different socket path. *** Created a small shell shim for docker-compose -> docker compose translation that will live on /usr/local/bin/docker-compose replacing the python tool *** Podman cannot emit logs to syslog but can emit them to journald which seems to be roughly equivalent *** docker compose changes container names so docker exec's that use container names are being updated to use docker-compose and logical names instead this gives us forward and backward compatibility. *** Updated how we check for new images from docker-compose pull. Now we list image ids, pull, and relist image ids and compare rather than inspecting docker-compose output as docker compose output isn't as verbose and doesn't contain necessary info. *** In theory once a service has updated to Noble we can rewrite config management to use podman commands instead of docker commands and docker compose commands instead of docker-compose commands. ** Rax-ord Noble Nodes With 1 VCPU (clarkb 20241210) *** Occasionally Ubuntu Noble nodes booted in rax-ord have a single vcpu *** Best I can tell nodepool didn't use the wrong flavor. Instead it seems possibly related to an older Xen release being used to boot Noble *** Not sure where the bug is (Xen or Linux) but maybe we can get Rax to do a hypervisor audit and upgrade those behind or remove them from being able to schedule things? ** Ubuntu-ports mirror fixups (fungi 20241217) *** fungi discovered that the ubuntu-ports mirror was stale and not being updated due to errors with the reprepro berkley db *** Fixing the reprepro berkley db didn't end up fixing mirroring as there was a new issue with a temporary file growing to infinity with recording of the files. *** Latest attempt to fixup is to start over: delete everything and rerun reprepro. * Open discussion

1 0

Team Meeting Agenda for December 10, 2024
by Clark Boylan 10 Dec '24

10 Dec '24

We will meet on December 10, 2024 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** Raw image handling is being sped up through the use of zlib *** Can still use additional image build jobs for the other images we care about. ** Backup Server Pruning (clarkb 20241008) *** The smaller of the two backup servers is needing manual pruning multiple times a month now (the interval is between 2 and 3 weeks) *** There are older server backup dirs present on that server that we could delete (as well as removal of the corresponding login user for that server) **** ask01, ethercalc02, etherpad01, gitea01, lists, review-dev01, and review01 **** https://review.opendev.org/c/opendev/system-config/+/937040 Purge retired backups for the above list on the vexxhost backup server. **** Manual prune post retirement looked good. We should rerun prune after purging to check the script is still happy in that state. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Docker Hub's New Rate Limits (clarkb 20241119) *** https://www.docker.com/blog/november-2024-updated-plans-announcement/ *** 10 pulls per IP per hour *** Disabling the use of the caching proxy server seems to have helped though we still hit rate limits. ** Docker compose plugin with podman service for servers (corvus 20241001) *** https://review.opendev.org/923084 is a demo (in the Zuul repo) of using docker compose v2 plugin with system podman service *** Looking at modifying the docker and docker compose installation process for Noble and newer to set up docker compose (no -) with podman on the backend. ** Gerrit 3.10 Upgrade Planning (clarkb 20241105) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.10 *** Upgrade is complete. There were no obvious issue during the upgrade *** topic:upgrade-gerrit-3.10 has a number of followups. The 3.9 image build removal should be safe once we're happy with 3.10 and unlikely to revert. *** Related Gerrit stable-3.10 has the updated openid redirect code in it now. An image rebuild of 3.10 should be accompanied with testing and a planned gerrit restart to pick this up. ** Diskimage Builder Testing and Release (clarkb 20241210) *** OpenEuler testing in DIB is failing due to the replacement of mirror content for OpenEuler **** Switching to not use our mirror also fails. It seems we must explicitly point at OpenEuler package content *** Do we want to disable OpenEuler image builds for now / make them non voting? Or try and fix it? Any volunteers? *** A request has been made to release DIB which will pull in gentoo fixes but there is an svc-map bugfix that would be good to land too held up by the openeuler testing issues. ** Rax-ord Noble Nodes With 1 VCPU (clarkb 20241210) *** Occasionally Ubuntu Noble nodes booted in rax-ord have a single vcpu *** Best I can tell nodepool didn't use the wrong flavor. Instead it seems possibly related to an older Xen release being used to boot Noble *** Not sure where the bug is (Xen or Linux) but maybe we can get Rax to do a hypervisor audit and upgrade those behind or remove them from being able to schedule things? * Open discussion

1 0

Team Meeting Agenda for December 3, 2024
by Clark Boylan 03 Dec '24

03 Dec '24

We will meet with this agenda on December 3, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** Raw image handling is being sped up through the use of zlib *** Can still use additional image build jobs for the other images we care about. ** Backup Server Pruning (clarkb 20241008) *** The smaller of the two backup servers is needing manual pruning multiple times a month now (the interval is between 2 and 3 weeks) *** There are older server backup dirs present on that server that we could delete (as well as removal of the corresponding login user for that server) **** ask01, ethercalc02, etherpad01, gitea01, lists, review-dev01, and review01 **** https://review.opendev.org/c/opendev/system-config/+/936203 retire these old unneeded backups in prep for eventual purging. **** We should run the manual prune script before we purge to ensure that script is still happy with the retirement process. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Docker Hub's New Rate Limits (clarkb 20241119) *** https://www.docker.com/blog/november-2024-updated-plans-announcement/ *** 10 pulls per IP per hour *** Disabling the use of the caching proxy server seems to have helped though we still hit rate limits. ** Docker compose plugin with podman service for servers (corvus 20241001) *** https://review.opendev.org/923084 is a demo (in the Zuul repo) of using docker compose v2 plugin with system podman service *** Looking at modifying the docker and docker compose installation process for Noble and newer to set up docker compose (no -) with podman on the backend. ** Gerrit 3.10 Upgrade Planning (clarkb 20241105) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.10 *** Feedback welcome on the process and listed concerns as well as the mitigations or assertions they don't affect us. *** Upgrade will occur at 1600UTC December 6, 2024 *** We just updated to 3.9.8 in production in order to test the 3.9.8 to 3.10.3 upgrade path. Clarkb will run through this testing soon and update the etherpad if necessary *** From 3.9.8 and onward there is a new sign in bug that prevents Gerrit from redirecting back to the page you clicked sign in on once logged in during some situations. **** We can live with this and Clarkb has engaged upstream to see if we can fix it. ** Upgrading Gitea to 1.22.4 (clarkb 20241126) *** https://review.opendev.org/c/opendev/system-config/+/936198 * Open discussion

1 0