We will meet with this agenda on December 17, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** December 17, 2024 is the last meeting of the year. We will resume our weekly meeting on January 7,2025 * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** Raw image handling is being sped up through the use of zlib *** Can still use additional image build jobs for the other images we care about. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Docker Hub's New Rate Limits (clarkb 20241119) *** https://www.docker.com/blog/november-2024-updated-plans-announcement/ *** 10 pulls per IP per hour *** Disabling the use of the caching proxy server seems to have helped though we still hit rate limits. *** We discovered last week that IPv6 /64s are treated as a single IP for rate limit purposes. This is almost certainly making our situation worse. **** Do we want to shim in ipv4 records for docker hub hosts in /etc/hosts to force the use of ipv4 until we're sufficiently off of dockerhub? ** Docker compose plugin with podman service for servers (corvus 20241001) *** topic:podman-prep in system-config *** Modified our install-docker role to install docker-compose-v2 and podman from distro repos on Ubuntu Noble (older systems are unchanged) *** podman.socket is configured to listen on docker's socket pat and docker.service and docker.socket are disabled and docker.socket's path is changed to avoid conflict if reenabled. *** This makes docker and docker compose commands all work talking to podman on the backend. Podman commands also appear to work and see resources created by docker even though podman appears to use a different socket path. *** Created a small shell shim for docker-compose -> docker compose translation that will live on /usr/local/bin/docker-compose replacing the python tool *** Podman cannot emit logs to syslog but can emit them to journald which seems to be roughly equivalent *** docker compose changes container names so docker exec's that use container names are being updated to use docker-compose and logical names instead this gives us forward and backward compatibility. *** Updated how we check for new images from docker-compose pull. Now we list image ids, pull, and relist image ids and compare rather than inspecting docker-compose output as docker compose output isn't as verbose and doesn't contain necessary info. *** In theory once a service has updated to Noble we can rewrite config management to use podman commands instead of docker commands and docker compose commands instead of docker-compose commands. ** Rax-ord Noble Nodes With 1 VCPU (clarkb 20241210) *** Occasionally Ubuntu Noble nodes booted in rax-ord have a single vcpu *** Best I can tell nodepool didn't use the wrong flavor. Instead it seems possibly related to an older Xen release being used to boot Noble *** Not sure where the bug is (Xen or Linux) but maybe we can get Rax to do a hypervisor audit and upgrade those behind or remove them from being able to schedule things? ** Ubuntu-ports mirror fixups (fungi 20241217) *** fungi discovered that the ubuntu-ports mirror was stale and not being updated due to errors with the reprepro berkley db *** Fixing the reprepro berkley db didn't end up fixing mirroring as there was a new issue with a temporary file growing to infinity with recording of the files. *** Latest attempt to fixup is to start over: delete everything and rerun reprepro. * Open discussion