- service-discuss - lists.opendev.org

Team Meeting Agenda for January 28, 2025
by Clark Boylan 27 Jan '25

27 Jan '25

== Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** https://review.opendev.org/q/hashtag:niz+status:open is next set of work happening in Zuul ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** Thoughts on using Noble **** Rely on borg 1.2.8 backups to existing servers until we have new backup servers running borg 1.4.x **** Some docker compose setups may need to be edited to set a restart policy of always **** Podman package updates don't appear to require container restarts. But we should observe this behavior if possible. ** Switch to quay.io/opendevmirror images where possible (clarkb 20250128) *** Mariadb is an easy switch but updating the image from one location to another (even if the hash is the same) causes a service restart *** Other changes to consider: Pulling our python images from quay.io. We lose speculative testing of the base image but that may be a reasonable compromise. ** Unpinning our Grafana deployment (clarkb 20250128) *** https://review.opendev.org/c/opendev/system-config/+/940073 *** Testing shows this may still exhibit the problem we saw that caused us to pin *** Trying to use a held node to test directly. ** Increasing Exim's Acceptable Line Length (clarkb 20250128) *** https://review.opendev.org/c/opendev/system-config/+/940248 *** Bounces were being generated by our own Exim system on mailman due to line lengths being too long. This is configurable and we can avoid the problem entirely. ** Running certcheck on bridge (clarkb 20250114) *** Any concern running this service in the bastion? *** Would better match how testing works. *** We would need to disable it on the cacti server. ** Service Coordinator Election (clarkb 20250107) *** Nominations Open From February 4, 2025 to February 18, 2025 *** Voting February 19, 2025 to February 26, 2025 *** All times and dates will be UTC based. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Beginning of the Year (Virtual) Meetup Recap (clarkb 20250107) *** https://etherpad.opendev.org/p/opendev-january-2025-meetup *** Tried to condense the notes down into a TODO list for the next $TIME * Open discussion

1 0

February 2025 OpenDev Service Coordinator Election
by Clark Boylan 21 Jan '25

21 Jan '25

It is that time of year again where we need to elect an OpenDev Service Coordinator [0] for the next six months. Nominations will be open from February 4, 2025 to February 18, 2025. If we have more than one candidate we will vote for our preferred candidate from February 19, 2025 to February 26, 2025. All dates are based on a UTC clock. Please send your self nominations to this mailing list during the nomination time frame. You are eligible to run and vote if you have contributed to OpenDev git repositories in the last 12 months (this period will be February 4, 2024 to February 3, 2025). I continue to encourage anyone who may be interested to reach out and ask questions. I am happy to support someone new in the role. [0] https://docs.opendev.org/opendev/system-config/latest/project.html#governan… Clark

1 0

Team Meeting Agenda for January 21, 2025
by Clark Boylan 20 Jan '25

20 Jan '25

== Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** https://review.opendev.org/q/hashtag:niz+status:open is next set of work happening in Zuul ** Deploying new Noble Servers (clarkb 20250107) *** New paste server is up and running under podman and docker compose *** Ran into problems with our pinned version of borg on Noble with python3.12 *** Updated borg for noble servers does seem to work ok. **** https://review.opendev.org/c/opendev/system-config/+/939667 fixup for warnings treated as errors with new borg spamming email ** Deploying Lodgeit entirely without Docker Hub (clarkb 20250121) *** In theory, now that we're using podman as the container runtime for paste, we can get speculative image testing without hosting on Docker Hub. *** https://review.opendev.org/c/opendev/lodgeit/+/939385 Publish lodgeit image to quay.io ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Gerrit 3.10.4 Update (clarkb 20250114) *** https://review.opendev.org/c/opendev/system-config/+/939167 New bugfix releases available *** Good opportunity to bundle up the h2 compaction change with a version update. **** https://review.opendev.org/c/opendev/system-config/+/938000 ** Running certcheck on bridge (clarkb 20250114) *** Any concern running this service in the bastion? *** Would better match how testing works. *** We would need to disable it on the cacti server. ** Service Coordinator Election (clarkb 20250107) *** Proposing this schedule: **** Nominations Open From February 4, 2025 to February 18, 2025 **** Voting February 19, 2025 to February 26, 2025 **** All times and dates will be UTC based. ** Beginning of the Year (Virtual) Meetup (clarkb 20250107) *** Consider this a spiritual successor to last year's Pre PTG. *** Idea is to take some time, sync up on needs and priorities, and maybe even do some hackathon type work to flush out the backlog *** Starts not long after this meeting see you there. *** https://etherpad.opendev.org/p/opendev-january-2025-meetup * Open discussion

1 0

Team Meeting Agenda for January 14, 2025
by Clark Boylan 13 Jan '25

13 Jan '25

== Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** https://review.opendev.org/q/hashtag:niz+status:open is next set of work happening in Zuul ** Deploying new Noble Servers (clarkb 20250107) *** Replacement paste server is getting deployed as we speak. *** Discovered that restart: always is the only container restart policy podman will respect on reboot. Others result in no containers on boot. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Mirroring Useful Container Images (clarkb 20250107) *** Initial set of images have been mirrored. *** We rediscovered that quay images pushed via docker image protocol are private by default but jobs have been updated to mark new images public *** Time to start adding useful images and tags **** https://review.opendev.org/c/opendev/system-config/+/939169 ** Gerrit 3.10.4 Update (clarkb 20250114) *** https://review.opendev.org/c/opendev/system-config/+/939167 New bugfix releases available *** Good opportunity to bundle up the h2 compaction change with a version update. **** https://review.opendev.org/c/opendev/system-config/+/938000 ** Running certcheck on bridge (clarkb 20250114) *** Any concern running this service in the bastion? *** Would better match how testing works. *** We would need to disable it on the cacti server. ** Service Coordinator Election (clarkb 20250107) *** Proposing this schedule: **** Nominations Open From February 4, 2025 to February 18, 2025 **** Voting February 19, 2025 to February 26, 2025 **** All times and dates will be UTC based. ** Beginning of the Year (Virtual) Meetup (clarkb 20250107) *** Consider this a spiritual successor to last year's Pre PTG. *** Idea is to take some time, sync up on needs and priorities, and maybe even do some hackathon type work to flush out the backlog *** January 21-23. Need to decide on specific times and days. *** https://etherpad.opendev.org/p/opendev-january-2025-meetup * Open discussion

1 0

Team Meeting Agenda for January 7, 2025
by Clark Boylan 06 Jan '25

06 Jan '25

== Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** Raw image handling is being sped up through the use of zlib *** Can still use additional image build jobs for the other images we care about. ** Deploying new Noble Servers (clarkb 20250107) *** Changes to manage containers with podman and docker compose on Noble have landed *** Next step is deploying a Noble server for a straightforward service (paste most likely) and observing how that goes. *** Ultimately would like to deploy a new review server on Noble using this system. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Mirroring Useful Container Images (clarkb 20250107) *** One approach to mitigating Docker Hub rate limits is to mirror useful images to another registry like quay.io. *** https://review.opendev.org/c/opendev/system-config/+/938508 Initial change to mirror images that are generically useful ** Gerrit H2 Cache File Growth (clarkb 20250107) *** Gerrit's git_file_diff and gerrit_file_diff caches are implemented as H2 databases on disk. *** These H2 databases are backed with files on disk that only compact when Gerrit is shutdown. *** This leads to files growing quite large which impacts Gerrit startup *** https://review.opendev.org/c/opendev/system-config/+/938000 Suggested workaround from Hashar improves compaction when we do shutdown *** Should we also revert our change to increase the logical size of the cache? This may slow down growth on disk as daily pruning would trim things back regularly (but not compact). ** Rax-ord Noble Nodes With 1 VCPU (clarkb 20241210) *** Occasionally Ubuntu Noble nodes booted in rax-ord have a single vcpu *** Best I can tell nodepool didn't use the wrong flavor. Instead it seems possibly related to an older Xen release being used to boot Noble *** Not sure where the bug is (Xen or Linux) but maybe we can get Rax to do a hypervisor audit and upgrade those behind or remove them from being able to schedule things? *** On our side of things we could have an early base job check for >1 vcpu if Xen is in use. ** Service Coordinator Election (clarkb 20250107) *** Proposing this schedule: **** Nominations Open From February 4, 2025 to February 18, 2025 **** Voting February 19, 2025 to February 26, 2025 **** All times and dates will be UTC based. ** Beginning of the Year (Virtual) Meetup (clarkb 20250107 *** Consider this a spiritual successor to last year's Pre PTG. *** Idea is to take some time, sync up on needs and priorities, and maybe even do some hackathon type work to flush out the backlog *** January 21-23? Open to ideas on timing. * Open discussion

1 0

Team Meeting Agenda for December 17, 2024
by Clark Boylan 16 Dec '24

16 Dec '24

We will meet with this agenda on December 17, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** December 17, 2024 is the last meeting of the year. We will resume our weekly meeting on January 7,2025 * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** Raw image handling is being sped up through the use of zlib *** Can still use additional image build jobs for the other images we care about. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Docker Hub's New Rate Limits (clarkb 20241119) *** https://www.docker.com/blog/november-2024-updated-plans-announcement/ *** 10 pulls per IP per hour *** Disabling the use of the caching proxy server seems to have helped though we still hit rate limits. *** We discovered last week that IPv6 /64s are treated as a single IP for rate limit purposes. This is almost certainly making our situation worse. **** Do we want to shim in ipv4 records for docker hub hosts in /etc/hosts to force the use of ipv4 until we're sufficiently off of dockerhub? ** Docker compose plugin with podman service for servers (corvus 20241001) *** topic:podman-prep in system-config *** Modified our install-docker role to install docker-compose-v2 and podman from distro repos on Ubuntu Noble (older systems are unchanged) *** podman.socket is configured to listen on docker's socket pat and docker.service and docker.socket are disabled and docker.socket's path is changed to avoid conflict if reenabled. *** This makes docker and docker compose commands all work talking to podman on the backend. Podman commands also appear to work and see resources created by docker even though podman appears to use a different socket path. *** Created a small shell shim for docker-compose -> docker compose translation that will live on /usr/local/bin/docker-compose replacing the python tool *** Podman cannot emit logs to syslog but can emit them to journald which seems to be roughly equivalent *** docker compose changes container names so docker exec's that use container names are being updated to use docker-compose and logical names instead this gives us forward and backward compatibility. *** Updated how we check for new images from docker-compose pull. Now we list image ids, pull, and relist image ids and compare rather than inspecting docker-compose output as docker compose output isn't as verbose and doesn't contain necessary info. *** In theory once a service has updated to Noble we can rewrite config management to use podman commands instead of docker commands and docker compose commands instead of docker-compose commands. ** Rax-ord Noble Nodes With 1 VCPU (clarkb 20241210) *** Occasionally Ubuntu Noble nodes booted in rax-ord have a single vcpu *** Best I can tell nodepool didn't use the wrong flavor. Instead it seems possibly related to an older Xen release being used to boot Noble *** Not sure where the bug is (Xen or Linux) but maybe we can get Rax to do a hypervisor audit and upgrade those behind or remove them from being able to schedule things? ** Ubuntu-ports mirror fixups (fungi 20241217) *** fungi discovered that the ubuntu-ports mirror was stale and not being updated due to errors with the reprepro berkley db *** Fixing the reprepro berkley db didn't end up fixing mirroring as there was a new issue with a temporary file growing to infinity with recording of the files. *** Latest attempt to fixup is to start over: delete everything and rerun reprepro. * Open discussion

1 0

Team Meeting Agenda for December 10, 2024
by Clark Boylan 09 Dec '24

09 Dec '24

We will meet on December 10, 2024 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** Raw image handling is being sped up through the use of zlib *** Can still use additional image build jobs for the other images we care about. ** Backup Server Pruning (clarkb 20241008) *** The smaller of the two backup servers is needing manual pruning multiple times a month now (the interval is between 2 and 3 weeks) *** There are older server backup dirs present on that server that we could delete (as well as removal of the corresponding login user for that server) **** ask01, ethercalc02, etherpad01, gitea01, lists, review-dev01, and review01 **** https://review.opendev.org/c/opendev/system-config/+/937040 Purge retired backups for the above list on the vexxhost backup server. **** Manual prune post retirement looked good. We should rerun prune after purging to check the script is still happy in that state. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Docker Hub's New Rate Limits (clarkb 20241119) *** https://www.docker.com/blog/november-2024-updated-plans-announcement/ *** 10 pulls per IP per hour *** Disabling the use of the caching proxy server seems to have helped though we still hit rate limits. ** Docker compose plugin with podman service for servers (corvus 20241001) *** https://review.opendev.org/923084 is a demo (in the Zuul repo) of using docker compose v2 plugin with system podman service *** Looking at modifying the docker and docker compose installation process for Noble and newer to set up docker compose (no -) with podman on the backend. ** Gerrit 3.10 Upgrade Planning (clarkb 20241105) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.10 *** Upgrade is complete. There were no obvious issue during the upgrade *** topic:upgrade-gerrit-3.10 has a number of followups. The 3.9 image build removal should be safe once we're happy with 3.10 and unlikely to revert. *** Related Gerrit stable-3.10 has the updated openid redirect code in it now. An image rebuild of 3.10 should be accompanied with testing and a planned gerrit restart to pick this up. ** Diskimage Builder Testing and Release (clarkb 20241210) *** OpenEuler testing in DIB is failing due to the replacement of mirror content for OpenEuler **** Switching to not use our mirror also fails. It seems we must explicitly point at OpenEuler package content *** Do we want to disable OpenEuler image builds for now / make them non voting? Or try and fix it? Any volunteers? *** A request has been made to release DIB which will pull in gentoo fixes but there is an svc-map bugfix that would be good to land too held up by the openeuler testing issues. ** Rax-ord Noble Nodes With 1 VCPU (clarkb 20241210) *** Occasionally Ubuntu Noble nodes booted in rax-ord have a single vcpu *** Best I can tell nodepool didn't use the wrong flavor. Instead it seems possibly related to an older Xen release being used to boot Noble *** Not sure where the bug is (Xen or Linux) but maybe we can get Rax to do a hypervisor audit and upgrade those behind or remove them from being able to schedule things? * Open discussion

1 0

Team Meeting Agenda for December 3, 2024
by Clark Boylan 02 Dec '24

02 Dec '24

We will meet with this agenda on December 3, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** Raw image handling is being sped up through the use of zlib *** Can still use additional image build jobs for the other images we care about. ** Backup Server Pruning (clarkb 20241008) *** The smaller of the two backup servers is needing manual pruning multiple times a month now (the interval is between 2 and 3 weeks) *** There are older server backup dirs present on that server that we could delete (as well as removal of the corresponding login user for that server) **** ask01, ethercalc02, etherpad01, gitea01, lists, review-dev01, and review01 **** https://review.opendev.org/c/opendev/system-config/+/936203 retire these old unneeded backups in prep for eventual purging. **** We should run the manual prune script before we purge to ensure that script is still happy with the retirement process. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Docker Hub's New Rate Limits (clarkb 20241119) *** https://www.docker.com/blog/november-2024-updated-plans-announcement/ *** 10 pulls per IP per hour *** Disabling the use of the caching proxy server seems to have helped though we still hit rate limits. ** Docker compose plugin with podman service for servers (corvus 20241001) *** https://review.opendev.org/923084 is a demo (in the Zuul repo) of using docker compose v2 plugin with system podman service *** Looking at modifying the docker and docker compose installation process for Noble and newer to set up docker compose (no -) with podman on the backend. ** Gerrit 3.10 Upgrade Planning (clarkb 20241105) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.10 *** Feedback welcome on the process and listed concerns as well as the mitigations or assertions they don't affect us. *** Upgrade will occur at 1600UTC December 6, 2024 *** We just updated to 3.9.8 in production in order to test the 3.9.8 to 3.10.3 upgrade path. Clarkb will run through this testing soon and update the etherpad if necessary *** From 3.9.8 and onward there is a new sign in bug that prevents Gerrit from redirecting back to the page you clicked sign in on once logged in during some situations. **** We can live with this and Clarkb has engaged upstream to see if we can fix it. ** Upgrading Gitea to 1.22.4 (clarkb 20241126) *** https://review.opendev.org/c/opendev/system-config/+/936198 * Open discussion

1 0

Team Meeting Agenda for November 26, 2024
by Clark Boylan 25 Nov '24

25 Nov '24

We will meet with this agenda on November 26, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** US Holiday this week. * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** Raw image handling is being sped up through the use of zlib *** Can still use additional image build jobs for the other images we care about. ** Backup Server Pruning (clarkb 20241008) *** The smaller of the two backup servers is needing manual pruning multiple times a month now (the interval is between 2 and 3 weeks) *** There are older server backup dirs present on that server that we could delete (as well as removal of the corresponding login user for that server) **** ask01, ethercalc02, etherpad01, gitea01, lists, review-dev01, and review01 **** https://review.opendev.org/c/opendev/system-config/+/936203 retire these old unneeded backups in prep for eventual purging. **** We should run the manual prune script before we purge to ensure that script is still happy with the retirement process. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Docker Hub's New Rate Limits (clarkb 20241119) *** https://www.docker.com/blog/november-2024-updated-plans-announcement/ *** 10 pulls per IP per hour *** Disabling the use of the caching proxy server seems to have helped though we still hit rate limits. ** Docker compose plugin with podman service for servers (corvus 20241001) *** https://review.opendev.org/923084 is a demo (in the Zuul repo) of using docker compose v2 plugin with system podman service *** Looking at modifying the docker and docker compose installation process for Noble and newer to set up docker compose (no -) with podman on the backend. ** Gerrit 3.10 Upgrade Planning (clarkb 20241105) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.10 *** Feedback welcome on the process and listed concerns as well as the mitigations or assertions they don't affect us. *** Pencilled in 1600UTC December 6, 2024 for the upgrade to occur. ** Renaming mailman's openinfra.dev vhost to openinfra.org (fungi 20241126) *** Happening December 2, 2024 *** https://etherpad.opendev.org/p/lists-openinfra-org-migration ** Upgrading Gitea to 1.22.4 (clarkb 20241126) *** https://review.opendev.org/c/opendev/system-config/+/936198 * Open discussion

1 0

Team Meeting Agenda for November 19, 2024
by Clark Boylan 18 Nov '24

18 Nov '24

We will meet with this agenda on November 19, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Next week is a Holiday week for many in the US. Expect slower responses. * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** Need jobs to build additional images. Currently only x86 Debian Bullseye is being built ** Backup Server Pruning (clarkb 20241008) *** The smaller of the two backup servers is needing manual pruning multiple times a month now (the interval is between 2 and 3 weeks) *** There are older server backup dirs present on that server that we could delete (as well as removal of the corresponding login user for that server) **** ask01, ethercalc02, etherpad01, gitea01, lists, review-dev01, and review01 **** https://review.opendev.org/c/opendev/system-config/+/933700 Backup deletions managed through Ansible **** https://review.opendev.org/c/opendev/system-config/+/934768 Handle backup verification for purged backups ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Docker Hub's New Rate Limits (clarkb 20241119) *** https://www.docker.com/blog/november-2024-updated-plans-announcement/ *** 10 pulls per IP per hour *** https://access.redhat.com/solutions/6218921 Quay's rate limiting statement for comparison ** Docker compose plugin with podman service for servers (corvus 20241001) *** https://review.opendev.org/923084 is a demo (in the Zuul repo) of using docker compose v2 plugin with system podman service *** This would let us host opendev images on quay.io (or any non-dockerhub site) and use speculative images *** Some caveats (seen in the change): **** unconfined apparmor profile to work around https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2040483 **** buildx startup probably not an issue since we don't use docker compose image builds: https://github.com/docker/buildx/issues/344 **** need to set DOCKER_HOST env variable (system-wide bashrc? maybe a docker context? other options?) *** Do we want to proceed? Any blockers? **** Plan is to test the switch with a less consequential service (paste maybe). Upgrade the hosting server to Noble first if necessary. ** Enabling mailman3 bounce processing (frickler 20241030) *** This has been enabled on a number of lists manually. **** Did we update openstack-discuss yet? *** Need to configure new lists to turn this on by default. Likely using mailman3 list styles to enforce them by default. ** Intermediate Insecure CI Registry Pruning (clarkb 20241112) *** Pruning is still in progress. Estimated total runtime will be around 6 days. *** We found an additional bug (lack of swift pagination in the object listing performed by zuul-registry) that slightly delayed us *** https://review.opendev.org/c/opendev/system-config/+/935542 Enable daily pruning after this bulk prune is complete. ** Gerrit 3.10 Upgrade Planning (clarkb 20241105) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.10 *** Feedback welcome on the process and listed concerns as well as the mitigations or assertions they don't affect us. *** Pencilled in 1600UTC December 6, 2024 for the upgrade to occur. ** mirror.sjc3.raxflex.opendev.org cinder volume issues (clarkb 20241119) *** Another occurrence of sector 0 being unreadable on the device. *** Filesystems remounted RO and AFS is non functional *** Disabling cloud in nodepool for now while we sort out if we replace the device again. * Open discussion

1 0