We will meet with this agenda on January 9, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenInfra Foundation Individual Board member election is happening now. Look for your ballot via email and vote. ** OpenInfra Live will feature OpenDev January 18, 2024. * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Python container updates (tonyb 20230718) *** https://review.opendev.org/c/opendev/system-config/+/905018 Drop Bullseye python3.11 images *** zuul-operator is the last hold out now **** https://review.opendev.org/c/zuul/zuul-operator/+/881245 is the change we need to get landed. ** Updating Zuul's database server (clarkb 20231121) *** Currently this is an older mysql 5.7 trove instance *** We can move it to a self hosted instance (maybe on a dedicated host?) running out of docker like many of our other services and get it more up to date. *** Are there other services we should consider this for as well? *** Research/Planning questions: https://etherpad.opendev.org/p/opendev-zuul-mysql-upgrade ** EMS discontinuing legacy/consumer hosting plans (fungi 20231219) *** We have until 2024-02-07 to upgrade to a business hosting plan (prepaying a year at 10x the current price) or move elsewhere. ** Followup on 20231216 incident (frickler 20231217) *** Do we want to pin external images like haproxy and only bump them after testing? (Not sure that would've helped for the current issue though) *** Use docker prune less aggressively for easier rollback? **** We do so for some services, like https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/..., might want to duplicate for all containers? Bump the hold time to 7d? *** Add timestamps to zuul_reboot.log? **** https://opendev.org/opendev/system-config/src/branch/master/playbooks/servic... **** Also this is running on Saturdays (weekday: 6), do we want to fix the comment or the dow? *** Do we want to document or implement a procedure for rolling back zuul upgrades? Or do we assume that issues can always be fixed in a forward going way? ** AFS quota issues (frickler 20231217) *** mirror.openeuler has reached its quota limit and the mirror job seems to be failing since two weeks. I'm also a bit worried that they seem do have doubled their volume over the last 12 months *** ubuntu mirrors are also getting close, but we might have another couple of months time there *** mirror.centos-stream seems to have a steep increase in the last two months and might also run into quota limits soon *** project.zuul with the latest releases is getting close to its tight limit of 1GB (sic), I suggest to simply double that ** Broken wheel build issues (frickler 20231217) *** wheel builds for centos >=8 seem broken, with nobody maintaining these it might be better to drop them? ** Gitea repo-archives filling server disk (clarkb 20240109) *** https://review.opendev.org/c/opendev/system-config/+/904868 update robots.txt on upstream's suggestion *** https://review.opendev.org/c/opendev/system-config/+/904874 Run weekly removal of all cached repo archives ** OpenDev Service Coordinator Election happening soon (clarkb 20240109) * Open discussion