November 2024 - service-discuss

Team Meeting Agenda for November 26, 2024
by Clark Boylan 26 Nov '24

26 Nov '24

We will meet with this agenda on November 26, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** US Holiday this week. * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** Raw image handling is being sped up through the use of zlib *** Can still use additional image build jobs for the other images we care about. ** Backup Server Pruning (clarkb 20241008) *** The smaller of the two backup servers is needing manual pruning multiple times a month now (the interval is between 2 and 3 weeks) *** There are older server backup dirs present on that server that we could delete (as well as removal of the corresponding login user for that server) **** ask01, ethercalc02, etherpad01, gitea01, lists, review-dev01, and review01 **** https://review.opendev.org/c/opendev/system-config/+/936203 retire these old unneeded backups in prep for eventual purging. **** We should run the manual prune script before we purge to ensure that script is still happy with the retirement process. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Docker Hub's New Rate Limits (clarkb 20241119) *** https://www.docker.com/blog/november-2024-updated-plans-announcement/ *** 10 pulls per IP per hour *** Disabling the use of the caching proxy server seems to have helped though we still hit rate limits. ** Docker compose plugin with podman service for servers (corvus 20241001) *** https://review.opendev.org/923084 is a demo (in the Zuul repo) of using docker compose v2 plugin with system podman service *** Looking at modifying the docker and docker compose installation process for Noble and newer to set up docker compose (no -) with podman on the backend. ** Gerrit 3.10 Upgrade Planning (clarkb 20241105) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.10 *** Feedback welcome on the process and listed concerns as well as the mitigations or assertions they don't affect us. *** Pencilled in 1600UTC December 6, 2024 for the upgrade to occur. ** Renaming mailman's openinfra.dev vhost to openinfra.org (fungi 20241126) *** Happening December 2, 2024 *** https://etherpad.opendev.org/p/lists-openinfra-org-migration ** Upgrading Gitea to 1.22.4 (clarkb 20241126) *** https://review.opendev.org/c/opendev/system-config/+/936198 * Open discussion

1 0

Team Meeting Agenda for November 19, 2024
by Clark Boylan 19 Nov '24

19 Nov '24

We will meet with this agenda on November 19, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Next week is a Holiday week for many in the US. Expect slower responses. * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** Need jobs to build additional images. Currently only x86 Debian Bullseye is being built ** Backup Server Pruning (clarkb 20241008) *** The smaller of the two backup servers is needing manual pruning multiple times a month now (the interval is between 2 and 3 weeks) *** There are older server backup dirs present on that server that we could delete (as well as removal of the corresponding login user for that server) **** ask01, ethercalc02, etherpad01, gitea01, lists, review-dev01, and review01 **** https://review.opendev.org/c/opendev/system-config/+/933700 Backup deletions managed through Ansible **** https://review.opendev.org/c/opendev/system-config/+/934768 Handle backup verification for purged backups ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Docker Hub's New Rate Limits (clarkb 20241119) *** https://www.docker.com/blog/november-2024-updated-plans-announcement/ *** 10 pulls per IP per hour *** https://access.redhat.com/solutions/6218921 Quay's rate limiting statement for comparison ** Docker compose plugin with podman service for servers (corvus 20241001) *** https://review.opendev.org/923084 is a demo (in the Zuul repo) of using docker compose v2 plugin with system podman service *** This would let us host opendev images on quay.io (or any non-dockerhub site) and use speculative images *** Some caveats (seen in the change): **** unconfined apparmor profile to work around https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2040483 **** buildx startup probably not an issue since we don't use docker compose image builds: https://github.com/docker/buildx/issues/344 **** need to set DOCKER_HOST env variable (system-wide bashrc? maybe a docker context? other options?) *** Do we want to proceed? Any blockers? **** Plan is to test the switch with a less consequential service (paste maybe). Upgrade the hosting server to Noble first if necessary. ** Enabling mailman3 bounce processing (frickler 20241030) *** This has been enabled on a number of lists manually. **** Did we update openstack-discuss yet? *** Need to configure new lists to turn this on by default. Likely using mailman3 list styles to enforce them by default. ** Intermediate Insecure CI Registry Pruning (clarkb 20241112) *** Pruning is still in progress. Estimated total runtime will be around 6 days. *** We found an additional bug (lack of swift pagination in the object listing performed by zuul-registry) that slightly delayed us *** https://review.opendev.org/c/opendev/system-config/+/935542 Enable daily pruning after this bulk prune is complete. ** Gerrit 3.10 Upgrade Planning (clarkb 20241105) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.10 *** Feedback welcome on the process and listed concerns as well as the mitigations or assertions they don't affect us. *** Pencilled in 1600UTC December 6, 2024 for the upgrade to occur. ** mirror.sjc3.raxflex.opendev.org cinder volume issues (clarkb 20241119) *** Another occurrence of sector 0 being unreadable on the device. *** Filesystems remounted RO and AFS is non functional *** Disabling cloud in nodepool for now while we sort out if we replace the device again. * Open discussion

1 0

Team Meeting Agenda for November 12, 2024
by Clark Boylan 12 Nov '24

12 Nov '24

We will meet on November 12, 2024 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** Need jobs to build additional images. Currently only x86 Debian Bullseye is being built ** Backup Server Pruning (clarkb 20241008) *** The smaller of the two backup servers is needing manual pruning multiple times a month now (the interval is between 2 and 3 weeks) *** There are older server backup dirs present on that server that we could delete (as well as removal of the corresponding login user for that server) **** ask01, ethercalc02, etherpad01, gitea01, lists, review-dev01, and review01 **** https://review.opendev.org/c/opendev/system-config/+/933700 Backup deletions managed through Ansible **** https://review.opendev.org/c/opendev/system-config/+/934768 Handle backup verification for purged backups ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Docker compose plugin with podman service for servers (corvus 20241001) *** https://review.opendev.org/923084 is a demo (in the Zuul repo) of using docker compose v2 plugin with system podman service *** This would let us host opendev images on quay.io (or any non-dockerhub site) and use speculative images *** Some caveats (seen in the change): **** unconfined apparmor profile to work around https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2040483 **** buildx startup probably not an issue since we don't use docker compose image builds: https://github.com/docker/buildx/issues/344 **** need to set DOCKER_HOST env variable (system-wide bashrc? maybe a docker context? other options?) *** Do we want to proceed? Any blockers? **** Plan is to test the switch with a less consequential service (paste maybe). Upgrade the hosting server to Noble first if necessary. ** Enabling mailman3 bounce processing (frickler 20241030) *** This would help getting rid of subscriptions that are no longer valid *** Example configuration page: https://lists.openstack.org/mailman3/lists/openstack-discuss.lists.openstac… (needs admin access) *** Enabled on service-discuss without changing any defaults. Next step is to enable elsewhere once we are happy service-discuss isn't negatively impacted. ** Intermediate Insecure CI Registry Pruning (clarkb 20241112) *** Backing swift container hasn't been pruned in many years *** Recent bugfix as well as addition to dry run pruning have landed in zuul-registry's prune command *** Announced date for Friday pruning. Do we want to start a dry run sooner (tomorrow?) ** Gerrit 3.10 Upgrade Planning (clarkb 20241105) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.10 *** Feedback welcome on the process and listed concerns as well as the mitigations or assertions they don't affect us. *** Pencilled in 1600UTC December 6, 2024 for the upgrade to occur. ** RTD Build Trigger Requests (clarkb 20241005) *** RTD build requests are failing but when manually reproduced seem to work. Appears to be some problem specific to the Ansible uri module. *** New theory is that it could be client fingerprinting by the CDN. *** https://review.opendev.org/c/zuul/zuul-jobs/+/934243 switch to curl instead of ansible uri module ** promote-openstack-manuals-developer fails with Ansible errors (clarkb 20241112) *** promote-openstack-manuals-developer job promotes developer documentation that live in a different location from the typical openstack docs *** https://zuul.opendev.org/t/openstack/build/1a84db5d173c4777b9d730923721b04a Example failure * Open discussion

1 0

Team Meeting Agenda for November 5, 2024
by Clark Boylan 05 Nov '24

05 Nov '24

We will meet on November 5, 2024 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** Need jobs to build additional images. Currently only x86 Debian Bullseye is being built ** Backup Server Pruning (clarkb 20241008) *** The smaller of the two backup servers is needing manual pruning multiple times a month now (the interval is between 2 and 3 weeks) *** There are older server backup dirs present on that server that we could delete (as well as removal of the corresponding login user for that server) **** ask01, ethercalc02, etherpad01, gitea01, lists, review-dev01, and review01 **** https://review.opendev.org/c/opendev/system-config/+/933354 Documentation of cleanup process applied to ethercalc02 backups. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Docker compose plugin with podman service for servers (corvus 20241001) *** https://review.opendev.org/923084 is a demo (in the Zuul repo) of using docker compose v2 plugin with system podman service *** This would let us host opendev images on quay.io (or any non-dockerhub site) and use speculative images *** Some caveats (seen in the change): **** unconfined apparmor profile to work around https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2040483 **** buildx startup probably not an issue since we don't use docker compose image builds: https://github.com/docker/buildx/issues/344 **** need to set DOCKER_HOST env variable (system-wide bashrc? maybe a docker context? other options?) *** Do we want to proceed? Any blockers? **** Plan is to test the switch with a less consequential service (paste maybe). Upgrade the hosting server to Noble first if necessary. ** Enabling mailman3 bounce processing (frickler 20241030) *** This would help getting rid of subscriptions that are no longer valid *** Example configuration page: https://lists.openstack.org/mailman3/lists/openstack-discuss.lists.openstac… (needs admin access) ** Failures of insecure registry (frickler 20241030) *** Symptoms were build jobs ending in timeout while pushing tags *** See e.g. https://zuul.opendev.org/t/openstack/builds?job_name=osc-build-image&result… *** We upgraded cheroot (the http server) as well as container contents and things seem to be more stable for now. *** We should plan cleanup of some sort for the backing store. Either test the prune command or swap to a new container and cleanup the old one. ** Gerrit 3.10 Upgrade Planning (clarkb 20241105) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.10 *** Pencilled in 1600UTC December 6, 2024 for the upgrade to occur. ** RTD Build Trigger Requests (clarkb 20241005) *** RTD build requests are failing but when manually reproduced seem to work. Appears to be some problem specific to the Ansible uri module. * Open discussion

1 0