- service-discuss - lists.opendev.org

Team Meeting Agenda for November 12, 2024
by Clark Boylan 12 Nov '24

12 Nov '24

We will meet on November 12, 2024 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** Need jobs to build additional images. Currently only x86 Debian Bullseye is being built ** Backup Server Pruning (clarkb 20241008) *** The smaller of the two backup servers is needing manual pruning multiple times a … [View More]month now (the interval is between 2 and 3 weeks) *** There are older server backup dirs present on that server that we could delete (as well as removal of the corresponding login user for that server) **** ask01, ethercalc02, etherpad01, gitea01, lists, review-dev01, and review01 **** https://review.opendev.org/c/opendev/system-config/+/933700 Backup deletions managed through Ansible **** https://review.opendev.org/c/opendev/system-config/+/934768 Handle backup verification for purged backups ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Docker compose plugin with podman service for servers (corvus 20241001) *** https://review.opendev.org/923084 is a demo (in the Zuul repo) of using docker compose v2 plugin with system podman service *** This would let us host opendev images on quay.io (or any non-dockerhub site) and use speculative images *** Some caveats (seen in the change): **** unconfined apparmor profile to work around https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2040483 **** buildx startup probably not an issue since we don't use docker compose image builds: https://github.com/docker/buildx/issues/344 **** need to set DOCKER_HOST env variable (system-wide bashrc? maybe a docker context? other options?) *** Do we want to proceed? Any blockers? **** Plan is to test the switch with a less consequential service (paste maybe). Upgrade the hosting server to Noble first if necessary. ** Enabling mailman3 bounce processing (frickler 20241030) *** This would help getting rid of subscriptions that are no longer valid *** Example configuration page: https://lists.openstack.org/mailman3/lists/openstack-discuss.lists.openstac… (needs admin access) *** Enabled on service-discuss without changing any defaults. Next step is to enable elsewhere once we are happy service-discuss isn't negatively impacted. ** Intermediate Insecure CI Registry Pruning (clarkb 20241112) *** Backing swift container hasn't been pruned in many years *** Recent bugfix as well as addition to dry run pruning have landed in zuul-registry's prune command *** Announced date for Friday pruning. Do we want to start a dry run sooner (tomorrow?) ** Gerrit 3.10 Upgrade Planning (clarkb 20241105) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.10 *** Feedback welcome on the process and listed concerns as well as the mitigations or assertions they don't affect us. *** Pencilled in 1600UTC December 6, 2024 for the upgrade to occur. ** RTD Build Trigger Requests (clarkb 20241005) *** RTD build requests are failing but when manually reproduced seem to work. Appears to be some problem specific to the Ansible uri module. *** New theory is that it could be client fingerprinting by the CDN. *** https://review.opendev.org/c/zuul/zuul-jobs/+/934243 switch to curl instead of ansible uri module ** promote-openstack-manuals-developer fails with Ansible errors (clarkb 20241112) *** promote-openstack-manuals-developer job promotes developer documentation that live in a different location from the typical openstack docs *** https://zuul.opendev.org/t/openstack/build/1a84db5d173c4777b9d730923721b04a Example failure * Open discussion [View Less]

1 0

Team Meeting Agenda for November 5, 2024
by Clark Boylan 05 Nov '24

05 Nov '24

We will meet on November 5, 2024 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** Need jobs to build additional images. Currently only x86 Debian Bullseye is being built ** Backup Server Pruning (clarkb 20241008) *** The smaller of the two backup servers is needing manual pruning multiple times a … [View More]month now (the interval is between 2 and 3 weeks) *** There are older server backup dirs present on that server that we could delete (as well as removal of the corresponding login user for that server) **** ask01, ethercalc02, etherpad01, gitea01, lists, review-dev01, and review01 **** https://review.opendev.org/c/opendev/system-config/+/933354 Documentation of cleanup process applied to ethercalc02 backups. ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Docker compose plugin with podman service for servers (corvus 20241001) *** https://review.opendev.org/923084 is a demo (in the Zuul repo) of using docker compose v2 plugin with system podman service *** This would let us host opendev images on quay.io (or any non-dockerhub site) and use speculative images *** Some caveats (seen in the change): **** unconfined apparmor profile to work around https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2040483 **** buildx startup probably not an issue since we don't use docker compose image builds: https://github.com/docker/buildx/issues/344 **** need to set DOCKER_HOST env variable (system-wide bashrc? maybe a docker context? other options?) *** Do we want to proceed? Any blockers? **** Plan is to test the switch with a less consequential service (paste maybe). Upgrade the hosting server to Noble first if necessary. ** Enabling mailman3 bounce processing (frickler 20241030) *** This would help getting rid of subscriptions that are no longer valid *** Example configuration page: https://lists.openstack.org/mailman3/lists/openstack-discuss.lists.openstac… (needs admin access) ** Failures of insecure registry (frickler 20241030) *** Symptoms were build jobs ending in timeout while pushing tags *** See e.g. https://zuul.opendev.org/t/openstack/builds?job_name=osc-build-image&result… *** We upgraded cheroot (the http server) as well as container contents and things seem to be more stable for now. *** We should plan cleanup of some sort for the backing store. Either test the prune command or swap to a new container and cleanup the old one. ** Gerrit 3.10 Upgrade Planning (clarkb 20241105) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.10 *** Pencilled in 1600UTC December 6, 2024 for the upgrade to occur. ** RTD Build Trigger Requests (clarkb 20241005) *** RTD build requests are failing but when manually reproduced seem to work. Appears to be some problem specific to the Ansible uri module. * Open discussion [View Less]

1 0

Team Meeting Agenda for October 29, 2024
by Clark Boylan 29 Oct '24

29 Oct '24

We will meet with this agenda on October 29, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** https://www.socallinuxexpo.org/scale/22x/events/open-infra-days CFP for Open Infra Days event at SCaLE is open until November 1 * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** Need jobs to build additional images. Currently only x86 Debian Bullseye is being … [View More]built ** Backup Server Pruning (clarkb 20241008) *** The smaller of the two backup servers is needing manual pruning multiple times a month now (the interval is between 2 and 3 weeks) *** There are older server backup dirs present on that server that we could delete (as well as removal of the corresponding login user for that server) **** ask01, ethercalc02, etherpad01, gitea01, lists, review-dev01, and review01 **** https://review.opendev.org/c/opendev/system-config/+/933354 Documentation of cleanup process applied to ethercalc02 backups. ** Updating Gerrit Cache Sizes (clarkb 20241022) *** Noticed Gerrit pruning its caches on startup after a recent upgrade. Some of the sizes were quite large compared to their configured maximum *** https://review.opendev.org/c/opendev/system-config/+/932763 increase sizes for four gerrit caches ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Docker compose plugin with podman service for servers (corvus 20241001) *** https://review.opendev.org/923084 is a demo (in the Zuul repo) of using docker compose v2 plugin with system podman service *** This would let us host opendev images on quay.io (or any non-dockerhub site) and use speculative images *** Some caveats (seen in the change): **** unconfined apparmor profile to work around https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2040483 **** buildx startup probably not an issue since we don't use docker compose image builds: https://github.com/docker/buildx/issues/344 **** need to set DOCKER_HOST env variable (system-wide bashrc? maybe a docker context? other options?) *** Do we want to proceed? Any blockers? **** Plan is to test the switch with a less consequential service (paste maybe). Upgrade the hosting server to Noble first if necessary. ** Etherpad 2.2.6 Upgrade (clarkb 20241029) *** https://review.opendev.org/c/opendev/system-config/+/933618 Upgrade Etherpad to version 2.2.6 * Open discussion [View Less]

1 0

Team Meeting Agenda for October 22, 2024
by Clark Boylan 22 Oct '24

22 Oct '24

We will meet with this agenda on October 22, 2024 at 19:00 UTC in #opendev-meeting. I know that this is a busy week for many with the PTG, but we missed our meeting last week. I want to make sure we've got a bit of time to catch up on anything important that we may be ignoring due to distractions like the PTG. == Agenda for next meeting == * Announcements ** https://www.socallinuxexpo.org/scale/22x/events/open-infra-days CFP for Open Infra Days event at SCaLE is open until November 1 ** PTG … [View More]is happening this week. * Actions from last meeting * Specs Review * Topics ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** https://review.opendev.org/c/opendev/zuul-jobs/+/929141 Build a debian bullseye image with dib in a zuul job ** OpenStack OpenAPI spec publishing (fungi 20240924) *** https://review.opendev.org/921934 ** Backup Server Pruning (clarkb 20241008) *** The smaller of the two backup servers is needing manual pruning multiple times a month now (the interval is between 2 and 3 weeks) *** There are older server backup dirs present on that server that we could delete (as well as removal of the corresponding login user for that server) **** ask01, ethercalc02, etherpad01, gitea01, lists, review-dev01, and review01 *** An alternative would be to attach a new empty volume and shuffle mounts around. Then let new backups fill the new space and leave old backups off to the side. ** Updating Gerrit Cache Sizes (clarkb 20241022) *** Noticed Gerrit pruning its caches on startup after a recent upgrade. Some of the sizes were quite large compared to their configured maximum *** https://review.opendev.org/c/opendev/system-config/+/932763 increase sizes for four gerrit caches ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Docker compose plugin with podman service for servers (corvus 20241001) *** https://review.opendev.org/923084 is a demo (in the Zuul repo) of using docker compose v2 plugin with system podman service *** This would let us host opendev images on quay.io (or any non-dockerhub site) and use speculative images *** Some caveats (seen in the change): **** unconfined apparmor profile to work around https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2040483 **** buildx startup probably not an issue since we don't use docker compose image builds: https://github.com/docker/buildx/issues/344 **** need to set DOCKER_HOST env variable (system-wide bashrc? maybe a docker context? other options?) *** Do we want to proceed? Any blockers? **** Plan is to test the switch with a less consequential service (paste maybe). Upgrade the hosting server to Noble first if necessary. * Open discussion [View Less]

1 0

Cancelling the October 15, 2024 OpenDev Team Meeting
by Clark Boylan 09 Oct '24

09 Oct '24

We are cancelling next week's team meeting that would normally occur on October 15, 2024. A number of us will be unavailable that day. It doesn't make sense to try and organize a meeting for one or two people. As usual feel free to bring up any items for discussion on this mailing list or on IRC in the #opendev channel. We can discuss things, as able, outside of the meeting via these channels. Clark

1 0

Team Meeting Agenda for October 8, 2024
by Clark Boylan 08 Oct '24

08 Oct '24

== Agenda for next meeting == * Announcements ** https://www.socallinuxexpo.org/scale/22x/events/open-infra-days CFP for Open Infra Days event at SCaLE is open until November 1 * Actions from last meeting * Specs Review * Topics ** Switching OpenDev Zuul Tenants to Ansible 9 by Default (clarkb 20241008) *** Some tenants (opendev and zuul) have already switched. There were some minor issues that needed to be addressed *** Testing with devstack + tempest looks good: https://review.opendev.org/… [View More]c/openstack/tempest/+/929684 *** https://review.opendev.org/c/openstack/project-config/+/931320 switches the remaining tenants (including openstack) to Ansible 9 by default. ** Rocky Package Mirror Creation (clarkb 20240924) *** No rsync script change pushed yet but one is expected soon. ** Rackspace's Flex Cloud (clarkb 20240604) *** Application credentials and general container/object creation works. *** Bug in openstack ansible modules prevents multipart uploads from reporting success within ansible (though it does upload behind the scenes) **** https://review.opendev.org/c/openstack/ansible-collections-openstack/+/9265… ** Updating ansible+ansible-lint versions in our repos (clarkb 20240827) *** https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/926970 is the last current open change related to this work ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** https://review.opendev.org/c/opendev/zuul-jobs/+/929141 Build a debian bullseye image with dib in a zuul job ** OpenStack OpenAPI spec publishing (fungi 20240924) *** https://review.opendev.org/921934 ** Backup Server Pruning (clarkb 20241008) *** The smaller of the two backup servers is needing manual pruning multiple times a month now (the interval is between 2 and 3 weeks) *** There are older server backup dirs present on that server that we could delete (as well as removal of the corresponding login user for that server) **** ask01, ethercalc02, etherpad01, gitea01, lists, review-dev01, and review01 *** An alternative would be to attach a new empty volume and shuffle mounts around. Then let new backups fill the new space and leave old backups off to the side. ** Mailman 3 Upgrade (clarkb 20241008) *** https://review.opendev.org/c/opendev/system-config/+/930236 will upgrade our installation of mailman 3 to the latest versions *** The containers themselves should handle required db migrations on startup ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Docker compose plugin with podman service for servers (corvus 20241001) *** https://review.opendev.org/923084 is a demo (in the Zuul repo) of using docker compose v2 plugin with system podman service *** This would let us host opendev images on quay.io (or any non-dockerhub site) and use speculative images *** Some caveats (seen in the change): **** unconfined apparmor profile to work around https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2040483 **** buildx startup probably not an issue since we don't use docker compose image builds: https://github.com/docker/buildx/issues/344 **** need to set DOCKER_HOST env variable (system-wide bashrc? maybe a docker context? other options?) *** Do we want to proceed? Any blockers? **** Plan is to test the switch with a less consequential service (paste maybe). Upgrade the hosting server to Noble first if necessary. * Open discussion [View Less]

1 0

Team Meeting Agenda for October 1, 2024
by Clark Boylan 01 Oct '24

01 Oct '24

We will meet with this agenda on October 1, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** https://www.socallinuxexpo.org/scale/22x/events/open-infra-days CFP for Open Infra Days event at SCaLE is open until November 1 * Actions from last meeting * Specs Review * Topics ** OpenStack Release Wednesday (clarkb 20241001) *** OpenStack will be publishing its next release on Wednesday from approximately 1000-1500 UTC. *** https://review.opendev.org/c/… [View More]

1 0

Team Meeting Agenda for September 24, 2024
by Clark Boylan 24 Sep '24

24 Sep '24

We will meet with this agenda on September 24, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** https://www.socallinuxexpo.org/scale/22x/events/open-infra-days CFP for Open Infra Days event at SCaLE is open until November 1 * Actions from last meeting * Specs Review * Topics ** Rocky Package Mirror Creation (clarkb 20240924) *** No rsync script change pushed yet but one is expected soon. ** Rackspace's Flex Cloud (clarkb 20240604) *** Next up is … [View More]

1 0

Team Meeting Agenda for September 17, 2024
by Clark Boylan 17 Sep '24

17 Sep '24

We will meet on September 17, 2024 at 19:00 UTC with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** AFS Mirror cleanups (clarkb … [View More]

1 0

Team Meeting Agenda for September 10, 2024
by Clark Boylan 10 Sep '24

10 Sep '24

We will meet with this agenda on September 10, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** AFS Mirror … [View More]

1 0