September 2024 - service-discuss

Team Meeting Agenda for October 1, 2024
by Clark Boylan 01 Oct '24

01 Oct '24

We will meet with this agenda on October 1, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** https://www.socallinuxexpo.org/scale/22x/events/open-infra-days CFP for Open Infra Days event at SCaLE is open until November 1 * Actions from last meeting * Specs Review * Topics ** OpenStack Release Wednesday (clarkb 20241001) *** OpenStack will be publishing its next release on Wednesday from approximately 1000-1500 UTC. *** https://review.opendev.org/c/openstack/project-config/+/930709 will be landed prior to the release to remove a semaphore that slows things down. ** Rocky Package Mirror Creation (clarkb 20240924) *** No rsync script change pushed yet but one is expected soon. ** Rackspace's Flex Cloud (clarkb 20240604) *** Next up is figuring out how to use this cloud for swift storage of CI job logs. ** Updating ansible+ansible-lint versions in our repos (clarkb 20240827) *** https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/926970 is the last current open change related to this work ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** https://review.opendev.org/c/opendev/zuul-jobs/+/929141 Build a debian bullseye image with dib in a zuul job ** OpenStack OpenAPI spec publishing (fungi 20240924) *** https://review.opendev.org/921934 ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** Docker compose plugin with podman service for servers (corvus 20241001) *** https://review.opendev.org/923084 is a demo (in the Zuul repo) of using docker compose v2 plugin with system podman service *** This would let us host opendev images on quay.io (or any non-dockerhub site) and use speculative images *** Some caveats (seen in the change): **** unconfined apparmor profile to work around https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2040483 **** buildx startup probably not an issue since we don't use docker compose image builds: https://github.com/docker/buildx/issues/344 **** need to set DOCKER_HOST env variable (system-wide bashrc? maybe a docker context? other options?) *** Do we want to proceed? Any blockers? * Open discussion

1 0

Team Meeting Agenda for September 24, 2024
by Clark Boylan 24 Sep '24

24 Sep '24

We will meet with this agenda on September 24, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** https://www.socallinuxexpo.org/scale/22x/events/open-infra-days CFP for Open Infra Days event at SCaLE is open until November 1 * Actions from last meeting * Specs Review * Topics ** Rocky Package Mirror Creation (clarkb 20240924) *** No rsync script change pushed yet but one is expected soon. ** Rackspace's Flex Cloud (clarkb 20240604) *** Next up is figuring out how to use this cloud for swift storage of CI job logs. ** Implementing DNS over TLS on Test Nodes (clarkb 20240924) *** OVN MitMs DNS traffic made in the clear this has created problems for some users **** https://serverfault.com/questions/1134180/how-to-disable-or-fix-openstack-i… *** We should be able to mitigate this using DNS over TLS now that some clouds appear to be using OVN *** https://review.opendev.org/c/opendev/base-jobs/+/929960 Adds this to configure-unbound role in base-jobs *** If that approach seems reasonable clarkb can split things up to go through a full round of base-test base job testing ** Etherpad 2.2.5 Upgrade (clarkb 20240820) *** Just wanted to close this out. We upgraded to a proper 2.2.5 release and meetpad seems to work. *** There is a new v2 API we can look into using but our tests indicate v1 still works for now. ** Updating ansible+ansible-lint versions in our repos (clarkb 20240827) *** https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/926970 is the last current open change related to this work ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** https://review.opendev.org/c/opendev/zuul-jobs/+/929141 Build a debian bullseye image with dib in a zuul job ** OpenStack OpenAPI spec publishing (fungi 20240924) *** https://review.opendev.org/921934 ** Gitea 1.22.2 Upgrade (clarkb 20240924) *** https://review.opendev.org/c/opendev/system-config/+/930217 Change to upgrade to 1.22.2 ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades * Open discussion

1 0

Team Meeting Agenda for September 17, 2024
by Clark Boylan 17 Sep '24

17 Sep '24

We will meet on September 17, 2024 at 19:00 UTC with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** CentOS 8 Stream EOLd and jobs can no longer successfully run there. Cleanup is happening under topic:drop-centos-8-stream *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Rackspace's Flex Cloud (clarkb 20240604) *** Continue to keep an eye out for networking symptoms similar to slow ssh hostkey scanning *** This is a floating IP cloud and jobs may need to adjust their use of IP addresses *** We've added this cloud to the nested virt labels (as well as openmetal) *** Next up I think we can figure out how to use swift in this cloud for job logs ** Etherpad 2.2.4 Upgrade (clarkb 20240820) *** We upgraded but that broken the meetpad integration. *** Upstream had already fixed the issue but hadn't released it yet. We have deployed a development version to address the problem. *** Ideally we'll get a 2.2.5 release soon that we can quickly update to. ** Updating ansible+ansible-lint versions in our repos (clarkb 20240827) *** After updating our default nodeset to Noble we discovered that ansible-lint with all of our pins cannot run on Noble due to incompatibilities with python 3.12. *** Updating to latest ansible-lint and the current version of Zuul's Ansible install (>=8,<9) does work but introduces new rules and lint errors *** https://review.opendev.org/c/openstack/project-config/+/926848 *** https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/926970 ** Zuul-launcher image builds (corvus 20240910) *** Configs are going in opendev/zuul-jobs *** https://review.opendev.org/c/opendev/zuul-jobs/+/929141 Build a debian bullseye image with dib in a zuul job * Open discussion

1 0

Team Meeting Agenda for September 10, 2024
by Clark Boylan 10 Sep '24

10 Sep '24

We will meet with this agenda on September 10, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** CentOS 8 Stream EOLd and jobs can no longer successfully run there. Cleanup is happening under topic:drop-centos-8-stream *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Rackspace's Flex Cloud (clarkb 20240604) *** This cloud is now in use with max-servers set to 32. *** We've seen issues scanning ssh hostkeys which may be related to networking *** At least one job discovered the nodes use floating IPs that hard way trying to bind to the fip address. ** Etherpad 2.2.4 Upgrade (clarkb 20240820) *** Etherpad 2.2.2 has been released which breaks compatibility with our ep_headings plugin. **** ep_headings2 appears to work though *** https://review.opendev.org/c/opendev/system-config/+/926078 ** OSUOSL ARM Cloud Issues (clarkb 20240827) *** Kolla has discovered that jobs take significantly more time on OSUOSL nodes than they did on linaro and seem to fail more than they succeed. Other projects are having fewer issues. **** IO throughput seems to be the main culprit. OSUOSL has been improving the nodes and now kolla-build-debian-aarch64 builds succeed occasionally (not not consistently). ** Updating ansible+ansible-lint versions in our repos (clarkb 20240827) *** After updating our default nodeset to Noble we discovered that ansible-lint with all of our pins cannot run on Noble due to incompatibilities with python 3.12. *** Updating to latest ansible-lint and the current version of Zuul's Ansible install (>=8,<9) does work but introduces new rules and lint errors *** https://review.opendev.org/c/openstack/project-config/+/926848 *** https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/926970 ** Zuul-launcher image builds (corvus 20240910) *** DIB roles/job landed in zuul-jobs *** In what repo should we define our central image build jobs? * Open discussion

1 0