- service-discuss - lists.opendev.org

Team Meeting Agenda for October 11, 2022
by Clark Boylan 10 Oct '22

10 Oct '22

We will meet with this agenda on October 11, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** PTG next week * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221011) *** Move ansible to a venv. *** Replace bridge host once existing host is running ansible out of a venv *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? *** https://review.opendev.org/c/opendev/system-config/+/855472 Disable writing console log files on bridge and static **** Needs a second change pushed to opendev/base-jobs/playbooks/infra-prod/setup-keys.yaml *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221011) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Need to retest launching a Jammy node now that user management is updated. ** Mailman 3 (clarkb 20221011) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** https://review.opendev.org/c/opendev/system-config/+/860157 Forking the upstream images due to lack of attention on issues and PRs we've filed ** Switch base job nodeset to ubuntu-jammy (frickler 20221007) *** Ubuntu 22.04.1 is out for some months now and thus can be considered stable, no need to wait a full year like we did with 20.04 *** Patch to switch base-test job: https://review.opendev.org/c/opendev/base-jobs/+/860686/2 *** Plan for further steps? Timetable, which repos to test, announcements *** Devstack currently inherits the base nodeset (via multinode) without pinning * Open discussion

1 0

Team Meeting Agenda for October 4, 2022
by Clark Boylan 03 Oct '22

03 Oct '22

We will meet on October 4, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenStack Release this week. * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221004) *** Move ansible to a venv. *** Replace bridge host once existing host is running ansible out of a venv *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? *** https://review.opendev.org/c/opendev/system-config/+/855472 *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221004) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/c/opendev/system-config/+/860112 Update to launch node to handle jammy hosts ** Mailman 3 (clarkb 20221004) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** https://review.opendev.org/c/opendev/system-config/+/860157 Forking the upstream images due to lack of attention on issues and PRs we've filed ** Gitea connectivity issues (clarkb 20221004) *** Users in certain geographies have reported trouble with Gitea access *** Gitea changed http libraries which broke our ability to trace requests from haproxy -> apache -> gitea **** Partially worked around that by removing X-Forwarded-For headers. * Open discussion

1 0

Team Meeting Agenda for September 27, 2022
by Clark Boylan 26 Sep '22

26 Sep '22

We will meet with this agenda on September 27, 2022 at 19:00 UTC in #opendev-meeting. This agenda is a slightly edited version compared to the wiki as I anticipate we'll have less participation this week. == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Mailman 3 (clarkb 20220927) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** Remaining issue with migration is size of some fields in the db getting exceeded. Can just update old list side. *** Pipermail redirects need testing on held node. ** Jaeger tracing server (for Zuul) (corvus 20220927) *** Change has landed. ** Nodepool Builder Disk utilization (clarkb 20220927) *** Have we expanded disk space on existing builders yet? * Open discussion

1 0

Team Meeting Agenda for September 20, 2022
by Clark Boylan 19 Sep '22

19 Sep '22

We will meet with this agenda on September 20, 2022 in #opendev-meeting at 19:00 UTC: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20220920) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? *** https://review.opendev.org/c/zuul/zuul/+/855309/ *** https://review.opendev.org/c/opendev/system-config/+/855472 ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220920) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220920) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** Remaining issue with migration is size of some fields in the db getting exceeded. Can just update old list side. *** Pipermail redirects need testing on held node. ** Jaeger tracing server (for Zuul) (corvus 20220920) *** Change has landed. ** Fedora 36 Rollout (clarkb 20220920) *** Now not booting reliably. ** Improving Ansible task runtime (clarkb 20220920) *** Ansible tasks can be slow on some test nodes. *** https://review.opendev.org/c/opendev/system-config/+/857232 *** Possibly related to Ansible 5 breaking pipelining in ssh connection specific config *** https://review.opendev.org/c/opendev/system-config/+/857239 ** Nodepool Builder Disk utilization (clarkb 20220920) *** We're using more of the disk now which makes us more prone to filling the disk if something goes wrong. *** Can we trim any images out? *** Do we need more or bigger builders? * Open discussion

1 0

Updates to meetpad's Jitsi Meet configuration
by Jeremy Stanley 19 Sep '22

19 Sep '22

Last week, we refreshed our configuration defaults for the Jitsi Meet services which provide the teleconferencing features of our meetpad.opendev.org service. The primary behavior change you're likely to notice is that meeting room URLs first take you to a pre-join page now. This is a workaround for some browsers, particularly Firefox, treating the audio stream as unsolicited and automatically blocking it. Previously, that was responsible for causing confusing "I can't hear anyone else" problems for many users. If you had trouble with sound on our meetpad service before, please do try it again at your convenience and let us know if it helped (or more importantly, if it didn't). -- Jeremy Stanley

1 1

Team Meeting Agenda for September 13, 2022
by Clark Boylan 12 Sep '22

12 Sep '22

Hello, We will meet with this agenda on September 13, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20220913) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? *** https://review.opendev.org/c/zuul/zuul/+/855309/ *** https://review.opendev.org/c/opendev/system-config/+/855472 ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220913) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220913) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** Bulk migration identified a few things that have since been updated. Need another pass to check. *** Pipermail redirects need testing on held node. ** Jaeger tracing server (for Zuul) (corvus 20220913) *** https://review.opendev.org/c/opendev/system-config/+/855983 Adds ansible and docker deployment for Jaeger ** Fedora 36 Rollout (clarkb 20220913) *** https://review.opendev.org/c/zuul/nodepool/+/853914 Remove fedora 35 testing from nodepool ** Jitsi Meet Updates for Meetpad (clarkb 20220913) *** https://review.opendev.org/c/opendev/system-config/+/856553 Update to use colibri websockets and scale out JVBs *** Not sure if the java keystore in use by the JVBs need to share a CA or otherwise be trusted by something else. ** Stability of Zuul reboot playbook (clarkb 20220913) *** New instability identified caused by race between container shutdown and docker exec for graceful command *** https://review.opendev.org/c/opendev/system-config/+/857209 Fix for this race ** Python container image updates (clarkb 20220913) *** https://review.opendev.org/c/opendev/system-config/+/856537 *** Updates to latest python point releases. *** This pulls in newer glibc allowing us to remove the backport in Zuul images ** Improving Ansible task runtime (clarkb 20220913) *** Ansible tasks can be slow on some test nodes. *** Particularly problematic when loops are used that serially process a large set of items *** https://review.opendev.org/c/zuul/zuul-jobs/+/857228 *** https://review.opendev.org/c/opendev/system-config/+/857232 *** Be on the lookout for problematic tasks and updates to speed them up. * Open discussion

1 0

Team Meeting Agenda for September 6, 2022
by Clark Boylan 06 Sep '22

06 Sep '22

Hello, This agenda is getting sent out a bit late due to the holiday yesterday. We will meet at 19:00 UTC in #opendev-meeting with this agenda today, September 6, 2022: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20220906) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220906) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220906) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** Migration testing looking good. Need to test updated database settings for large attachments (including that db backups function) and redirects for old pipermail archives. ** Jaeger tracing server (for Zuul) (corvus 20220906) *** Any changes need review yet? ** Fedora 36 Rollout (clarkb 20220906) *** Status Update ** Jitsi Meet Updates for Meetpad (clarkb 20220906) *** Configs updated to start with landing page for meetings. Should fix Firefox auto mute of auto playing audio *** Discovered that JVBs rely on newer colibri websocket system for scaling out. Our extra JVBs need to be updated to function under this system. ** Stability of Zuul reboot playbook (clarkb 20220906) *** Crashes if unattended-upgrades is running due to conflict over apt lock. *** Potentially crashes if services were already stopped outside of the playbook run. * Open discussion

1 0

Team Meeting Agenda for August 30, 2022
by Clark Boylan 29 Aug '22

29 Aug '22

We will meet with this agenda on August 30, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack feature freeze this week. * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20220830) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220830) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220830) *** https://review.opendev.org/c/opendev/system-config/+/851248 Still very much WIP but worth a look at this point *** Working through migration testing now. ** Gerrit 08:00 UTC load issues (clarkb 20220830) *** Bumping up the http thread count *** Fallout of blocking IPs (users noticed) *** We should work through this slowly as the OpenStack release is approaching. ** Jaeger tracing server (for Zuul) (corvus 20220830) *** Zuul will be adding OpenTelemetry tracing support. Should OpenDev run a Jaeger tracing server? *** See https://zuul-ci.org/docs/zuul/latest/developer/specs/tracing.html ** Fedora 36 Rollout (clarkb 20220830) *** Do we need to wait for OpenStack to release first? *** We'll update the nodeset and most jobs will switch automatically. *** Will we remove Fedora 35 next? * Open discussion

1 0

Team Meeting Agenda for August 23, 2022
by Clark Boylan 22 Aug '22

22 Aug '22

Hello, We will meet with this agenda on August 23, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Clark appears to have been the only Service Coordinator nominee. ** OpenStack release is fast approaching. https://releases.openstack.org/zed/schedule.html feature freeze September 2 ish. * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20220823) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220823) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220823) *** https://review.opendev.org/c/opendev/system-config/+/851248 Still very much WIP but worth a look at this point *** The test deployment is largely there now. The next step is testing data migration from mm2 to mm3. ** Gitea 1.17 upgrade (clarkb 20220823) *** https://review.opendev.org/c/opendev/system-config/+/847204 *** This should be ready to go now that 1.17.1 has been released. Just need to decide on a good window of time to deploy it. ** Gerrit 08:00 UTC load issues (clarkb 20220823) *** Bumping up the http thread count *** Fallout of blocking IPs (users noticed) *** We should work through this slowly as the OpenStack release is approaching. ** Jaeger tracing server (for Zuul) (corvus 20220823) *** Zuul will be adding OpenTelemetry tracing support. Should OpenDev run a Jaeger tracing server? *** See https://zuul-ci.org/docs/zuul/latest/developer/specs/tracing.html *** Implementation should be straightforward; does this warrant an OpenDev spec, or should I just propose the change? * Open discussion

1 0

Self Nomination for OpenDev Service Coordinator
by Clark Boylan 17 Aug '22

17 Aug '22

Hello, I am submitting my nomination for OpenDev Service Coordinator. I plan to continue to focus on building sustainable services with the current resources we have. Upgrading servers to Jammy and replacing old services like mailman 2 are high on my todo list. I have been doing this for a while and don't have a lot of new things to say. I think it is important that these community services/resources continue to be accessible to the community, and that requires ensuring they are sustainable. If you have been happy with the last six months then you can expect more of the same. I am still happy for someone else to volunteer as well. You have a few more hours if anyone else would like to give it a go. I think having a group that can share these responsibilities is a good thing. But I am happy to continue as well. Clark

2 1