- service-discuss - lists.opendev.org

Team Meeting Agenda for June 25, 2024
by Clark Boylan 25 Jun '24

25 Jun '24

We will meet with this agenda on June 25, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades **** tonyb expects to try a simple focal -> noble upgrade this week ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** CentOS 8 Stream EOLd and jobs can no longer successfully run there. Cleanup is happening under topic:drop-centos-8-stream *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Gitea 1.22 Upgrade Planning (clarkb 20240528) *** There is a Gitea 1.22.0 release now. Once we have the general upgrade working we can test the doctor tool to fixup the DB case sensitivity. *** https://review.opendev.org/c/opendev/system-config/+/920580 ** Improving mailman email throughput (clarkb 20240604) *** We've noticed slowness delivering emails for openstack-discuss *** Increasing the number of out runners hasn't alleviated this issue (it should prevent openstack-discuss from holding up other lists though) *** Next we need to modify the mailman email batch count and exim verp queue sizes. ** OpenMetal Cloud Rebuild (clarkb 20240604) *** Things were working fairly well with max-servers 50 until they weren't *** Looks like problems may be related to available disk space and how ceph is used by various services (not enough are using ceph) *** Also need to look into increasing the number of placement groups per OSD as we're far below the recommendation of 100 PGs per OSD **** https://docs.ceph.com/en/latest/dev/placement-group/ These docs also note this ration can vary without noticeable impact so it may be fine. ** Testing Rackspace's New Cloud Offering (clarkb 20240604) *** Rax reached out via a ticket to our nodepool account indicating we can test out their new offering. They are specifically looking for feedback *** Clarkb is working on some dialogue after Rax reached out a bit more directly. * Open discussion

1 0

Team Meeting Agenda for June 11, 2024
by Clark Boylan 11 Jun '24

11 Jun '24

We will meet with this agenda on June 11, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** clarkb out June 14-19. Will miss the June 18 meeting if we have one. * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades **** tonyb expects to try a simple focal -> noble upgrade this week ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Gitea 1.22 Upgrade Planning (clarkb 20240528) *** There is a Gitea 1.22.0 release now. Once we have the general upgrade working we can test the doctor tool to fixup the DB case sensitivity. *** https://review.opendev.org/c/opendev/system-config/+/920580 ** Fixing Cloud Launcher Ansible (clarkb 20240604) *** Git updates broke our cloud launcher playbook even after Frickler addressed security group redundancy errors *** https://review.opendev.org/c/opendev/system-config/+/921685 workaround the Git issue *** We should know on the day of the meeting if this is working again. ** Improving mailman email throughput (clarkb 20240604) *** We've noticed slowness delivering emails for openstack-discuss *** Increasing the number of out runners hasn't alleviated this issue (it should prevent openstack-discuss from holding up other lists though) *** Next we need to modify the mailman email batch count and exim verp queue sizes. ** OpenMetal Cloud Rebuild (clarkb 20240604) *** The old cloud is gone and the new cloud is getting configured. *** https://etherpad.opendev.org/p/openmetal-cloud-bootstrapping captures our todo list for bootstrapping the new cloud ** Testing Rackspace's New Cloud Offering (clarkb 20240604) *** Rax reached out via a ticket to our nodepool account indicating we can test out their new offering. They are specifically looking for feedback *** Probably need to reach out to rax and find out a bit more information before determining if we are the target audience, but if so this is a good opportunity for more test nodes and improved clouds. * Open discussion

1 0

Email account for opendev.org Zuul Job
by Senthil Vasudevan 05 Jun '24

05 Jun '24

Hello, I have a query regarding the ubuntu account associated with the opendev job. Can we have one ubuntu account for many opendev zuul jobs. As of now our 3 Zuul CI jobs associated with Opendev.org which are using three different ubuntu accounts. Is there any possibility to use one to many mapping and use one email account for 3 different jobs. Please clarify. Thanks Senthil V -- ****** DISCLAIMER - MSysTechnologies LLC ****** This email message, contents and its attachments may contain confidential, proprietary or legally privileged information and is intended solely for the use of the individual or entity to whom it is actually intended. If you have erroneously received this message, please permanently delete it immediately and notify the sender. If you are not the intended recipient of the email message,you are notified strictly not to disseminate,distribute or copy this e-mail.E-mail transmission cannot be guaranteed to be secure or error-free as Information could be intercepted, corrupted, lost, destroyed, incomplete or contain viruses and MSysTechnologies LLC accepts no liability for the contents and integrity of this mail or for any damage caused by the limitations of the e-mail transmission.

2 1

Team Meeting Agenda for June 4, 2024
by Clark Boylan 04 Jun '24

04 Jun '24

We will meet with this meeting agenda on June 4, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades **** tonyb expects to try a simple focal -> noble upgrade this week ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Gerrit 3.9 Upgrade Planning (clarkb 20240423) *** Gerrit has been upgraded *** https://review.opendev.org/c/opendev/system-config/+/920938 is ready when we think we won't revert to 3.8 **** Followup changes to ^ add 3.10 image builds and testing ** Gitea 1.22 Upgrade Planning (clarkb 20240528) *** There is a Gitea 1.22.0 release now. Once we have the general upgrade working we can test the doctor tool to fixup the DB case sensitivity. *** https://review.opendev.org/c/opendev/system-config/+/920580 ** Fixing Cloud Launcher Ansible (clarkb 20240604) *** Git updates broke our cloud launcher playbook even after Frickler addressed security group redundancy errors *** https://review.opendev.org/c/opendev/system-config/+/921061 workaround the Git issue ** Increase Mailman 3 Out Runner Count (clarkb 20240604) *** We've seen outbound mail delivery for large lists take noticeable amounts of time *** https://review.opendev.org/c/opendev/system-config/+/920765 Increase the number of out runners to speed up outbound mail delivery ** OpenMetal Cloud Rebuild (clarkb 20240604) *** Yuriys has been in touch via email indicating things are ready on their side to rebuild this cloud on their new platform *** We need to gracefully shutdown and remove the existing cloud resources from Nodepool *** OpenMetal will work with us to build the new cloud, then we can add a mirror node and add the new cloud to Nodepool *** https://review.opendev.org/c/openstack/project-config/+/921072 Nodepool cleanups *** https://review.opendev.org/c/opendev/system-config/+/921075 System-config cleanups ** Testing Rackspace's New Cloud Offering (clarkb 20240604) *** Rax reached out via a ticket to our nodepool account indicating we can test out their new offering. They are specifically looking for feedback *** Probably need to reach out to rax and find out a bit more information before determining if we are the target audience, but if so this is a good opportunity for more test nodes and improved clouds. * Open discussion

1 0

Team Meeting Agenda for May 28, 2024
by Clark Boylan 28 May '24

28 May '24

We will meet with this agenda on May 28, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenInfra Summit CFP ends May 29, 2024 * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 *** Tonyb looking at cacti and the wiki as next items. ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Gerrit 3.9 Upgrade Planning (clarkb 20240423) *** Upgrade prep notes are going in https://etherpad.opendev.org/p/gerrit-upgrade-3.9 *** https://www.gerritcodereview.com/3.9.html Release Notes *** Planning to upgrade the Gerrit server on May 31, 2024 from 1600-1700 UTC *** infra-root please take a minute to look over the upgrade document and call out any concerns before we get too close to the upgrade date. ** Linaro Cloud SSL Cert Renewal (clarkb 20240521) *** It has been a couple of months and we need to renew the cert again. Should be documented now. Any volunteers? ** Gitea 1.22 Upgrade Planning (clarkb 20240528) *** There is a Gitea 1.22.0 release now. Once we have the general upgrade working we can test the doctor tool to fixup the DB case sensitivity. *** https://review.opendev.org/c/opendev/system-config/+/920580 * Open discussion

1 0

Team Meeting Agenda for May 21, 2024
by Clark Boylan 21 May '24

21 May '24

We will meet with this agenda on May 21, 2024 (today) at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenInfra Summit CFP ends May 29, 2024 * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 *** Tonyb looking at cacti and the wiki as next items. ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Building Ubuntu Noble Nodes (clarkb 20240416) *** Mirrors for x86_64 and arm64 are in place. We should be able to add image builds to nodepool. ** Gerrit 3.9 Upgrade Planning (clarkb 20240423) *** Upgrade prep notes are going in https://etherpad.opendev.org/p/gerrit-upgrade-3.9 *** https://www.gerritcodereview.com/3.9.html Release Notes *** Planning to upgrade the Gerrit server on May 31, 2024 from 1600-1700 UTC ** Linaro Cloud SSL Cert Renewal (clarkb 20240521) *** It has been a couple of months and we need to renew the cert again. Should be documented now. Any volunteers? * Open discussion

1 0

Team Meeting Agenda for May 14, 2024
by Clark Boylan 14 May '24

14 May '24

We will meet with this agenda on May 14, 2024 at 19:00 UTC in #opendev-meeting: * Announcements ** Clarkb out 5/17 and 5/20 * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 *** Meetpad is done. Just need to clean up old servers when we feel confident we won't rollback. *** What is next? ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Building Ubuntu Noble Nodes (clarkb 20240416) *** Next step is to add Noble to our mirrors ** Gerrit 3.9 Upgrade Planning (clarkb 20240423) *** Upgrade prep notes are going in https://etherpad.opendev.org/p/gerrit-upgrade-3.9 *** https://www.gerritcodereview.com/3.9.html Release Notes *** Downgrade is possible with an offline reindex using the old war. *** Things to think about: **** Do we want to enable diff3 diffs on merge changes? **** Change topic limit seems to only apply to open changes. Being over the limit doesn't seem to impact the upgrade process **** https://gerrit-review.googlesource.com/c/gerrit/+/424404 Upstream change to have release war build target without webfonts **** There is a new batched ref updated event that collects multiple ref updated events in a single event. This is disabled by default, but could one day impact zuul. Work sorting out likely. *** Server could stand to be upgraded as well but I think it is probably best to decouple these concerns. ** Cleaning up the openstack.org DNS zone (fungi 20240514) *** Proposed for deletion: https://paste.opendev.org/show/bVHJchKcKBnlgTNRHLVK/

1 0

Team Meeting Agenda for May 7, 2024
by Clark Boylan 07 May '24

07 May '24

We will meet with this agenda on May 7, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Everything but Gerrit is done now. When do we think we want to do Gerrit? ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Building Ubuntu Noble Nodes (clarkb 20240416) *** Next step is to add Noble to our mirrors ** Gerrit 3.9 Upgrade Planning (clarkb 20240423) *** Upgrade prep notes are going in https://etherpad.opendev.org/p/gerrit-upgrade-3.9 *** https://www.gerritcodereview.com/3.9.html Release Notes *** https://review.opendev.org/c/opendev/system-config/+/918333 Need updated and promoted 3.9 image before doing more testing. *** Downgrade is possible with an offline reindex using the old war. *** Things to think about: **** Do we want to enable diff3 diffs on merge changes? **** Change topic limit seems to only apply to open changes. Being over the limit doesn't seem to impact the upgrade process **** https://gerrit-review.googlesource.com/c/gerrit/+/424404 Upstream change to have release war build target without webfonts **** There is a new batched ref updated event that collects multiple ref updated events in a single event. This is disabled by default, but could one day impact zuul. Work sorting out likely. *** Server could stand to be upgraded as well but I think it is probably best to decouple these concerns. ** Wiki SSL Cert Renewal (clarkb 20240423) *** Clarkb will renew Friday ish (May 10) * Open discussion

1 0

Team Meeting Agenda for April 30, 2024
by Clark Boylan 30 Apr '24

30 Apr '24

We will meet with this agenda on April 30, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Everything but Gerrit is done now. When do we think we want to do Gerrit? ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** Do we want to prefer removing projects from zuul's tenant config or propose changes to projects to cleanup Xenial usage then if/when they break we did our best? *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Building Ubuntu Noble Nodes (clarkb 20240416) *** Changes needed in opendev nodepool elements and glean have all merged and DIB is testing Noble builds now *** Next step is to add Noble to our mirrors (do we have room within the existing Ubuntu Volume? Is that Volume still small enough for OpenAFS) then we can add image builds to Nodepool. ** Etherpad 2.0.3 Upgrade planning (clarkb 20240416) *** https://review.opendev.org/c/opendev/system-config/+/914119 *** Etherpad 2.0.2 removes the old APIKEY API auth method and now uses Oauth2.0 client_credentials instead. The above change updates docs and testing to accommodate this change. ** Gerrit 3.9 Upgrade Planning (clarkb 20240423) *** Upgrade prep notes are going in https://etherpad.opendev.org/p/gerrit-upgrade-3.9 *** https://www.gerritcodereview.com/3.9.html Release Notes *** Downgrade is possible with an offline reindex using the old war. *** Things to think about: **** Do we want to enable diff3 diffs on merge changes? **** Do we think 5000 changes per topic is too few? If so we need to update a new config option to set a larger value than the default. **** We can build the docs without external fonts, but the release target doesn't do this for us. I've asked how we might do that without forking the bazel configs. **** There is a new batched ref updated event that collects multiple ref updated events in a single event. This is disabled by default, but could one day impact zuul. Work sorting out likely. *** Server could stand to be upgraded as well but I think it is probably best to decouple these concerns. ** Wiki SSL Cert Renewal (clarkb 20240423) *** We have just under 4 weeks to renew the cert. This is one of the last (maybe the only) manually managed ssl cert. *** Will probably delay until we have less than a week left before processing it. * Open discussion

1 0

Team Meeting Agenda for April 23, 2024
by Clark Boylan 23 Apr '24

23 Apr '24

We will meet with this agenda on April 23, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Etherpad and mailman are done. Gerrit and Gitea are all that remain. ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial is next but currently busy with PTG, Release, and other tasks. *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Building Ubuntu Noble Nodes (clarkb 20240416) *** https://review.opendev.org/c/opendev/glean/+/915907 Glean updates for python3.12 support *** Appears we will need a debootstrap from debian unstable to build noble ** Etherpad 2.0.x Upgrade planning (clarkb 20240416) *** https://review.opendev.org/c/opendev/system-config/+/914119 *** Etherpad 2.0.2 removes the old APIKEY API auth method and now uses Oauth2.0 client_credentials instead. This new system is not yet released, but we are working to determine how to use it pre release. ** Add reference to the project-team-guide in the fail ci msg (fungi 20240423) *** https://review.opendev.org/914189 ** Gerrit 3.9 Upgrade Planning (clarkb 20240423) *** https://www.gerritcodereview.com/3.9.html Release Notes *** Downgrade is possible with an offline reindex using the old war. *** Need to compile an upgrade checklist (go over breaking changes and also test the downgrade). *** Server could stand to be upgraded as well but I think it is probably best to decouple these concerns. ** Wiki SSL Cert Renewal (clarkb 20240423) *** We have just under 4 weeks to renew the cert. This is one of the last (maybe the only) manually managed ssl cert. *** Will probably delay until we have less than a week left before processing it. * Open discussion

1 0