- service-discuss - lists.opendev.org

Team Meeting Agenda For August 27, 2024
by Clark Boylan 26 Aug '24

26 Aug '24

== Agenda for next meeting == * Announcements ** Clarkb will miss the next meeting (September 3, 2024) due to the Summit. * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** CentOS 8 Stream EOLd and jobs can no longer successfully run there. Cleanup is happening under topic:drop-centos-8-stream *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Rackspace's Flex Cloud (clarkb 20240604) *** Fungi and Clarkb spoke to rax about this and tl;dr is it is an entirely new cloud generation and we can use our existing accounts *** Fungi got clouds.yaml updated and is able to talk to the new cloud region. *** Need to enroll the cloud into our cloud launcher to configure networks, subnets, routers, security groups, and ssh keys. **** https://review.opendev.org/c/opendev/system-config/+/927214 *** Need to figure out flavors and quotas. *** Optionally upload the upstream Ubuntu Noble Cloud Image ** Etherpad 2.2.2 Upgrade (clarkb 20240820) *** Etherpad 2.2.2 has been released which breaks compatibility with our ep_headings plugin. *** fungi tested etherpad 2.2.2 + ep_headings2 against our prod database and found that it seems to work. We can probably go ahead and upgrade but maybe be extra cautious and do a db backup before the upgrade? *** https://review.opendev.org/c/opendev/system-config/+/926078 ** Service Coordinator Election (clarkb 20240820) *** clarkb was the only nomination. Is Service Coordinator for six more months by default *** If a nomination was missed now is the time to bring that up. ** OSUOSL ARM Cloud Issues (clarkb 20240827) *** nb04 ran out disk on /opt which broke image builds *** Cleaning up /opt/dib_tmp didn't produce working builds because losetup was also out of device space *** Current status is server has been rebooted to clear out stale losetup loopback devices and leaked mounts with another round of disk cleaning in progress. *** Separately Kolla has discovered that jobs take significantly more time on OSUOSL nodes than they did on linaro and seem to fail more than they succeed. Other projects are having fewer issues. **** Not sure yet what the underlying issue is but hunches are that it is related to IO (since Neutron unittests don't seem super bothered but Kolla container builds are). ** Updating ansible+ansible-lint versions in our repos (clarkb 20240827) *** After updating our default nodeset to Noble we discovered that ansible-lint with all of our pins cannot run on Noble due to incompatibilities with python 3.12. *** Updating to latest ansible-lint and the current version of Zuul's Ansible install (>=8,<9) does work but introduces new rules and lint errors *** https://review.opendev.org/c/openstack/project-config/+/926848 *** https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/926970 **** This change fails due to an inability to build openafs packages on CentOS 9 Stream arm64 images due to out of date kernels. Hopefully fixing those image builds will correct this problem. * Open discussion

1 0

August 2024 OpenDev Service Coordinator Election
by Clark Boylan 20 Aug '24

20 Aug '24

Hello everyone, It is that time of year again where we need to elect a new Service Coordinator. Last election held its candidate nomination period from February 6 to 20, 2024; due to having only a single candidate no voting was necessary. Six months from that time period is August 6 to 20, 2024. The August 2024 OpenDev Service Coordinator election will run on this timeline: candidate nominations open from August 6 to August 20, 2024. Then if we have more than one candidate we will vote to select the winning candidate from August 21 to August 28, 2024. The electorate will be contributors to OpenDev git repos since August 20, 2023 (basically the year prior to the election). Candidates can respond to this email thread indicating their self nomination or start a new thread on this mailing list to do so. We will start and end days at midnight UTC to avoid any confusion around specific timezones. I would encourage anyone who is interested in running to do so. I can answer questions about the role, and am happy to support a new Service Coordinator. Finally, we've got a bit of time before August 6. If there are questions, concerns or feedback on this process please feel free to bring them up here on the mailing list or more directly if necessary. Clark

1 1

OpenDev Team Meeting Agenda for August 20, 2024
by Clark Boylan 19 Aug '24

19 Aug '24

We will meet with this agenda on August 20, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Clarkb will miss the September 3, 2024 meeting due to the Summit. * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades **** tonyb expects to try a simple focal -> noble upgrade this week ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** CentOS 8 Stream EOLd and jobs can no longer successfully run there. Cleanup is happening under topic:drop-centos-8-stream *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Testing Rackspace's New Cloud Offering (clarkb 20240604) *** Clarkb is trying to meet with rax this week. ** Etherpad 2.2.2 Upgrade (clarkb 20240820) *** Etherpad 2.2.2 has been released which breaks compatibility with our ep_headings plugin. There is an ep_headings2 alternative plugin that is likely not compatible with ep_headings content, but it isn't clear if that will break etherpad entirely or just force people to re apply heading formatting to pads they care about. *** We need to do more testing of the behaviors so that we are not surprised if/when we upgrade. *** https://review.opendev.org/c/opendev/system-config/+/926078 ** Service Coordinator Nominations End August 20 (Day of the Meeting) (clarkb 20240820) *** No nominations have arrived yet. Is clarkb it by default again? *** clarkb can submit a nomination after the meeting to make it official if that is the path we're taking ** Updating the default Zuul Nodeset to Noble on August 21, 2024 (clarkb 20240820) *** https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… *** https://review.opendev.org/c/opendev/base-jobs/+/926360 *** Just a reminder that we are planning to make this change soon. Be on the lookout for unexpected fallout. ** openstack.org DNS Hosting Moving to CloudFlare (clarkb 20240820) *** Once this move is done we won't be able to update the zone in rax and have that reflect in global DNS for us. *** Currently fungi can make edits to the new zone contents. clarkb should be able to at some point after getting access. ** OpenMetal Cloud Cert Expiring Soon (clarkb 20240820) *** This cloud was deployed using Kolla's LetsEncrypt integration *** It is likely that we just need to retrigger kolla to request a new cert. *** Can we put that on a cron job? Maybe another item to feedback to OpenMetal so that this works for everyone and not just us. * Open discussion

1 0

Forgejo instead of Gitea
by Radosław Piliszek 10 Aug '24

10 Aug '24

Hi OpenDev friends, I wonder if Forgejo has ever been mentioned as a really free (as in speech) alternative to Gitea. If not, I think this is the best time because the two projects started to diverge (as of early 2024) and the further down the path of Gitea upgrades, the harder it might be to migrate. I think Forgejo presents itself as a better incarnation of Gitea for OpenDev because it is more aligned in spirit and has higher transparency than Gitea. All the reasoning behind Forgejo and related relevant details are summarised in [1] Please let me know if this could be welcome on OpenDev. If so, I can offer my time for being a bridge between the communities and helping with the possible migration. Kind regards, Radek [1] https://forgejo.org/compare-to-gitea/

2 4

Team Meeting Agenda For July 30, 2024
by Clark Boylan 29 Jul '24

29 Jul '24

We will meet with this agenda on July 30, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Clarkb was bad at planning and has family obligations the next two Tuesdays (August 6 and 13) * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades **** tonyb expects to try a simple focal -> noble upgrade this week ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** CentOS 8 Stream EOLd and jobs can no longer successfully run there. Cleanup is happening under topic:drop-centos-8-stream *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Testing Rackspace's New Cloud Offering (clarkb 20240604) *** Clarkb is still waiting to hear back from rax on whether or not a meeting works to discuss this further and if so when. ** Reconsider queue depth for integrated gate (frickler 20240705) *** During the rush to get fixes for the recent CVE merged, a lot of gate failures happened *** Chances that patch #20 in the pipeline gets merged without any issue in the 19 patches before it seems neglegible *** Suggestion: Reduce depth to 10 for now, which still seems pretty optimistic *** Alternative proposal: Update openstack tenant jobs to report early failures so that resources can be allocated to mergable changes more quickly. **** This has been implemented for Tempest jobs. This can expand further to openstack tox jobs if we feel that is also helpful. ** Removing the Works on Arm Linaro Cloud from Nodepool (clarkb 20240730) *** The Works on Arm program contacted us late last week to let us know the hardware resources this cloud runs on will go away on August 10 *** We have begun the process of removing this cloud from Nodepool. *** Will try to work with Kevin on cleaning up the cloud side as much as is necessary. * Open discussion

1 0

Team Meeting Agenda for July 23, 2024
by Clark Boylan 22 Jul '24

22 Jul '24

We will meet with this agenda on July 23, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades **** tonyb expects to try a simple focal -> noble upgrade this week ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** CentOS 8 Stream EOLd and jobs can no longer successfully run there. Cleanup is happening under topic:drop-centos-8-stream *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Gitea DB Doctoring (clarkb 20240723) *** Gitea 1.22 upgrade is complete *** https://etherpad.opendev.org/p/hcz4yMxUIsAWGgyoHKeZ proposed plan for db updates on each gitea node. ** Testing Rackspace's New Cloud Offering (clarkb 20240604) *** Clarkb is still waiting to hear back from rax on whether or not a meeting works to discuss this further and if so when. ** Drop x/* projects with config errors from zuul (frickler 20240706) *** Patch proposed at https://review.opendev.org/c/openstack/project-config/+/923509 *** Announcement was sent for this. Now need an up to date list then we can merge a cleanup. ** Reconsider queue depth for integrated gate (frickler 20240705) *** During the rush to get fixes for the recent CVE merged, a lot of gate failures happened *** Chances that patch #20 in the pipeline gets merged without any issue in the 19 patches before it seems neglegible *** Suggestion: Reduce depth to 10 for now, which still seems pretty optimistic *** Alternative proposal: Update openstack tenant jobs to report early failures so that resources can be allocated to mergable changes more quickly. * Open discussion

1 0

Re: [service-announce] Removal of Inactive Projects from Zuul's Tenant Configuration
by Radosław Piliszek 11 Jul '24

11 Jul '24

Hi Clark and the rest of the OpenDev team, On Tue, 9 Jul 2024 at 23:40, Clark Boylan <cboylan(a)sapwetik.org> wrote: > The OpenDev team is going to start removing inactive x/* namespace projects from Zuul's tenant configuration. The two primary motivations behind this are to remove projects that already have broken configurations within Zuul (see the Zuul config errors page [0]), and to avoid projects entering this state in the future as we clean up test nodes like Ubuntu Xenial or CentOS 8 Stream or otherwise modernize the CI system. On that note, you could also merge the removal of the NebulOuS tenant. [1] [1] https://review.opendev.org/c/openstack/project-config/+/921725 Kind regards, Radek

1 0

Team Meeting Agenda for July 9, 2024
by Clark Boylan 08 Jul '24

08 Jul '24

We will meet with this agenda on July 9, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Clarkb out next week through Wednesday. * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades **** tonyb expects to try a simple focal -> noble upgrade this week ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** CentOS 8 Stream EOLd and jobs can no longer successfully run there. Cleanup is happening under topic:drop-centos-8-stream **** What do we think about a forceful removal of centos 8 stream fips jobs at this point? *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Gitea 1.22 Upgrade Planning (clarkb 20240528) *** https://review.opendev.org/c/opendev/system-config/+/920580 *** https://104.130.219.4:3081/opendev/system-config 1.22.1 Held node *** Clarkb is thinking we can upgrade to 1.22.1 then figure out the db correcting after the upgrade. ** Etherpad 2.1.1 Upgrade (clarkb 20240709) *** https://review.opendev.org/c/opendev/system-config/+/923661 ** Testing Rackspace's New Cloud Offering (clarkb 20240604) *** Clarkb is still waiting to hear back from rax on whether or not a meeting works to discuss this further and if so when. ** Drop x/* projects with config errors from zuul (frickler 20240706) *** Patch proposed at https://review.opendev.org/c/openstack/project-config/+/923509 *** Send a final warning to the ML first? Or is there a volunteer to chase down individual projects? ** zuul db performance issues (frickler 20240705) *** https://zuul.opendev.org/t/openstack/buildsets times out without showing results *** https://zuul.opendev.org/t/openstack/buildsets?pipeline=gate takes 20-30s *** Makes investigating gate status more complicated *** Not sure if our deployment is reaching its limits or Zuul needs some more query optimization? ** Reconsider queue depth for integrated gate (frickler 20240705) *** During the rush to get fixes for the recent CVE merged, a lot of gate failures happened *** Chances that patch #20 in the pipeline gets merged without any issue in the 19 patches before it seems neglegible *** Suggestion: Reduce depth to 10 for now, which still seems pretty optimistic * Open discussion

1 0

Team Meeting Agenda for July 2, 2024
by Clark Boylan 01 Jul '24

01 Jul '24

We will meet with this agenda on July 2, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Holiday at the end of the week for some of us. * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades **** tonyb expects to try a simple focal -> noble upgrade this week ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** CentOS 8 Stream EOLd and jobs can no longer successfully run there. Cleanup is happening under topic:drop-centos-8-stream **** Projects like glance are still holding onto these old jobs despite no hope of them ever succeeding. Do we want to say Monday July 8 we're force merging cleanups on our side and projects will have to deal with the fallout since they aren't fixing things now? *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Gitea 1.22 Upgrade Planning (clarkb 20240528) *** There is a Gitea 1.22.0 release now. Once we have the general upgrade working we can test the doctor tool to fixup the DB case sensitivity. *** https://review.opendev.org/c/opendev/system-config/+/920580 ** OpenMetal Cloud Rebuild (clarkb 20240604) *** Waiting on a response from OpenMetal about the best way to proceed. I don't want to try and fix things and step on their toes or vice versa. ** Testing Rackspace's New Cloud Offering (clarkb 20240604) *** Clarkb is still waiting to hear back from rax on whether or not a meeting works to discuss this further and if so when. ** Nodepool in Zuul (corvus 20240702) *** Nodepool in Zuul work is proceeding *** Creating Opendev image build jobs *** Running zuul-launcher in shadow mode ** Collating backlog items from $everyone (tonyb 20240107) *** Review of approved specs: https://docs.opendev.org/opendev/infra-specs/latest/ Seems like that could do with some love *** Would it be helpful to have a '#noteit' IRC action that'd record the timestamp for context and add it to an editable list? *** There is a pretty solid risk of this becoming long and write only :/ * Open discussion

1 0

Team Meeting Agenda for June 25, 2024
by Clark Boylan 24 Jun '24

24 Jun '24

We will meet with this agenda on June 25, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Upgrading Old Servers (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades **** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade **** tonyb looking at cacti after wiki *** https://etherpad.opendev.org/p/opendev-focal-server-upgrades **** tonyb expects to try a simple focal -> noble upgrade this week ** AFS Mirror cleanups (clarkb 20240220) *** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial *** CentOS 8 Stream EOLd and jobs can no longer successfully run there. Cleanup is happening under topic:drop-centos-8-stream *** Can followup with webserver log processing to determine which other mirrors may be dead. ** Gitea 1.22 Upgrade Planning (clarkb 20240528) *** There is a Gitea 1.22.0 release now. Once we have the general upgrade working we can test the doctor tool to fixup the DB case sensitivity. *** https://review.opendev.org/c/opendev/system-config/+/920580 ** Improving mailman email throughput (clarkb 20240604) *** We've noticed slowness delivering emails for openstack-discuss *** Increasing the number of out runners hasn't alleviated this issue (it should prevent openstack-discuss from holding up other lists though) *** Next we need to modify the mailman email batch count and exim verp queue sizes. ** OpenMetal Cloud Rebuild (clarkb 20240604) *** Things were working fairly well with max-servers 50 until they weren't *** Looks like problems may be related to available disk space and how ceph is used by various services (not enough are using ceph) *** Also need to look into increasing the number of placement groups per OSD as we're far below the recommendation of 100 PGs per OSD **** https://docs.ceph.com/en/latest/dev/placement-group/ These docs also note this ration can vary without noticeable impact so it may be fine. ** Testing Rackspace's New Cloud Offering (clarkb 20240604) *** Rax reached out via a ticket to our nodepool account indicating we can test out their new offering. They are specifically looking for feedback *** Clarkb is working on some dialogue after Rax reached out a bit more directly. * Open discussion

1 0