May 2023 - service-discuss - lists.opendev.org

Informal Summit Gathering for Zuul and OpenDev
by Clark Boylan 30 May '23

30 May '23

Hello everyone, I'd like to take advantage of the Summit to get together with those of us work on Zuul and OpenDev if possible. I think less formal is probably a good thing so that people can come and go as they are able. The beer garden in Berlin worked really well for this. Stanley Park Brewing [0] appears to be similar with outdoor patio seating in Stanley Park. This location is just under 3km from the convention center which makes it a long walk or a short ride in $vehicle. Looking at a schedule I think Thursday evening after the Summit is going to be the best time for most people. Let's meet there at 6pm on Thursday (June 15) and hang out. If you intend on being there can you throw your name onto this etherpad, https://etherpad.opendev.org/zuul-opendev-yvr-2023, so that we can get a rough idea for how large of a group we'll be? I don't expect we'll need reservations and instead try to grab one or more of their outdoor tables. You may want to bring sunscreen. If the weather turns out to be terrible I'll update this thread with any change of plans. [0] https://www.stanleyparkbrewing.com/ Thanks, Clark

1 0

Team Meeting Agenda for May 30, 2023
by Clark Boylan 30 May '23

30 May '23

Hello, We will meet with this agenda on May 30, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** No Meeting June 13, 2023 due to Summit ** Clarkb will be unable to attend June 20, 2023 due to travel * Actions from last meeting * Specs Review * Topics ** Migrating to Quay (clarkb 20230530) *** This has been reverted. We'll focus instead on moving services to newer Ubuntu versions and podman from there. *** https://review.opendev.org/c/opendev/system-config/+/883311 A role to install podman in system-config ** Bastion host (ianw 20230530) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230530) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit replication leaked tasks (clarkb 20230530) *** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230530) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Next up mirrors, meetpad, and ci registry. *** Please grab a node or two. ** Fedora cleanup (clarkb 20230523) *** Removal of Fedora 37 mirror impacted DIB jobs. DIB can fetch from upstream. https://review.opendev.org/c/openstack/diskimage-builder/+/883798 *** Proposed the removal of Fedora mirrors and/or images https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… **** Tonyb has been looking at moving existing fedora test nodes off of our mirrors. Not straightforward as it requires zuul-jobs role updates. ** Quo vadis storyboard (frickler 20230530) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG * Open discussion

1 0

Incorrect "StaringX Documentation" menu
by ronald.stone＠windriver.com 25 May '23

25 May '23

The dark blue header on the TOC at the left of each page of StarlingX documentation (docs.starlingx.io) has a drop down menu titled "StarlingX Documentation", with a triangular 'down' arrow to its right. This menu has only two static entries, "Install Guides" and "API Guides". The Install Guides entry points to the root of the doc set, not the Install Guide. Neither entry updates correctly when switching version, e.g. to release 7. We don't see any value in this menu given that complete information is available in the TOC and would like to replace it with a static header. I do not see any definition for it in conf.py. How can we change this behavior? Thanks.

2 1

Re: [service-announce] Re: opendevorg docker hub images moving to quay.io
by Radosław Piliszek 24 May '23

24 May '23

On Wed, 24 May 2023 at 20:12, Clark Boylan <cboylan(a)sapwetik.org> wrote: > The workarounds available are not great Hi Clark, If I only care about Ubuntu Jammy+ and podman (and quay.io of course), should I expect any issues? I am open to collaborating on this aspect. Kind regards, Radek -yoctozepto

3 4

Team Meeting Agenda for May 23, 2023
by Clark Boylan 23 May '23

23 May '23

We will meet with this agenda on May 23, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Migrating to Quay (clarkb 20230523) *** https://etherpad.opendev.org/p/opendev-quay-migration-2023 Plan/TODO list *** Discovered a deficiency in Docker impacting speculative testing of container images **** https://etherpad.opendev.org/p/3anTDDTht91wLwohumzW **** A number of issues have been found when setting up services with podman ***** No reliable source of packages for Ubuntu older than Jammy ***** No syslog support **** Migration to podman still a reasonable goal but a longer term one. In the interim suggestion is we revert quay.io image moves for opendev back to docker hub to keep speculative image testing. Then work towards a future where we use not docker and can switch to not docker.io more easily. ** Bastion host (ianw 20230523) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230523) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit replication leaked tasks (clarkb 20230523) *** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230523) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Next up mirrors, meetpad, and ci registry. *** Please grab a node or two. ** Fedora cleanup (clarkb 20230523) *** Removal of Fedora 37 mirror impacted DIB jobs. DIB can fetch from upstream. *** Proposed the removal of Fedora mirrors and/or images https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… **** Proceed with removal as it seems no one is asking for Fedora specifically. ** Quo vadis storyboard (frickler 20230523) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG * Open discussion

1 0

Team Meeting Agenda for May 16, 2023
by Clark Boylan 16 May '23

16 May '23

We will meet with this agenda on May 16, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Migrating to Quay (clarkb 20230516) *** https://etherpad.opendev.org/p/opendev-quay-migration-2023 Plan/TODO list *** Discovered a deficiency in Docker impacting speculative testing of container images **** https://etherpad.opendev.org/p/3anTDDTht91wLwohumzW **** clarkb's suggestion: we build all container images with buildx (it doesn't have this problem) and use skopeo to grab the correct images: https://review.opendev.org/c/opendev/system-config/+/882977 ** Bastion host (ianw 20230516) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230516) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit replication leaked tasks (clarkb 20230516) *** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230516) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Next up mirrors, meetpad, and ci registry. *** Please grab a node or two. ** AFS volume quotas and utilization (clarkb 20230516) *** Utilization climbing again. *** https://review.opendev.org/c/opendev/system-config/+/883083 Cleanup unused fedora mirror content *** Proposed the removal of Fedora mirrors and/or images https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Quo vadis storyboard (frickler 20230516) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG ** Wiki cert (clarkb 20230516) *** clarkb will try to get this done in a few days once there is about a week of validity remaining. * Open discussion

1 0

Future of Fedora and CentOS Stream Test Images
by Clark Boylan 10 May '23

10 May '23

Hello everyone, Over the last couple of years we've grown support for a number of RHEL like or RHEL adjacent distros in our CI system. We have added OpenEuler, Rocky Linux, and CentOS Stream (it replaced CentOS 8). At the same time we've fallen behind producing and updating images for Fedora. Currently we have disk images and test nodes for Fedora 35 and 36. We mirror packages for Fedora 36 and 37. In the past our intention was always to primarily have a single up to date Fedora image and mirror setup for our CI system which would be Fedora 38 as of April 18, 2023. One option we have is to drop Fedora 35 and 37 from Nodepool and our mirrors, add Fedora 38, and then drop Fedora 36. This would basically maintain the status quo of the last few years. That said I think this is probably a good opportunity to rethink our approach, and we are looking for your feedback. Currently the RHEL like distros are overlapping with each other a bit and for this reason we added Rocky Linux without adding extra mirroring for it. So far this seems to be stable enough as we don't run significant numbers of jobs on Rocky Linux. I think one option available to us would be to drop Fedora mirroring entirely, then add Fedora 38 images/nodes, and then drop Fedora 36 images/nodes. One upside to this approach is it will free up disk space on our OpenAFS fileservers which are under a space crunch currently. I also wonder if we can drop Fedora entirely. It seems like CentOS Stream now occupies a similar role within our CI system. It gives us forward looking updates that will eventually be in RHEL. This allows us to test against a future state in preparation for RHEL changes. The upside to this is that Fedora's support lifetime is about 13 months but CentOS Stream releases appear to have several years of support. This should reduce the amount of mirror and image work we need to perform. Are there specific use cases that Fedora serves that CentOS Stream does not within our CI system? If so are they important enough to continue to try and run on the Fedora release treadmill? If we do want to try and keep up with Fedora do we have any concerns dropping our local mirroring of Fedora packages? Please let us know, Clark

7 14

Team Meeting Agenda for May 9, 2023
by Clark Boylan 09 May '23

09 May '23

We will meet with this agenda on May 9, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Migrating to Quay (clarkb 20230509) *** https://etherpad.opendev.org/p/opendev-quay-migration-2023 Plan/TODO list *** About half of the images that need moving have changes pushed and are largely ready to merge. *** Will use the halfway point to pause and make sure everything is going well and we aren't missing anything important. ** Bastion host (ianw 20230509) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230509) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Upgrade and Project Renames (ianw 20230509) *** https://review.opendev.org/c/openstack/project-config/+/882075 list little bit of ACL cleanup work *** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 *** We should restart on updated image to pick up container move as well as theming changes. ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230509) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Next up mirrors, meetpad, and ci registry. *** Please grab a node or two. ** AFS volume quotas and utilization (clarkb 20230509) *** Utilization is slightly down compared to last week. *** Proposed the removal of Fedora mirrors https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Quo vadis storyboard (frickler 20230509) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG * Open discussion

1 0

Re: [dev][infra][qa][tact-sig] Future of Fedora and CentOS Stream Test Images
by Kashyap Chamarthy 05 May '23

05 May '23

On Wed, May 03, 2023 at 03:28:00PM +0000, Jeremy Stanley wrote: > tl;dr is that the OpenDev Collaboratory is looking at potentially > scaling back on Fedora-based test platform support. Several options > are presented in this service-discuss post: > > https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… > > If you have an opinion on using Fedora in our CI jobs, please follow > up to the discussion there. Hi, Jeremy. FWIW, I think it is still valuable to have at least the latest Fedora-only image available. Often times kernel and virtualization fixes are first fixed on Fedora -- many upstream kernel developers happen to use Fedora. Another reason I wanted to mention is the 'virt-preview' repo[1]. But I see that 'virt-preview' is also available for CentOS stream. So that's a less strong reason. That said, if dropping Fedora is the compromise we need to make to reduce the workload of "mirroring and image management, then I don't insist. [1] https://copr.fedorainfracloud.org/coprs/g/virtmaint-sig/virt-preview/ -- /kashyap

4 6

Team Meeting Agenda for May 2, 2023
by Clark Boylan 02 May '23

02 May '23

We will meet with this agenda on May 2, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Migrating to Quay (clarkb 20230502) *** zookeeper-statsd is now publishing to quay.io showing that this all seems to work for OpenDev *** Need to decide on an order of operations. **** Do we do python-builder/python-base/uwsgi-base first then the consuming images? This is likely less work, but does open us to potential problems if we need quick updates of the base images and rebuilds of everything. However we are effectively rebuilding anything anyway so maybe we do this and do our best to do it in a short timeframe. **** Maybe we can pick a couple of days to sprint this and try to get as much done as possible? **** Probably a good idea to write down a plan and todo list as soon as we have an approach to track progress. Clarkb can do this. **** Don't forget to resync any images from docker hub to quay.io prior to publishing to quay.io if images have updated since the initial sync. **** We may need to manually clean up the old docker hub images on hosts after we move images. Zuul and Nodepool at least appear to need this. **** Do we need to announce this move to service-announce? We may have external image consumers. ** Bastion host (ianw 20230502) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230502) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Upgrade and Project Renames (ianw 20230502) *** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 *** Gerrit 3.8 images exist now and we don't build 3.6. A couple of changes were landed to 3.7 as well to make 3.8 happy should we restart on the newer image? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230502) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Nameserver replacement largely complete. **** Do we still have cleanup todo? Removing old nodes from inventory, deleting them from the cloud, AAAA glue records? **** https://etherpad.opendev.org/p/2023-opendev-dns *** Need help with the next servers in the list. ** AFS volume quotas and utilization (clarkb 20230502) *** Fedora 36 to be cleaned up in the near future. ** Gitea 1.19 Upgrade (clarkb 20230502) *** https://review.opendev.org/c/opendev/system-config/+/877541 Upgrade opendev.org to 1.19.2 ** Quo vadis storyboard (frickler 20230502) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG * Open discussion

1 0