February 2024 - service-discuss

Team Meeting Agenda for February 27, 2024
by Clark Boylan 27 Feb '24

27 Feb '24

We will meet with this agenda on February 27, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Service Coordinator Election Results (clarkb 20240227) *** Single nominee was clarkb *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Paste, Etherpad, Gitea, Gerrit, Refstack, and Mailman could use upgrades. Starting with Paste due to its simplicity. *** https://review.opendev.org/c/opendev/system-config/+/909471 Upgrade paste.o.o mariadb to 10.11 ** AFS Mirror cleanups (clarkb 20240220) *** OpenSUSE Leap has been removed **** topic:drop-opensuse *** Debian Buster is in progress **** topic:drop-buster *** CentOS 7 and Ubuntu Xenial still to do. *** Can followup with webserver log processing to determine which other mirrors may be dead. ** OpenDev email hosting (clarkb 20240220) *** Would be good to come to a consensus on whether or not we want ot host this ourselves or just pay someone to do it. ** Gitea 1.21.7 Upgrade (clarkb 20240227) *** https://review.opendev.org/c/opendev/system-config/+/909941 ** git-review vendoring the commit message hook (clarkb 20240227) *** https://review.opendev.org/c/opendev/git-review/+/910275 ** Project Renames (clarkb 20240227) *** Do we need to update the tooling to move the replication waiting queue aside during the rename process? * Open discussion

1 0

Clarkb's Service Coordinator Nomination for February 2024
by Clark Boylan 20 Feb '24

20 Feb '24

As previously mentioned on this list [0] it is Service Coordinator nomination time. In fact the nomination period ends about two hours after I send this email out. There have been no other nominees, and I am willing to continue operating in this capacity. Consider this my self nomination for the Service coordinator position. I continue to think it would be great to have others in the role, and I will happily support whoever that may be in the future should we have additional volunteers. Until that time, you can expect I'll continue to try and push for the sustainability of our services. We have been able to lean on Zuul, Ansible, Containers, and CI testing to reliably keep our services up to date, and we should continue to do this or similar. The consistent deployment practices reduces mental overhead understanding how things operate, and our CI testing gives us confidence when we make changes. [0] https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… Thank you for your consideration, Clark

1 0

Team Meeting Agenda for February 20, 2024
by Clark Boylan 20 Feb '24

20 Feb '24

We will meet with this agenda on February 20, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Service coordinator nominations open February 6, 2024 - February 20, 2024 *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… *** No nominees as of the time the agenda was put together. Nominations stay open 4 hours beyond the end of our meeting. * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** https://review.opendev.org/c/opendev/system-config/+/908512 Ensure new servers have sufficient CPU version levels. *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 *** Hound looks to be a fairly simple service with indexed/ached data on disk. It's currently running Focal with "bookworm" containers Any reason to stay there? *** I think there was a suggestion we could implement prometheus and once were happy with that drop cacti, Is that correct? ** PrePTG Summary (clarkb 20240220) *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… *** Code review practices are worth reviewing for accuracy *** Happy to followup on other topics or have others write up summaries from their perspective. ** MariaDB Upgrades (clarkb 20240220) *** Relying on the container image MARIADB_AUTO_UPGRADE flag *** Paste, Etherpad, Gitea, Gerrit, Refstack, and Mailman could use upgrades. Starting with Paste due to its simplicity. ** AFS Mirror cleanups (clarkb 20240220) *** We will remove OpenSUSE Leap, Debian Buster, CentOS 7, and Ubuntu Xenial *** Can followup with webserver log processing to determine which other mirrors may be dead. ** OpenDev email hosting (clarkb 20240220) *** Would be good to come to a consensus on whether or not we want ot host this ourselves or just pay someone to do it. * Open discussion

1 0

Summary of Discussion from the OpenDev PrePTG
by Clark Boylan 16 Feb '24

16 Feb '24

This week's OpenDev PrePTG event resulted in a lot of successful discussion. We captured notes in the agenda Etherpad [0] alongside the agenda items and those interested should probably refer to that document after reading my summary. I'll try to keep this summary as brief as possible and will focus on actionable outcomes as a result. At a high level we are comfortable with the code review practices we have been employing and will not be making any drastic changes. We did note that we should codify these practices a bit more than we have though. My first attempt at this would be: * OpenDev code reviewers should be empowered to exercise judgement when applying single core reviewer approvals. It is OK to single core approve changes when reviewers are comfortable the change is low risk or low impact. * Holding off on approval despite a +2 is appropriate if the reviewer is unable to follow the change through merging and deployment. * It is generally a good idea to follow up on approved changes you have made to ensure they have deployed successfully. * Code pushed by another core reviewer is assumed to be implicitly approved by that approver unless they explicitly annotate the change with an indication it should not be merged (WIP/DNM/-2/etc). We also discussed the possibility of adding new code reviewers through the TaCT SIG. Hopefully, a larger group of reviewers looking at OpenStack related configs will enable more time for OpenDev work from OpenDev core reviewers. In the short term there are a number of tasks that we identified we could be working on: * Clarkb will be reaching out to our cloud providers to reengage with them. We'll use this as an opportunity to determine if there might be new tools we can take advantage of (Kubernetes as a service for example), or if expected updates have been made (has OpenMetal's deployment tooling been updated). * We will proceed with upgrading MariaDB containers using the MARIADB_AUTO_UPGRADE env var flag. MariaDB seems confident that upgrades between the versions we run should be quick and painless. * The planned Prometheus service deployment can be picked up and completed. This would be a good task for a new contribute. * The Grafana container image can be unpinned. This is also a good task for a new contributor. * OpenDev git repo migration into the OpenDev Zuul tenant should continue. We think that the vast majority of our repos can either be moved or retired. Then we'll coordinate moves for system-config, project-config, and the DNS zone repos. * We can clean up Nodepool images and mirror content for OpenSUSE LEAP, Debian Buster, CentOS 7, and Ubuntu Xenial. On a number of topics we realized that more investigation is required: * The Podman, Docker, Docker Compose situation needs to be further investigated and tested. In particular OpenDev is interested in finding the least painful way to host images off of docker hub while still enjoying speculative container testing within Zuul jobs. Corvus is hoping to be able to look into this again. * Fungi will check if storyboard's container deployments can be resurrected in order to redeploy storyboard on the latest version of storyboard. Once that is decided we will be sending email about plans to sunset and retire the storyboard service. * We can use mirror web server logs to determine which mirror paths are most actively used. This information can then be used to decide on additional mirror content to prune. We discussed a number of other topics without any solid conclusion. I want to call out two in particular that I would like the team to followup on, but please read the Etherpad nodes for complete coverage. First, is the question of how we want to handle email hosting for opendev.org. We can either host it ourselves or pay for hosted services which appear to be very reasonably priced. Second, is the management of some sort of backlog with indicators for new contributor and possible internship opportunities on the backlog items. Tonyb indicated a willingness to help mentor students involved in the Foundation's university partnerships through projects if we have good opportunities for them. I am happy to answer any questions related to the event that you might have, or followup on agenda items I haven't already covered above. There was more discussion than I managed to cover in this one email, but I think it is long enough already. It might also be worthwhile to get other perspectives as mine is sure to have biases related to my work within OpenDev that are different than yours. [0] https://etherpad.opendev.org/p/opendev-preptg-202402 Clark

1 0

Team Meeting Agenda for February 13, 2024
by Clark Boylan 13 Feb '24

13 Feb '24

We will meet with this agenda on February 13, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Service coordinator nominations open February 6, 2024 - February 20, 2024 *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** OpenDev PrePTG February 14+15 at 14:00-16:00UTC and 22:00-00:00UTC both days on our meetpad service. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** https://review.opendev.org/c/opendev/system-config/+/908512 Ensure new servers have sufficient CPU version levels. *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 *** Hound looks to be a fairly simple service with indexed/ached data on disk. It's currently running Focal with "bookwork" containers Any reason to stay there? *** I think there was a suggestion we could implement prometheus and once were happy with that drop cacti, Is that correct? ** AFS quota issues (frickler 20231217) *** mirror.openeuler has reached its quota limit and the mirror job seems to be failing since two weeks. I'm also a bit worried that they seem do have doubled their volume over the last 12 months *** ubuntu mirrors are also getting close, but we might have another couple of months time there *** mirror.centos-stream seems to have a steep increase in the last two months and might also run into quota limits soon *** We should be able to clean up older arm64 ubuntu-ports and possibly also older debian arm mirroring. ** Keycloak upgrade struggles (clarkb 20240130) *** A new keycloak03 server is up and ready for use. This server is a redeployment of latest keycloak and a real database backend. ** Etherpad 1.9.7 Upgrade (clarkb 20240206) *** https://review.opendev.org/c/opendev/system-config/+/908328 ** Zuul Circular Dependency Refactor (clarkb 20240213) *** This has been deployed to OpenDev in a prerelease form. If you notice problems please let the Zuul project know *** Zuul has temporarily enabled this functionality within its tenant for testing/dogfooding purposes, but we don't expect OpenDev to enable this generally. * Open discussion

1 0

Team Meeting Agenda for February 6, 2024
by Clark Boylan 06 Feb '24

06 Feb '24

We will meet with this agenda on February 6, 2024 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Service coordinator nominations open February 6, 2024 - February 20, 2024 *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… ** OpenDev PrePTG February 14+15 at 14:00-16:00UTC and 22:00-00:00UTC both days on our meetpad service. *** https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… * Actions from last meeting * Specs Review * Topics ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/q/topic:jitsi_meet-jammy-update *** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58 *** Hound looks to be a fairly simple service with indexed/ached data on disk. It's currently running Focal with "bookwork" containers Any reason to stay there? *** I think there was a suggestion we could implement prometheus and once were happy with that drop cacti, Is that correct? ** Python container updates (tonyb 20230718) *** zuul-operator is the last hold out now **** https://review.opendev.org/c/zuul/zuul-operator/+/881245 is the change we need to get landed. ** AFS quota issues (frickler 20231217) *** mirror.openeuler has reached its quota limit and the mirror job seems to be failing since two weeks. I'm also a bit worried that they seem do have doubled their volume over the last 12 months *** ubuntu mirrors are also getting close, but we might have another couple of months time there *** mirror.centos-stream seems to have a steep increase in the last two months and might also run into quota limits soon *** We should be able to clean up older arm64 ubuntu-ports and possibly also older debian arm mirroring. ** Keycloak upgrade struggles (clarkb 20240130) *** Keycloak converted its runtime from wildfly to quarkus. This led to legacy wildfly images and modern quarkus images that are not super compatible with each other *** A switch from the legacy wildfly to quarkus image failed because the h2 database isn't very portable between the two. (Different file paths, filenames and db credentials). *** Pivoting to a new latest release deployment using a mariadb. Will require us to rebuild the zuul auth stuff if we can't port the db content across. ** Gitea Upgrade and DB Confg Changes (clarkb 20240206) *** https://review.opendev.org/c/opendev/system-config/+/907500 Increase DB connection limits after HTTP 500 errors due to hitting the limit *** https://review.opendev.org/c/opendev/system-config/+/907472 Upgrade to the latest bugfix release ** Etherpad 1.9.7 Upgrade (clarkb 20240206) *** No change yet. The changelog indicates this update impacts how plugins are installed. We should make sure the plugins we rely on function in our images builds. * Open discussion

1 0