- service-discuss - lists.opendev.org

Infra Team Meeting Agenda for February 15, 2022
by Clark Boylan 15 Feb '22

15 Feb '22

We will meet Tuesday, February 15, 2022 at 19:00 UTC in #opendev-meeting with this agenda: * Announcements ** Service Coordinator nomination time has begun. You have until February 15, 2022 UTC end of day. That is today. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220215) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220215) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning our Nodepool images (clarkb 20220215) *** http://lists.opendev.org/pipermail/service-announce/2021-December/000029.ht… cleanup announcement *** https://review.opendev.org/c/opendev/system-config/+/829115 Removes fedora 34 mirror content ** Spring cleaning for old reviews? (frickler 20220215) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** https://etherpad.opendev.org/p/opendev-repo-retirements List of repos to retire. Please review for any that are inappropriate to remove. *** https://review.opendev.org/c/opendev/system-config/+/829119 Removes unused repos from integration testing. *** https://review.opendev.org/c/openstack/project-config/+/829121 Sets noop jobs on unused repos so that they can be retired. ** Gitea 1.16.1 (clarkb 20220215) *** https://review.opendev.org/c/opendev/system-config/+/828184 Change to upgrade to 1.16.1 when we are ready *** https://104.130.74.7:3081/opendev/system-config Test site via held node here *** There are changes to how docker container processes are started. In particular change log calls out this may affect ssh. We update our testing to push content via ssh to simulate gerrit replication to cover this. ** Gerrit gitea weblinks (clarkb 20220215) *** Fixes in progress upstream **** https://gerrit-review.googlesource.com/c/gerrit/+/329279 **** https://gerrit-review.googlesource.com/c/plugins/gitiles/+/330361 *** Not sure why the bazel test runs fail complaining about java runtimes. This happens for me too on the gerrit repo but only happens to gitiles upstream (not gerrit). * Open discussion

1 0

Team Meeting Agenda for February 8, 2022
by Clark Boylan 08 Feb '22

08 Feb '22

We will meet with this agenda on February 8, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenInfra Summit CFP needs your input: https://openinfra.dev/summit/ Tomorrow (the 9th) is the last day to get your entries in. ** Service Coordinator nomination time is ending. You have until February 8, 2022. That is today. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220208) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220208) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning our Nodepool images (clarkb 20220208) *** http://lists.opendev.org/pipermail/service-announce/2021-December/000029.ht… cleanup announcement *** CentOS 8 is now gone. Any progress on Fedora 34 removal? ** Spring cleaning for old reviews? (frickler 20220208) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** https://etherpad.opendev.org/p/opendev-repo-retirements List of repos to retire. Please review for any that are inappropriate to remove. ** Gerrit mergeabilityComputationBehavior (frickler 20220127) *** https://gerrit-documentation.storage.googleapis.com/Documentation/3.5.0/con… *** Frickler volunteered to write a change for this. * Open discussion

1 0

Team Meeting Agenda for February 1, 2022
by Clark Boylan 01 Feb '22

01 Feb '22

We will meet at 19:00 UTC on February 1, 2022 in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenInfra Summit CFP needs your input: https://openinfra.dev/summit/ ** Service Coordinator nomination time has begun. You have until February 8, 2022 * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220201) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220201) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning our Nodepool images (clarkb 20220201) *** http://lists.opendev.org/pipermail/service-announce/2021-December/000029.ht… cleanup announcement *** Changes to remove CentOS 8 are up. Are we ready to go ahead with this? In order they should be merged: **** https://review.opendev.org/c/openstack/project-config/+/827183 **** https://review.opendev.org/c/opendev/base-jobs/+/827181 **** https://review.opendev.org/c/openstack/project-config/+/827184 **** https://review.opendev.org/c/opendev/system-config/+/827186 ** Spring cleaning for old reviews? (frickler 20220201) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** Do we see value in keeping those or could we run some script to auto-abandon those with a helpful comment? *** Neutron has a script that could be easily adopted for this task **** https://opendev.org/openstack/neutron/src/branch/master/tools/abandon_old_r… *** Clarkb to produce a list of repos that can be retired. We can then retire the repos and abandon their open reviews as step 0 ** Gerrit mergeabilityComputationBehavior (frickler 20220127) *** https://gerrit-documentation.storage.googleapis.com/Documentation/3.5.0/con… *** This was disabled while upgrading to 3.4.0 since the default is "NEVER" *** We could set the option to "API_REF_UPDATED_AND_CHANGE_REINDEX" to restore the previous behavior *** It doesn't look like we have to have any concerns in regard to CPU load http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=70348… ** Gerrit bugs/issues that we are tracking (clarkb 20220201) *** ACLs for pushing signed tags are ignored. Should be fixed in 3.4 now. https://review.opendev.org/c/opendev/system-config/+/827153 will update our images. *** URL substititions for gitweb urls does not properly include the hash which is needed by gitea. https://bugs.chromium.org/p/gerrit/issues/detail?id=15589 *** Backlog of ssh git pulls against Gerrit slows Zuul down. https://bugs.chromium.org/p/gerrit/issues/detail?id=15649 * Open discussion

1 0

Infra Team Meeting Agenda for January 25, 2022
by Clark Boylan 25 Jan '22

25 Jan '22

Hello, We will meet with this agenda on January 25, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenInfra Summit CFP and programming committee need your input: https://openinfra.dev/summit/ ** Service Coordinator nomination time has begun. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220125) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220125) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Buster to Bullseye updates are complete. *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning our Nodepool images (clarkb 20220125) *** http://lists.opendev.org/pipermail/service-announce/2021-December/000029.ht… cleanup announcement *** Plan to remove CentOS 8 from Nodepool and Zuul first thing next week. ** Spring cleaning for old reviews? (frickler 20220125) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** Do we see value in keeping those or could we run some script to auto-abandon those with a helpful comment? *** Neutron has a script that could be easily adopted for this task **** https://opendev.org/openstack/neutron/src/branch/master/tools/abandon_old_r… *** (clarkb) Considering that we had a big shift in approach to config management about 2 years ago I would say abandoning anything prior to that is probably fine. I have a change or two since then that might be good to keep around. Maybe we start at 2 years and see what that looks like? We can always restore changes too. ** Updating Grafana (clarkb 20220125) *** https://review.opendev.org/c/opendev/grafyaml/+/825990 needed to fix api use against newer grafana *** https://review.opendev.org/c/opendev/system-config/+/825410 update to latest grafana * Open discussion

1 0

Team Meeting Agenda for January 18, 2022
by Clark Boylan 18 Jan '22

18 Jan '22

We will meet January 18, 2022 at 19:00 UTC in #opendev-meeting with this agenda: * Announcements ** OpenInfra Summit CFP and programming committee need your input: https://openinfra.dev/summit/ * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220118) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220118) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Buster to Bullseye updates are complete. *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning our Nodepool images (clarkb 20220118) *** http://lists.opendev.org/pipermail/service-announce/2021-December/000029.ht… cleanup announcement *** Tumbleweed has been removed *** CentOS 8 removal process has begun. Gave projects until the end of the month before we remove the nodeset and nodepool configs. ** Scheduling Gerrit Project Renames (clarkb 20220118) *** Target January 24 around 22:00 UTC? ** Spring cleaning for old reviews? (frickler 20220113) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** Do we see value in keeping those or could we run some script to auto-abandon those with a helpful comment? *** Neutron has a script that could be easily adopted for this task **** https://opendev.org/openstack/neutron/src/branch/master/tools/abandon_old_r… * Open discussion

1 0

Re: [service-announce] Changes to OpenDev's stable of CI images
by Clark Boylan 12 Jan '22

12 Jan '22

On Tue, Dec 7, 2021, at 8:38 AM, Clark Boylan wrote: > Hello, > > CentOS 8 reaches end of life on December 31, 2021. This means now is > the time for you to convert jobs to CentOS 8|9 Stream or some other > appropriate platform. The OpenDev team will plan to remove CentOS 8 > images and mirrors from our infrastructure in early January, hopefully > by January 14, 2022. Again, please update your jobs now so that you are > prepared. These changes are necessary as the distro will no longer be > supported, and there is no way for us to ensure that the images will be > secure or that we'll even be able to build them. > > We've also got Fedora 35 lined up to replace Fedora 34. Due to Fedora's > short support window we typically swap out one Fedora release with the > next when the next is ready. If you are using the fedora-34 label > please update it to fedora-35. If you are using fedora-latest we'll be > updating that to use fedora-35 and the swap will happen automatically > for you. We expect to have Fedora 34 cleaned up by the end of the year. > > Additionally both OpenSUSE 15 and Tumbleweed could use some help. For > OpenSUSE 15 we need to update it to 15.3 as well as updating the > associated mirrors if we plan to keep it around. I'm happy to keep this > image around if we can get volunteers in the community that can help > with that. If there isn't interest in modernizing the OpenSUSE 15 > images I'd like to propose we clean them up in January along with the > CentOS 8 images. For Tumbleweed, the images haven't received much use > or attention. I'd like to propose that we simply retire this image and > clean it up. We had thought it might be a good platform for monitoring > upcoming changes in the Linux world, but the reality is that only works > if you've got people active caring for it and adjusting to those > changes. The Tumbleweed cleanup will likely begin before the end of the > year. > > Finally, our Gentoo images bounce between being buildable and broken > with more time spent being broken than not. I'd like to propose that > this image also get cleaned up in the near future. Again, if there is > interest in the community for keeping this alive I think we can make > that happen. But as is this image is not in a sustainable place and it > would be better for us to stop running jobs on it. > > Let us know if you have any questions or comments, To followup on this we have found volunteers for Gentoo and OpenSUSE Leap 15 maintenance. This means those images are not immediately up for removal. However, if those volunteers find they aren't sustainable for one reason or another they may still be removed. OpenSUSE Tumbleweed has been removed and as far as we could tell nothing was using it when we removed it. Shouldn't create any problems. CentOS 8 removal has begun with a number of changes [0] to start removing some of the infrastructure bits (like wheel mirrors and openafs packages) that supported it. I will be reaching out to OpenStack as well as they seem to be the primary users of the centos-8 label. Hopefully we'll be able to remove all of the existing centos-8 jobs cleanly, but if not then we'll wait until a reasonable amount of time and cleanup has passed (a couple of weeks?) before force merging the removal of the nodeset and label from our zuul configs. Fedora 34 cleanup is still in progress, but should pick up again as people return from holiday. [0] https://review.opendev.org/q/topic:%22remove-centos-8%22+status:open

2 1

Infra Team Meeting Agenda for January 11, 2022
by Clark Boylan 11 Jan '22

11 Jan '22

Our first meeting of the year will be held January 11, 2022 at 19:00 UTC in #opendev-meeting. I expect it will still be pretty quiet due to holidays so we might get through it quickly. This is our agenda: == Agenda for next meeting == * Announcements ** OpenInfra Foundation Board Election happening this week. Don't forget to vote. * Actions from last meeting * Specs Review * Topics ** ansible-lint failures (clarkb 20220111) *** An unpinned dep (rich) in old ansible-lint has updated and broken ansible-lint. Need to pin rich or update ansible-lint. *** clarkb has a number of these changes up for review now. topic:ansible-lint ** Improving OpenDev's CD throughput (clarkb 20220111) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220111) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Updating Buster images to Bullseye **** https://review.opendev.org/q/hashtag:%22bullseye-image-update%22+status:open The uwsgi image update in particular could use review. *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning our Nodepool images (clarkb 20220111) *** http://lists.opendev.org/pipermail/service-announce/2021-December/000029.ht… cleanup announcement *** CentOS 8 and OpenSUSE Tumbleweed need cleanup this month. ** Scheduling Gerrit Project Renames (clarkb 20220111) * Open discussion

1 0

No Team Meeting December 28, 2021
by Clark Boylan 28 Dec '21

28 Dec '21

Just a heads up that things have been quiet (as expected) the last week. I think we can go ahead and cancel tomorrow's meeting. See you all in a couple weeks as we return to normal scheduling in the new year. Clark

1 0

Team Meeting Agenda for December 14, 2021
by Clark Boylan 14 Dec '21

14 Dec '21

We will meet with this agenda on December 14, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Meetings Cancelled December 21, 2021 and January 4, 2022 for Holidays ** We may have a meeting on December 28, 2021 depending on who is around. * Actions from last meeting * Specs Review * Topics ** Log4j vulnerability fun (clarkb 20211214) *** Discuss what we've learned about this bug and its impact. ** Improving OpenDev's CD throughput (clarkb 20211214) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? ** Container maintenance (clarkb 20211214) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Updating Buster images to Bullseye *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Adding a lists.openinfra.dev mailman site (fungi 20211214) *** Ansible newlist invocations still struggling *** Stack at https://review.opendev.org/c/opendev/system-config/+/820900/ should correct this. ** Spring cleaning our Nodepool images (clarkb 20211214) *** CentOS 8, Fedora 34, Gentoo, OpenSUSE 15, and OpenSUSE tumbleweed are either EOL, going EOL, or lacking care necessary to keep building up to date images. *** http://lists.opendev.org/pipermail/service-announce/2021-December/000029.ht… cleanup announcement *** Will need volunteers for the work. Probably ok if this proceeds a little slower than advertised. * Open discussion

1 0

Re: [service-announce] Changes to OpenDev's stable of CI images
by Jack Morgan 08 Dec '21

08 Dec '21

Hey Clark, I'd like to help out / volunteer on whatever needs to be done. It doesn't have to be specific to image clean up. I'm not sure how to get started so thought we could meet for coffee but open to other suggestions. On 12/7/21 08:38, Clark Boylan wrote: > Hello, > > CentOS 8 reaches end of life on December 31, 2021. This means now is the time for you to convert jobs to CentOS 8|9 Stream or some other appropriate platform. The OpenDev team will plan to remove CentOS 8 images and mirrors from our infrastructure in early January, hopefully by January 14, 2022. Again, please update your jobs now so that you are prepared. These changes are necessary as the distro will no longer be supported, and there is no way for us to ensure that the images will be secure or that we'll even be able to build them. > > We've also got Fedora 35 lined up to replace Fedora 34. Due to Fedora's short support window we typically swap out one Fedora release with the next when the next is ready. If you are using the fedora-34 label please update it to fedora-35. If you are using fedora-latest we'll be updating that to use fedora-35 and the swap will happen automatically for you. We expect to have Fedora 34 cleaned up by the end of the year. > > Additionally both OpenSUSE 15 and Tumbleweed could use some help. For OpenSUSE 15 we need to update it to 15.3 as well as updating the associated mirrors if we plan to keep it around. I'm happy to keep this image around if we can get volunteers in the community that can help with that. If there isn't interest in modernizing the OpenSUSE 15 images I'd like to propose we clean them up in January along with the CentOS 8 images. For Tumbleweed, the images haven't received much use or attention. I'd like to propose that we simply retire this image and clean it up. We had thought it might be a good platform for monitoring upcoming changes in the Linux world, but the reality is that only works if you've got people active caring for it and adjusting to those changes. The Tumbleweed cleanup will likely begin before the end of the year. > > Finally, our Gentoo images bounce between being buildable and broken with more time spent being broken than not. I'd like to propose that this image also get cleaned up in the near future. Again, if there is interest in the community for keeping this alive I think we can make that happen. But as is this image is not in a sustainable place and it would be better for us to stop running jobs on it. Thanks, -- Jack Morgan

2 1