August 2022 - service-discuss - lists.opendev.org

OpenSSH 8.8, RSA keys, and Gerrit
by Clark Boylan 13 Oct '22

13 Oct '22

Hello, About a year ago Fedora 33 released and gave us a preview of OpenSSH's sha1 + RSA key deprecation fallout. Fedora 33 users noticed they could no longer use SSH RSA keys to connect to our Gerrit at review.opendev.org. This happens because Fedora 33's OpenSSH packaging has deprecated sha1 hashes for RSA, and despite both the client and server supporting rsa-sha2-* variants they couldn't negotiate their use between them. OpenSSH 8.8 released recently and did similar in the upstream software which means users with up to date OpenSSH installations are noticing similar problems (Arch Linux for example). There are a couple of workarounds that you can use. Probably the best option is to use an ed25519 or ecdsa key with our Gerrit. Modern clients and our Gerrit SSHD negotiate these keys without issue. Less optimal is to manually re-enable the use of the ssh-rsa hash, but we recommend against this as your software providers have decided this is no longer secure enough. On our end we've brought this up with the MINA SSHD devs [0] with the hope that the SSH implementation that Gerrit uses can be updated to negotiate the sha2 hashes properly. Also, the rsa-sha2 RFC indicates [1] clients may fallback to a sha2 variant instead of the sha1 variant which would workaround MINA's lack of support for negotiation in the protocol. If you are an OpenSSH>=8.8 or Fedora>=33 user you might consider filing bugs against your ssh clients to change the default fallback to a sha2 variant on your platforms. [0] https://issues.apache.org/jira/browse/SSHD-1141 [1] https://datatracker.ietf.org/doc/html/rfc8332#section-3.3 Hopefully I've put enough keywords in this email that the various search engines will index it, and the next time someone runs into these problems they'll find this explanation. Clark

1 1

Team Meeting Agenda for August 30, 2022
by Clark Boylan 29 Aug '22

29 Aug '22

We will meet with this agenda on August 30, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack feature freeze this week. * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20220830) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220830) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220830) *** https://review.opendev.org/c/opendev/system-config/+/851248 Still very much WIP but worth a look at this point *** Working through migration testing now. ** Gerrit 08:00 UTC load issues (clarkb 20220830) *** Bumping up the http thread count *** Fallout of blocking IPs (users noticed) *** We should work through this slowly as the OpenStack release is approaching. ** Jaeger tracing server (for Zuul) (corvus 20220830) *** Zuul will be adding OpenTelemetry tracing support. Should OpenDev run a Jaeger tracing server? *** See https://zuul-ci.org/docs/zuul/latest/developer/specs/tracing.html ** Fedora 36 Rollout (clarkb 20220830) *** Do we need to wait for OpenStack to release first? *** We'll update the nodeset and most jobs will switch automatically. *** Will we remove Fedora 35 next? * Open discussion

1 0

Team Meeting Agenda for August 23, 2022
by Clark Boylan 22 Aug '22

22 Aug '22

Hello, We will meet with this agenda on August 23, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Clark appears to have been the only Service Coordinator nominee. ** OpenStack release is fast approaching. https://releases.openstack.org/zed/schedule.html feature freeze September 2 ish. * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20220823) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220823) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220823) *** https://review.opendev.org/c/opendev/system-config/+/851248 Still very much WIP but worth a look at this point *** The test deployment is largely there now. The next step is testing data migration from mm2 to mm3. ** Gitea 1.17 upgrade (clarkb 20220823) *** https://review.opendev.org/c/opendev/system-config/+/847204 *** This should be ready to go now that 1.17.1 has been released. Just need to decide on a good window of time to deploy it. ** Gerrit 08:00 UTC load issues (clarkb 20220823) *** Bumping up the http thread count *** Fallout of blocking IPs (users noticed) *** We should work through this slowly as the OpenStack release is approaching. ** Jaeger tracing server (for Zuul) (corvus 20220823) *** Zuul will be adding OpenTelemetry tracing support. Should OpenDev run a Jaeger tracing server? *** See https://zuul-ci.org/docs/zuul/latest/developer/specs/tracing.html *** Implementation should be straightforward; does this warrant an OpenDev spec, or should I just propose the change? * Open discussion

1 0

Self Nomination for OpenDev Service Coordinator
by Clark Boylan 17 Aug '22

17 Aug '22

Hello, I am submitting my nomination for OpenDev Service Coordinator. I plan to continue to focus on building sustainable services with the current resources we have. Upgrading servers to Jammy and replacing old services like mailman 2 are high on my todo list. I have been doing this for a while and don't have a lot of new things to say. I think it is important that these community services/resources continue to be accessible to the community, and that requires ensuring they are sustainable. If you have been happy with the last six months then you can expect more of the same. I am still happy for someone else to volunteer as well. You have a few more hours if anyone else would like to give it a go. I think having a group that can share these responsibilities is a good thing. But I am happy to continue as well. Clark

2 1

Team Meeting Agenda for August 16, 2022
by Clark Boylan 15 Aug '22

15 Aug '22

We will meet August 16, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenDev Service Coordinator Nomination period happening now until August 16, 2022 * Actions from last meeting * Specs Review * Topics ** Improving Grafana management tooling (clarkb 20220816) *** https://lists.opendev.org/pipermail/service-discuss/2022-July/000342.html *** https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/851955 Last remaining cleanup change ** Bastion host (ianw 20220816) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220816) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220816) *** https://review.opendev.org/c/opendev/system-config/+/851248 Still very much WIP but worth a look at this point *** The test deployment is largely there now. The next step is testing data migration from mm2 to mm3. ** Gitea 1.17 upgrade (clarkb 20220816) *** https://review.opendev.org/c/opendev/system-config/+/847204 *** https://github.com/go-gitea/gitea/milestone/122 1.17.1 Milestone is in progress * Open discussion

1 0

Team Meeting Agenda for August 9, 2022
by Clark Boylan 08 Aug '22

08 Aug '22

We will meet on August 9, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenDev Service Coordinator Nomination period happening now until August 16, 2022 * Actions from last meeting * Specs Review * Topics ** Improving Grafana management tooling (clarkb 20220809) *** https://lists.opendev.org/pipermail/service-discuss/2022-July/000342.html *** https://review.opendev.org/q/topic:grafana-json ** Bastion host (ianw 20220809) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220809) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220809) *** https://review.opendev.org/c/opendev/system-config/+/851248 Still very much WIP but worth a look at this point *** The service deployment is just about all there. Need to figure out configuring the service and lists properly now. ** Gitea 1.17 upgrade (clarkb 20220809) *** https://review.opendev.org/c/opendev/system-config/+/847204 *** https://github.com/go-gitea/gitea/milestone/122 1.17.1 Milestone is in progress * Open discussion

1 0

[all] Dynamic Zuul results table in Gerrit 3
by Radosław Piliszek 04 Aug '22

04 Aug '22

Hello Fellow OpenStack and OpenDev Folks! TL;DR click on [3] and enjoy. I am starting this thread to not hijack the discussion happening on [1]. First of all, I would like to thank gibi (Balazs Gibizer) for hacking a way to get the place to render the table in the first place (pun intended). I have been a long-time-now user of [2]. I have improved and customised it for myself but never really got to share back the changes I made. The new Gerrit obviously broke the whole script so it was of no use to share at that particular state. However, inspired by gibi's work, I decided to finally sit down and fix it to work with Gerrit 3 and here it comes: [3]. Works well on Chrome with Tampermonkey. Not tested others. I hope you will enjoy this little helper (I do). I know the script looks super fugly but it generally boils down to a mix of styles of 3 people and Gerrit having funky UI rendering. Finally, I'd also like to thank hrw (Marcin Juszkiewicz) for linking me to the original Michel's script in 2019. [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-November/019051… [2] https://opendev.org/x/coats/src/commit/444c95738677593dcfed0cfd9667d4c4f0d5… [3] https://gist.github.com/yoctozepto/7ea1271c299d143388b7c1b1802ee75e Kind regards, -yoctozepto

7 11

Service Coordinator Election August 2022 Edition
by Clark Boylan 02 Aug '22

02 Aug '22

It is almost that time again. Back in February I said that we'd have a Service Coordinator Election nomination period that runs from August 2, 2022 to August 16, 2022 [0]. If you'd like to run (I'm more than happy for someone else to do it) now is the time to start thinking about that. I'm giving everyone a head up a week in advance today, and will send a notice on August 2nd that the nomination period is beginning. If you'd like to take this on you can send your nomination as a reply to this email thread or in a new email thread to the service-discuss list. [0] https://lists.opendev.org/pipermail/service-discuss/2022-February/000318.ht… Clark

1 1

Infra Team Meeting Agenda for August 2, 2022
by Clark Boylan 01 Aug '22

01 Aug '22

We will meet with this agenda on August 2, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenDev Service Coordinator Nomination time begins now (August 2, 2022) and ends August 16, 2022 * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220802) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Improving Grafana management tooling (clarkb 20220802) *** https://lists.opendev.org/pipermail/service-discuss/2022-July/000342.html *** https://review.opendev.org/q/topic:grafana-json ** Bastion host (ianw 20220802) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220802) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/c/opendev/system-config/+/851094/2 Run system-config-run-base against Jammy *** https://review.opendev.org/c/opendev/system-config/+/851266/1 Fix install-docker role for Jammy *** https://review.opendev.org/c/opendev/system-config/+/851248 WIP change to run mailman3 on Jammy ** Gitea 1.17 upgrade (clarkb 20220802) *** https://review.opendev.org/c/opendev/system-config/+/847204 ** Rocky images no longer booting (clarkb 20220802) *** Lack of machine ID broke installation of kernels to /boot which broke our Grub config *** https://review.opendev.org/c/openstack/diskimage-builder/+/851687 * Open discussion

1 0