September 2022 - service-discuss

OpenSSH 8.8, RSA keys, and Gerrit
by Clark Boylan 14 Oct '22

14 Oct '22

Hello, About a year ago Fedora 33 released and gave us a preview of OpenSSH's sha1 + RSA key deprecation fallout. Fedora 33 users noticed they could no longer use SSH RSA keys to connect to our Gerrit at review.opendev.org. This happens because Fedora 33's OpenSSH packaging has deprecated sha1 hashes for RSA, and despite both the client and server supporting rsa-sha2-* variants they couldn't negotiate their use between them. OpenSSH 8.8 released recently and did similar in the upstream software which means users with up to date OpenSSH installations are noticing similar problems (Arch Linux for example). There are a couple of workarounds that you can use. Probably the best option is to use an ed25519 or ecdsa key with our Gerrit. Modern clients and our Gerrit SSHD negotiate these keys without issue. Less optimal is to manually re-enable the use of the ssh-rsa hash, but we recommend against this as your software providers have decided this is no longer secure enough. On our end we've brought this up with the MINA SSHD devs [0] with the hope that the SSH implementation that Gerrit uses can be updated to negotiate the sha2 hashes properly. Also, the rsa-sha2 RFC indicates [1] clients may fallback to a sha2 variant instead of the sha1 variant which would workaround MINA's lack of support for negotiation in the protocol. If you are an OpenSSH>=8.8 or Fedora>=33 user you might consider filing bugs against your ssh clients to change the default fallback to a sha2 variant on your platforms. [0] https://issues.apache.org/jira/browse/SSHD-1141 [1] https://datatracker.ietf.org/doc/html/rfc8332#section-3.3 Hopefully I've put enough keywords in this email that the various search engines will index it, and the next time someone runs into these problems they'll find this explanation. Clark

1 1

Team Meeting Agenda for September 27, 2022
by Clark Boylan 27 Sep '22

27 Sep '22

We will meet with this agenda on September 27, 2022 at 19:00 UTC in #opendev-meeting. This agenda is a slightly edited version compared to the wiki as I anticipate we'll have less participation this week. == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Mailman 3 (clarkb 20220927) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** Remaining issue with migration is size of some fields in the db getting exceeded. Can just update old list side. *** Pipermail redirects need testing on held node. ** Jaeger tracing server (for Zuul) (corvus 20220927) *** Change has landed. ** Nodepool Builder Disk utilization (clarkb 20220927) *** Have we expanded disk space on existing builders yet? * Open discussion

1 0

Team Meeting Agenda for September 20, 2022
by Clark Boylan 20 Sep '22

20 Sep '22

We will meet with this agenda on September 20, 2022 in #opendev-meeting at 19:00 UTC: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20220920) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? *** https://review.opendev.org/c/zuul/zuul/+/855309/ *** https://review.opendev.org/c/opendev/system-config/+/855472 ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220920) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220920) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** Remaining issue with migration is size of some fields in the db getting exceeded. Can just update old list side. *** Pipermail redirects need testing on held node. ** Jaeger tracing server (for Zuul) (corvus 20220920) *** Change has landed. ** Fedora 36 Rollout (clarkb 20220920) *** Now not booting reliably. ** Improving Ansible task runtime (clarkb 20220920) *** Ansible tasks can be slow on some test nodes. *** https://review.opendev.org/c/opendev/system-config/+/857232 *** Possibly related to Ansible 5 breaking pipelining in ssh connection specific config *** https://review.opendev.org/c/opendev/system-config/+/857239 ** Nodepool Builder Disk utilization (clarkb 20220920) *** We're using more of the disk now which makes us more prone to filling the disk if something goes wrong. *** Can we trim any images out? *** Do we need more or bigger builders? * Open discussion

1 0

Updates to meetpad's Jitsi Meet configuration
by Jeremy Stanley 19 Sep '22

19 Sep '22

Last week, we refreshed our configuration defaults for the Jitsi Meet services which provide the teleconferencing features of our meetpad.opendev.org service. The primary behavior change you're likely to notice is that meeting room URLs first take you to a pre-join page now. This is a workaround for some browsers, particularly Firefox, treating the audio stream as unsolicited and automatically blocking it. Previously, that was responsible for causing confusing "I can't hear anyone else" problems for many users. If you had trouble with sound on our meetpad service before, please do try it again at your convenience and let us know if it helped (or more importantly, if it didn't). -- Jeremy Stanley

1 1

Team Meeting Agenda for September 13, 2022
by Clark Boylan 13 Sep '22

13 Sep '22

Hello, We will meet with this agenda on September 13, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20220913) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? *** https://review.opendev.org/c/zuul/zuul/+/855309/ *** https://review.opendev.org/c/opendev/system-config/+/855472 ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220913) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220913) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** Bulk migration identified a few things that have since been updated. Need another pass to check. *** Pipermail redirects need testing on held node. ** Jaeger tracing server (for Zuul) (corvus 20220913) *** https://review.opendev.org/c/opendev/system-config/+/855983 Adds ansible and docker deployment for Jaeger ** Fedora 36 Rollout (clarkb 20220913) *** https://review.opendev.org/c/zuul/nodepool/+/853914 Remove fedora 35 testing from nodepool ** Jitsi Meet Updates for Meetpad (clarkb 20220913) *** https://review.opendev.org/c/opendev/system-config/+/856553 Update to use colibri websockets and scale out JVBs *** Not sure if the java keystore in use by the JVBs need to share a CA or otherwise be trusted by something else. ** Stability of Zuul reboot playbook (clarkb 20220913) *** New instability identified caused by race between container shutdown and docker exec for graceful command *** https://review.opendev.org/c/opendev/system-config/+/857209 Fix for this race ** Python container image updates (clarkb 20220913) *** https://review.opendev.org/c/opendev/system-config/+/856537 *** Updates to latest python point releases. *** This pulls in newer glibc allowing us to remove the backport in Zuul images ** Improving Ansible task runtime (clarkb 20220913) *** Ansible tasks can be slow on some test nodes. *** Particularly problematic when loops are used that serially process a large set of items *** https://review.opendev.org/c/zuul/zuul-jobs/+/857228 *** https://review.opendev.org/c/opendev/system-config/+/857232 *** Be on the lookout for problematic tasks and updates to speed them up. * Open discussion

1 0

Team Meeting Agenda for September 6, 2022
by Clark Boylan 06 Sep '22

06 Sep '22

Hello, This agenda is getting sent out a bit late due to the holiday yesterday. We will meet at 19:00 UTC in #opendev-meeting with this agenda today, September 6, 2022: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20220906) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220906) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220906) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** Migration testing looking good. Need to test updated database settings for large attachments (including that db backups function) and redirects for old pipermail archives. ** Jaeger tracing server (for Zuul) (corvus 20220906) *** Any changes need review yet? ** Fedora 36 Rollout (clarkb 20220906) *** Status Update ** Jitsi Meet Updates for Meetpad (clarkb 20220906) *** Configs updated to start with landing page for meetings. Should fix Firefox auto mute of auto playing audio *** Discovered that JVBs rely on newer colibri websocket system for scaling out. Our extra JVBs need to be updated to function under this system. ** Stability of Zuul reboot playbook (clarkb 20220906) *** Crashes if unattended-upgrades is running due to conflict over apt lock. *** Potentially crashes if services were already stopped outside of the playbook run. * Open discussion

1 0