Hello,
About a year ago Fedora 33 released and gave us a preview of OpenSSH's sha1 + RSA key deprecation fallout. Fedora 33 users noticed they could no longer use SSH RSA keys to connect to our Gerrit at review.opendev.org. This happens because Fedora 33's OpenSSH packaging has deprecated sha1 hashes for RSA, and despite both the client and server supporting rsa-sha2-* variants they couldn't negotiate their use between them. OpenSSH 8.8 released recently and did similar in the upstream software which means users with up to date OpenSSH installations are noticing similar problems (Arch Linux for example).
There are a couple of workarounds that you can use. Probably the best option is to use an ed25519 or ecdsa key with our Gerrit. Modern clients and our Gerrit SSHD negotiate these keys without issue. Less optimal is to manually re-enable the use of the ssh-rsa hash, but we recommend against this as your software providers have decided this is no longer secure enough.
On our end we've brought this up with the MINA SSHD devs [0] with the hope that the SSH implementation that Gerrit uses can be updated to negotiate the sha2 hashes properly. Also, the rsa-sha2 RFC indicates [1] clients may fallback to a sha2 variant instead of the sha1 variant which would workaround MINA's lack of support for negotiation in the protocol. If you are an OpenSSH>=8.8 or Fedora>=33 user you might consider filing bugs against your ssh clients to change the default fallback to a sha2 variant on your platforms.
[0] https://issues.apache.org/jira/browse/SSHD-1141
[1] https://datatracker.ietf.org/doc/html/rfc8332#section-3.3
Hopefully I've put enough keywords in this email that the various search engines will index it, and the next time someone runs into these problems they'll find this explanation.
Clark
We will meet with this agenda on September 27, 2022 at 19:00 UTC in #opendev-meeting. This agenda is a slightly edited version compared to the wiki as I anticipate we'll have less participation this week.
== Agenda for next meeting ==
* Announcements
* Actions from last meeting
* Specs Review
* Topics
** Mailman 3 (clarkb 20220927)
*** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point
*** https://etherpad.opendev.org/p/mm3migration
*** Remaining issue with migration is size of some fields in the db getting exceeded. Can just update old list side.
*** Pipermail redirects need testing on held node.
** Jaeger tracing server (for Zuul) (corvus 20220927)
*** Change has landed.
** Nodepool Builder Disk utilization (clarkb 20220927)
*** Have we expanded disk space on existing builders yet?
* Open discussion
We will meet with this agenda on September 20, 2022 in #opendev-meeting at 19:00 UTC:
== Agenda for next meeting ==
* Announcements
* Actions from last meeting
* Specs Review
* Topics
** Bastion host (ianw 20220920)
*** worth moving ansible/openstacksdk to a venv? system-config jobs first then production
*** c.f. https://review.opendev.org/c/opendev/system-config/+/847700
*** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work?
*** https://review.opendev.org/c/zuul/zuul/+/855309/
*** https://review.opendev.org/c/opendev/system-config/+/855472
** Upgrading Bionic servers to Focal/Jammy (clarkb 20220920)
*** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades
** Mailman 3 (clarkb 20220920)
*** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point
*** https://etherpad.opendev.org/p/mm3migration
*** Remaining issue with migration is size of some fields in the db getting exceeded. Can just update old list side.
*** Pipermail redirects need testing on held node.
** Jaeger tracing server (for Zuul) (corvus 20220920)
*** Change has landed.
** Fedora 36 Rollout (clarkb 20220920)
*** Now not booting reliably.
** Improving Ansible task runtime (clarkb 20220920)
*** Ansible tasks can be slow on some test nodes.
*** https://review.opendev.org/c/opendev/system-config/+/857232
*** Possibly related to Ansible 5 breaking pipelining in ssh connection specific config
*** https://review.opendev.org/c/opendev/system-config/+/857239
** Nodepool Builder Disk utilization (clarkb 20220920)
*** We're using more of the disk now which makes us more prone to filling the disk if something goes wrong.
*** Can we trim any images out?
*** Do we need more or bigger builders?
* Open discussion
Last week, we refreshed our configuration defaults for the Jitsi
Meet services which provide the teleconferencing features of our
meetpad.opendev.org service.
The primary behavior change you're likely to notice is that meeting
room URLs first take you to a pre-join page now. This is a
workaround for some browsers, particularly Firefox, treating the
audio stream as unsolicited and automatically blocking it.
Previously, that was responsible for causing confusing "I can't hear
anyone else" problems for many users.
If you had trouble with sound on our meetpad service before, please
do try it again at your convenience and let us know if it helped (or
more importantly, if it didn't).
--
Jeremy Stanley
Hello,
This agenda is getting sent out a bit late due to the holiday yesterday. We will meet at 19:00 UTC in #opendev-meeting with this agenda today, September 6, 2022:
== Agenda for next meeting ==
* Announcements
* Actions from last meeting
* Specs Review
* Topics
** Bastion host (ianw 20220906)
*** worth moving ansible/openstacksdk to a venv? system-config jobs first then production
*** c.f. https://review.opendev.org/c/opendev/system-config/+/847700
*** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work?
** Upgrading Bionic servers to Focal/Jammy (clarkb 20220906)
*** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades
** Mailman 3 (clarkb 20220906)
*** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point
*** Migration testing looking good. Need to test updated database settings for large attachments (including that db backups function) and redirects for old pipermail archives.
** Jaeger tracing server (for Zuul) (corvus 20220906)
*** Any changes need review yet?
** Fedora 36 Rollout (clarkb 20220906)
*** Status Update
** Jitsi Meet Updates for Meetpad (clarkb 20220906)
*** Configs updated to start with landing page for meetings. Should fix Firefox auto mute of auto playing audio
*** Discovered that JVBs rely on newer colibri websocket system for scaling out. Our extra JVBs need to be updated to function under this system.
** Stability of Zuul reboot playbook (clarkb 20220906)
*** Crashes if unattended-upgrades is running due to conflict over apt lock.
*** Potentially crashes if services were already stopped outside of the playbook run.
* Open discussion