June 2022 - service-discuss - lists.opendev.org

OpenSSH 8.8, RSA keys, and Gerrit
by Clark Boylan 13 Oct '22

13 Oct '22

Hello, About a year ago Fedora 33 released and gave us a preview of OpenSSH's sha1 + RSA key deprecation fallout. Fedora 33 users noticed they could no longer use SSH RSA keys to connect to our Gerrit at review.opendev.org. This happens because Fedora 33's OpenSSH packaging has deprecated sha1 hashes for RSA, and despite both the client and server supporting rsa-sha2-* variants they couldn't negotiate their use between them. OpenSSH 8.8 released recently and did similar in the upstream software which means users with up to date OpenSSH installations are noticing similar problems (Arch Linux for example). There are a couple of workarounds that you can use. Probably the best option is to use an ed25519 or ecdsa key with our Gerrit. Modern clients and our Gerrit SSHD negotiate these keys without issue. Less optimal is to manually re-enable the use of the ssh-rsa hash, but we recommend against this as your software providers have decided this is no longer secure enough. On our end we've brought this up with the MINA SSHD devs [0] with the hope that the SSH implementation that Gerrit uses can be updated to negotiate the sha2 hashes properly. Also, the rsa-sha2 RFC indicates [1] clients may fallback to a sha2 variant instead of the sha1 variant which would workaround MINA's lack of support for negotiation in the protocol. If you are an OpenSSH>=8.8 or Fedora>=33 user you might consider filing bugs against your ssh clients to change the default fallback to a sha2 variant on your platforms. [0] https://issues.apache.org/jira/browse/SSHD-1141 [1] https://datatracker.ietf.org/doc/html/rfc8332#section-3.3 Hopefully I've put enough keywords in this email that the various search engines will index it, and the next time someone runs into these problems they'll find this explanation. Clark

1 1

[all] Dynamic Zuul results table in Gerrit 3
by Radosław Piliszek 04 Aug '22

04 Aug '22

Hello Fellow OpenStack and OpenDev Folks! TL;DR click on [3] and enjoy. I am starting this thread to not hijack the discussion happening on [1]. First of all, I would like to thank gibi (Balazs Gibizer) for hacking a way to get the place to render the table in the first place (pun intended). I have been a long-time-now user of [2]. I have improved and customised it for myself but never really got to share back the changes I made. The new Gerrit obviously broke the whole script so it was of no use to share at that particular state. However, inspired by gibi's work, I decided to finally sit down and fix it to work with Gerrit 3 and here it comes: [3]. Works well on Chrome with Tampermonkey. Not tested others. I hope you will enjoy this little helper (I do). I know the script looks super fugly but it generally boils down to a mix of styles of 3 people and Gerrit having funky UI rendering. Finally, I'd also like to thank hrw (Marcin Juszkiewicz) for linking me to the original Michel's script in 2019. [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-November/019051… [2] https://opendev.org/x/coats/src/commit/444c95738677593dcfed0cfd9667d4c4f0d5… [3] https://gist.github.com/yoctozepto/7ea1271c299d143388b7c1b1802ee75e Kind regards, -yoctozepto

7 11

Team Meeting Agenda for June 28, 2022
by Clark Boylan 27 Jun '22

27 Jun '22

Hello, We will meet with this agenda on June 28, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** clarkb missing July 12 meeting * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220628) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog *** Auto Zuul upgrade and reboots didn't run due to a flock issue. Should be fixed for this next weekend. ** Gerrit 3.5 upgrade (ianw 20220628) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** Upgrade completed. *** Setting a change to the WorkInProgress state seems to unconditionally mark it as having merge conflicts in change listings (but not on the change page itself) *** #link https://review.opendev.org/c/opendev/system-config/+/847035 Removing Gerrit 3.4 and adding Gerrit 3.6 images and testing. ** Improving Grafana management tooling (clarkb 20220628) *** Grafyaml doesn't properly support setting the color thresholds on graphs anymore (this makes failed states show red and happy states show green, we always seen green now) *** https://review.opendev.org/q/topic:grafana-screenshots Improved testing of our dashboards to see that they render as expected. ** Run a custom URL shortener service (frickler 20220628) *** Many people use bit.ly or similar in IRC channel topics and elsewhere *** https://opensource.com/article/18/7/apache-url-shortener shows an easy solution that could be git-based *** Should be easy to with some new DNS record on static.o.o *** Data could be managed in a single file (maybe in project-config) or one file per URL ** Zuul job POST_FAILURES (clarkb 20220628) *** TripleO and OSA are both seeing a higher than usual number of POST_FAILURES *** The cause seems to be post-run timeouts during swift log uploads *** Both OSA and tripleo upload quite a number of log files. It could be related to this, but we're yet to find a root cause due to difficulty of debugging things that don't log properly. ** Bastion host (ianw 20220628) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? * Open discussion

1 0

Team Meeting Agenda for June 21, 2022
by Clark Boylan 20 Jun '22

20 Jun '22

Hello, We will meet with this agenda on June 21, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220621) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog *** https://review.opendev.org/c/opendev/system-config/+/846195 Running Zuul cluster upgrade playbook automatically. ** Container maintenance (clarkb 20220621) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB **** review02's mariadb was upgraded to 10.6 from 10.4 using the process described in https://etherpad.opendev.org/p/gerrit-upgrade-3.5 ** Gerrit 3.5 upgrade (ianw 20220621) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** Upgrade completed. *** Couple of issues discovered **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16018 **** Setting a change to the WorkInProgress state seems to unconditionally mark it as having merge conflicts in change listings (but not on the change page itself) *** Time to look at removing our 3.4 image builds and adding 3.6 image builds with updated upgrade testing. ** Enable webapp on nodepool launchers? (frickler 20220621) *** Would help with checking for image freshness / builds failing *** Also look into pushing stats into grafana? ** Run a custom URL shortener service (frickler 20220621) *** Many people use bit.ly or similar in IRC channel topics and elsewhere *** https://opensource.com/article/18/7/apache-url-shortener shows an easy solution that could be git-based *** Should be easy to with some new DNS record on static.o.o *** Data could be managed in a single file (maybe in project-config) or one file per URL * Open discussion

1 0

Team meeting agenda for June 14, 2022
by Clark Boylan 13 Jun '22

13 Jun '22

We will meet on June 14, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220614) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog *** Running Zuul cluster upgrade playbook automatically. ** Container maintenance (clarkb 20220614) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Gerrit 3.5 upgrade planning (ianw 20220614) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** http://lists.opendev.org/pipermail/service-announce/2022-May/000039.html Scheduled for 20:00 UTC June 19, 2022 ** Zuul changing default Ansible version to v5 soon (clarkb 20220614) *** Now that Zuul supports Ansible v5 the next step is to default to v5 *** Need to send an announcement for the default change in OpenDev happening at the end of June. ** Enable webapp on nodepool launchers? (frickler 20220611) *** Would help with checking for image freshness / builds failing *** Also look into pushing stats into grafana? ** Run a custom URL shortener service (frickler 20220611) *** Many people use bit.ly or similar in IRC channel topics and elsewhere *** https://opensource.com/article/18/7/apache-url-shortener shows an easy solution that could be git-based *** Should be easy to with some new DNS record on static.o.o *** Data could be managed in a single file (maybe in project-config) or one file per URL * Open discussion

1 0