October 2021 - service-discuss

OpenSSH 8.8, RSA keys, and Gerrit
by Clark Boylan 13 Oct '22

13 Oct '22

Hello, About a year ago Fedora 33 released and gave us a preview of OpenSSH's sha1 + RSA key deprecation fallout. Fedora 33 users noticed they could no longer use SSH RSA keys to connect to our Gerrit at review.opendev.org. This happens because Fedora 33's OpenSSH packaging has deprecated sha1 hashes for RSA, and despite both the client and server supporting rsa-sha2-* variants they couldn't negotiate their use between them. OpenSSH 8.8 released recently and did similar in the upstream software which means users with up to date OpenSSH installations are noticing similar problems (Arch Linux for example). There are a couple of workarounds that you can use. Probably the best option is to use an ed25519 or ecdsa key with our Gerrit. Modern clients and our Gerrit SSHD negotiate these keys without issue. Less optimal is to manually re-enable the use of the ssh-rsa hash, but we recommend against this as your software providers have decided this is no longer secure enough. On our end we've brought this up with the MINA SSHD devs [0] with the hope that the SSH implementation that Gerrit uses can be updated to negotiate the sha2 hashes properly. Also, the rsa-sha2 RFC indicates [1] clients may fallback to a sha2 variant instead of the sha1 variant which would workaround MINA's lack of support for negotiation in the protocol. If you are an OpenSSH>=8.8 or Fedora>=33 user you might consider filing bugs against your ssh clients to change the default fallback to a sha2 variant on your platforms. [0] https://issues.apache.org/jira/browse/SSHD-1141 [1] https://datatracker.ietf.org/doc/html/rfc8332#section-3.3 Hopefully I've put enough keywords in this email that the various search engines will index it, and the next time someone runs into these problems they'll find this explanation. Clark

1 1

[all] Dynamic Zuul results table in Gerrit 3
by Radosław Piliszek 04 Aug '22

04 Aug '22

Hello Fellow OpenStack and OpenDev Folks! TL;DR click on [3] and enjoy. I am starting this thread to not hijack the discussion happening on [1]. First of all, I would like to thank gibi (Balazs Gibizer) for hacking a way to get the place to render the table in the first place (pun intended). I have been a long-time-now user of [2]. I have improved and customised it for myself but never really got to share back the changes I made. The new Gerrit obviously broke the whole script so it was of no use to share at that particular state. However, inspired by gibi's work, I decided to finally sit down and fix it to work with Gerrit 3 and here it comes: [3]. Works well on Chrome with Tampermonkey. Not tested others. I hope you will enjoy this little helper (I do). I know the script looks super fugly but it generally boils down to a mix of styles of 3 people and Gerrit having funky UI rendering. Finally, I'd also like to thank hrw (Marcin Juszkiewicz) for linking me to the original Michel's script in 2019. [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-November/019051… [2] https://opendev.org/x/coats/src/commit/444c95738677593dcfed0cfd9667d4c4f0d5… [3] https://gist.github.com/yoctozepto/7ea1271c299d143388b7c1b1802ee75e Kind regards, -yoctozepto

7 11

Team Meeting Agenda for October 26, 2021
by Clark Boylan 25 Oct '21

25 Oct '21

We will meet on October 26, 2021 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review ** Mailman 3 spec https://review.opendev.org/810990 * Topics ** Improving OpenDev's CD throughput (clarkb 20211026) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211026) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Fedora 34 test node booting problems (clarkb 20211026) *** Changes to Fedora's kernel packaging broke Xen *** Not yet sure if that may have also somehow broken OVH and iweb. ** Begin planning for Gerrit 3.4 upgrade (clarkb 20211026) *** Read through release notes and identify areas of concern. **** https://www.gerritcodereview.com/3.4.html *** Do we need to modify system-config-run-review-3.4 or system-config-upgrade-review jobs to test any specific behaviors? *** Probably worth holding a 3.4 test node and doing a skim for any unexpected behaviors. * Open discussion

1 0

Team Meeting Agenda for October 19, 2021
by Clark Boylan 18 Oct '21

18 Oct '21

We will meet October 19, 2021 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** PTG this week. *** OpenDev session Wednesday October 20, 2021 at 14:00 - 16:00 UTC in https://meetpad.opendev.org/oct2021-ptg-opendev *** Zuul session Thursday October 21, 2021 at 14:00 UTC in https://meetpad.opendev.org/zuul-2021-10-21 * Actions from last meeting * Specs Review ** Mailman 3 spec https://review.opendev.org/810990 * Topics ** Improving OpenDev's CD throughput (clarkb 20211019) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211019) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Gerrit project renames (clarkb 20211019) *** Overall went well. *** Possible trouble renaming secrets and doing ZK secrets backups? *** We accidentally updated all Gitea projects. Should we just do that by default: https://review.opendev.org/c/opendev/system-config/+/814443 ? ** Improve zuul restarts (frickler 20211014) *** Docs at https://docs.opendev.org/opendev/system-config/latest/zuul.html#restarting-… need updating **** What to restart (scheduler,web,fingergw?) and how (docker restart vs. docker-compose) **** When to run the re-enqueue **** Collecting debug information *** Don't reenqueue periodic jobs * Open discussion Note the PTG is happening, if there are conflicts we can probably focus on the most important items (zuul restart docs and post project rename stuff) and keep the meeting short.

1 0

Gerrit 3.3 upgrade: dashboards and attention sets
by Ian Wienand 13 Oct '21

13 Oct '21

Hello You may have noticed the recent upgrade of Gerrit to version 3.3 on review.opendev.org. We note two things users should be aware of - Dashboard links If your dashboard uses https://review.opendev.org/#/dashboard/... it may no longer work, despite being the documented way to create dashboards. We are currently unclear if this is a bug or a feature [1] but in the mean time, removing the "/#/" (i.e. just use https://review.opendev.org/dashboard/...) makes things work. - Attention sets You have hopefully seen a new section on the main page "Your Turn". This is the Gerrit "Attention Set" feature [2]. This is an optional feature, but we feel it is worth spending some time with it to see how this can fit into our workflows. One thing to be aware of is that adding a vote by default adds the uploader in the attention set. While this makes sense in some models (particuarly non-gating where you are expected to merge yourself), in our environment usually adding a +1 or +2 with no comments is not something that needs the attention of the uploader -- it's either waiting for more review, or Zuul will merge it after gating tests. At the bottom of the modal box where you are inputting your votes you can easily modify the attention set. Look out for it next time you're reviewing! Consider removing others if your comments do not require new attention. If we pollute other people's attention lists with changes that don't actually need their attention the feature will not be useful. So it is worth us all trying to work with the system and be mindful of who and when we add people (we have raised an issue to discuss our workflow with this feature, but it will not be an instant fix [3]). Thanks! OpenDev Admins [1] https://groups.google.com/g/repo-discuss [2] https://gerrit-review.googlesource.com/Documentation/user-attention-set.html [3] https://bugs.chromium.org/p/gerrit/issues/detail?id=15154

1 0

OpenDev PTG Planning
by Clark Boylan 12 Oct '21

12 Oct '21

Hello, The OpenDev team is participating in next week's PTG by holding office hours during a two hour block of time. Join us October 20, 2021 at 14:00 - 16:00 UTC in https://meetpad.opendev.org/oct2021-ptg-opendev to discuss OpenDev related questions, concerns, services and whatever else you might want to talk about. We do ask that you add your topics to our etherpad, https://etherpad.opendev.org/p/oct2021-ptg-opendev, to help us prepare, prioritize, and do simple scheduling during our block of time. See you there, Clark

1 0

Team Meeting Agenda for October 12, 2021
by Clark Boylan 11 Oct '21

11 Oct '21

We will meet on October 12, 2021 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review ** Prometheus spec https://review.opendev.org/c/opendev/infra-specs/+/804122 *** This appears to have the votes. Plan to merge it Thursday 23:00UTC ish if no objections show up. ** Mailman 3 spec https://review.opendev.org/810990 *** Can use more reviews, but looks good to clarkb. * Topics ** Improving OpenDev's CD throughput (clarkb 20211012) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211012) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Scheduling Gerrit project renames (clarkb 20211012) *** October 15 at 18:00UTC. *** Now is the time to make sure projects have their changes in order and that we are ready. *** Updating the process to update project metadata on renamed projects ** Upgrading to Gerrit 3.3 (ianw 20211012) *** Upgrade happened over the weekend. Went very smoothly *** clarkb has been using hashtag:gerrit-3.3 to track post upgrade changes that we want to land related to the upgrade *** Are we ready to drop the 3.2 images? https://review.opendev.org/c/opendev/system-config/+/813074 *** Next up 3.4! * Open discussion

1 0

Team Meeting Agenda for October 5, 2021
by Clark Boylan 04 Oct '21

04 Oct '21

Hello, we will meet with this agenda on October 5, 2021 at 19:00UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack Release tomorrow * Actions from last meeting * Specs Review ** Prometheus spec https://review.opendev.org/c/opendev/infra-specs/+/804122 *** Need to update to propose running node exporter from their shipped binary. ** Mailman 3 spec https://review.opendev.org/810990 * Topics ** Improving OpenDev's CD throughput (clarkb 20211005) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211005) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Debian Buster to Bullseye Updates (clarkb 20211005) *** https://review.opendev.org/c/opendev/system-config/+/809269 Gitea bullseye update *** https://review.opendev.org/c/opendev/system-config/+/809286 Gerrit bullseye update ** Gitea 1.15.3 Upgrade (clarkb 20211005) *** https://review.opendev.org/c/opendev/system-config/+/803231 ** Upgrading to Gerrit 3.3 (ianw 20211005) *** 3.4 is a much bigger jump and will need more care. Clarkb suggests we get to 3.3 soon then plan 3.4. ** Scheduling Gerrit project renames (clarkb 20211005) *** Penciled in the week of October 11-15 *** Updating the process to update project metadata on renamed projects * Open discussion

1 0