May 2021 - service-discuss - lists.opendev.org

[all] Dynamic Zuul results table in Gerrit 3
by Radosław Piliszek 04 Aug '22

04 Aug '22

Hello Fellow OpenStack and OpenDev Folks! TL;DR click on [3] and enjoy. I am starting this thread to not hijack the discussion happening on [1]. First of all, I would like to thank gibi (Balazs Gibizer) for hacking a way to get the place to render the table in the first place (pun intended). I have been a long-time-now user of [2]. I have improved and customised it for myself but never really got to share back the changes I made. The new Gerrit obviously broke the whole script so it was of no use to share at that particular state. However, inspired by gibi's work, I decided to finally sit down and fix it to work with Gerrit 3 and here it comes: [3]. Works well on Chrome with Tampermonkey. Not tested others. I hope you will enjoy this little helper (I do). I know the script looks super fugly but it generally boils down to a mix of styles of 3 people and Gerrit having funky UI rendering. Finally, I'd also like to thank hrw (Marcin Juszkiewicz) for linking me to the original Michel's script in 2019. [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-November/019051… [2] https://opendev.org/x/coats/src/commit/444c95738677593dcfed0cfd9667d4c4f0d5… [3] https://gist.github.com/yoctozepto/7ea1271c299d143388b7c1b1802ee75e Kind regards, -yoctozepto

7 11

Reevaluating choice of IRC network for our service bots
by Jeremy Stanley 29 May '21

29 May '21

As most people reading this are likely aware, there has been a recent change in management for the Freenode IRC network. According to public statements[0], the former staff have all left and the network is in entirely new hands. Setting aside arguments about which parties are in the wrong, further stability of the service we provide is once again brought into question. We've seen evidence of (much) lower levels of abuse mitigation over the past few days, with a marked increase in registered accounts engaging in harassment over extended periods of time across our channels without getting banned from the network. Due to disagreements between Freenode and other large IRC networks and frequent related attacks against Freenode's servers, our communities have experienced many extended periods of instability there. In order to insure against a potential collapse of the network, the OpenDev Collaboratory (and the OpenStack Project Infrastructure before it) has for many years maintained a sort of evacuation plan: For the multitude of channels overseen by OpenDev, we also keep equivalent channel registrations on the Open and Free Technology Community (OFTC) IRC network[1], and have communicated with its operators on multiple occasions about their willingness and ability to host our communities there should the need arise. This week's events are, unfortunately, not an isolated incident. The systems administrators for OpenDev have encouraged a move to OFTC on multiple occasions, with the first formal proposal in March 2014[2]. In prior cases, the impact to the community for such a switch outweighed the perceived benefits, but as we've heard growing displeasure with Freenode from representatives of many of the projects we serve, it is time once more to revisit whether we should enact our longstanding evacuation plans. Because the OpenDev Collaboratory exists to serve the projects it hosts, input from project representatives and the Advisory Council is critical in deciding major changes to services. If projects were to move their channels to OFTC, the transition would not be seamless. In particular, differences in authentication (no SASL support, but you can authenticate your connection with a certificate[3] if you don't want to identify to the NickServ after connecting), permissions model (OFTC has coarse-grained RBAC designed to reduce channel mismanagement), and NickServ and ChanServ command syntax are among the challenges they're likely to face. The same IRC nicks may also not be available in some cases, though due to the generally smaller size of OFTC compared to Freenode this hopefully won't come up for too many users. On a positive note, we may be able to go back to not requiring nick registration just to join channels, easing onboarding for newcomers. If there is a consensus to move OpenDev's services to OFTC, or any other IRC network for that matter, this will entail a bit of development effort in order to accommodate the differences mentioned above. Please do note, however, that moving to a network other than OFTC would additionally mean we can't guarantee the availability of the same IRC channel names either, so that needs to be weighed in any decision. Some discussion[4][5][6] is already underway within the OpenStack community as to what they'd prefer, but we haven't heard much from other projects yet, so please do respond with your thoughts on the matter. [0] https://fuchsnet.ch/freenode-resign-letter.txt [1] https://www.oftc.net/ [2] http://lists.openstack.org/pipermail/openstack-dev/2014-March/028783.html [3] https://www.oftc.net/NickServ/CertFP/ [4] http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022468.html [5] http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022539.html [6] http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23openstack-tc.2021… -- Jeremy Stanley

6 14

Team Meeting Agenda for May 25, 2021
by Clark Boylan 24 May '21

24 May '21

We will meet May 25, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account inconsistencies **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** More "dangerous" list has been generated. Should still be safe-ish particularly if we disable the accounts first. *** Configuration tuning **** Reduce the number of ssh threads. Possibly create bot/batch user groups and thread counts as part of this. **** https://groups.google.com/g/repo-discuss/c/BQKxAfXBXuo Upstream conversation with people struggling with similar problems. * General topics ** Refreshing non LE certs that expire in just under a month (clarkb 20210525) *** Shutdown/remove: ask.o.o **** https://review.opendev.org/c/opendev/system-config/+/792789 *** Refresh as yearly cert or try to LE: ethercalc, wiki, translate, storyboard **** https://review.opendev.org/c/opendev/system-config/+/792708 *** Meeting with foundation to discuss: openstackid and openstackid-dev **** They are interested in taking on the hosting but due to timing we probably need to provision a cert for it. Will look into LE for this as well. ** Potentially Migrating away from Freenode to OFTC (clarkb 20210525) *** Freenode is under new management *** Freenode policies are changing *** http://lists.opendev.org/pipermail/service-discuss/2021-May/000236.html ** Switch Vexxhost to provide only specialized labels in Nodepool (clarkb 20210525) ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210525) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul is done. Mailman next **** Need to snapshot the server then perform in place upgrades on a test node based on the snapshot. ** Scheduling project renames (clarkb 20210525) *** Our playbook(s) that do renames likely need updating since the last gerrit upgrade. **** We can test this with our functional testing of gerrit too. * Open discussion

1 0

Updating configuration management for our listservs
by Clark Boylan 19 May '21

19 May '21

Hello everyone, Just a quick update that we are updating the configuration management for our listservs. This email serves as a quick sanity check that everything is continuing to function as expected. If you notice anything odd please let us know. Clark

1 0

Team meeting agenda for May 18, 2021
by Clark Boylan 17 May '21

17 May '21

We will meet with this agenda on May 18, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account inconsistencies **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** More "dangerous" list has been generated. Should still be safe-ish particularly if we disable the accounts first. *** Configuration tuning **** Reduce the number of ssh threads. Possibly create bot/batch user groups and thread counts as part of this. **** https://groups.google.com/g/repo-discuss/c/BQKxAfXBXuo Upstream conversation with people struggling with similar problems. *** Update our base job's nodeset **** https://review.opendev.org/789098 * General topics ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210518) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul is done. Mailman next **** https://review.opendev.org/c/opendev/system-config/+/789622 ** Refreshing non LE certs that expire in just under a month (clarkb 20210518) *** Shutdown/remove: ask.o.o *** Refresh as yearly cert or try to LE: ethercalc, wiki, translate, storyboard *** Meeting with foundation to discuss: openstackid and openstackid-dev ** Cleanup of too small swap setups on newer servers (clarkb 20210518) *** make_swap.sh was updated to accidentally limit swap size to 8MB when we wanted an 8GB limit ** Removing registration requirement from our IRC channels (clarkb 20210518) *** TheJulia asks if we think this is still necessary. *** Looking at the last month or so of eavesdrop logs it seems we had ~1.5 spam attempts in the unregistered channel *** https://review.opendev.org/c/openstack/project-config/+/791818 ** "Toggle CI" in Gerrit web UI (rosmaita 20210518) *** I thought this was on the [https://etherpad.opendev.org/p/gerrit-3.2-post-upgrade-notes 3.2 post-upgrade etherpad] but I can't find it. We used to have this button in the old web interface *** The Cinder team is looking for a solution to Change Log pollution from our 3rd party CI systems **** example: https://review.opendev.org/c/openstack/cinder/+/790796/ *** actual reviewer comments get lost in the Change Log *** the "Only comments" toggle hides all the "Added to cc:" lines, which is nice. It also hides the Zuul "build succeeded/failed" comments ... what do we need to do so this will work for other CI responses? **** The "Only Comments" toggle and the "Zuul Summary" tab are what you should be using. To filter with only comments the CI system needs to set an autogenerated tag on its comments. Zuul does this unconditionally if you set it up to report over http. For the Zuul Summary tab the CI system must have a Full Name in gerrit of "foo name CI" and the result format must match Zuul's. ** Scheduling project renames (clarkb 20210518) *** Our playbook(s) that do renames likely need updating since the last gerrit upgrade. **** We can test this with our functional testing of gerrit too. * Open discussion

1 0

Team Meeting on May 11, 2021 Cancelled
by Clark Boylan 10 May '21

10 May '21

Hello everyone, Last week I mentioned that I won't be able to run the meeting tomorrow and others seemed to think skipping it this week would be fine. I have not heard any desire to have a meeting since, so I'll make it official and cancel the meeting now. We will see you next week on May 18, 2021 at 19:00UTC in #opendev-meeting. Clark

1 0

[devstack][infra] POST_FAILURE on export-devstack-journal : Export journal
by Martin Kopec 09 May '21

09 May '21

Hi, one of our jobs (python-tempestconf project) is frequently failing with POST_FAILURE [1] during the following task: export-devstack-journal : Export journal I'm bringing this to a broader audience as we're not sure where exactly the issue might be. Did you encounter a similar issue lately or in the past? [1] https://zuul.opendev.org/t/openstack/builds?job_name=python-tempestconf-tem… Thanks for any advice, -- Martin Kopec

4 10

Re: [service-announce] Switching default nodeset in base job to ubuntu-focal
by Sorin Sbarnea 05 May '21

05 May '21

I support the move as I am aware of several issues that are sorted (or simplified) by using a newer release. It worth mentioning that GHA has a nice way to announce pending upgrade of their “nodeset” upgrades: they include a visible warning on each executed job that use a node that is pending a change, something like: “This job is using ubuntu-latest which will soon be switched from x to y. Use specific labels if you want to avoid such a switch.” They do no mention the exact date of the switch but apparently these warnings to stay on for at least one month. On Tue, 4 May 2021 at 22:49, Jeremy Stanley <fungi(a)yuggoth.org> wrote: > The OpenDev Collaboratory is normally somewhat conservative about > when it changes its Zuul node "default" (that is, the nodeset in its > base job). It's been ubuntu-bionic since at least July of 2019 when > we first moved our base job into its own repository, and > ubuntu-focal nodes have been available in our deployment since > before Ubuntu's 20.04 LTS was officially released more than a year > ago. All that is to say, updating our default is overdue. > > On Tuesday, 2021-05-18 (two weeks from today) we'll approve > https://review.opendev.org/789098 to update the nodeset to > ubuntu-focal for any jobs which don't specify their own or inherit > from another job which does. We've already merged a change which > updates our base-test job to reflect this, so you can test either by > explicitly setting your jobs to use an ubuntu-focal nodeset or to > inherit from base-test (though we don't recommend merging changes > for the latter, just rely on speculative execution to identify > problems which you may encounter when the default changes). > > Please reply to the service-discuss mailing list with any questions > or concerns. You should also feel free to leave comments on the > change linked above. > -- > Jeremy Stanley > _______________________________________________ > service-announce mailing list > service-announce(a)lists.opendev.org > http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce > -- -- /zbr

1 0

Team Meeting Agenda for May 4, 2021
by Clark Boylan 03 May '21

03 May '21

We will meet at 19:00 UTC in #opendev-meeting on May 4, 2021 with this agenda: == Agenda for next meeting == * Announcements ** Clark out during May 11 meeting time. Need a volunteer chair or can cancel that meeting. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account inconsistencies **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** More "dangerous" list has been generated. Should still be safe-ish particularly if we disable the accounts first. *** Configuration tuning **** Reduce the number of ssh threads. Possibly create bot/batch user groups and thread counts as part of this. **** https://groups.google.com/g/repo-discuss/c/BQKxAfXBXuo Upstream conversation with people struggling with similar problems. *** Update our base job's nodeset **** https://review.opendev.org/789098 *** Gerrit global config simplification **** https://review.opendev.org/789383 * General topics ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210504) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Zuul scheduler is last remaining server that needs an upgrade in the zuul cluster. ** openEuler patches (ianw 20210504) *** TC level, or being too officious ** InMotion Cloud reorganization (clarkb 20210504) *** Currently limited by number of IP addresses available *** We could deploy an executor in the cloud then set up test nodes without direct external access *** Any held nodes would need floating IPs attached to them to be accessible ** Removing registration requirement from our IRC channels (clarkb 20210504) *** TheJulia asks if we think this is still necessary. *** Looking at the last month or so of eavesdrop logs it seems we had ~1.5 spam attempts in the unregistered channel ** Switching artifact signing keys from RSA to ECC *** https://review.opendev.org/789062 * Open discussion

1 0