- service-discuss - lists.opendev.org

Team meeting agenda for June 29, 2021
by Clark Boylan 29 Jun '21

29 Jun '21

We will meet on June 29, 2021 at 19:00 UTC in #opendev-meeting on OFTC with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Topics ** Eavesdrop and Limnoria updates (clarkb 20210629) *** Fixed crontab bug for text to html conversion of logs. This may be the (last?) fix we needed for lag in logs. ** Gerrit Account cleanup (clarkb 20210629) *** Need review on more dangerous list. Can disable those accounts then clean them up if they look ok. ** review upgrade (ianw 20210628) *** upgrade checklist ready for review *** https://etherpad.opendev.org/p/gerrit-upgrade-2021 *** date/time for upgrade? 19th july maybe, .au time? ** Server Upgrades (clarkb 20210629) *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Mailman next **** Will use semi recent snapshot as the test platform. ** Draft matrix spec (corvus 20210629) *** https://review.opendev.org/796156 *** Reaching out to ESM ** Scheduling project renames (clarkb 20210629) *** Our playbook(s) that do renames likely need updating since the last gerrit upgrade. **** We can test this with our functional testing of gerrit too. ** Gitea01 backups to one of two servers are unhappy (clarkb 202106029) * Open discussion

1 0

StarlingX Docs promote job failing
by Camp, MaryX 24 Jun '21

24 Jun '21

Hi, Since the security updates to Zuul today, the StarlingX docs promote job is failing on every review. [1] The last successful promote job ran yesterday. [2] Here is the error: The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'targets' The error appears to be in '/var/lib/zuul/builds/e93712cdb47b4b519d2f9f1202a7f2e1/trusted/project_0/opendev.org/opendev/base-jobs/playbooks/docs/promote.yaml': line 51, column 7, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: target_dict: "{{ afs.targets.master }}" - name: Set target directory if not master ^ here Is anyone else affected by this? thanks, Mary Camp Kelly Services Technical Writer | maryx.camp(a)intel.com<mailto:maryx.camp@intel.com> [1] https://review.opendev.org/q/project:starlingx/docs+status:closed [2] https://review.opendev.org/c/starlingx/docs/+/797756

2 2

FLOSS MOOC ML
by Kendall Nelson 22 Jun '21

22 Jun '21

Hello :) I was wondering what the process was for getting a ML setup? Ildiko and I have been attending regular meetings and advising on the development of a MOOC on FLOSS[1] and they were interested in starting an accompanying ML. I mentioned opendev maybe being a place for that and explained that they could be setup as admins to manage the ML, etc. I am all ears about what to do to get started! -Kendall Nelson (diablo_rojo) [1] https://gitlab.com/mooc-floss/mooc-floss

3 3

Team Meeting Agenda for June 15, 2021
by Clark Boylan 15 Jun '21

15 Jun '21

We will meet on June 15, 2021 at 19:00UTC in #opendev-meeting on OFTC with this agenda: == Agenda for next meeting == * Announcements ** clarkb out next week. Will need a volunteer meeting chair or we can skip. * Actions from last meeting * Specs approval * Topics ** Eavesdrop and Limnoria updates (clarkb 20210615) *** Fixed a bug upstream when joining many channels *** Still running a fork to include setuptools updates ** Gerrit Account cleanup (clarkb 20210615) *** Need review on more dangerous list. Can disable those accounts then clean them up if they look ok. ** Server Upgrades (clarkb 20210615) *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Mailman next **** Will use semi recent snapshot as the test platform. *** Review02 getting into shape ** Cleaning up PPC packages in our CentOS 8 mirror (clarkb 20210615) ** Draft matrix spec (corvus 20210615) *** https://review.opendev.org/796156 ** Scheduling project renames (clarkb 20210615) *** Our playbook(s) that do renames likely need updating since the last gerrit upgrade. **** We can test this with our functional testing of gerrit too. * Open discussion

1 0

Team Meeting Agenda for June 8, 2021
by Clark Boylan 08 Jun '21

08 Jun '21

We will meet with this agenda on June 8, 2021 at 19:00 UTC in #opendev-meeting on OFTC: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Topics ** Switch to OFTC now largely complete (clarkb 20210608) *** https://review.opendev.org/q/topic:%22limnoria%22+status:open Limnoria bot rewrite *** https://review.opendev.org/c/openstack/ptgbot/+/793792 Update PTGbot for OFTC *** Is PGTbot python3 capable? We also need to register a nick for it. ** Gerrit Account cleanup (clarkb 20210608) *** Need review on more dangerous list. Can disable those accounts then clean them up if they look ok. ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210608) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Mailman next **** Will use semi recent snapshot as the test platform. ** Cleaning up PPC packages in our CentOS 8 mirror (clarkb 20210608) ** Scheduling project renames (clarkb 20210608) *** Our playbook(s) that do renames likely need updating since the last gerrit upgrade. **** We can test this with our functional testing of gerrit too. * Open discussion

1 0

Team Meeting Agenda for June 1, 2021
by Clark Boylan 01 Jun '21

01 Jun '21

Hello everyone. We will meet at 19:00 UTC on June 1, 2021 (today) in #opendev-meeting on OFTC. Apologies for getting this agenda out late. Yesterday was a holiday and I ended up being distracted. Here is the agenda we will use today: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Topics ** Switch to OFTC now largely complete (clarkb 20210601) *** https://review.opendev.org/q/topic:%22limnoria%22+status:open Limnoria bot rewrite ** Gerrit Account cleanup (clarkb 20210601) *** Need review on more dangerous list. Can disable those accounts then clean them up if they look ok. ** Refreshing non LE certs that expire in just under a month (clarkb 20210601) *** Wiki is the last remaining service needing an update. Clarkb on it. ** Switch Vexxhost to provide only specialized labels in Nodepool (clarkb 20210601) *** https://review.opendev.org/c/openstack/project-config/+/785769 ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210601) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Mailman next **** Need to snapshot the server then perform in place upgrades on a test node based on the snapshot. ** Scheduling project renames (clarkb 20210601) *** Our playbook(s) that do renames likely need updating since the last gerrit upgrade. **** We can test this with our functional testing of gerrit too. * Open discussion

1 0

Reevaluating choice of IRC network for our service bots
by Jeremy Stanley 30 May '21

30 May '21

As most people reading this are likely aware, there has been a recent change in management for the Freenode IRC network. According to public statements[0], the former staff have all left and the network is in entirely new hands. Setting aside arguments about which parties are in the wrong, further stability of the service we provide is once again brought into question. We've seen evidence of (much) lower levels of abuse mitigation over the past few days, with a marked increase in registered accounts engaging in harassment over extended periods of time across our channels without getting banned from the network. Due to disagreements between Freenode and other large IRC networks and frequent related attacks against Freenode's servers, our communities have experienced many extended periods of instability there. In order to insure against a potential collapse of the network, the OpenDev Collaboratory (and the OpenStack Project Infrastructure before it) has for many years maintained a sort of evacuation plan: For the multitude of channels overseen by OpenDev, we also keep equivalent channel registrations on the Open and Free Technology Community (OFTC) IRC network[1], and have communicated with its operators on multiple occasions about their willingness and ability to host our communities there should the need arise. This week's events are, unfortunately, not an isolated incident. The systems administrators for OpenDev have encouraged a move to OFTC on multiple occasions, with the first formal proposal in March 2014[2]. In prior cases, the impact to the community for such a switch outweighed the perceived benefits, but as we've heard growing displeasure with Freenode from representatives of many of the projects we serve, it is time once more to revisit whether we should enact our longstanding evacuation plans. Because the OpenDev Collaboratory exists to serve the projects it hosts, input from project representatives and the Advisory Council is critical in deciding major changes to services. If projects were to move their channels to OFTC, the transition would not be seamless. In particular, differences in authentication (no SASL support, but you can authenticate your connection with a certificate[3] if you don't want to identify to the NickServ after connecting), permissions model (OFTC has coarse-grained RBAC designed to reduce channel mismanagement), and NickServ and ChanServ command syntax are among the challenges they're likely to face. The same IRC nicks may also not be available in some cases, though due to the generally smaller size of OFTC compared to Freenode this hopefully won't come up for too many users. On a positive note, we may be able to go back to not requiring nick registration just to join channels, easing onboarding for newcomers. If there is a consensus to move OpenDev's services to OFTC, or any other IRC network for that matter, this will entail a bit of development effort in order to accommodate the differences mentioned above. Please do note, however, that moving to a network other than OFTC would additionally mean we can't guarantee the availability of the same IRC channel names either, so that needs to be weighed in any decision. Some discussion[4][5][6] is already underway within the OpenStack community as to what they'd prefer, but we haven't heard much from other projects yet, so please do respond with your thoughts on the matter. [0] https://fuchsnet.ch/freenode-resign-letter.txt [1] https://www.oftc.net/ [2] http://lists.openstack.org/pipermail/openstack-dev/2014-March/028783.html [3] https://www.oftc.net/NickServ/CertFP/ [4] http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022468.html [5] http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022539.html [6] http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23openstack-tc.2021… -- Jeremy Stanley

6 14

Team Meeting Agenda for May 25, 2021
by Clark Boylan 25 May '21

25 May '21

We will meet May 25, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account inconsistencies **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** More "dangerous" list has been generated. Should still be safe-ish particularly if we disable the accounts first. *** Configuration tuning **** Reduce the number of ssh threads. Possibly create bot/batch user groups and thread counts as part of this. **** https://groups.google.com/g/repo-discuss/c/BQKxAfXBXuo Upstream conversation with people struggling with similar problems. * General topics ** Refreshing non LE certs that expire in just under a month (clarkb 20210525) *** Shutdown/remove: ask.o.o **** https://review.opendev.org/c/opendev/system-config/+/792789 *** Refresh as yearly cert or try to LE: ethercalc, wiki, translate, storyboard **** https://review.opendev.org/c/opendev/system-config/+/792708 *** Meeting with foundation to discuss: openstackid and openstackid-dev **** They are interested in taking on the hosting but due to timing we probably need to provision a cert for it. Will look into LE for this as well. ** Potentially Migrating away from Freenode to OFTC (clarkb 20210525) *** Freenode is under new management *** Freenode policies are changing *** http://lists.opendev.org/pipermail/service-discuss/2021-May/000236.html ** Switch Vexxhost to provide only specialized labels in Nodepool (clarkb 20210525) ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210525) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul is done. Mailman next **** Need to snapshot the server then perform in place upgrades on a test node based on the snapshot. ** Scheduling project renames (clarkb 20210525) *** Our playbook(s) that do renames likely need updating since the last gerrit upgrade. **** We can test this with our functional testing of gerrit too. * Open discussion

1 0

Updating configuration management for our listservs
by Clark Boylan 19 May '21

19 May '21

Hello everyone, Just a quick update that we are updating the configuration management for our listservs. This email serves as a quick sanity check that everything is continuing to function as expected. If you notice anything odd please let us know. Clark

1 0

Team meeting agenda for May 18, 2021
by Clark Boylan 18 May '21

18 May '21

We will meet with this agenda on May 18, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account inconsistencies **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** More "dangerous" list has been generated. Should still be safe-ish particularly if we disable the accounts first. *** Configuration tuning **** Reduce the number of ssh threads. Possibly create bot/batch user groups and thread counts as part of this. **** https://groups.google.com/g/repo-discuss/c/BQKxAfXBXuo Upstream conversation with people struggling with similar problems. *** Update our base job's nodeset **** https://review.opendev.org/789098 * General topics ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210518) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul is done. Mailman next **** https://review.opendev.org/c/opendev/system-config/+/789622 ** Refreshing non LE certs that expire in just under a month (clarkb 20210518) *** Shutdown/remove: ask.o.o *** Refresh as yearly cert or try to LE: ethercalc, wiki, translate, storyboard *** Meeting with foundation to discuss: openstackid and openstackid-dev ** Cleanup of too small swap setups on newer servers (clarkb 20210518) *** make_swap.sh was updated to accidentally limit swap size to 8MB when we wanted an 8GB limit ** Removing registration requirement from our IRC channels (clarkb 20210518) *** TheJulia asks if we think this is still necessary. *** Looking at the last month or so of eavesdrop logs it seems we had ~1.5 spam attempts in the unregistered channel *** https://review.opendev.org/c/openstack/project-config/+/791818 ** "Toggle CI" in Gerrit web UI (rosmaita 20210518) *** I thought this was on the [https://etherpad.opendev.org/p/gerrit-3.2-post-upgrade-notes 3.2 post-upgrade etherpad] but I can't find it. We used to have this button in the old web interface *** The Cinder team is looking for a solution to Change Log pollution from our 3rd party CI systems **** example: https://review.opendev.org/c/openstack/cinder/+/790796/ *** actual reviewer comments get lost in the Change Log *** the "Only comments" toggle hides all the "Added to cc:" lines, which is nice. It also hides the Zuul "build succeeded/failed" comments ... what do we need to do so this will work for other CI responses? **** The "Only Comments" toggle and the "Zuul Summary" tab are what you should be using. To filter with only comments the CI system needs to set an autogenerated tag on its comments. Zuul does this unconditionally if you set it up to report over http. For the Zuul Summary tab the CI system must have a Full Name in gerrit of "foo name CI" and the result format must match Zuul's. ** Scheduling project renames (clarkb 20210518) *** Our playbook(s) that do renames likely need updating since the last gerrit upgrade. **** We can test this with our functional testing of gerrit too. * Open discussion

1 0