October 2020 - service-discuss

Re: [service-announce] October 20 Gerrit Outage Update
by Mohammed Naser 29 Oct '20

29 Oct '20

Hi everyone, I'm happy to see that things are back in order, however, I *hate* to be that person, but I think there are still some hard questions that we need to answer together transparently. I am especially concerned because of how this affects the workflow of developers overall but also the security measures we have in place, should something like Zuul be targeted instead. I've added some questions in-line below Thanks, Mohammed On Tue, Oct 20, 2020 at 8:33 PM Ian Wienand <iwienand(a)redhat.com> wrote: > > As of this mail, Gerrit access has been restored. Please read on for > important information, especially around change verification. > > Background > ----------- > > On 2020-10-20 at 01:30 a user unexpectedly added a workflow approval > to a change that they were not expected to have access to. At 02:06 > UTC an alert was raised via IRC. Administrators found the account had > added themselves to a core group and made the +W vote. The account > was disabled, and removed from the groups it had added itself to by > 02:55 UTC. Administrators began to analyse the situation and Gerrit > was taken offline at 04:02 UTC to preserve state and allow for > analysis. > > From this time, administrators were working on log collection and > analysis, along with restoring backups for comparison purposes. > > By around 08:45 UTC it was clear that the privilege escalation had > been achieved by gaining control of a Launchpad SSO account with > Gerrit administrator privileges. By this time, we had ruled out > software vulnerabilities. Logs showed the first unauthorized access > of the administrator account in Gerrit on 2020-10-06. Communication > with Launchpad admins agrees with this analysis. We saw one session > opened as the administrator user to StoryBoard on this same day, but > logs show no data was modified or hidden stories viewed. So, just to be clear, someone who had root access to our Gerrit installation had their account compromised which resulted in this (and not something that occurred as a by-product of some other service -- say storyboard -- leaking some sort of information?) I see two issues in this at the moment: - There is no need for us to have anyone with admin powers to Gerrit at all times, we've done enough automation to sustain us and a manual 'circuit breaker' of adding a user *IF* necessary should be put in place. - If the above is not possible, anyone who is part of this group should have 2FA enabled inside Launchpad's SSO. I would very much prefer the first option rather than the second one. If it was an individual's account that was accessed and not a system account, have we audited that there are not other things that might have been accessed such as resources relating to Zuul, other systems and potentially rotating/auditing all our infrastructure? > Analysis has been performed on the Gerrit database and git trees from > October 1st, pre-dating any known unauthorized access. > > Access was restored at around 2020-10-21 00:00 UTC > > Outcomes > ----------- > > The following has been verified: > > The administrator account used has been disabled and credentials > updated > > We have verified that all group and user addition/removals since > Oct 1 are valid. The only invalid additions were made by the > compromised administrator account to add a single user account to > the Administrators group; and then that account added itself to > another known group. > > The account given administrator privilege has been removed from > the groups it added itself to and is disabled. > > There is no evidence of any unauthorized access via methods other > than Gerrit HTTP and Gerrit SSH access. > > No commits have been pushed to git trees bypassing code review. > Every git tree has been compared to the Oct 1 version and all > commits have been correctly inserted via Gerrit changes. I saw this artifact, I have no idea if it was put into consideration, but, food for thought: https://review.opendev.org/#/c/758881/ > The version of Gerrit we use stores HTTP API passwords in > plain-text. We know that a limited number of passwords were > gathered via the HTTP API and it is possible passwords were > gathered via the database. We thus have assumed that all HTTP API > passwords have been disclosed. This password needs to be > explicitly enabled by users, and many users do not have it > enabled. > > Remediation > ----------- > > This leaves us with the following remediation actions: > > Users should double-check their Launchpad recent activity at > https://login.launchpad.net/activity for any suspicious logins. If > found, please notify the OpenDev admins in Freenode #opendev and > Launchpad admins in #launchpad immediately. > > All HTTP API passwords have been cleared. If you push changes via > HTTPS (instead of typical SSH), are a gertty user, or run a CI > system or something else that communicates with the Gerrit HTTP > API, you will need to regenerate a password. > > Any SSH keys added to accounts since 2020-10-01 have been removed. > This affects only a limited number of accounts. This is done in > an abundance of caution, and we do not believe any accounts had > unauthorized SSH keys added > > We should audit all changes for projects since 2020-10-01. > > We have no evidence that any account had its ssh keys compromised, > thus we can rule out any unauthorized changes being uploaded via SSH. > However we can not conclusively rule out that compromised HTTP API > passwords were used to push a change through Gerrit. For example, a > change could be uploaded that looks like it came from a user, or the > API key of a core team member may have been used to approve a change > without authorization. > > Given our extensive analysis we consider it exceedingly unlikely that > this vector was used. We have had no notifications of users seeing > unexpected changes either uploaded by them, or approved by them in > projects they work on. This said, we believe it is important to > inform the community of this very unlikely, but still possible, > vulnerability of the source code. > > To this end, we have prepared a list of all changes from the known > affected period which should be audited for correctness. These are > available at > > https://static.opendev.org/project/opendev.org/gerrit-diffs/ > > Team members should browse these changes and make sure they were > correctly approved in Gerrit. If any change looks suspicious you > should notify OpenDev administrators in Freenode #opendev immediately. > > Further actions > ---------------- > > We are planning the following for the short term future: > > The Opendev administrators will be looking at alternative models > for Gerrit admin account management. > > We are already well into planning and testing a coming upgrade to > a version of Gerrit which does not store plain-text API keys. > > Longer term, we've written a spec for replacing Launchpad SSO as > our authentication provider. > > We thank you for your patience during this trying time, and we look > forward to returning to supporting the community doing what it does > best -- working together to create great things. > Thank you for this. I'd also like to raise the question of moving forward, how to be able to track these things. We had a user that had full root access to our Gerrit installation for ~2 weeks without our knowledge entirely, only uncovered when they did something (that, in the grand scheme of things, was relatively trivial, compared to what could have happened). What can we do to set up the necessary infrastructure to ensure that these things are monitored. OpenDev is considered to be critical infrastructure for this entire community and there's not much that an outsider can do other than the 'keyholders' for the resources. We've historically refused to have any monitoring and now things like this have slipped up, I'm just worried that we have a big looming thing coming up ahead of us that will catch us off guard and we'll be completely unprepared for it... > > _______________________________________________ > service-announce mailing list > service-announce(a)lists.opendev.org > http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce -- Mohammed Naser VEXXHOST, Inc.

4 7

Re: [service-announce] review.opendev.org Gerrit outage and upgrade 15:00UTC November 20 to 01:00UTC November 23, 2020
by Radosław Piliszek 28 Oct '20

28 Oct '20

On Tue, Oct 27, 2020 at 10:17 PM Clark Boylan <cboylan(a)sapwetik.org> wrote: > The OpenDev team is planning a long weekend Gerrit outage on review.opendev.org starting 15:00UTC November 20 and running to 01:00UTC November 23, 2020 in order to upgrade to Gerrit 3.2. <3 Finally! <snip> > * Gerrit 3.2 requires git 2.2.0 or newer to use the Change ID commit hook. This may be a problem for RHEL/CentOS 7 users. I thought it depends on git-review itself rather than a particular Gerrit version? <snip> > How can I help? > Once the upgrade is complete you'll want to confirm the basic functionality you rely on is there. We know there will be differences or missing features. Patience as we figure out how to address those on a new Gerrit installation is much appreciated. If you're interested in hacking on Java and Javascript we'd love help with the plugins necessary to address the known problems. You should be able to build this out locally without any special access. Please let us know if you are interested and we can help you bootstrap. Count me in if there are things to be done, might be able to spare some time for Gerrit. :-) We might want to port the live Zuul status reporter as well. -yoctozepto

2 2

Cancelling the team meeting today
by Clark Boylan 27 Oct '20

27 Oct '20

Hello everyone, We'll cancel the team meeting today since many of us are distracted by the PTG. We'll see you next week! Clark

1 0

Re: Some of your OpenDev Gerrit SSH keys have been deleted
by App Support 22 Oct '20

22 Oct '20

Hi Jeremy, Would the deletion affect the login to https://review.opendev.org/? <https://review.opendev.org/?> Somehow I got an error when trying to login. I am new to OpenStack and just started contributing to documentations. Thank you, Qi Zhang > On Oct 22, 2020, at 2:36 PM, Jeremy Stanley <fungi(a)yuggoth.org> wrote: > > In response to a security breach for the Gerrit Code Review service > at https://review.opendev.org/ I have deleted one or more SSH public > keys uploaded to your account between October 1 and October 19, > 2020. If you added any SSH public keys during that time, you will > need to reupload them in your user preferences before they can be > used to authenticate to the service. For more information on the > breach, see this announcement: > > http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html > > If you have any questions, feel free to reach out to OpenDev's > systems administrators on the service-discuss(a)lists.opendev.org > mailing list or in the #opendev channel on the Freenode IRC network. > -- > Jeremy Stanley, OpenDev Systems Administrator

2 2

Re: [service-announce] October 20 Gerrit Outage
by Jonathan Bryce 20 Oct '20

20 Oct '20

Thanks Clark and the rest of the OpenDev infra crew for literally working around the clock on this issue! Appreciate the effort to verify everything. Also wanted to share that the updates are being posted here if people want to see the history for completeness: https://review.opendev.org/maintenance.html Jonathan On October 20, 2020 12:50:32 "Clark Boylan" <cboylan(a)sapwetik.org> wrote: > Hello everyone, > > By now most of you have probably noticed that we took Gerrit offline > recently. The reason for that is we believe an admin account in Gerrit was > compromised allowing an attacker to escalate privileges within Gerrit. > > Around 02:00 UTC October 20 suspicious review activity was noticed, and we > were made aware of it shortly afterwards. The involved account was disabled > and removed from privileged Gerrit groups. After further investigation we > decided that we needed to stop the service, this happened at about 04:00 UTC. > > After the service was stopped we shifted focus to identifying the source of > the issue as well as investigating impact. We believe this originated on > October 6th with at least two compromised Ubuntu One accounts. One of which > was a Gerrit admin account. These accounts, like the one that initially > tipped us off, have been dealt with at this point. > > In order to evaluate impact we are using backups from October 1 to find > configuration, database, and git repo changes that have been made. We have > identified 97 accounts that updated ssh keys after that point in time. > These ssh keys are being removed as we can't be sure the changes were valid > changes made by the user. If you are one of these users you will need to > add your key(s) back in. We will also attempt to reach out to the affected > users directly by email soon. We will be checking openid urls and group > membership changes as well. We will determine what actions make sense for > these items once we have evaluated the impact to them. > > All Gerrit HTTP API tokens will be deleted. You will need to generate new > ones if you are an API user. Sorry, gertty fans. > > On the git repo side of things there are a few things that we will need to > check. Using our October 1 state we will generate lists of commits that > have landed since then for each branch on each repo. We will verify that > the latest commit can reach the last known good commit in the git DAG. For > non merge commits we will also correlate these to Gerrit changes. We will > then ask that you help us by verifying the commits on your projects are as > reviewed and not malicious. We will also need to check git tags which > should all be signed and can be verified that way. > > This is a good reminder to check activity on your online accounts and > identities for anything unexpected. > > We understand that an inaccessible Gerrit is not fun. We are trying to go > as quickly as we can while also not sacrificing caution and care. > > Clark > > _______________________________________________ > service-announce mailing list > service-announce(a)lists.opendev.org > http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce

1 0

Cancelling the team meeting October 20, 2020
by Clark Boylan 19 Oct '20

19 Oct '20

Hello everyone, I've been swamped with summit things. Initially I thought that we could do a quieter meeting during the summit, but I've been too distracted to even look at an agenda. For that reason I think we should cancel the meeting this week. We've got the PTG next week. Please register and check the schedule if you plan to attend. We'll see you there as well as at next week's meeting (I hope). Clark

1 0

Meeting Agenda for October 13, 2020
by Clark Boylan 12 Oct '20

12 Oct '20

We will meet in #opendev-meeting at 19:00 UTC on October 13, 2020 with this agenda: == Agenda for next meeting == * Announcements ** OpenStack Release October 14 ** Summit next week. PTG the week after. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Preparing to upgrade Gerrit from 2.13 to 3.2 **** review-test.opendev.org is an upgraded snapshot of production from October 1. Please check it out **** Basic functionality seems to be working ***** logging in, git review -s, git review to push, commenting on changes, ICLA signing, replication, change searching, and so on. **** jeepyb bug/spec update hooks and the welcome message hook rely on database access and will need to be updated or sunsetted **** Upgrade Process ***** Backup then upgrade from 2.13 to 2.16. This is our fallback midpoint checkpoint ***** Backup again then migrate to notedb on 2.16 ***** Upgrade to 3.2 ***** Upgrade to 2.16 along with backups should be doable in a day. Then notedb migration can happen overnight with 3.2 upgrade happening on day two. **** Unknowns ***** Storyboard integration **** Can we start talking about scheduling the outage and upgrade? *** Luca has offered to do a conference call with us. Let me know if interested and I'll include you for scheduling if/when that happens. * General topics ** PTG PLanning (clarkb 20200929) *** October PTG registration is now open: https://www.openstack.org/ptg/ *** OpenDev planning stats here: https://etherpad.opendev.org/opendev-ptg-planning-oct-2020 ** Bup and Borg Backups (clarkb 20200929) *** Ethercalc to be the first borg backed up service ** Splitting puppet else into specific infra-prod jobs (clarkb 20200929) *** Should be mostly mechanical *** Does it make sense to try and sprint this? Have several people work on getting it done in a short period of time? ** Trusty Upgrade Progress (clarkb 20200929) *** Wiki updates * Open discussion

1 0

Testing the results of Gerrit upgrades on review-test.opendev.org
by Clark Boylan 09 Oct '20

09 Oct '20

Hello everyone, Recently fungi and I managed to make review-test.opendev.org a copy of production Gerrit (review.opendev.org) as of October 1. We have since upgraded that installation to Gerrit 2.16. The 2.16 version is a potential upgrade target for our production Gerrit as we have to stop there and do a required notedb migration step. What isn't yet clear is if we'll expose the 2.16 installation or continue on to upgrade to a 3.x version. In order to figure that out we'll be doing the notedb migration and the 3.x upgrades in the near future. Once we've gone through the process we should have a good idea of what makes sense as far as upgrades go for us. In any case the server is up for testing now if people want to take a look at 2.16 and offer feedback. I'll followup on this thread when we get upgraded to 3.x and ask for feedback on that version too. Head to https://review-test.opendev.org and check it out. There are some minor known issues. The biggest that people will probably notice is that our hacky javascript CI results table does not work in the old gerrit UI or the new polygerrit UI. The commentlinks for zuul results are also not working on polygerrit UI. We're still investigating, but we're going to be careful that we don't worry about every single issue like that pre upgrade otherwise we'll never get the upgrade done. Instead we'll gather feedback and prioritize then sort out what needs fixing now and what we can sort out later. Let us know what you think, Clark

1 1

Team Meeting Agenda for October 6, 2020
by Clark Boylan 05 Oct '20

05 Oct '20

We will meet at 19:00 UTC in #opendev-meeting on October 6, 2020 with this agenda: == Agenda for next meeting == * Announcements ** OpenStack Release next week. ** Rax hosted db outages around 03:00-05:00 UTC Friday including those for review and grafana * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Where is review-test is now a mimic of production Gerrit as of October 1 **** Testing of the upgrade process to begin shortly. *** Luca has offered to do a conference call with us. Let me know if interested and I'll include you for scheduling if/when that happens. * General topics ** PTG PLanning (clarkb 20200929) *** October PTG registration is now open: https://www.openstack.org/ptg/ *** OpenDev planning stats here: https://etherpad.opendev.org/opendev-ptg-planning-oct-2020 ** Rehoming tarballs (ianw 20200929) *** large selection of things in tarballs.openstack.org/openstack/... that should be under tenant dirs opendev/ x/ zuul/ etc. *** https://review.opendev.org/#/c/754257/ -- script to generate script that generates a script to cleanup *** http://paste.openstack.org/show/798368/ -- resulting moves *** symlink old directories? apache redirects (files gone on afs)? just move but notifiy lists? ** Splitting puppet else into specific infra-prod jobs (clarkb 20200929) *** Should be mostly mechanical *** Does it make sense to try and sprint this? Have several people work on getting it done in a short period of time? ** Bup and Borg Backups (clarkb 20200929) *** https://review.opendev.org/741366 is ready to land when we are ready. ** Trusty Upgrade Progress (clarkb 20200929) *** Wiki updates * Open discussion

1 0