[service-announce] October 20 Gerrit Outage

[Jonathan Bryce]

Thanks Clark and the rest of the OpenDev infra crew for literally working 
around the clock on this issue! Appreciate the effort to verify everything. 
Also wanted to share that the updates are being posted here if people want 
to see the history for completeness: 
https://review.opendev.org/maintenance.html

Jonathan


On October 20, 2020 12:50:32 "Clark Boylan" <cboylan at sapwetik.org> wrote:

> Hello everyone,
>
> By now most of you have probably noticed that we took Gerrit offline 
> recently. The reason for that is we believe an admin account in Gerrit was 
> compromised allowing an attacker to escalate privileges within Gerrit.
>
> Around 02:00 UTC October 20 suspicious review activity was noticed, and we 
> were made aware of it shortly afterwards. The involved account was disabled 
> and removed from privileged Gerrit groups. After further investigation we 
> decided that we needed to stop the service, this happened at about 04:00 UTC.
>
> After the service was stopped we shifted focus to identifying the source of 
> the issue as well as investigating impact. We believe this originated on 
> October 6th with at least two compromised Ubuntu One accounts. One of which 
> was a Gerrit admin account. These accounts, like the one that initially 
> tipped us off, have been dealt with at this point.
>
> In order to evaluate impact we are using backups from October 1 to find 
> configuration, database, and git repo changes that have been made. We have 
> identified 97 accounts that updated ssh keys after that point in time. 
> These ssh keys are being removed as we can't be sure the changes were valid 
> changes made by the user. If you are one of these users you will need to 
> add your key(s) back in. We will also attempt to reach out to the affected 
> users directly by email soon. We will be checking openid urls and group 
> membership changes as well. We will determine what actions make sense for 
> these items once we have evaluated the impact to them.
>
> All Gerrit HTTP API tokens will be deleted. You will need to generate new 
> ones if you are an API user. Sorry, gertty fans.
>
> On the git repo side of things there are a few things that we will need to 
> check. Using our October 1 state we will generate lists of commits that 
> have landed since then for each branch on each repo. We will verify that 
> the latest commit can reach the last known good commit in the git DAG. For 
> non merge commits we will also correlate these to Gerrit changes.  We will 
> then ask that you help us by verifying the commits on your projects are as 
> reviewed and not malicious. We will also need to check git tags which 
> should all be signed and can be verified that way.
>
> This is a good reminder to check activity on your online accounts and 
> identities for anything unexpected.
>
> We understand that an inaccessible Gerrit is not fun. We are trying to go 
> as quickly as we can while also not sacrificing caution and care.
>
> Clark
>
> _______________________________________________
> service-announce mailing list
> service-announce at lists.opendev.org
> http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opendev.org/pipermail/service-discuss/attachments/20201020/8300141b/attachment.html>