[service-announce] October 20 Gerrit Outage

[Clark Boylan]

Hello everyone,

By now most of you have probably noticed that we took Gerrit offline recently. The reason for that is we believe an admin account in Gerrit was compromised allowing an attacker to escalate privileges within Gerrit.

Around 02:00 UTC October 20 suspicious review activity was noticed, and we were made aware of it shortly afterwards. The involved account was disabled and removed from privileged Gerrit groups. After further investigation we decided that we needed to stop the service, this happened at about 04:00 UTC.

After the service was stopped we shifted focus to identifying the source of the issue as well as investigating impact. We believe this originated on October 6th with at least two compromised Ubuntu One accounts. One of which was a Gerrit admin account. These accounts, like the one that initially tipped us off, have been dealt with at this point.

In order to evaluate impact we are using backups from October 1 to find configuration, database, and git repo changes that have been made. We have identified 97 accounts that updated ssh keys after that point in time. These ssh keys are being removed as we can't be sure the changes were valid changes made by the user. If you are one of these users you will need to add your key(s) back in. We will also attempt to reach out to the affected users directly by email soon. We will be checking openid urls and group membership changes as well. We will determine what actions make sense for these items once we have evaluated the impact to them.

All Gerrit HTTP API tokens will be deleted. You will need to generate new ones if you are an API user. Sorry, gertty fans.

On the git repo side of things there are a few things that we will need to check. Using our October 1 state we will generate lists of commits that have landed since then for each branch on each repo. We will verify that the latest commit can reach the last known good commit in the git DAG. For non merge commits we will also correlate these to Gerrit changes.  We will then ask that you help us by verifying the commits on your projects are as reviewed and not malicious. We will also need to check git tags which should all be signed and can be verified that way.

This is a good reminder to check activity on your online accounts and identities for anything unexpected.

We understand that an inaccessible Gerrit is not fun. We are trying to go as quickly as we can while also not sacrificing caution and care.

Clark