Thanks Clark and the rest of the OpenDev infra crew for literally working
around the clock on this issue! Appreciate the effort to verify everything.
Also wanted to share that the updates are being posted here if people want
to see the history for completeness:
https://review.opendev.org/maintenance.html
Jonathan
On October 20, 2020 12:50:32 "Clark Boylan" <cboylan@sapwetik.org> wrote:
Hello everyone,
By now most of you have probably noticed that we took Gerrit offline
recently. The reason for that is we believe an admin account in Gerrit was
compromised allowing an attacker to escalate privileges within Gerrit.
Around 02:00 UTC October 20 suspicious review activity was noticed, and we
were made aware of it shortly afterwards. The involved account was disabled
and removed from privileged Gerrit groups. After further investigation we
decided that we needed to stop the service, this happened at about 04:00 UTC.
After the service was stopped we shifted focus to identifying the source of
the issue as well as investigating impact. We believe this originated on
October 6th with at least two compromised Ubuntu One accounts. One of which
was a Gerrit admin account. These accounts, like the one that initially
tipped us off, have been dealt with at this point.
In order to evaluate impact we are using backups from October 1 to find
configuration, database, and git repo changes that have been made. We have
identified 97 accounts that updated ssh keys after that point in time.
These ssh keys are being removed as we can't be sure the changes were valid
changes made by the user. If you are one of these users you will need to
add your key(s) back in. We will also attempt to reach out to the affected
users directly by email soon. We will be checking openid urls and group
membership changes as well. We will determine what actions make sense for
these items once we have evaluated the impact to them.
All Gerrit HTTP API tokens will be deleted. You will need to generate new
ones if you are an API user. Sorry, gertty fans.
On the git repo side of things there are a few things that we will need to
check. Using our October 1 state we will generate lists of commits that
have landed since then for each branch on each repo. We will verify that
the latest commit can reach the last known good commit in the git DAG. For
non merge commits we will also correlate these to Gerrit changes. We will
then ask that you help us by verifying the commits on your projects are as
reviewed and not malicious. We will also need to check git tags which
should all be signed and can be verified that way.
This is a good reminder to check activity on your online accounts and
identities for anything unexpected.
We understand that an inaccessible Gerrit is not fun. We are trying to go
as quickly as we can while also not sacrificing caution and care.
Clark
_______________________________________________
service-announce mailing list
service-announce@lists.opendev.org
http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce