- service-discuss - lists.opendev.org

Re: Opendev review crawler
by Chouchen, Moataz 04 Nov '20

04 Nov '20

Hi Jeremy, Thank you for your quick answer and for your support we will be more than happy to share our research with you once published! For this purpose of study, we need to collect the whole Opendev data. Right now, we collected data from 2020 to 2015 and still need the other years. As per requested we will add more waiting time to my process to let the server catch up. If you also have any other suggestions that will help, we will be happy to try it. We will be happy to receive an answer from you Kind regards, Moataz ________________________________ From: Jeremy Stanley Sent: Tuesday, November 3, 2020 10:42 AM To: service-discuss(a)lists.opendev.org Cc: Chouchen, Moataz; Ouni, Ali; Laurin, François Subject: Re: Opendev review crawler On 2020-11-03 17:43:53 +0000 (+0000), Chouchen, Moataz wrote: > I am Moataz Chouchen a Phd Student in Ets Montréal under the > supervision of Professor Ali Ouni. I am sending this email to you > to explain my requests of the Opendev data to respond to your > request regarding Opendev code review data crawling. Thanks so much for getting back to us! I'm sorry we ended up blocking access to your various systems, but we were at a loss for how to otherwise get in contact with whoever was running them since the queries were being performed anonymously (at least until we finally saw similar API requests originating from your university). > In fact, my Phd subject consists of the identification, the > localization, and the understanding of Modern Code Review (MCR) > problems (also called antipatterns). For this purpose, we need to > crawl code review data from different projects (including Opendev) > since this data will help us to study the process of MCR in these > projects and their associated anti-patterns. Specifically, we need > the whole data of Opendev to perform data analysis methods on them > (including code review metrics distributions, social graph > analysis etc) and study them in more depth. For this reason, we > created a script to crawl data from Opendev and use them in our > empirical study on MCR antipatterns in Opendev. This sounds really interesting, and I'm sure this mailing list would also love to receive a link to your research once you've completed and published it. I'm thrilled you have an interest in looking into these topics, especially with regards to the usage patterns and trends for a service we're operating. > I would like to thank you for your interest to understand the > issues behind our requests and your intention to solve the issue > for both sides. I am looking forward to collaborating with you and > I would like to know if there are any rules that I should follow > when requesting data from your servers and I will be more than > happy If you can help me crawling the data and provide an access > for me. I am also at your disposal for further clarifications > regarding the requests. I hope I hear from you soon. Mainly what I think we need to know is whether you've already collected the data you need from our Gerrit API, or how much you still need to query. The server wants to cache requests in memory for the sake of efficiency, but as you're aware we've got quite a lot of data. When queries for older data consume available memory on the server it begins to spend a lot more time trying to aggressively garbage collect cache contents, and that degrades its ability to serve requests for you and for other users. If you still have a lot left to query, you might try to introduce a bit of a pause between each batch of paginated results to give the server some time to catch up freeing memory for caching other requests. Also if obtaining the data in a different form would be useful to you, we might be able to look into performing a dump of specific database tables so you don't have to slowly trickle it out of the API. Please do reply to the service-discuss mailing list at your earliest convenience and let us know how we can better help you with your research. Thanks again! -- Jeremy Stanley

2 1

opendev/gear reviews
by Antoine Musso 04 Nov '20

04 Nov '20

Hello, The opendev/gear is a pure python implementation of the Gearman protocol, written for Zuul 2.x. The project has a few pending changes that could use a final approval, one is required to unbreak the CI jobs. I am interested in helping on the maintenance, though I don't know which process I can start to eventually be blessed with W+1 rights on this repository. I will be more than happy to help on that front. Meanwhile, here is a summary of the changes that could certainly be approved immediately: "use python3 as context for build-python-release" https://review.opendev.org/#/c/742165/ Without it, the job fails for all proposed change. The same fix has been made to Zuul via https://review.opendev.org/#/c/742761/ "Bump crypto requirement to accommodate security standards" https://review.opendev.org/#/c/742117/ Raise the RSA security certificate bits from 1024 to 2048 and sign the key with sha256 instead of sha1. Without it, it seems pyOpenSSL complains the key is too small and the hash system too weak. "wakeConnections: Randomize connections before scanning them" https://review.opendev.org/#/c/747119/ That is an issue we had for a while at Wikimedia, the server wakes connection in the order they get registered. The connection that registered first ends up executing way more work than the last registered one. By shuffling the list of connections before waking them up, that helps spreading the load. "Modify connection timeout process" https://review.opendev.org/#/c/531059/ Resolves a potential deadlock, a connection timeout would not release the connection lock. "Move handleDisconnect into BaseClientServer" https://review.opendev.org/#/c/714709/ Really a noop, it is just moving a method to the parent class which is the sole use of it. "Added Docker image builds" https://review.opendev.org/#/c/688446/ Generates a Docker image to run the gear daemon. cheers, -- Antoine "hashar" Musso

1 0

Opendev review crawler
by Chouchen, Moataz 03 Nov '20

03 Nov '20

Dear Opendev administrators, I am Moataz Chouchen a Phd Student in Ets Montréal under the supervision of Professor Ali Ouni. I am sending this email to you to explain my requests of the Opendev data to respond to your request regarding Opendev code review data crawling. In fact, my Phd subject consists of the identification, the localization, and the understanding of Modern Code Review (MCR) problems (also called antipatterns). For this purpose, we need to crawl code review data from different projects (including Opendev) since this data will help us to study the process of MCR in these projects and their associated anti-patterns. Specifically, we need the whole data of Opendev to perform data analysis methods on them (including code review metrics distributions, social graph analysis etc) and study them in more depth. For this reason, we created a script to crawl data from Opendev and use them in our empirical study on MCR antipatterns in Opendev. I would like to thank you for your interest to understand the issues behind our requests and your intention to solve the issue for both sides. I am looking forward to collaborating with you and I would like to know if there are any rules that I should follow when requesting data from your servers and I will be more than happy If you can help me crawling the data and provide an access for me. I am also at your disposal for further clarifications regarding the requests. I hope I hear from you soon. Thank you Please stay safe, Moataz

3 2

Team meeting Agenda for November 3, 2020
by Clark Boylan 02 Nov '20

02 Nov '20

We will meet November 3, 2020 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Wallaby cycle signing key has been activated https://review.opendev.org/760364 *** Please sign if you haven't yet https://docs.opendev.org/opendev/system-config/latest/signing.html ** Note the Daylight Savings Time switch has happened in many parts of the world. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Preparing to upgrade Gerrit from 2.13 to 3.2 **** Need to rerun upgrade testing on review-test and revive it for usage testing **** We announced the upgrade is happening November 20 - 22 **** jeepyb bug/spec update hooks and the welcome message hook rely on database access and will need to be updated or sunsetted **** Upgrade Process ***** Backup then upgrade from 2.13 to 2.16. This is our fallback midpoint checkpoint ***** Backup again then migrate to notedb on 2.16 ***** Upgrade to 3.2 ***** Upgrade to 2.16 along with backups should be doable in a day. Then notedb migration can happen overnight with 3.2 upgrade happening on day two. ***** Looking into using a surrogate Gerrit running on a very performance vexxhost flavor to do the upgrade/data migrations which may speed things up. Needs testing. **** Unknowns ***** Storyboard integration *** Luca has offered to do a conference call with us. Let me know if interested and I'll include you for scheduling if/when that happens. * General topics ** PTG followups (clarkb 20201103) ** meetpad was not useable for (some?) participants from China (frickler 20201030) *** neutron went back to using Zoom for their last sessions because of this *** can we (maybe via help from the foundation involving their local members) work on improving this? *** would be sad to see open tools not being able to be used because of this ** Bup and Borg Backups (clarkb 20200929) *** Ethercalc to be the first borg backed up service ** Long term plans for running openstackid.org (clarkb 20201103) ** Splitting puppet else into specific infra-prod jobs (clarkb 20200929) *** Should be mostly mechanical *** Does it make sense to try and sprint this? Have several people work on getting it done in a short period of time? ** Trusty Upgrade Progress (clarkb 20200929) *** Wiki updates * Open discussion

1 0

Re: [service-announce] October 20 Gerrit Outage Update
by Mohammed Naser 29 Oct '20

29 Oct '20

Hi everyone, I'm happy to see that things are back in order, however, I *hate* to be that person, but I think there are still some hard questions that we need to answer together transparently. I am especially concerned because of how this affects the workflow of developers overall but also the security measures we have in place, should something like Zuul be targeted instead. I've added some questions in-line below Thanks, Mohammed On Tue, Oct 20, 2020 at 8:33 PM Ian Wienand <iwienand(a)redhat.com> wrote: > > As of this mail, Gerrit access has been restored. Please read on for > important information, especially around change verification. > > Background > ----------- > > On 2020-10-20 at 01:30 a user unexpectedly added a workflow approval > to a change that they were not expected to have access to. At 02:06 > UTC an alert was raised via IRC. Administrators found the account had > added themselves to a core group and made the +W vote. The account > was disabled, and removed from the groups it had added itself to by > 02:55 UTC. Administrators began to analyse the situation and Gerrit > was taken offline at 04:02 UTC to preserve state and allow for > analysis. > > From this time, administrators were working on log collection and > analysis, along with restoring backups for comparison purposes. > > By around 08:45 UTC it was clear that the privilege escalation had > been achieved by gaining control of a Launchpad SSO account with > Gerrit administrator privileges. By this time, we had ruled out > software vulnerabilities. Logs showed the first unauthorized access > of the administrator account in Gerrit on 2020-10-06. Communication > with Launchpad admins agrees with this analysis. We saw one session > opened as the administrator user to StoryBoard on this same day, but > logs show no data was modified or hidden stories viewed. So, just to be clear, someone who had root access to our Gerrit installation had their account compromised which resulted in this (and not something that occurred as a by-product of some other service -- say storyboard -- leaking some sort of information?) I see two issues in this at the moment: - There is no need for us to have anyone with admin powers to Gerrit at all times, we've done enough automation to sustain us and a manual 'circuit breaker' of adding a user *IF* necessary should be put in place. - If the above is not possible, anyone who is part of this group should have 2FA enabled inside Launchpad's SSO. I would very much prefer the first option rather than the second one. If it was an individual's account that was accessed and not a system account, have we audited that there are not other things that might have been accessed such as resources relating to Zuul, other systems and potentially rotating/auditing all our infrastructure? > Analysis has been performed on the Gerrit database and git trees from > October 1st, pre-dating any known unauthorized access. > > Access was restored at around 2020-10-21 00:00 UTC > > Outcomes > ----------- > > The following has been verified: > > The administrator account used has been disabled and credentials > updated > > We have verified that all group and user addition/removals since > Oct 1 are valid. The only invalid additions were made by the > compromised administrator account to add a single user account to > the Administrators group; and then that account added itself to > another known group. > > The account given administrator privilege has been removed from > the groups it added itself to and is disabled. > > There is no evidence of any unauthorized access via methods other > than Gerrit HTTP and Gerrit SSH access. > > No commits have been pushed to git trees bypassing code review. > Every git tree has been compared to the Oct 1 version and all > commits have been correctly inserted via Gerrit changes. I saw this artifact, I have no idea if it was put into consideration, but, food for thought: https://review.opendev.org/#/c/758881/ > The version of Gerrit we use stores HTTP API passwords in > plain-text. We know that a limited number of passwords were > gathered via the HTTP API and it is possible passwords were > gathered via the database. We thus have assumed that all HTTP API > passwords have been disclosed. This password needs to be > explicitly enabled by users, and many users do not have it > enabled. > > Remediation > ----------- > > This leaves us with the following remediation actions: > > Users should double-check their Launchpad recent activity at > https://login.launchpad.net/activity for any suspicious logins. If > found, please notify the OpenDev admins in Freenode #opendev and > Launchpad admins in #launchpad immediately. > > All HTTP API passwords have been cleared. If you push changes via > HTTPS (instead of typical SSH), are a gertty user, or run a CI > system or something else that communicates with the Gerrit HTTP > API, you will need to regenerate a password. > > Any SSH keys added to accounts since 2020-10-01 have been removed. > This affects only a limited number of accounts. This is done in > an abundance of caution, and we do not believe any accounts had > unauthorized SSH keys added > > We should audit all changes for projects since 2020-10-01. > > We have no evidence that any account had its ssh keys compromised, > thus we can rule out any unauthorized changes being uploaded via SSH. > However we can not conclusively rule out that compromised HTTP API > passwords were used to push a change through Gerrit. For example, a > change could be uploaded that looks like it came from a user, or the > API key of a core team member may have been used to approve a change > without authorization. > > Given our extensive analysis we consider it exceedingly unlikely that > this vector was used. We have had no notifications of users seeing > unexpected changes either uploaded by them, or approved by them in > projects they work on. This said, we believe it is important to > inform the community of this very unlikely, but still possible, > vulnerability of the source code. > > To this end, we have prepared a list of all changes from the known > affected period which should be audited for correctness. These are > available at > > https://static.opendev.org/project/opendev.org/gerrit-diffs/ > > Team members should browse these changes and make sure they were > correctly approved in Gerrit. If any change looks suspicious you > should notify OpenDev administrators in Freenode #opendev immediately. > > Further actions > ---------------- > > We are planning the following for the short term future: > > The Opendev administrators will be looking at alternative models > for Gerrit admin account management. > > We are already well into planning and testing a coming upgrade to > a version of Gerrit which does not store plain-text API keys. > > Longer term, we've written a spec for replacing Launchpad SSO as > our authentication provider. > > We thank you for your patience during this trying time, and we look > forward to returning to supporting the community doing what it does > best -- working together to create great things. > Thank you for this. I'd also like to raise the question of moving forward, how to be able to track these things. We had a user that had full root access to our Gerrit installation for ~2 weeks without our knowledge entirely, only uncovered when they did something (that, in the grand scheme of things, was relatively trivial, compared to what could have happened). What can we do to set up the necessary infrastructure to ensure that these things are monitored. OpenDev is considered to be critical infrastructure for this entire community and there's not much that an outsider can do other than the 'keyholders' for the resources. We've historically refused to have any monitoring and now things like this have slipped up, I'm just worried that we have a big looming thing coming up ahead of us that will catch us off guard and we'll be completely unprepared for it... > > _______________________________________________ > service-announce mailing list > service-announce(a)lists.opendev.org > http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce -- Mohammed Naser VEXXHOST, Inc.

4 7

Re: [service-announce] review.opendev.org Gerrit outage and upgrade 15:00UTC November 20 to 01:00UTC November 23, 2020
by Radosław Piliszek 28 Oct '20

28 Oct '20

On Tue, Oct 27, 2020 at 10:17 PM Clark Boylan <cboylan(a)sapwetik.org> wrote: > The OpenDev team is planning a long weekend Gerrit outage on review.opendev.org starting 15:00UTC November 20 and running to 01:00UTC November 23, 2020 in order to upgrade to Gerrit 3.2. <3 Finally! <snip> > * Gerrit 3.2 requires git 2.2.0 or newer to use the Change ID commit hook. This may be a problem for RHEL/CentOS 7 users. I thought it depends on git-review itself rather than a particular Gerrit version? <snip> > How can I help? > Once the upgrade is complete you'll want to confirm the basic functionality you rely on is there. We know there will be differences or missing features. Patience as we figure out how to address those on a new Gerrit installation is much appreciated. If you're interested in hacking on Java and Javascript we'd love help with the plugins necessary to address the known problems. You should be able to build this out locally without any special access. Please let us know if you are interested and we can help you bootstrap. Count me in if there are things to be done, might be able to spare some time for Gerrit. :-) We might want to port the live Zuul status reporter as well. -yoctozepto

2 2

Cancelling the team meeting today
by Clark Boylan 27 Oct '20

27 Oct '20

Hello everyone, We'll cancel the team meeting today since many of us are distracted by the PTG. We'll see you next week! Clark

1 0

Re: Some of your OpenDev Gerrit SSH keys have been deleted
by App Support 22 Oct '20

22 Oct '20

Hi Jeremy, Would the deletion affect the login to https://review.opendev.org/? <https://review.opendev.org/?> Somehow I got an error when trying to login. I am new to OpenStack and just started contributing to documentations. Thank you, Qi Zhang > On Oct 22, 2020, at 2:36 PM, Jeremy Stanley <fungi(a)yuggoth.org> wrote: > > In response to a security breach for the Gerrit Code Review service > at https://review.opendev.org/ I have deleted one or more SSH public > keys uploaded to your account between October 1 and October 19, > 2020. If you added any SSH public keys during that time, you will > need to reupload them in your user preferences before they can be > used to authenticate to the service. For more information on the > breach, see this announcement: > > http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html > > If you have any questions, feel free to reach out to OpenDev's > systems administrators on the service-discuss(a)lists.opendev.org > mailing list or in the #opendev channel on the Freenode IRC network. > -- > Jeremy Stanley, OpenDev Systems Administrator

2 2

Re: [service-announce] October 20 Gerrit Outage
by Jonathan Bryce 20 Oct '20

20 Oct '20

Thanks Clark and the rest of the OpenDev infra crew for literally working around the clock on this issue! Appreciate the effort to verify everything. Also wanted to share that the updates are being posted here if people want to see the history for completeness: https://review.opendev.org/maintenance.html Jonathan On October 20, 2020 12:50:32 "Clark Boylan" <cboylan(a)sapwetik.org> wrote: > Hello everyone, > > By now most of you have probably noticed that we took Gerrit offline > recently. The reason for that is we believe an admin account in Gerrit was > compromised allowing an attacker to escalate privileges within Gerrit. > > Around 02:00 UTC October 20 suspicious review activity was noticed, and we > were made aware of it shortly afterwards. The involved account was disabled > and removed from privileged Gerrit groups. After further investigation we > decided that we needed to stop the service, this happened at about 04:00 UTC. > > After the service was stopped we shifted focus to identifying the source of > the issue as well as investigating impact. We believe this originated on > October 6th with at least two compromised Ubuntu One accounts. One of which > was a Gerrit admin account. These accounts, like the one that initially > tipped us off, have been dealt with at this point. > > In order to evaluate impact we are using backups from October 1 to find > configuration, database, and git repo changes that have been made. We have > identified 97 accounts that updated ssh keys after that point in time. > These ssh keys are being removed as we can't be sure the changes were valid > changes made by the user. If you are one of these users you will need to > add your key(s) back in. We will also attempt to reach out to the affected > users directly by email soon. We will be checking openid urls and group > membership changes as well. We will determine what actions make sense for > these items once we have evaluated the impact to them. > > All Gerrit HTTP API tokens will be deleted. You will need to generate new > ones if you are an API user. Sorry, gertty fans. > > On the git repo side of things there are a few things that we will need to > check. Using our October 1 state we will generate lists of commits that > have landed since then for each branch on each repo. We will verify that > the latest commit can reach the last known good commit in the git DAG. For > non merge commits we will also correlate these to Gerrit changes. We will > then ask that you help us by verifying the commits on your projects are as > reviewed and not malicious. We will also need to check git tags which > should all be signed and can be verified that way. > > This is a good reminder to check activity on your online accounts and > identities for anything unexpected. > > We understand that an inaccessible Gerrit is not fun. We are trying to go > as quickly as we can while also not sacrificing caution and care. > > Clark > > _______________________________________________ > service-announce mailing list > service-announce(a)lists.opendev.org > http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce

1 0

Cancelling the team meeting October 20, 2020
by Clark Boylan 19 Oct '20

19 Oct '20

Hello everyone, I've been swamped with summit things. Initially I thought that we could do a quieter meeting during the summit, but I've been too distracted to even look at an agenda. For that reason I think we should cancel the meeting this week. We've got the PTG next week. Please register and check the schedule if you plan to attend. We'll see you there as well as at next week's meeting (I hope). Clark

1 0