- service-announce - lists.opendev.org

Take care when deleting email addresses in Gerrit
by Clark Boylan 14 Jul '21

14 Jul '21

Hello everyone, We have discovered a bug in Gerrit [0] that allows you to delete your account's Ubuntu One OpenID association if you delete the email from your account associated with that OpenID. When you do this the next time you log in Gerrit will create a new account for you. Fixing this after the fact is currently difficult. We are working to correct this but the required corrections take time. We would like to avoid these issues in the first place. Please avoid deleting email addresses from your accounts unless you are sure they are not associated with your OpenID. If you would like help double checking that for you please let us know and one of our Gerrit admins can double check. We are working to make this fixable after the fact, and will try to work with the Gerrit upstream to prevent you from deleting Ubuntu One OpenID associations from your account in the first place. Until then we apologize for the inconvenience. Finally, this message and others like it are sent to service-announce(a)lists.opendev.org. I am sending this email to a much broader set of mailing lists because the service-announce list membership is quite small. All of our users should be subscribed to that list to get important notices like this one. We keep the traffic low and only send mail there when we feel it is important for you to see it. [0] https://bugs.chromium.org/p/gerrit/issues/detail?id=14776 Clark

1 0

OpenDev IRC services are moving to OFTC this weekend
by Jeremy Stanley 26 May '21

26 May '21

As a majority of our constituent projects have voiced a preference for enacting our long-standing evacuation plan, the OpenDev Collaboratory's IRC service bots will be switching from Freenode to the OFTC network this weekend (May 29-30, 2021). We understand this is short notice, but multiple projects have requested that we act quickly. Please expect some gaps in channel logging and notifications from our various bots over the course of the weekend. I have provided a much more detailed writeup to the service-discuss mailing list, and encourage anyone with questions to read it and follow up there if needed. Subsequent updates will be sent only to service-discuss, in order to limit noise for individual project lists and keep further discussion focused in one place as much as possible: http://lists.opendev.org/pipermail/service-discuss/2021-May/000249.html -- Jeremy Stanley

1 0

Switching default nodeset in base job to ubuntu-focal
by Jeremy Stanley 18 May '21

18 May '21

The OpenDev Collaboratory is normally somewhat conservative about when it changes its Zuul node "default" (that is, the nodeset in its base job). It's been ubuntu-bionic since at least July of 2019 when we first moved our base job into its own repository, and ubuntu-focal nodes have been available in our deployment since before Ubuntu's 20.04 LTS was officially released more than a year ago. All that is to say, updating our default is overdue. On Tuesday, 2021-05-18 (two weeks from today) we'll approve https://review.opendev.org/789098 to update the nodeset to ubuntu-focal for any jobs which don't specify their own or inherit from another job which does. We've already merged a change which updates our base-test job to reflect this, so you can test either by explicitly setting your jobs to use an ubuntu-focal nodeset or to inherit from base-test (though we don't recommend merging changes for the latter, just rely on speculative execution to identify problems which you may encounter when the default changes). Please reply to the service-discuss mailing list with any questions or concerns. You should also feel free to leave comments on the change linked above. -- Jeremy Stanley

1 1

git-review 2.1.0
by Jeremy Stanley 23 Apr '21

23 Apr '21

It's my pleasure to announce the release of git-review 2.1.0. These are the corresponding release notes since the previous release: New Features ------------ * Add support for --no-thin which is passed to git push. This is useful to work around situations where Gerrit’s JGit and git-review’s C git implementations cannot agree on the pack file contents used to transfer a push. When they disagree you see this as unpack failures due to missing trees. Using --no-thin avoids complicated negotiations and works around this problem. As always, the latest version of git-review is available from PyPI: https://pypi.org/project/git-review Find the updated documentation at its new home: https://docs.opendev.org/opendev/git-review And source code can be obtained through the OpenDev Collaboratory: https://opendev.org/opendev/git-review Please don't hesitate to follow up through the service-discuss(a)lists.opendev.org mailing list or in the #opendev channel on the Freenode IRC network if you have any questions. -- Jeremy Stanley

1 0

git-review 2.0.0
by Jeremy Stanley 03 Mar '21

03 Mar '21

It's my pleasure to announce the release of git-review 2.0.0. These are the corresponding release notes since the previous release: New Features ------------ * A new gitreview.branchauthor option allows configuring the change author (owner) name part of the autocreated branch name when downloading changes. * A new gitreview.notopic configuration setting makes git-review always behave as if the --notopic command line option was supplied. Upgrade Notes ------------- * Not really a deprecation so much as a removal, Gerrit 2.15 (released in 2017) ceased providing a draft state for changes, and many large sites were disabling support for it far earlier than that. As of now, git-review has removed the -D and --draft options entirely, and so can no longer push draft changes on old Gerrit deployments which still allowed them. * As of this release, git-review no longer supports running under the Python 2.7 interpreter. The major version component has been increased accordingly. Users wishing to use git-review with Python 2.7 can cap their installations like pip install "git-review<2", though pip version 9 and later will use python_requires package metadata to avoid installing git-review on an unsupported interpreter. Future removal of support for Python 3 minor versions will only result in a minor version increase for corresponding git-review releases. Bug Fixes --------- * The Change-Id generating commit hook is now installed into any detected Git submodules, so that it will take effect when creating changes within a submodule and not just in the parent repository. * Encoding was corrected for Git credentials, which was previously causing HTTP authentication to fail under some Python 3 versions. * Change download functions now preserve any unapplied local edits on the branch when attempting to reset the worktree. * The --no-follow-tags option to git push is now used for all pushes, so that git-review won’t error when encountering a local tag which is absent from Gerrit. * For safety, attempts to push a commit with unstaged or uncommitted changes in the worktree will be caught and an error reported, rather than leaving it up to git rebase to spot them. This addresses a situation where users enabling rebase.autostash would otherwise experience data loss when the test rebase is subsequently reset. As always, the latest version of git-review is available from PyPI: https://pypi.org/project/git-review Find the updated documentation at its new home: https://docs.opendev.org/opendev/git-review And source code can be obtained through the OpenDev Collaboratory: https://opendev.org/opendev/git-review Please don't hesitate to follow up through the service-discuss(a)lists.opendev.org mailing list or in the #opendev channel on the Freenode IRC network if you have any questions. -- Jeremy Stanley

1 0

Recommended change to Zuul pipeline config for all tenants
by jim＠acmegating.com 15 Feb '21

15 Feb '21

Hi, If you manage the Zuul pipeline definitions for a tenant in OpenDev's Zuul, please make the following change to the pipeline definitions as soon as possible: * Remove any reference to the "mysql" reporter. The current version of Zuul now automatically adds the SQL reporter to every pipeline definition, so the explicit configuration is unnecessary. It is now marked as deprecated and will produce an error in a future release of Zuul, and it produces an warning message now. Here is an example change which I performed for the OpenStack tenant: https://review.opendev.org/775716 Thanks, Jim

1 0

bindep 2.9.0
by Jeremy Stanley 06 Feb '21

06 Feb '21

It's my pleasure to announce the release of bindep 2.9.0. These are the corresponding release notes since the previous release: New Features ------------ * Bindep can now be imported as a Python module, useful when making plugins for other Python-based tools or experimenting in the REPL. * Basic support for Oracle Linux is now included. * A new set of virtual platform profiles, platform:base-py2 and platform:base-py3 can be used to differentiate packages on Debian derivatives (including Ubuntu), Red Hat derivatives (including CentOS and Fedora), and MacOS X/Darwin. Bug Fixes --------- * Improved parsing of warning and error output from the apk tool in alpine. * Now handles parsing pacman 1.5.2 and later output so –brief works on newer releases of Archlinux. * If a local file has the same name as a package, this no longer confuses parsing of pacman output on Arch Linux. Find the updated documentation at its new home: https://docs.opendev.org/opendev/bindep/ Please don't hesitate to follow up through the service-discuss(a)lists.opendev.org mailing list or in the #opendev channel on the Freenode IRC network if you have any questions. -- Jeremy Stanley

1 0

review.opendev.org Gerrit outage and upgrade 15:00UTC November 20 to 01:00UTC November 23, 2020
by Clark Boylan 23 Nov '20

23 Nov '20

The OpenDev team is planning a long weekend Gerrit outage on review.opendev.org starting 15:00UTC November 20 and running to 01:00UTC November 23, 2020 in order to upgrade to Gerrit 3.2. The upgrade has two major portions. First we will incrementally move from our current version 2.13 to 2.16. Each point release requires a database migration and git indexing operations that take the majority of the time. The second major part, once at version 2.16, is to convert Gerrit to use the new NoteDB backend. This stores reviews together with code in the git trees, rather than in a separate database. There are known problems converting to NoteDB from any version prior to 2.16, which is why we need the initial upgrade steps. This is this slowest portion of the upgrade process, and the one most prone to unforeseen issues despite extensive pre-testing. We will be creating snapshots to facilitate quick fallback if required. Once this is complete, we can move to the 3.x series and incrementally upgrade from 3.0 to 3.2. Timing wise we expect the first 2.13 -> 2.16 phase to be completed in about 8 hours. Next we'll start the notedb migration and let that run overnight. When we return the next morning the NoteDB migration should be complete and we can finish with the last set of upgrades to 3.2. Hopefully that means we're done in the first half of the outage window, but if we have problems, things are slower than we expect, or if we have to work through any details after finishing the upgrade we'd like to have the extra buffer in place. Some important things to know: * Gerrit's web UI will be changing. We don't have a lot of control over this. As a result we'll lose some existing CI result rendering niceness that we have on 2.13 (in particular the summary table and CI results toggle will go away). We hope that once we've upgraded we can investigate solving this through Gerrit plugins. * All of our existing integrations with launchpad and storyboard may not be fully functional. The changes to the database system impact aspects of how we lookup user details in order to update bugs. * Gerrit 3.2 requires git 2.2.0 or newer to use the Change ID commit hook. This may be a problem for RHEL/CentOS 7 users. * This upgrade will fix the openssh problems with host keys that Fedora 33 users have experienced. * Q&A * If this upgrade isn't perfect why are we doing it anyway? We've come to the realization that if we don't make imperfect progress we'll never make any progress. We have decided that the benefits outweigh the known drawbacks and we'll do our best to work on those issues after the upgrade. Why now? This is long overdue and the quiet time at the end of the year tends to be a good time for these big upgrades Can I install my favorite Gerrit plugin? We're hoping to complete the upgrade then start looking at things like this. One big reason for it is finishing the upgrade simplifies the number of Gerrit docker images we need to care for. Will my third party CI system continue to work? We expect they will. We know Zuul is tested against newer Gerrit already. How can I help? Once the upgrade is complete you'll want to confirm the basic functionality you rely on is there. We know there will be differences or missing features. Patience as we figure out how to address those on a new Gerrit installation is much appreciated. If you're interested in hacking on Java and Javascript we'd love help with the plugins necessary to address the known problems. You should be able to build this out locally without any special access. Please let us know if you are interested and we can help you bootstrap. Finally, we're continuing to test the upgrade and trying to sort out if we can get this upgrade completed in a shorter period of time. We'll keep you up to date if further testing forces us to postpone or if the outage window timeframe changes at all. As always feel free to send your questions our way and we'll do our best to answer them. On behalf of the OpenDev team, Clark

2 2

October 20 Gerrit Outage Update
by Ian Wienand 21 Oct '20

21 Oct '20

As of this mail, Gerrit access has been restored. Please read on for important information, especially around change verification. Background ----------- On 2020-10-20 at 01:30 a user unexpectedly added a workflow approval to a change that they were not expected to have access to. At 02:06 UTC an alert was raised via IRC. Administrators found the account had added themselves to a core group and made the +W vote. The account was disabled, and removed from the groups it had added itself to by 02:55 UTC. Administrators began to analyse the situation and Gerrit was taken offline at 04:02 UTC to preserve state and allow for analysis. >From this time, administrators were working on log collection and analysis, along with restoring backups for comparison purposes. By around 08:45 UTC it was clear that the privilege escalation had been achieved by gaining control of a Launchpad SSO account with Gerrit administrator privileges. By this time, we had ruled out software vulnerabilities. Logs showed the first unauthorized access of the administrator account in Gerrit on 2020-10-06. Communication with Launchpad admins agrees with this analysis. We saw one session opened as the administrator user to StoryBoard on this same day, but logs show no data was modified or hidden stories viewed. Analysis has been performed on the Gerrit database and git trees from October 1st, pre-dating any known unauthorized access. Access was restored at around 2020-10-21 00:00 UTC Outcomes ----------- The following has been verified: The administrator account used has been disabled and credentials updated We have verified that all group and user addition/removals since Oct 1 are valid. The only invalid additions were made by the compromised administrator account to add a single user account to the Administrators group; and then that account added itself to another known group. The account given administrator privilege has been removed from the groups it added itself to and is disabled. There is no evidence of any unauthorized access via methods other than Gerrit HTTP and Gerrit SSH access. No commits have been pushed to git trees bypassing code review. Every git tree has been compared to the Oct 1 version and all commits have been correctly inserted via Gerrit changes. The version of Gerrit we use stores HTTP API passwords in plain-text. We know that a limited number of passwords were gathered via the HTTP API and it is possible passwords were gathered via the database. We thus have assumed that all HTTP API passwords have been disclosed. This password needs to be explicitly enabled by users, and many users do not have it enabled. Remediation ----------- This leaves us with the following remediation actions: Users should double-check their Launchpad recent activity at https://login.launchpad.net/activity for any suspicious logins. If found, please notify the OpenDev admins in Freenode #opendev and Launchpad admins in #launchpad immediately. All HTTP API passwords have been cleared. If you push changes via HTTPS (instead of typical SSH), are a gertty user, or run a CI system or something else that communicates with the Gerrit HTTP API, you will need to regenerate a password. Any SSH keys added to accounts since 2020-10-01 have been removed. This affects only a limited number of accounts. This is done in an abundance of caution, and we do not believe any accounts had unauthorized SSH keys added We should audit all changes for projects since 2020-10-01. We have no evidence that any account had its ssh keys compromised, thus we can rule out any unauthorized changes being uploaded via SSH. However we can not conclusively rule out that compromised HTTP API passwords were used to push a change through Gerrit. For example, a change could be uploaded that looks like it came from a user, or the API key of a core team member may have been used to approve a change without authorization. Given our extensive analysis we consider it exceedingly unlikely that this vector was used. We have had no notifications of users seeing unexpected changes either uploaded by them, or approved by them in projects they work on. This said, we believe it is important to inform the community of this very unlikely, but still possible, vulnerability of the source code. To this end, we have prepared a list of all changes from the known affected period which should be audited for correctness. These are available at https://static.opendev.org/project/opendev.org/gerrit-diffs/ Team members should browse these changes and make sure they were correctly approved in Gerrit. If any change looks suspicious you should notify OpenDev administrators in Freenode #opendev immediately. Further actions ---------------- We are planning the following for the short term future: The Opendev administrators will be looking at alternative models for Gerrit admin account management. We are already well into planning and testing a coming upgrade to a version of Gerrit which does not store plain-text API keys. Longer term, we've written a spec for replacing Launchpad SSO as our authentication provider. We thank you for your patience during this trying time, and we look forward to returning to supporting the community doing what it does best -- working together to create great things.

1 0

October 20 Gerrit Outage
by Clark Boylan 20 Oct '20

20 Oct '20

Hello everyone, By now most of you have probably noticed that we took Gerrit offline recently. The reason for that is we believe an admin account in Gerrit was compromised allowing an attacker to escalate privileges within Gerrit. Around 02:00 UTC October 20 suspicious review activity was noticed, and we were made aware of it shortly afterwards. The involved account was disabled and removed from privileged Gerrit groups. After further investigation we decided that we needed to stop the service, this happened at about 04:00 UTC. After the service was stopped we shifted focus to identifying the source of the issue as well as investigating impact. We believe this originated on October 6th with at least two compromised Ubuntu One accounts. One of which was a Gerrit admin account. These accounts, like the one that initially tipped us off, have been dealt with at this point. In order to evaluate impact we are using backups from October 1 to find configuration, database, and git repo changes that have been made. We have identified 97 accounts that updated ssh keys after that point in time. These ssh keys are being removed as we can't be sure the changes were valid changes made by the user. If you are one of these users you will need to add your key(s) back in. We will also attempt to reach out to the affected users directly by email soon. We will be checking openid urls and group membership changes as well. We will determine what actions make sense for these items once we have evaluated the impact to them. All Gerrit HTTP API tokens will be deleted. You will need to generate new ones if you are an API user. Sorry, gertty fans. On the git repo side of things there are a few things that we will need to check. Using our October 1 state we will generate lists of commits that have landed since then for each branch on each repo. We will verify that the latest commit can reach the last known good commit in the git DAG. For non merge commits we will also correlate these to Gerrit changes. We will then ask that you help us by verifying the commits on your projects are as reviewed and not malicious. We will also need to check git tags which should all be signed and can be verified that way. This is a good reminder to check activity on your online accounts and identities for anything unexpected. We understand that an inaccessible Gerrit is not fun. We are trying to go as quickly as we can while also not sacrificing caution and care. Clark

1 0