[service-announce] review.opendev.org Gerrit outage and upgrade 15:00UTC November 20 to 01:00UTC November 23, 2020

[Clark Boylan]

The OpenDev team is planning a long weekend Gerrit outage on review.opendev.org starting 15:00UTC November 20 and running to 01:00UTC November 23, 2020 in order to upgrade to Gerrit 3.2.

The upgrade has two major portions.  First we will incrementally move from our current version 2.13 to 2.16.  Each point release requires a database migration and git indexing operations that take the majority of the time.  The second major part, once at version 2.16, is to convert Gerrit to use the new NoteDB backend.  This stores reviews together with code in the git trees, rather than in a separate database.  There are known problems converting to NoteDB from any version prior to 2.16, which is why we need the initial upgrade steps.  This is this slowest portion of the upgrade process, and the one most prone to unforeseen issues despite extensive pre-testing.  We will be creating snapshots to facilitate quick fallback if required. Once this is complete, we can move to the 3.x series and incrementally upgrade from 3.0 to 3.2.

Timing wise we expect the first 2.13 -> 2.16 phase to be completed in about 8 hours. Next we'll start the notedb migration and let that run overnight. When we return the next morning the NoteDB migration should be complete and we can finish with the last set of upgrades to 3.2. Hopefully that means we're done in the first half of the outage window, but if we have problems, things are slower than we expect, or if we have to work through any details after finishing the upgrade we'd like to have the extra buffer in place.

Some important things to know:
* Gerrit's web UI will be changing. We don't have a lot of control over this. As a result we'll lose some existing CI result rendering niceness that we have on 2.13 (in particular the summary table and CI results toggle will go away). We hope that once we've upgraded we can investigate solving this through Gerrit plugins.
* All of our existing integrations with launchpad and storyboard may not be fully functional. The changes to the database system impact aspects of how we lookup user details in order to update bugs.
* Gerrit 3.2 requires git 2.2.0 or newer to use the Change ID commit hook. This may be a problem for RHEL/CentOS 7 users.
* This upgrade will fix the openssh problems with host keys that Fedora 33 users have experienced.

* Q&A *
If this upgrade isn't perfect why are we doing it anyway?
  We've come to the realization that if we don't make imperfect progress we'll never make any progress. We have decided that the benefits outweigh the known drawbacks and we'll do our best to work on those issues after the upgrade.

Why now?
  This is long overdue and the quiet time at the end of the year tends to be a good time for these big upgrades

Can I install my favorite Gerrit plugin?
  We're hoping to complete the upgrade then start looking at things like this. One big reason for it is finishing the upgrade simplifies the number of Gerrit docker images we need to care for.

Will my third party CI system continue to work?
  We expect they will. We know Zuul is tested against newer Gerrit already.

How can I help?
  Once the upgrade is complete you'll want to confirm the basic functionality you rely on is there. We know there will be differences or missing features. Patience as we figure out how to address those on a new Gerrit installation is much appreciated. If you're interested in hacking on Java and Javascript we'd love help with the plugins necessary to address the known problems. You should be able to build this out locally without any special access. Please let us know if you are interested and we can help you bootstrap.

Finally, we're continuing to test the upgrade and trying to sort out if we can get this upgrade completed in a shorter period of time. We'll keep you up to date if further testing forces us to postpone or if the outage window timeframe changes at all.

As always feel free to send your questions our way and we'll do our best to answer them.

On behalf of the OpenDev team,
Clark