Re: [service-announce] review.opendev.org Gerrit outage and upgrade 15:00UTC November 20 to 01:00UTC November 23, 2020

[Clark Boylan]

On Thu, Nov 5, 2020, at 7:16 AM, Antoine Musso wrote:
> Hello,
> 
> Wikimedia has upgraded from 2.16 to 3.2 in June 2020 conducted by
> Christian Aistleitner.  He wrote a nice report on the upstream mailling 
> list: https://groups.google.com/g/repo-discuss/c/G5wucKJg9Ag
> 
> 
> 
> On 27/10/2020 22:16, Clark Boylan wrote:
> > The OpenDev team is planning a long weekend Gerrit outage on review.opendev.org starting 15:00UTC November 20 and running to 01:00UTC November 23, 2020 in order to upgrade to Gerrit 3.2.
> > 
> > The upgrade has two major portions.  First we will incrementally move from our current version 2.13 to 2.16.  Each point release requires a database migration and git indexing operations that take the majority of the time.
> 
> 
>  From Christian report, you should be able to skip the git indexing 
> between each minor upgrades and only do a full reindexing once you are 
> upgraded to 3.2.
> 
> The git indexing had an issue which is that the changes to index were 
> split by repository. In the worse case scenario, if you have a thousand 
> of small repositories and one very large one, the later was only 
> processed by a single thread.  Christian added code to chunk by changes 
> regardless of the repository, that has dramatically speed up the indexing.
> 
> If I am not mistaken, the commit is 20784548c3fb and it has been 
> released in 2.16.22, 3.0.12, 3.1.8 and 3.2.3.
> 
> Given you should be able to skip reindexing between upgrades. Make sure 
> to use 3.2.3 to benefit from the faster indexing.

The actual process is:
* Stop Gerrit
* git gc --aggressive all repos
* gerrit init on 2.14
* gerrit init on 2.15
* gerrit init on 2.16
* Perform complete offline reindex. We do this here in order to have a working midpoint snapshot And it doesn't take too long as an individual step.
* Stop gerrit
* git gc --aggressive
* Perform offline notedb migration
* git gc --aggressive
* gerrit init on 3.0
* gerrit init on 3.1
* gerrit init on 3.2
* Perform complete offline reindex.
* Start gerrit

We can probably optimize a few of those git gc steps away but they appear to have a large impact on steps like reindexing so we're just doing them. But every little step of db migrations, reindexing, gc'ing, and doing the notedb migration adds up. Note I tested doing gerrit init to skip versions but it doesn't seem to work properly when you do that.

> 
> 
> 
> > The second major part, once at version 2.16, is to convert Gerrit to use the new NoteDB backend.  This stores reviews together with code in the git trees, rather than in a separate database.  There are known problems converting to NoteDB from any version prior to 2.16, which is why we need the initial upgrade steps.  This is this slowest portion of the upgrade process, and the one most prone to unforeseen issues despite extensive pre-testing.  We will be creating snapshots to facilitate quick fallback if required. Once this is complete, we can move to the 3.x series and incrementally upgrade from 3.0 to 3.2.
> 
> 2.16 has received improvements for the NoteDB migration. Again make sure 
> to use the latest patch release (2.16.23 at this time).

We actually build our images off the tip of the branches and not from release tags. But this is a good reminder to rebuild to ensure we've got the latest releases.

> 
> <snip>
> 

<snip>

Thanks for the feedback, it helps to have reminders like this as well as general input ensuring we're on the right change.