[service-announce] review.opendev.org Gerrit outage and upgrade 15:00UTC November 20 to 01:00UTC November 23, 2020
hashar at free.fr
Thu Nov 5 15:16:49 UTC 2020
Wikimedia has upgraded from 2.16 to 3.2 in June 2020 conducted by
Christian Aistleitner. He wrote a nice report on the upstream mailling
On 27/10/2020 22:16, Clark Boylan wrote:
> The OpenDev team is planning a long weekend Gerrit outage on review.opendev.org starting 15:00UTC November 20 and running to 01:00UTC November 23, 2020 in order to upgrade to Gerrit 3.2.
> The upgrade has two major portions. First we will incrementally move from our current version 2.13 to 2.16. Each point release requires a database migration and git indexing operations that take the majority of the time.
From Christian report, you should be able to skip the git indexing
between each minor upgrades and only do a full reindexing once you are
upgraded to 3.2.
The git indexing had an issue which is that the changes to index were
split by repository. In the worse case scenario, if you have a thousand
of small repositories and one very large one, the later was only
processed by a single thread. Christian added code to chunk by changes
regardless of the repository, that has dramatically speed up the indexing.
If I am not mistaken, the commit is 20784548c3fb and it has been
released in 2.16.22, 3.0.12, 3.1.8 and 3.2.3.
Given you should be able to skip reindexing between upgrades. Make sure
to use 3.2.3 to benefit from the faster indexing.
> The second major part, once at version 2.16, is to convert Gerrit to use the new NoteDB backend. This stores reviews together with code in the git trees, rather than in a separate database. There are known problems converting to NoteDB from any version prior to 2.16, which is why we need the initial upgrade steps. This is this slowest portion of the upgrade process, and the one most prone to unforeseen issues despite extensive pre-testing. We will be creating snapshots to facilitate quick fallback if required. Once this is complete, we can move to the 3.x series and incrementally upgrade from 3.0 to 3.2.
2.16 has received improvements for the NoteDB migration. Again make sure
to use the latest patch release (2.16.23 at this time).
> Some important things to know:
> * Gerrit's web UI will be changing. We don't have a lot of control over this. As a result we'll lose some existing CI result rendering niceness that we have on 2.13 (in particular the summary table and CI results toggle will go away). We hope that once we've upgraded we can investigate solving this through Gerrit plugins.
https://www.polymer-project.org/ ) and relies on the REST API to fetch data.
For the summary table, Wikimedia has a light version of the one you have
which got borrowed from somewhere else.
See "gr-test-result-table-module" at:
It is hopefully not too complicated to implement.
CI results toggle
That is now built-in Gerrit, the UI will have a "Only comments" toggle
which filters out any message with a tag prefixed by "autogenerated:".
Thus in Wikimedia good old Zuul 2.5 we have simply tagged with
"autogenerated:ci" by using:
Result: https://phabricator.wikimedia.org/T48148#6294913 (there are a
few more details on that task).
Another one is that the SUCCESS/FAILURE message can no more be
highlighted via rewriting the message and using CSS. The way text is
parsed has completely changed and its now impossible to do it via the
commentlinks Gerrit settings.
As a result on Zuul 2.5 we have set job_name_in_report = false and lack
the nice formatting and green/red coloring. I am not sure whether that
still applies to Zuul 3.x though.
Wikimedia task: https://phabricator.wikimedia.org/T256575
plugin that would parse the message and prettify the results comment on
the client side.
> * Q&A *
> If this upgrade isn't perfect why are we doing it anyway?
> We've come to the realization that if we don't make imperfect progress we'll never make any progress. We have decided that the benefits outweigh the known drawbacks and we'll do our best to work on those issues after the upgrade.
I would add a couple user facing improvements:
The new UI alone is definitely worth the upgrade. It is way more
pleasant than the old one. Albeit it needs some days to adjust, you will
surely never look back :]
The cherry on the cake is that with the upgrade comes support for Git
protocol version 2. In short, when enabled, it makes fetches from big
repository an order of magnitude faster.
I wrote a quick blog post about git protocol v2 at:
And Google announcement with lot more details:
I wish you the very best for the maintenance. Thank you for making it
Antoine "hashar" Musso
More information about the service-discuss