Testing the results of Gerrit upgrades on review-test.opendev.org

Clark Boylan cboylan at sapwetik.org
Fri Oct 9 18:46:03 UTC 2020

On Tue, Oct 6, 2020, at 4:41 PM, Clark Boylan wrote:
> Hello everyone,
> Recently fungi and I managed to make review-test.opendev.org a copy of 
> production Gerrit (review.opendev.org) as of October 1. We have since 
> upgraded that installation to Gerrit 2.16. The 2.16 version is a 
> potential upgrade target for our production Gerrit as we have to stop 
> there and do a required notedb migration step. What isn't yet clear is 
> if we'll expose the 2.16 installation or continue on to upgrade to a 
> 3.x version. In order to figure that out we'll be doing the notedb 
> migration and the 3.x upgrades in the near future. Once we've gone 
> through the process we should have a good idea of what makes sense as 
> far as upgrades go for us.
> In any case the server is up for testing now if people want to take a 
> look at 2.16 and offer feedback. I'll followup on this thread when we 
> get upgraded to 3.x and ask for feedback on that version too. Head to 
> https://review-test.opendev.org and check it out.
> There are some minor known issues. The biggest that people will 
> probably notice is that our hacky javascript CI results table does not 
> work in the old gerrit UI or the new polygerrit UI. The commentlinks 
> for zuul results are also not working on polygerrit UI. We're still 
> investigating, but we're going to be careful that we don't worry about 
> every single issue like that pre upgrade otherwise we'll never get the 
> upgrade done. Instead we'll gather feedback and prioritize then sort 
> out what needs fixing now and what we can sort out later.
> Let us know what you think,
> Clark

Quick update. Fungi and I got the server upgraded from 2.16 to 3.2 over the last day and a half. Unfortunately the notedb migration is a significant chunk of time which will prolong the upgrade process, but it succeeded and the server is back up and running on the latest release. Feel free to test the server out and give feedback.

As far as upgrade planning goes, we have a few important things to test like replication behavior and sizes post notedb migration. I'm working on spinning up a test gitea server to replicate into from review-test for that. Otherwise basic functionality seems to work: I can log in, git review -s is able to grab the hook script, git review can push changes, I am able to leave top level and inline comments on changes, and so on.

The upgrade itself will likely take about two days with Gerrit being inaccessible the whole time. On day one we'll upgrade from 2.13 to 2.16 pre notedb migration. This will act as a checkpoint which we can fallback to if the 3.x upgrade has any problems. Then between day one and day two we can run the long notedb migration. When we return to the upgrading on day two we can upgrade from 2.16 to 3.2, then apply config updates to services like zuul as well as other related cleanups.

Actual details to be determined after more testing. I don't want to commit to such exact plans just yet.


More information about the service-discuss mailing list