From cboylan at sapwetik.org Sat Nov 21 23:53:55 2020 From: cboylan at sapwetik.org (Clark Boylan) Date: Sat, 21 Nov 2020 15:53:55 -0800 Subject: [service-announce] =?utf-8?q?review=2Eopendev=2Eorg_Gerrit_outage?= =?utf-8?q?_and_upgrade_15=3A00UTC_November_20_to_01=3A00UTC_November_23?= =?utf-8?q?=2C_2020?= In-Reply-To: <410ac6e8-6381-49d8-836e-302fe166b47a@www.fastmail.com> References: <410ac6e8-6381-49d8-836e-302fe166b47a@www.fastmail.com> Message-ID: <5762736f-da9d-4a6f-8edd-7f31fb91b5ca@www.fastmail.com> Hello everyone, By now you've probably noticed that the Gerrit upgrade appears complete. And for most users it probably is. Gerrit 3.2 is up and running, Zuul is talking to it and running jobs, and you can push/review changes. On the OpenDev side of things we're continuing to identify issues and fix them as we can. We are also still in the process of returning our Zuul driven continuous deployment to normal operations. I hesitate to call this completely done as we may need to restart Gerrit periodically to apply config updates to address issues that we find. During our announced upgrade period (which ends at 01:00 UTC November 23) we will do these without announcements. Don't be surprised if you find Gerrit is offline during this period. Once this block of time has passed we may still need to do restarts, but we'll do our best to send notice of them on IRC and in our status log. The only major unexpected functional issue we've found is in the git hosting for x/foo-project repos in Gerrit. The new Gerrit UI web endpoints conflict with our use of the x/ namespace, meaning "git clone http://review.opendev.org/x/foo-project" is being incorrectly interpreted as a UI request, rather than as a git clone. We've worked around that and have started to engage upstream to see if we can find a permanent fix for these projects. But I want to call this out early because there is a chance we may need to rename the entirety of x/* to notx/* in the future. We recognize this is not ideal, and we didn't catch it in testing. We'll do our best to communicate and plan for this with you if it becomes necessary. The crowd sourcing of feedback has begun at: https://etherpad.opendev.org/p/gerrit-3.2-post-upgrade-notes Feel free to add things you notice as well. Thank you for your patience. Clark on behalf of the OpenDev team From fungi at yuggoth.org Mon Nov 23 00:41:22 2020 From: fungi at yuggoth.org (Jeremy Stanley) Date: Mon, 23 Nov 2020 00:41:22 +0000 Subject: [service-announce] review.opendev.org Gerrit outage and upgrade 15:00UTC November 20 to 01:00UTC November 23, 2020 In-Reply-To: <5762736f-da9d-4a6f-8edd-7f31fb91b5ca@www.fastmail.com> Message-ID: <20201123004122.mqipwhsqj65rghos@yuggoth.org> Many thanks for your patience, our Gerrit upgrade maintenance has now concluded. We consider services to be mostly functional at this point. Here are a few updates you may want to consider as you venture back into code review: 1. If you encounter any changes updated in the past three days which lack verified votes from Zuul, you'll probably want to leave "recheck" comments on them to get test results, since their upload events were likely lost in the shuffle. 2. Make sure you're using a compatible release of the git-review tool; you need 1.27 or newer to work with our new Gerrit version. 3. The version of Gerrit we're running now properly hashes REST API passwords, and as a result you need to use HTTP basic auth instead of digest auth with it (gertty users will want to set "auth-type: basic" in the server entry of their configuration). 4. For that reason, we also suggest generating a new REST API password in your Gerrit settings if you currently have one, since old copies were stored in the database in unencrypted plain text. 5. We're assembling notes on important changes and apparent regressions here, please review them and feel free to help expand on the information we've collected so far: https://etherpad.opendev.org/gerrit-3.2-post-upgrade-notes 6. For those interested in the details of our maintenance activity, the entire effort was coordinated in IRC and logged: http://eavesdrop.openstack.org/meetings/opendev_maint/2020/opendev_maint.2020-11-20-12.59.log.html 7. If you have any other input, please follow up on the service-discuss at lists.opendev.org mailing list or in the #opendev channel on the Freenode IRC network. -- Jeremy Stanley on behalf of the OpenDev Collaboratory sysadmins -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: