- service-discuss - lists.opendev.org

OpenDev git error: RPC failed; result=18, HTTP code = 200iB | 57.00 KiB/s
by Pengzhi 17 Dec '20

17 Dec '20

Hi OpenDev team, When trying to pull tripleo from opendev.org, git failed with Fatal Error 128. Can you please check and fix it? [ooo@VM-3O ~]$ git config http.postBuffer 24288000 [ooo@VM-3O ~]$ git clone https://opendev.org/openstack/tripleo-quickstart-extras/ Cloning into 'tripleo-quickstart-extras'... remote: Enumerating objects: 19996, done. remote: Counting objects: 100% (19996/19996), done. remote: Compressing objects: 100% (6835/6835), done. error: RPC failed; result=18, HTTP code = 200iB | 57.00 KiB/s     fatal: The remote end hung up unexpectedly fatal: early EOF fatal: index-pack failed The Error is reproducible with/without the following git configurations [ooo@VM-3O ~]$ git config --list core.compression=-1 http.postbuffer=24288000 Thanks!

3 4

powerkvm CI nova cloning
by Ian Wienand 11 Dec '20

11 Dec '20

Hello, We've identified that powerkvm CI seems to be in a loop attempting to clone nova. It would be helpful if we could get this to stop :) It's just a guess, but we found a similar problem in OpenDev where the executors were attempting to cache the repo, but their default timeout of 300 seconds wasn't long enough to complete the clone. They would kill their clone attempt and start over again immediately. In the Zuul executor logs you'll see something like 2020-12-10 02:02:08,578 ERROR zuul.ExecutorServer: [e: ...] [build:...] Got exception while updating repo gerrit/openstack/nova with a full traceback above. Our solution was to increase the git_timeout [1] to long enough for the clone to complete. The recent upgrade may be to blame. We're experimenting with different options for gc/repack, but cloning the repo from gerrit directly it close to 2gb. Another option is to convert this to cloning to https://opendev.org. Please reach out if there is anything we can do to help. Thanks, -i [1] https://review.opendev.org/c/opendev/system-config/+/766400

2 1

Re: Opendev review crawler
by Chouchen, Moataz 09 Dec '20

09 Dec '20

Dear Jeremy, Ian, Thank you for your kind support and for removing firewall rules that block my access. I will add more delay and use a User-Agent header with my contact details in my future requests as you recommended. I also agree with you that a dump of specific database tables would be great and can encourage other researchers to conduct more studies on code review and benefit from the available rich review data on Opendev projects. Kind regards, Moataz ________________________________ From: Jeremy Stanley Sent: Wednesday, November 4, 2020 12:12 PM To: service-discuss(a)lists.opendev.org Cc: Chouchen, Moataz; Ouni, Ali; Laurin, François Subject: Re: Opendev review crawler On 2020-11-04 00:02:33 +0000 (+0000), Chouchen, Moataz wrote: [...] > For this purpose of study, we need to collect the whole Opendev > data. Right now, we collected data from 2020 to 2015 and still > need the other years. As per requested we will add more waiting > time to my process to let the server catch up. If you also have > any other suggestions that will help, we will be happy to try it. [...] Just adding that delay/throttle will probably suffice, but also Ian's suggestion to set a custom user agent string with contact info on your API requests can help avoid confusion and make it easier for site administrators to reach you if there's a problem with your queries. I've gone ahead and removed the firewall rules we had temporarily blocking your IP addresses as well. If you have any questions, please don't hesitate to get in touch through our service-discuss mailing list. Good luck on the research too, I can't wait to read your findings! -- Jeremy Stanley

2 1

Meeting Agenda for December 8, 2020
by Clark Boylan 07 Dec '20

07 Dec '20

We will meet on December 8, 2020 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Need volunteer to run next week's meeting. Or we can cancel. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Post Gerrit Upgrade Updates **** Configuration tuning ***** Adding config to jgit.config ***** Using strong refs for jgit caches ***** Enabling git protocol v2 ***** Batch user groups and threads **** Zuul results table rendering (ianw 20201208) **** WIP changes (ianw 20201208) **** Zuul needs updates to properly support Gerrit 3.3.0 * General topics ** Bup and Borg Backups (clarkb 20201208) ** Docker rate limiting is being seen (at least in limestone) (clarkb 20201208) ** PTG followups (clarkb 20201208) ** meetpad was not useable for (some?) participants from China (frickler 20201208) *** clarkb trying to set up time with Horace to test a meetpad call with a participant in China. ** Splitting puppet else into specific infra-prod jobs (clarkb 20201208) *** Should be mostly mechanical *** Does it make sense to try and sprint this? Have several people work on getting it done in a short period of time? ** Trusty Upgrade Progress (clarkb 20200929) *** Wiki updates * Open discussion

1 0

Re: [all][infra] CI test result table in the new gerrit review UI
by Ian Wienand 02 Dec '20

02 Dec '20

On Fri, Nov 27, 2020 at 02:11:21PM +0000, Tristan Cacqueray wrote: > From what I understand, we can either use java and the polymer template > system, or the javascript api to implement the zuul results table. This was discussed today in the infra meeting @ [1] There was a couple of broad conclusions. There's no dispute that it works, for now. The maintenace of this in perpituity is the major concern. Some points: There are already two Zuul plugins in the upstream repos zuul : https://gerrit.googlesource.com/plugins/zuul -- This is rather badly namespaced, and is focused on showing the status of Depends-On: changes a cyclic dependencies zuul-status : https://gerrit.googlesource.com/plugins/zuul-status/ -- This is also using a broad namespace, and appears to be related to showing the ongoing job status in the UI directly, but does not show final results. An example I found at [2] from an old wikimedia thread Upstream have shown to take some interest in these plugins when undertaking upgrades to polymer, etc. and thus integrating there is seen as the best practice. Unless we have a strong reason not to, we would like to consume any plugins from upstream for this reason. It would be in our best interest to work with these existing plugins to clarify what they do more directly and come up with some better namespacing to make it more obvious as potential number of plugins grows. The current proposed implementation does not look like any of the upstream plugins. Admittedly, [3] shows this is not a strong eco-system; the documentation is mostly TODO and examples are thin. Implementing this with Reason [4] adds one more significant hurdle; we have no track record of maintaing this javascript dialect and it is not used upstream in any way. It also doesn't use the same build and test frameworks; this may not be a blocker but again creates hurdles by being different to upstream. One more concrete comment is that I'm pretty sure walking the shadow DOM to update things is [5] not quite as intended and we should be using endpoints. In trying to find the best place to start, I've pulled out bits of the checks plugin, image-diff plugin and codeeditor plugin into the simplest thing I could get to a tab with a table in it @ [6]. You can see this for now @ [7]. I think we're going to get more potential eyes using polymer components and making things look like the existing plugins. It's a matter of taste but I think the blank canvas of a separate tab serves the purpose well. I think with the Bazel build bits, we can ultimately write an upstream Zuul job to build this for us in gerrit's Zuul (corvus might know if that's right?) Tristan -- maybe as step one, we could try integrating what you have into that framework? Maybe we rewrite the ReasonML ... but having it something is step 1. Overall I think this is the broad strokes of what we would look at sending upstream. If they think it's too specific, Zuul tenant seems a logical home. We would also like to expand the gate testing to better handle testing plugins. This will involve us automatically adding a repo, sample change, and Zuul user comments during the gate test. A logical extension of this would be to take samples using a headless browser for reporting as artifacts; held nodes can also be used for advanced debugging. This will give us better confidence as we keep our Gerrit up-to-date. I'd like to solict some feedback on the checks plugin/API, which Zuul added support for with [8]. My understanding is that this has never really consolidated upstream and is under redevelopment [9]. I don't think there's much there for us at this point; even still seeing as we are so Zuul centric we might be able to do things with a specific plugin this API will never do. -i [1] http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-12-01-19.01.l… [2] https://imgur.com/a/uBk2oxQ [3] https://gerrit-review.googlesource.com/Documentation/pg-plugin-dev.html [4] https://reasonml.github.io/ [5] https://github.com/softwarefactory-project/zuul-results-gerrit-plugin/blob/… [6] https://github.com/ianw/gerrit-zuul-summary-status [7] https://104.130.172.52/c/openstack/diskimage-builder/+/554002 [8] https://opendev.org/zuul/zuul/commit/e78e948284392477d385d493fc9ec194d54448… [9] https://www.gerritcodereview.com/design-docs/ci-reboot.html

2 1

Meeting Agenda for Decemer 1, 2020
by Clark Boylan 30 Nov '20

30 Nov '20

We will meet with this agenda on Decemer 1, 2020 at 19:00UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Wallaby cycle signing key has been activated https://review.opendev.org/760364 *** Please sign if you haven't yet https://docs.opendev.org/opendev/system-config/latest/signing.html * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Post Gerrit Upgrade Updates **** Configuration tuning ***** Cache sizes ***** Autogc **** Zuul results table rendering **** Switch to Java 11 ***** Any concern about java version specific things in lucene or h2 caches? ***** Shoudl definitely test this on review-test * General topics ** Bup and Borg Backups (clarkb 20201201) ** Docker rate limiting is being seen (at least in limestone) (clarkb 20201201) ** PTG followups (clarkb 20201201) ** meetpad was not useable for (some?) participants from China (frickler 20201201) *** clarkb trying to set up time with Horace to test a meetpad call with a participant in China. ** Splitting puppet else into specific infra-prod jobs (clarkb 20201201) *** Should be mostly mechanical *** Does it make sense to try and sprint this? Have several people work on getting it done in a short period of time? ** Trusty Upgrade Progress (clarkb 20200929) *** Wiki updates * Open discussion

1 0

Third Party CI updates to treat review.opendev.org better
by Clark Boylan 30 Nov '20

30 Nov '20

Hello everyone, Since the Gerrit 3.2 upgrade we've noticed occasional high load from Gerrit. In debugging we've discovered there are cache and other setting tunables that we could configure better for our setup. We are in the process of making these changes to our Gerrit configuration. We have also noticed that a specific Zuul behavior contributes to the load as well. We are asking that third party CI systems running Zuul apply this change [0] or add Gerrit http configuration [1] to your configuration. We recommend the http configuration because it is available as far back as Zuul 3.3.0 and adds more functionality to your setup. This is also an excellent time to update your Third Party CI information [2] (including contact info) as we've noticed that many of these wiki entries may be stale. If you are still running Zuul v2 can you reach out to us and we'll see what we can do to help. [0] https://review.opendev.org/c/zuul/zuul/+/764069 [1] https://zuul-ci.org/docs/zuul/reference/drivers/gerrit.html#attr-%3Cgerrit%… [2] https://wiki.openstack.org/wiki/ThirdPartySystems Clark

1 0

Meeting Agenda for November 17, 2020
by Clark Boylan 16 Nov '20

16 Nov '20

We will meet Novemeber 17, 2020 at 19:00 in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Wallaby cycle signing key has been activated https://review.opendev.org/760364 *** Please sign if you haven't yet https://docs.opendev.org/opendev/system-config/latest/signing.html * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Preparing to upgrade Gerrit from 2.13 to 3.2 **** review-test is back up an running. Now with November 5th state. **** We announced the upgrade is happening November 20 - 22 **** jeepyb bug update and the welcome message hooks rely on database access and will need to be updated or sunsetted **** Upgrade Notes ***** Gave more threads to notedb conversion and it went quicker. Not fast but was an improvement. ***** Need to land the non WIP changes at https://review.opendev.org/#/q/status:open+topic:gerrit-upgrade-prep ***** The first WIP change in https://review.opendev.org/#/q/status:open+topic:gerrit-upgrade-prep is a number of squashed changes to simplify post upgrade config management sync up. ***** https://etherpad.opendev.org/p/opendev-gerrit-3.2-upgrade-plan the plan is here. A few items still need a little work. Please take a look. **** Unknowns ***** Storyboard integration * General topics ** Bup and Borg Backups (clarkb 20201117) ** PTG followups (clarkb 20201117) ** meetpad was not useable for (some?) participants from China (frickler 20201110) *** clarkb trying to set up time with Horace to test a meetpad call with a participant in China. ** Long term plans for running openstackid.org (clarkb 20201110) ** Splitting puppet else into specific infra-prod jobs (clarkb 20201110) *** Should be mostly mechanical *** Does it make sense to try and sprint this? Have several people work on getting it done in a short period of time? ** Trusty Upgrade Progress (clarkb 20200929) *** Wiki updates * Open discussion

1 0

Meeting Agenda for November 10, 2020
by Clark Boylan 09 Nov '20

09 Nov '20

We will meet with this agenda Novemer 10, 2020 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Wallaby cycle signing key has been activated https://review.opendev.org/760364 *** Please sign if you haven't yet https://docs.opendev.org/opendev/system-config/latest/signing.html * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Preparing to upgrade Gerrit from 2.13 to 3.2 **** review-test is back up an running. Now with November 5th state. **** We announced the upgrade is happening November 20 - 22 **** jeepyb bug update and the welcome message hooks rely on database access and will need to be updated or sunsetted **** Upgrade Notes ***** Gave more threads to notedb conversion and it went quicker. Not fast but was an improvement. ***** Don't think we will do the surrogate Gerrit as it increases complexity of testing and implementation. We only need to do this once. ***** Gerrit 3.2 seems to want email addresses set on accounts in order to modify them (eg add an ssh pub key). This affects our admin accounts and the project creator. ***** How do we safely land the stack at https://review.opendev.org/#/q/status:open+topic:gerrit-upgrade-prep since Zuul will try to run ansible in sequence for each change? **** Unknowns ***** Storyboard integration *** Luca has offered to do a conference call with us. Let me know if interested and I'll include you for scheduling if/when that happens. * General topics ** PTG followups (clarkb 20201110) ** meetpad was not useable for (some?) participants from China (frickler 20201110) *** clarkb trying to set up time with Horace to test a meetpad call with a participant in China. ** Bup and Borg Backups (clarkb 20201110) *** Borg backup all the things https://review.opendev.org/#/c/761855/ ** Long term plans for running openstackid.org (clarkb 20201110) ** Splitting puppet else into specific infra-prod jobs (clarkb 20201110) *** Should be mostly mechanical *** Does it make sense to try and sprint this? Have several people work on getting it done in a short period of time? ** Trusty Upgrade Progress (clarkb 20200929) *** Wiki updates * Open discussion

1 0

Re: [service-announce] review.opendev.org Gerrit outage and upgrade 15:00UTC November 20 to 01:00UTC November 23, 2020
by Antoine Musso 05 Nov '20

05 Nov '20

Hello, Wikimedia has upgraded from 2.16 to 3.2 in June 2020 conducted by Christian Aistleitner. He wrote a nice report on the upstream mailling list: https://groups.google.com/g/repo-discuss/c/G5wucKJg9Ag On 27/10/2020 22:16, Clark Boylan wrote: > The OpenDev team is planning a long weekend Gerrit outage on review.opendev.org starting 15:00UTC November 20 and running to 01:00UTC November 23, 2020 in order to upgrade to Gerrit 3.2. > > The upgrade has two major portions. First we will incrementally move from our current version 2.13 to 2.16. Each point release requires a database migration and git indexing operations that take the majority of the time. From Christian report, you should be able to skip the git indexing between each minor upgrades and only do a full reindexing once you are upgraded to 3.2. The git indexing had an issue which is that the changes to index were split by repository. In the worse case scenario, if you have a thousand of small repositories and one very large one, the later was only processed by a single thread. Christian added code to chunk by changes regardless of the repository, that has dramatically speed up the indexing. If I am not mistaken, the commit is 20784548c3fb and it has been released in 2.16.22, 3.0.12, 3.1.8 and 3.2.3. Given you should be able to skip reindexing between upgrades. Make sure to use 3.2.3 to benefit from the faster indexing. > The second major part, once at version 2.16, is to convert Gerrit to use the new NoteDB backend. This stores reviews together with code in the git trees, rather than in a separate database. There are known problems converting to NoteDB from any version prior to 2.16, which is why we need the initial upgrade steps. This is this slowest portion of the upgrade process, and the one most prone to unforeseen issues despite extensive pre-testing. We will be creating snapshots to facilitate quick fallback if required. Once this is complete, we can move to the 3.x series and incrementally upgrade from 3.0 to 3.2. 2.16 has received improvements for the NoteDB migration. Again make sure to use the latest patch release (2.16.23 at this time). <snip> > Some important things to know: > * Gerrit's web UI will be changing. We don't have a lot of control over this. As a result we'll lose some existing CI result rendering niceness that we have on 2.13 (in particular the summary table and CI results toggle will go away). We hope that once we've upgraded we can investigate solving this through Gerrit plugins. summary table ------------- The new Gerrit web UI is enterely JavaScript driven (using https://www.polymer-project.org/ ) and relies on the REST API to fetch data. For the summary table, Wikimedia has a light version of the one you have which got borrowed from somewhere else. See "gr-test-result-table-module" at: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/hea… It is hopefully not too complicated to implement. CI results toggle ----------------- That is now built-in Gerrit, the UI will have a "Only comments" toggle which filters out any message with a tag prefixed by "autogenerated:". Thus in Wikimedia good old Zuul 2.5 we have simply tagged with "autogenerated:ci" by using: success: gerrit: verified: 2 tag: autogenerated:ci Result: https://phabricator.wikimedia.org/T48148#6294913 (there are a few more details on that task). Another one is that the SUCCESS/FAILURE message can no more be highlighted via rewriting the message and using CSS. The way text is parsed has completely changed and its now impossible to do it via the commentlinks Gerrit settings. As a result on Zuul 2.5 we have set job_name_in_report = false and lack the nice formatting and green/red coloring. I am not sure whether that still applies to Zuul 3.x though. Wikimedia task: https://phabricator.wikimedia.org/T256575 That being said, it is definitely possible to write a Gerrit Javascript plugin that would parse the message and prettify the results comment on the client side. <snip> > * Q&A * > If this upgrade isn't perfect why are we doing it anyway? > We've come to the realization that if we don't make imperfect progress we'll never make any progress. We have decided that the benefits outweigh the known drawbacks and we'll do our best to work on those issues after the upgrade. I would add a couple user facing improvements: The new UI alone is definitely worth the upgrade. It is way more pleasant than the old one. Albeit it needs some days to adjust, you will surely never look back :] <snip> The cherry on the cake is that with the upgrade comes support for Git protocol version 2. In short, when enabled, it makes fetches from big repository an order of magnitude faster. I wrote a quick blog post about git protocol v2 at: https://phabricator.wikimedia.org/J199 And Google announcement with lot more details: https://opensource.googleblog.com/2018/05/introducing-git-protocol-version-… I wish you the very best for the maintenance. Thank you for making it happen! -- Antoine "hashar" Musso

3 2