November 2020 - service-discuss

Re: Opendev review crawler
by Chouchen, Moataz 10 Dec '20

10 Dec '20

Dear Jeremy, Ian, Thank you for your kind support and for removing firewall rules that block my access. I will add more delay and use a User-Agent header with my contact details in my future requests as you recommended. I also agree with you that a dump of specific database tables would be great and can encourage other researchers to conduct more studies on code review and benefit from the available rich review data on Opendev projects. Kind regards, Moataz ________________________________ From: Jeremy Stanley Sent: Wednesday, November 4, 2020 12:12 PM To: service-discuss(a)lists.opendev.org Cc: Chouchen, Moataz; Ouni, Ali; Laurin, François Subject: Re: Opendev review crawler On 2020-11-04 00:02:33 +0000 (+0000), Chouchen, Moataz wrote: [...] > For this purpose of study, we need to collect the whole Opendev > data. Right now, we collected data from 2020 to 2015 and still > need the other years. As per requested we will add more waiting > time to my process to let the server catch up. If you also have > any other suggestions that will help, we will be happy to try it. [...] Just adding that delay/throttle will probably suffice, but also Ian's suggestion to set a custom user agent string with contact info on your API requests can help avoid confusion and make it easier for site administrators to reach you if there's a problem with your queries. I've gone ahead and removed the firewall rules we had temporarily blocking your IP addresses as well. If you have any questions, please don't hesitate to get in touch through our service-discuss mailing list. Good luck on the research too, I can't wait to read your findings! -- Jeremy Stanley

2 1

Meeting Agenda for Decemer 1, 2020
by Clark Boylan 30 Nov '20

30 Nov '20

We will meet with this agenda on Decemer 1, 2020 at 19:00UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Wallaby cycle signing key has been activated https://review.opendev.org/760364 *** Please sign if you haven't yet https://docs.opendev.org/opendev/system-config/latest/signing.html * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Post Gerrit Upgrade Updates **** Configuration tuning ***** Cache sizes ***** Autogc **** Zuul results table rendering **** Switch to Java 11 ***** Any concern about java version specific things in lucene or h2 caches? ***** Shoudl definitely test this on review-test * General topics ** Bup and Borg Backups (clarkb 20201201) ** Docker rate limiting is being seen (at least in limestone) (clarkb 20201201) ** PTG followups (clarkb 20201201) ** meetpad was not useable for (some?) participants from China (frickler 20201201) *** clarkb trying to set up time with Horace to test a meetpad call with a participant in China. ** Splitting puppet else into specific infra-prod jobs (clarkb 20201201) *** Should be mostly mechanical *** Does it make sense to try and sprint this? Have several people work on getting it done in a short period of time? ** Trusty Upgrade Progress (clarkb 20200929) *** Wiki updates * Open discussion

1 0

Third Party CI updates to treat review.opendev.org better
by Clark Boylan 30 Nov '20

30 Nov '20

Hello everyone, Since the Gerrit 3.2 upgrade we've noticed occasional high load from Gerrit. In debugging we've discovered there are cache and other setting tunables that we could configure better for our setup. We are in the process of making these changes to our Gerrit configuration. We have also noticed that a specific Zuul behavior contributes to the load as well. We are asking that third party CI systems running Zuul apply this change [0] or add Gerrit http configuration [1] to your configuration. We recommend the http configuration because it is available as far back as Zuul 3.3.0 and adds more functionality to your setup. This is also an excellent time to update your Third Party CI information [2] (including contact info) as we've noticed that many of these wiki entries may be stale. If you are still running Zuul v2 can you reach out to us and we'll see what we can do to help. [0] https://review.opendev.org/c/zuul/zuul/+/764069 [1] https://zuul-ci.org/docs/zuul/reference/drivers/gerrit.html#attr-%3Cgerrit%… [2] https://wiki.openstack.org/wiki/ThirdPartySystems Clark

1 0

Meeting Agenda for November 17, 2020
by Clark Boylan 16 Nov '20

16 Nov '20

We will meet Novemeber 17, 2020 at 19:00 in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Wallaby cycle signing key has been activated https://review.opendev.org/760364 *** Please sign if you haven't yet https://docs.opendev.org/opendev/system-config/latest/signing.html * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Preparing to upgrade Gerrit from 2.13 to 3.2 **** review-test is back up an running. Now with November 5th state. **** We announced the upgrade is happening November 20 - 22 **** jeepyb bug update and the welcome message hooks rely on database access and will need to be updated or sunsetted **** Upgrade Notes ***** Gave more threads to notedb conversion and it went quicker. Not fast but was an improvement. ***** Need to land the non WIP changes at https://review.opendev.org/#/q/status:open+topic:gerrit-upgrade-prep ***** The first WIP change in https://review.opendev.org/#/q/status:open+topic:gerrit-upgrade-prep is a number of squashed changes to simplify post upgrade config management sync up. ***** https://etherpad.opendev.org/p/opendev-gerrit-3.2-upgrade-plan the plan is here. A few items still need a little work. Please take a look. **** Unknowns ***** Storyboard integration * General topics ** Bup and Borg Backups (clarkb 20201117) ** PTG followups (clarkb 20201117) ** meetpad was not useable for (some?) participants from China (frickler 20201110) *** clarkb trying to set up time with Horace to test a meetpad call with a participant in China. ** Long term plans for running openstackid.org (clarkb 20201110) ** Splitting puppet else into specific infra-prod jobs (clarkb 20201110) *** Should be mostly mechanical *** Does it make sense to try and sprint this? Have several people work on getting it done in a short period of time? ** Trusty Upgrade Progress (clarkb 20200929) *** Wiki updates * Open discussion

1 0

Meeting Agenda for November 10, 2020
by Clark Boylan 09 Nov '20

09 Nov '20

We will meet with this agenda Novemer 10, 2020 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Wallaby cycle signing key has been activated https://review.opendev.org/760364 *** Please sign if you haven't yet https://docs.opendev.org/opendev/system-config/latest/signing.html * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Preparing to upgrade Gerrit from 2.13 to 3.2 **** review-test is back up an running. Now with November 5th state. **** We announced the upgrade is happening November 20 - 22 **** jeepyb bug update and the welcome message hooks rely on database access and will need to be updated or sunsetted **** Upgrade Notes ***** Gave more threads to notedb conversion and it went quicker. Not fast but was an improvement. ***** Don't think we will do the surrogate Gerrit as it increases complexity of testing and implementation. We only need to do this once. ***** Gerrit 3.2 seems to want email addresses set on accounts in order to modify them (eg add an ssh pub key). This affects our admin accounts and the project creator. ***** How do we safely land the stack at https://review.opendev.org/#/q/status:open+topic:gerrit-upgrade-prep since Zuul will try to run ansible in sequence for each change? **** Unknowns ***** Storyboard integration *** Luca has offered to do a conference call with us. Let me know if interested and I'll include you for scheduling if/when that happens. * General topics ** PTG followups (clarkb 20201110) ** meetpad was not useable for (some?) participants from China (frickler 20201110) *** clarkb trying to set up time with Horace to test a meetpad call with a participant in China. ** Bup and Borg Backups (clarkb 20201110) *** Borg backup all the things https://review.opendev.org/#/c/761855/ ** Long term plans for running openstackid.org (clarkb 20201110) ** Splitting puppet else into specific infra-prod jobs (clarkb 20201110) *** Should be mostly mechanical *** Does it make sense to try and sprint this? Have several people work on getting it done in a short period of time? ** Trusty Upgrade Progress (clarkb 20200929) *** Wiki updates * Open discussion

1 0

Re: [service-announce] review.opendev.org Gerrit outage and upgrade 15:00UTC November 20 to 01:00UTC November 23, 2020
by Antoine Musso 05 Nov '20

05 Nov '20

Hello, Wikimedia has upgraded from 2.16 to 3.2 in June 2020 conducted by Christian Aistleitner. He wrote a nice report on the upstream mailling list: https://groups.google.com/g/repo-discuss/c/G5wucKJg9Ag On 27/10/2020 22:16, Clark Boylan wrote: > The OpenDev team is planning a long weekend Gerrit outage on review.opendev.org starting 15:00UTC November 20 and running to 01:00UTC November 23, 2020 in order to upgrade to Gerrit 3.2. > > The upgrade has two major portions. First we will incrementally move from our current version 2.13 to 2.16. Each point release requires a database migration and git indexing operations that take the majority of the time. From Christian report, you should be able to skip the git indexing between each minor upgrades and only do a full reindexing once you are upgraded to 3.2. The git indexing had an issue which is that the changes to index were split by repository. In the worse case scenario, if you have a thousand of small repositories and one very large one, the later was only processed by a single thread. Christian added code to chunk by changes regardless of the repository, that has dramatically speed up the indexing. If I am not mistaken, the commit is 20784548c3fb and it has been released in 2.16.22, 3.0.12, 3.1.8 and 3.2.3. Given you should be able to skip reindexing between upgrades. Make sure to use 3.2.3 to benefit from the faster indexing. > The second major part, once at version 2.16, is to convert Gerrit to use the new NoteDB backend. This stores reviews together with code in the git trees, rather than in a separate database. There are known problems converting to NoteDB from any version prior to 2.16, which is why we need the initial upgrade steps. This is this slowest portion of the upgrade process, and the one most prone to unforeseen issues despite extensive pre-testing. We will be creating snapshots to facilitate quick fallback if required. Once this is complete, we can move to the 3.x series and incrementally upgrade from 3.0 to 3.2. 2.16 has received improvements for the NoteDB migration. Again make sure to use the latest patch release (2.16.23 at this time). <snip> > Some important things to know: > * Gerrit's web UI will be changing. We don't have a lot of control over this. As a result we'll lose some existing CI result rendering niceness that we have on 2.13 (in particular the summary table and CI results toggle will go away). We hope that once we've upgraded we can investigate solving this through Gerrit plugins. summary table ------------- The new Gerrit web UI is enterely JavaScript driven (using https://www.polymer-project.org/ ) and relies on the REST API to fetch data. For the summary table, Wikimedia has a light version of the one you have which got borrowed from somewhere else. See "gr-test-result-table-module" at: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/hea… It is hopefully not too complicated to implement. CI results toggle ----------------- That is now built-in Gerrit, the UI will have a "Only comments" toggle which filters out any message with a tag prefixed by "autogenerated:". Thus in Wikimedia good old Zuul 2.5 we have simply tagged with "autogenerated:ci" by using: success: gerrit: verified: 2 tag: autogenerated:ci Result: https://phabricator.wikimedia.org/T48148#6294913 (there are a few more details on that task). Another one is that the SUCCESS/FAILURE message can no more be highlighted via rewriting the message and using CSS. The way text is parsed has completely changed and its now impossible to do it via the commentlinks Gerrit settings. As a result on Zuul 2.5 we have set job_name_in_report = false and lack the nice formatting and green/red coloring. I am not sure whether that still applies to Zuul 3.x though. Wikimedia task: https://phabricator.wikimedia.org/T256575 That being said, it is definitely possible to write a Gerrit Javascript plugin that would parse the message and prettify the results comment on the client side. <snip> > * Q&A * > If this upgrade isn't perfect why are we doing it anyway? > We've come to the realization that if we don't make imperfect progress we'll never make any progress. We have decided that the benefits outweigh the known drawbacks and we'll do our best to work on those issues after the upgrade. I would add a couple user facing improvements: The new UI alone is definitely worth the upgrade. It is way more pleasant than the old one. Albeit it needs some days to adjust, you will surely never look back :] <snip> The cherry on the cake is that with the upgrade comes support for Git protocol version 2. In short, when enabled, it makes fetches from big repository an order of magnitude faster. I wrote a quick blog post about git protocol v2 at: https://phabricator.wikimedia.org/J199 And Google announcement with lot more details: https://opensource.googleblog.com/2018/05/introducing-git-protocol-version-… I wish you the very best for the maintenance. Thank you for making it happen! -- Antoine "hashar" Musso

3 2

Re: Opendev review crawler
by Chouchen, Moataz 04 Nov '20

04 Nov '20

Hi Jeremy, Thank you for your quick answer and for your support we will be more than happy to share our research with you once published! For this purpose of study, we need to collect the whole Opendev data. Right now, we collected data from 2020 to 2015 and still need the other years. As per requested we will add more waiting time to my process to let the server catch up. If you also have any other suggestions that will help, we will be happy to try it. We will be happy to receive an answer from you Kind regards, Moataz ________________________________ From: Jeremy Stanley Sent: Tuesday, November 3, 2020 10:42 AM To: service-discuss(a)lists.opendev.org Cc: Chouchen, Moataz; Ouni, Ali; Laurin, François Subject: Re: Opendev review crawler On 2020-11-03 17:43:53 +0000 (+0000), Chouchen, Moataz wrote: > I am Moataz Chouchen a Phd Student in Ets Montréal under the > supervision of Professor Ali Ouni. I am sending this email to you > to explain my requests of the Opendev data to respond to your > request regarding Opendev code review data crawling. Thanks so much for getting back to us! I'm sorry we ended up blocking access to your various systems, but we were at a loss for how to otherwise get in contact with whoever was running them since the queries were being performed anonymously (at least until we finally saw similar API requests originating from your university). > In fact, my Phd subject consists of the identification, the > localization, and the understanding of Modern Code Review (MCR) > problems (also called antipatterns). For this purpose, we need to > crawl code review data from different projects (including Opendev) > since this data will help us to study the process of MCR in these > projects and their associated anti-patterns. Specifically, we need > the whole data of Opendev to perform data analysis methods on them > (including code review metrics distributions, social graph > analysis etc) and study them in more depth. For this reason, we > created a script to crawl data from Opendev and use them in our > empirical study on MCR antipatterns in Opendev. This sounds really interesting, and I'm sure this mailing list would also love to receive a link to your research once you've completed and published it. I'm thrilled you have an interest in looking into these topics, especially with regards to the usage patterns and trends for a service we're operating. > I would like to thank you for your interest to understand the > issues behind our requests and your intention to solve the issue > for both sides. I am looking forward to collaborating with you and > I would like to know if there are any rules that I should follow > when requesting data from your servers and I will be more than > happy If you can help me crawling the data and provide an access > for me. I am also at your disposal for further clarifications > regarding the requests. I hope I hear from you soon. Mainly what I think we need to know is whether you've already collected the data you need from our Gerrit API, or how much you still need to query. The server wants to cache requests in memory for the sake of efficiency, but as you're aware we've got quite a lot of data. When queries for older data consume available memory on the server it begins to spend a lot more time trying to aggressively garbage collect cache contents, and that degrades its ability to serve requests for you and for other users. If you still have a lot left to query, you might try to introduce a bit of a pause between each batch of paginated results to give the server some time to catch up freeing memory for caching other requests. Also if obtaining the data in a different form would be useful to you, we might be able to look into performing a dump of specific database tables so you don't have to slowly trickle it out of the API. Please do reply to the service-discuss mailing list at your earliest convenience and let us know how we can better help you with your research. Thanks again! -- Jeremy Stanley

2 1

opendev/gear reviews
by Antoine Musso 04 Nov '20

04 Nov '20

Hello, The opendev/gear is a pure python implementation of the Gearman protocol, written for Zuul 2.x. The project has a few pending changes that could use a final approval, one is required to unbreak the CI jobs. I am interested in helping on the maintenance, though I don't know which process I can start to eventually be blessed with W+1 rights on this repository. I will be more than happy to help on that front. Meanwhile, here is a summary of the changes that could certainly be approved immediately: "use python3 as context for build-python-release" https://review.opendev.org/#/c/742165/ Without it, the job fails for all proposed change. The same fix has been made to Zuul via https://review.opendev.org/#/c/742761/ "Bump crypto requirement to accommodate security standards" https://review.opendev.org/#/c/742117/ Raise the RSA security certificate bits from 1024 to 2048 and sign the key with sha256 instead of sha1. Without it, it seems pyOpenSSL complains the key is too small and the hash system too weak. "wakeConnections: Randomize connections before scanning them" https://review.opendev.org/#/c/747119/ That is an issue we had for a while at Wikimedia, the server wakes connection in the order they get registered. The connection that registered first ends up executing way more work than the last registered one. By shuffling the list of connections before waking them up, that helps spreading the load. "Modify connection timeout process" https://review.opendev.org/#/c/531059/ Resolves a potential deadlock, a connection timeout would not release the connection lock. "Move handleDisconnect into BaseClientServer" https://review.opendev.org/#/c/714709/ Really a noop, it is just moving a method to the parent class which is the sole use of it. "Added Docker image builds" https://review.opendev.org/#/c/688446/ Generates a Docker image to run the gear daemon. cheers, -- Antoine "hashar" Musso

1 0

Opendev review crawler
by Chouchen, Moataz 03 Nov '20

03 Nov '20

Dear Opendev administrators, I am Moataz Chouchen a Phd Student in Ets Montréal under the supervision of Professor Ali Ouni. I am sending this email to you to explain my requests of the Opendev data to respond to your request regarding Opendev code review data crawling. In fact, my Phd subject consists of the identification, the localization, and the understanding of Modern Code Review (MCR) problems (also called antipatterns). For this purpose, we need to crawl code review data from different projects (including Opendev) since this data will help us to study the process of MCR in these projects and their associated anti-patterns. Specifically, we need the whole data of Opendev to perform data analysis methods on them (including code review metrics distributions, social graph analysis etc) and study them in more depth. For this reason, we created a script to crawl data from Opendev and use them in our empirical study on MCR antipatterns in Opendev. I would like to thank you for your interest to understand the issues behind our requests and your intention to solve the issue for both sides. I am looking forward to collaborating with you and I would like to know if there are any rules that I should follow when requesting data from your servers and I will be more than happy If you can help me crawling the data and provide an access for me. I am also at your disposal for further clarifications regarding the requests. I hope I hear from you soon. Thank you Please stay safe, Moataz

3 2

Team meeting Agenda for November 3, 2020
by Clark Boylan 02 Nov '20

02 Nov '20

We will meet November 3, 2020 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Wallaby cycle signing key has been activated https://review.opendev.org/760364 *** Please sign if you haven't yet https://docs.opendev.org/opendev/system-config/latest/signing.html ** Note the Daylight Savings Time switch has happened in many parts of the world. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-… Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Preparing to upgrade Gerrit from 2.13 to 3.2 **** Need to rerun upgrade testing on review-test and revive it for usage testing **** We announced the upgrade is happening November 20 - 22 **** jeepyb bug/spec update hooks and the welcome message hook rely on database access and will need to be updated or sunsetted **** Upgrade Process ***** Backup then upgrade from 2.13 to 2.16. This is our fallback midpoint checkpoint ***** Backup again then migrate to notedb on 2.16 ***** Upgrade to 3.2 ***** Upgrade to 2.16 along with backups should be doable in a day. Then notedb migration can happen overnight with 3.2 upgrade happening on day two. ***** Looking into using a surrogate Gerrit running on a very performance vexxhost flavor to do the upgrade/data migrations which may speed things up. Needs testing. **** Unknowns ***** Storyboard integration *** Luca has offered to do a conference call with us. Let me know if interested and I'll include you for scheduling if/when that happens. * General topics ** PTG followups (clarkb 20201103) ** meetpad was not useable for (some?) participants from China (frickler 20201030) *** neutron went back to using Zoom for their last sessions because of this *** can we (maybe via help from the foundation involving their local members) work on improving this? *** would be sad to see open tools not being able to be used because of this ** Bup and Borg Backups (clarkb 20200929) *** Ethercalc to be the first borg backed up service ** Long term plans for running openstackid.org (clarkb 20201103) ** Splitting puppet else into specific infra-prod jobs (clarkb 20200929) *** Should be mostly mechanical *** Does it make sense to try and sprint this? Have several people work on getting it done in a short period of time? ** Trusty Upgrade Progress (clarkb 20200929) *** Wiki updates * Open discussion

1 0