On 2020-11-03 17:43:53 +0000 (+0000), Chouchen, Moataz wrote:
I am Moataz Chouchen a Phd Student in Ets Montréal under the supervision of Professor Ali Ouni. I am sending this email to you to explain my requests of the Opendev data to respond to your request regarding Opendev code review data crawling.
Thanks so much for getting back to us! I'm sorry we ended up blocking access to your various systems, but we were at a loss for how to otherwise get in contact with whoever was running them since the queries were being performed anonymously (at least until we finally saw similar API requests originating from your university).
In fact, my Phd subject consists of the identification, the localization, and the understanding of Modern Code Review (MCR) problems (also called antipatterns). For this purpose, we need to crawl code review data from different projects (including Opendev) since this data will help us to study the process of MCR in these projects and their associated anti-patterns. Specifically, we need the whole data of Opendev to perform data analysis methods on them (including code review metrics distributions, social graph analysis etc) and study them in more depth. For this reason, we created a script to crawl data from Opendev and use them in our empirical study on MCR antipatterns in Opendev.
This sounds really interesting, and I'm sure this mailing list would also love to receive a link to your research once you've completed and published it. I'm thrilled you have an interest in looking into these topics, especially with regards to the usage patterns and trends for a service we're operating.
I would like to thank you for your interest to understand the issues behind our requests and your intention to solve the issue for both sides. I am looking forward to collaborating with you and I would like to know if there are any rules that I should follow when requesting data from your servers and I will be more than happy If you can help me crawling the data and provide an access for me. I am also at your disposal for further clarifications regarding the requests. I hope I hear from you soon.
Mainly what I think we need to know is whether you've already collected the data you need from our Gerrit API, or how much you still need to query. The server wants to cache requests in memory for the sake of efficiency, but as you're aware we've got quite a lot of data. When queries for older data consume available memory on the server it begins to spend a lot more time trying to aggressively garbage collect cache contents, and that degrades its ability to serve requests for you and for other users.
If you still have a lot left to query, you might try to introduce a bit of a pause between each batch of paginated results to give the server some time to catch up freeing memory for caching other requests. Also if obtaining the data in a different form would be useful to you, we might be able to look into performing a dump of specific database tables so you don't have to slowly trickle it out of the API.
Please do reply to the service-discuss mailing list at your earliest convenience and let us know how we can better help you with your research. Thanks again!