On 1/20/22 6:58 PM, Xie, Zhiqi (NSB - CN/Hangzhou) wrote:

Hi,

Our teams are using zuul based on openstack zuul 2.x. We have hundreds of users and we had performance impact during rush hour.

There are many connections synced in zuul’s main loop, including gerrit query/review/git-upload-pack and gearman task sending. In rush hour our gerrit will be slow to response (at most it will take 2 secs to response the query and 1 min to response git-upload-pack), and we have hundreds of jobs per gerrit change and thus we have too many job-launch task and job-set-description task. These connection make the zuul rather slow in the rush hours. In the worst situation, zuul will go through a loop in 30 mins.

We have to make modification to solve this locally, to make all the connections asynced. Now the performance impact is eased.

And recently we are planning to go up to zuul version 4.x or 5.x, so we have some quick peek of current zuul v4. We find that gearman connection is gone but zookeeper connections comes instead. And zookeeper connection are still synced. And gerrit support http rest api, but the query is still synced. Therefore we are worried about performance in our environment. We want to keep pace with community so we would like some answers:

Zuul version 2 is very different from version 4/5, so extrapolating from v2 behavior may not be useful. It's probably best to start with the current state and go from there. Happily, many of the concerns you cite are explicitly what we seek to improve in v5, so I'm optimistic. :)

1.How will zuul v4/5 solve the slowness when gerrit is not that fast. We will encounter gerrit connection lasting for seconds in rush hour and don’t want zuul’s main thread go slow.

Gerrit event processing and queries are handled outside of Zuul's main scheduler loop, so event inputs don't slow down processing. Reporting to Gerrit is performed synchronously in the main loop to maintain strict sequencing, but even this is mitigated in v5 with multiple schedulers (see below).

2.If zookeeper’s connection is not fast, the zuul’s main thread will cost too much time in getting response from zookeeper. How to prevent this to happen? Will adding more nodes of zookeeper in a cluster help?

ZooKeeper is very fast and is not a bottleneck in any environment I've seen. Adding more nodes beyond 3 won't necessarily help.

3.Does zuul have plan to make the network I/O asynced to improve performance?

In v5, almost everything that can be performed asynchronously (by which I mean outside of the main scheduler loop) currently is. This means each pipeline is processed synchronously as quickly as possible. To scale beyond that, we add multiple schedulers so that multiple pipelines can be processed simultaneously. -Jim