Hi,

In OpenStack, we occasionally have periods where, for various reasons, one project, or a group of projects, uses a sufficiently large number of node resources that it significantly delays results for other projects. We happen to be in one of those periods, so I've had some time to think about ways to mitigate this.

First, let me say that one of the potential causes is long and unreliable tests. There are efforts underway to communicate that and hopefully address it. But that topic isn't really for this list.

Our concern is, when a system gets into this state, can or should we do anything in Zuul to mitigate it?

I would like to avoid (for now) the use of project-based quotas. I could certainly see us adopting tenant quotas in the future, but that doesn't help us in this case where all of the contention is within a single tenant. Within a tenant, project quotas just seem to be too much work. I wouldn't even begin to know how to set them fairly. And what we really want to do is use as many node resources are available, but use them in a fair manner.

I think we can accomplish this by augmenting our use of node request priority. Currently, we use a priority queue with very coarse granularity (high, normal, low) set by Zuul according to pipeline. I'd like to adjust the system so that each subsequent node request from a given project has a lower priority. In other words, at any given time, the first change from project A in gate has the highest priority, and the same is true for project B. The second change from A has a slightly lower priority, and so forth.

The problem with this is that the node request priorities much change over time as the contents of the pipeline queues change. Once the first change from A merges, the second change from A should now have the highest priority. That makes this incompatible with our current priority queue system.

Fortunately, Tobias has been working on adding caching to Nodepool's use of Zookeeper. If we start having node requests cached in-memory in the launchers, the in-memory records can be automatically updated by Zookeeper callbacks. This means that we can encode our additional priority information on an attribute in the Znode, updating it as necessary, and combine it with the already existing information from the priority queue to determine the order in which to fulfill requests.

To summarize, I propose the following:

* The current priority queue system for node requests will remain in place. I'll call that "precedence" to distinguish it from the new bits.

* Add a "priority" field to the node request znodes in Zookeeper. This is an integer which indicates the priority of this request relative to others at the same pipeline precedence level.

* Zuul will update the priority field after each time a pipeline queue processor runs, according to internal heuristics. I propose that initially we decrement the priority for each subsequent change from the same project, though we may also add an option to do so by project-group, as determined by membership in a shared change queue.

* We add temporary feature flags to Zuul so that we can try out these new features (including different options for how to determine the priority) before committing to future support. Whatever we decide to keep, we may also want to make optional, so these flags may eventually turn into permanent configuration flags.

* We implement a TreeCache for node requests in nodepool.

* When nodepool begins to iterate over all node requests in order to handle them, it first sorts by precedence, then priority. Items with the same precedence and priority will remain in their chronological order of submission.

I think that will help make node distribution more fair, and it builds on top of the current system without changing it so it should be forward/backward compatible, and even optional.

-Jim