Hi Zuulers,
We just discovered an (in our case) unpleasant behaviour of zuul and semaphores.
Scenario: We have a job ‘foo’ with a semaphore of 5. Now the queue grows and we have 30 items with job ‘foo’ in the queue. The current behaviour of zuul is that it first gets the nodes and then locks the semaphore. This means that 5 jobs run and 25 wait for the semaphore but already got and locked their nodes. In case the job is expensive and long running this can become quite a big resource waste problem if we’re in a quota constrained situation.
After some discussion on IRC this would be a proposal to solve this problem: We could make the order of ‘request nodes’ and ‘lock semaphore’ configurable per job with a default to the safe behaviour of ‘lock semaphore’ then ‘request nodes’. Jobs would still have the possibility to do the fast path by requesting the nodes first. This makes sense if you have lightweight jobs that should run in a fast consecutive sequence and you know that some waste of resources is ok.
What do the other semaphore users out there think about this?
Kind regards Tobias