Hello Zuul folk,

We have 2 projects using the same Queue defined as: - queue: name: Topic allow-circular-dependencies: true dependencies-by-topic: true per-branch: true

With this job and project setup:

- job: name: project1-build runs: ...

- job: name: project2-build runs: ... dependencies: project1-build

- project: name: project1 queue: Topic gate: jobs: - project1-build

- project: name: project2 queue: Topic gate: jobs: - project2-build

A developer gates a Gerrit change for project1, followed closely by a change for project2. Both changes share the same value in the Topic field, and the same branch.

When checking the web status page, the changes appear to enqueue in the right order, under the same Queue. The issue is that job project2-build is immediately started without waiting for the completion project1-build, leading to incorrect testing. What's more, the change for project1 is merged on successful testing despite failures in the jobs for project2. It seems Zuul simply does not treat these two changes as a set of circular dependencies, though, to my understanding, it should. We've tried checking the <tenant>/status API endpoint while this is happening, and noticed no "bundle_id" field is shown for either of the "items" in the queue - which we assume is a symptom of the same underlying issue.

Stranger still, we have a Zuul Staging environment where this issue does not reproduce, project2-build waits for project-build and a "bundle_id" does exist in the status JSON object every time. Both the Staging and the Production environments use the same version of Zuul (6.4.0), the only difference between the environments we can think of is that the Staging environment runs a single instance of scheduler/executor/web (on the same machine), while the Production environment uses multiple instances of scheduler / executor / merger (but only 1 web) spread across multiple hosts.

Is it possible we're hitting some race-condition, or some kind of cache (Gerrit / Zookeeper / Zuul itself)? Are we missing something here? Is there anything else we can try and do to figure this out?

Regards, Arichist