Parallel production jobs changes
iwienand at redhat.com
Tue Dec 7 05:58:44 UTC 2021
Thank you for prior reviews and sticking with this rather complicated
fiddling; we ended up with some failures and reverts that I hope have
Firstly, we have a typo in matching install playbooks for our current
roles that has propogated as we copy-paste; a perfunctory fix
I am then proposing we rename the job we currently call
infra-prod-install-ansible to infra-prod-bootstrap-bridge. This is a
back-to-the-future situation; as it was originally called something
like "install-bridge". I think this better reflects where it will end
up, as hopefully revealed in the following steps
After this, we need to run infra-prod-bootstrap-bridge as the base job
before all other production jobs. To recap -- this will be the
synchronisation point that puts Zuul's checkout of system-config on
the bastion host (bridge) that is then used to deploy all production
systems (eventually, in as parallel way as possible).
This revealed what I think is a problem with the original job -- it
runs the install-ansible role via
playbooks/zuul/run-production-playbook.yaml. This is a
chicken-and-egg problem -- run-production-playbook.yaml uses the
Ansible installed on bridge to run playbooks in system-config to
... install Ansible on bridge. Addressed with:
this makes infra-prod-bootstrap bridge a stand-alone job that should
* install the required version of Ansible on bridge
* setup system-config to Zuul's checkout for the buildset
For sanity, we keep the current parent of the infra-prod-* jobs the
same -- this means each infra-prod-* job will still be re-checking-out
system-config as it runs. We should validate
* the install of ansible/openstacksdk/etc. is actually idempotent;
(i.e. very run of infra-prod-bootstrap-bridge isn't reinstalling
* infra-prod-bootstrap-bridge always runs first in a deployment
buildset (i.e. dependencies are correct)
* infra-prod-bootstrap-bridge correctly puts the right checkout of
system-config on bridge (as mentioned, for now the other infra-prod
jobs will continue to overwrite it)
Once we have validated infra-prod-bootstrap-bridge is running as we
like, we can drop the other jobs checking-out code with:
and also cleanup base-jobs
At this point, we should be ready to run in parallel (touch wood...)
On Wed, Nov 17, 2021 at 04:04:33PM +1100, Ian Wienand wrote:
> To recap: currently production deployment jobs run sequentially. Zuul
> starts the job on an executor, which is setup to log into the bastion
> host. The job sets up the system-config playbooks on the bastion host
> and Ansible is run from there against the production server.
> To run in parallel, each job needs to not assume it owns the
> system-config playbooks on the bastion host.
> Each Zuul *buildset* can use the same system-config playbook checkout
> though. To achieve this we need to rework the dependencies; each
> production job needs to depend on a common source-setup job. Once the
> source is setup on the bastion host, the actual production jobs can
> run in parallel.
> To the changes...
> Firstly, I believe we're doing the setup steps for the executor to log
> into bridge twice:
> removes this duplication, and should be safe to merge.
> As pointed out in prior reviews when running in the periodic or hourly
> pipelines each job overrides that bastion host checkout to master.
> moves this step into base-jobs, in preparation for only being done
> once by the separate source-setup job. I believe this will be safe to
> merge; system-config will just do it again in an idempotent way,
> merges, which drops this step from system-config.
> We can then merge the system-config job dependency updates in
> This should mean that all jobs not only rely on the correct base jobs,
> but jobs that need certificates, etc. will be relying on the
> letsencrypt job, etc. This should be safe to merge as nothing should
> actually change, we just have stricter dependencies.
> After this, I think we are ready to refactor the base jobs into the
> two separate steps -- firstly setup the keys on the executor to log
> into the bastion host, then setup the source to use on the bastion
> This initial refactor should be safe to merge as it creates two new
> jobs, but the existing base job keeps running both steps as-is.
> Then we are ready for the penultimate change:
> This updates the system-config jobs to all depend on
> "infra-prod-setup-src" which will be the canonical job that sets up
> the source repository on bridge.o.o. All other jobs in the buildset
> will depend on this job, ensuring consistency for a run.
> This should also be safe, as it again doesn't actually change
> Once all this is in, we need the final change to enable parallel
> running (and think about correct semaphores between periodic/hourly
> and regular runs). That is yet to be written, but we have enough to
> get to that point!
More information about the service-discuss