Parallel production jobs changes

[Ian Wienand]

Hi,

To recap: currently production deployment jobs run sequentially.  Zuul
starts the job on an executor, which is setup to log into the bastion
host.  The job sets up the system-config playbooks on the bastion host
and Ansible is run from there against the production server.

To run in parallel, each job needs to not assume it owns the
system-config playbooks on the bastion host.

Each Zuul *buildset* can use the same system-config playbook checkout
though.  To achieve this we need to rework the dependencies; each
production job needs to depend on a common source-setup job.  Once the
source is setup on the bastion host, the actual production jobs can
run in parallel.

To the changes...

Firstly, I believe we're doing the setup steps for the executor to log
into bridge twice:

 https://review.opendev.org/c/opendev/system-config/+/818190

removes this duplication, and should be safe to merge.

As pointed out in prior reviews when running in the periodic or hourly
pipelines each job overrides that bastion host checkout to master.

 https://review.opendev.org/c/opendev/base-jobs/+/818189

moves this step into base-jobs, in preparation for only being done
once by the separate source-setup job.  I believe this will be safe to
merge; system-config will just do it again in an idempotent way,
until:

 https://review.opendev.org/c/opendev/system-config/+/818191

merges, which drops this step from system-config.

We can then merge the system-config job dependency updates in

 https://review.opendev.org/c/opendev/system-config/+/807672

This should mean that all jobs not only rely on the correct base jobs,
but jobs that need certificates, etc. will be relying on the
letsencrypt job, etc.  This should be safe to merge as nothing should
actually change, we just have stricter dependencies.

After this, I think we are ready to refactor the base jobs into the
two separate steps -- firstly setup the keys on the executor to log
into the bastion host, then setup the source to use on the bastion
host:

  https://review.opendev.org/c/opendev/base-jobs/+/807807

This initial refactor should be safe to merge as it creates two new
jobs, but the existing base job keeps running both steps as-is.

Then we are ready for the penultimate change:

  https://review.opendev.org/c/opendev/system-config/+/807808

This updates the system-config jobs to all depend on
"infra-prod-setup-src" which will be the canonical job that sets up
the source repository on bridge.o.o.  All other jobs in the buildset
will depend on this job, ensuring consistency for a run.

This should also be safe, as it again doesn't actually change
ordering.

Once all this is in, we need the final change to enable parallel
running (and think about correct semaphores between periodic/hourly
and regular runs).  That is yet to be written, but we have enough to
get to that point!

-i