Parallel production jobs changes

[Ian Wienand]

Thank you for prior reviews and sticking with this rather complicated
fiddling; we ended up with some failures and reverts that I hope have
been addressed:

Firstly, we have a typo in matching install playbooks for our current
roles that has propogated as we copy-paste; a perfunctory fix

 https://review.opendev.org/c/opendev/system-config/+/820281

I am then proposing we rename the job we currently call
infra-prod-install-ansible to infra-prod-bootstrap-bridge.  This is a
back-to-the-future situation; as it was originally called something
like "install-bridge".  I think this better reflects where it will end
up, as hopefully revealed in the following steps

 https://review.opendev.org/c/opendev/system-config/+/820282

After this, we need to run infra-prod-bootstrap-bridge as the base job
before all other production jobs.  To recap -- this will be the
synchronisation point that puts Zuul's checkout of system-config on
the bastion host (bridge) that is then used to deploy all production
systems (eventually, in as parallel way as possible).

This revealed what I think is a problem with the original job -- it
runs the install-ansible role via
playbooks/zuul/run-production-playbook.yaml.  This is a
chicken-and-egg problem -- run-production-playbook.yaml uses the
Ansible installed on bridge to run playbooks in system-config to
... install Ansible on bridge.  Addressed with:

 https://review.opendev.org/c/opendev/system-config/+/820320/

this makes infra-prod-bootstrap bridge a stand-alone job that should

 * install the required version of Ansible on bridge
 * setup system-config to Zuul's checkout for the buildset

For sanity, we keep the current parent of the infra-prod-* jobs the
same -- this means each infra-prod-* job will still be re-checking-out
system-config as it runs.  We should validate

 * the install of ansible/openstacksdk/etc. is actually idempotent;
   (i.e. very run of infra-prod-bootstrap-bridge isn't reinstalling
   everything).

 * infra-prod-bootstrap-bridge always runs first in a deployment
    buildset (i.e. dependencies are correct)

 * infra-prod-bootstrap-bridge correctly puts the right checkout of
   system-config on bridge (as mentioned, for now the other infra-prod
   jobs will continue to overwrite it)

Once we have validated infra-prod-bootstrap-bridge is running as we
like, we can drop the other jobs checking-out code with:

  https://review.opendev.org/c/opendev/system-config/+/820651

and also cleanup base-jobs

 https://review.opendev.org/c/opendev/base-jobs/+/820652

At this point, we should be ready to run in parallel (touch wood...)

-i


On Wed, Nov 17, 2021 at 04:04:33PM +1100, Ian Wienand wrote:
> Hi,
> 
> To recap: currently production deployment jobs run sequentially.  Zuul
> starts the job on an executor, which is setup to log into the bastion
> host.  The job sets up the system-config playbooks on the bastion host
> and Ansible is run from there against the production server.
> 
> To run in parallel, each job needs to not assume it owns the
> system-config playbooks on the bastion host.
> 
> Each Zuul *buildset* can use the same system-config playbook checkout
> though.  To achieve this we need to rework the dependencies; each
> production job needs to depend on a common source-setup job.  Once the
> source is setup on the bastion host, the actual production jobs can
> run in parallel.
> 
> To the changes...
> 
> Firstly, I believe we're doing the setup steps for the executor to log
> into bridge twice:
> 
>  https://review.opendev.org/c/opendev/system-config/+/818190
> 
> removes this duplication, and should be safe to merge.
> 
> As pointed out in prior reviews when running in the periodic or hourly
> pipelines each job overrides that bastion host checkout to master.
> 
>  https://review.opendev.org/c/opendev/base-jobs/+/818189
> 
> moves this step into base-jobs, in preparation for only being done
> once by the separate source-setup job.  I believe this will be safe to
> merge; system-config will just do it again in an idempotent way,
> until:
> 
>  https://review.opendev.org/c/opendev/system-config/+/818191
> 
> merges, which drops this step from system-config.
> 
> We can then merge the system-config job dependency updates in
> 
>  https://review.opendev.org/c/opendev/system-config/+/807672
> 
> This should mean that all jobs not only rely on the correct base jobs,
> but jobs that need certificates, etc. will be relying on the
> letsencrypt job, etc.  This should be safe to merge as nothing should
> actually change, we just have stricter dependencies.
> 
> After this, I think we are ready to refactor the base jobs into the
> two separate steps -- firstly setup the keys on the executor to log
> into the bastion host, then setup the source to use on the bastion
> host:
> 
>   https://review.opendev.org/c/opendev/base-jobs/+/807807
> 
> This initial refactor should be safe to merge as it creates two new
> jobs, but the existing base job keeps running both steps as-is.
> 
> Then we are ready for the penultimate change:
> 
>   https://review.opendev.org/c/opendev/system-config/+/807808
> 
> This updates the system-config jobs to all depend on
> "infra-prod-setup-src" which will be the canonical job that sets up
> the source repository on bridge.o.o.  All other jobs in the buildset
> will depend on this job, ensuring consistency for a run.
> 
> This should also be safe, as it again doesn't actually change
> ordering.
> 
> Once all this is in, we need the final change to enable parallel
> running (and think about correct semaphores between periodic/hourly
> and regular runs).  That is yet to be written, but we have enough to
> get to that point!
> 
> -i