On Fri, Jan 25, 2019 at 2:38 PM James E. Blair corvus@inaugust.com wrote:
Alternative 2: Use a Central Registry
In this approach, we would run a central registry to store speculative images. It would not store final images at all, and every image uploaded to it would be tagged only with a tag like "pipeline_check_change_1234_original_tag" to uniquely identify its build context.
At the end of the image build job, for every image that we built: Copy it from the per-change registry to the central registry, and tag it with pipeline and change id. zuul_return image location and tag.
At the start of a job which requires images: For each previous image (as provided by zuul_return): Copy it from the central registry to the local registry, and tag it with the original tag.
The advantages of this approach:
- Developers can fetch images from the central registry by pulling them with the crazy tag.
- Implementation is a little easier, and uses more standardized tools.
The disadvantages of this approach:
- An external service is required.
- We will need to prune old images from the registry.
This is more or less what we are doing today for building and promoting container images for RDO.
In our periodic pipeline, we have a parent job that builds container images and pushes them to an OpenShift standalone registry [1]. These images have a specific tag format that dependant jobs are able to use in order to pull and test them. We have an out-of-band process for promoting the images based on the results of the following jobs -- if the criteria for promotion is met, the images are pushed to DockerHub under an appropriate tag.
The container images on the registry are configured to be public and anyone can pull them but pushing requires a token from a service account in the namespace.
We prune images using a fairly generic script [2] which prunes tags older than N days that do not match a supplied whitelist. This deletes the tags but does not delete the container image blobs from the filesystem. To delete the (now orphaned) blobs from the filesystem, OpenShift provides a "oadm prune images" command [3] which we run after deleting the tags.
We initially chose OpenShift as our registry because it made authentication and pruning simple when we compared it to docker-registry and docker-distribution. We experimented with Atomic Registry which was a simplified implementation of the OpenShift standalone registry but Atomic Registry was deprecated [4].
The OpenShift deployment lives on a single node and contains only the necessary bits to expose the registry publicly. It is not meant to host applications.
The registry has been in use for over a year now and is currently hosting near 10 000 tags across 878 images in 4 different namespaces and it prunes around 1000 tags every day. It has been fairly stable with 4 cores, 8GB of RAM and ~500GB of disk.
[1]: https://docs.openshift.com/container-platform/3.9/install_config/install/sta... [2]: https://github.com/rdo-infra/rdo-infra-playbooks/blob/master/roles/rdo-infra... [3]: https://docs.openshift.com/container-platform/3.9/admin_guide/pruning_resour... [4]: https://www.projectatomic.io/blog/2017/05/oo-standalone-registry/
David Moreau Simard dmsimard = [irc, github, twitter]