On Fri, May 5, 2023, at 6:57 AM, Jeremy Stanley wrote:
On 2023-05-05 09:01:01 -0400 (-0400), Neal Gompa wrote: [...]
I'm confused why you're mirroring Fedora repos and building custom Fedora images. Is there something we could do on the Fedora Cloud side to make this workload easier? We already produce OpenStack-targeted Fedora Cloud images, as an example. [...]
There are two main reasons:
First, we want the images to be as minimal as possible. Take DevStack-based test jobs for example, they use pip to install Python libraries into the system context, and so any preinstalled distribution packages of Python libraries can cause conflicts which are hard to mitigate. This is, for example, a big part of why we developed our own minimal alternative to cloud-init (glean). If we used pre-built images we'd still need to maintain a cross-distro toolchain capable of altering images in order to cleanly uninstall packages which could cause conflicts.
Second, in order to speed up jobs, we pre-cache a lot of content onto the images we build (Git repositories for projects, frequently downloaded files like CirrOS images, et cetera). This means that, at a minimum, we need to maintain a cross-distro toolchain capable of altering images in order to embed the cache, so building the images from scratch isn't a major leap past there.
There is a third reason: automated consistency and updates. We have Debian, Ubuntu, Fedora, CentOS, Rocky, OpenEuler, and OpenSUSE images. For many of these we also build several versions of the distro. Alongside that we've got a number of clouds we operate in. Some of these clouds need vhd images, others qcow2 images, and others raw images. Some clouds are running on x86_64 hardware and others are arm64. Building our own images ensures that we can build images for any one of these distros, upload it to any one of these clouds, and have it operate in a consistent manner without being impacted by external choices made by the various upstream distros. Before we started building our own images in this way, we spent significant amounts of time debugging why one image was different to another despite having only two cloud and a single CPU architecture to worry about then. This also enables a number quality of life improvements. We build the images with a consistent Zuul user and don't have to worry about every distro's insistence for a special cloud image user that changes over time. Our images are rebuilt daily ensuring they are kept up to date. We can build our images before the distros do. This was the case when CentOS Stream started, and makes it possible to test new distros before they are released.