Re: [dev][infra][qa][tact-sig] Future of Fedora and CentOS Stream Test Images
On Wed, May 03, 2023 at 03:28:00PM +0000, Jeremy Stanley wrote:
tl;dr is that the OpenDev Collaboratory is looking at potentially scaling back on Fedora-based test platform support. Several options are presented in this service-discuss post:
https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/th...
If you have an opinion on using Fedora in our CI jobs, please follow up to the discussion there.
Hi, Jeremy.
FWIW, I think it is still valuable to have at least the latest Fedora-only image available. Often times kernel and virtualization fixes are first fixed on Fedora -- many upstream kernel developers happen to use Fedora.
Another reason I wanted to mention is the 'virt-preview' repo[1]. But I see that 'virt-preview' is also available for CentOS stream. So that's a less strong reason.
That said, if dropping Fedora is the compromise we need to make to reduce the workload of "mirroring and image management, then I don't insist.
[1] https://copr.fedorainfracloud.org/coprs/g/virtmaint-sig/virt-preview/
On Thu, May 4, 2023, at 5:52 AM, Kashyap Chamarthy wrote:
On Wed, May 03, 2023 at 03:28:00PM +0000, Jeremy Stanley wrote:
tl;dr is that the OpenDev Collaboratory is looking at potentially scaling back on Fedora-based test platform support. Several options are presented in this service-discuss post:
https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/th...
If you have an opinion on using Fedora in our CI jobs, please follow up to the discussion there.
Hi, Jeremy.
FWIW, I think it is still valuable to have at least the latest Fedora-only image available. Often times kernel and virtualization fixes are first fixed on Fedora -- many upstream kernel developers happen to use Fedora.
Once upon a time we had OpenSUSE Tumbleweed images too. These distros are in theory great for catching problems early, but in practice it seems very few people (if any) have the time and ability to address every problem that comes up in them.
Another reason I wanted to mention is the 'virt-preview' repo[1]. But I see that 'virt-preview' is also available for CentOS stream. So that's a less strong reason.
Right, it seems like CentOS Stream is filling a very similar role these days and importantly with a much longer runway of support. This means there is less work required to keep CentOS Stream images up compared to Fedora images.
That said, if dropping Fedora is the compromise we need to make to reduce the workload of "mirroring and image management, then I don't insist.
Currently there isn't anyone volunteering/able to add Fedora 38 images (which is latest). I think if someone wanted to do that we could proceed, but likely without locally hosted package mirrors. That would reduce a lot of the overhead and limit the work required to spinning up images with diskimage-builder. Based on other feedback [2], I suspect that we just don't have any interested users (and consequently no interested volunteers)? That said we'll wait a bit for more feedback and discussion before making a concrete decision.
[1] https://copr.fedorainfracloud.org/coprs/g/virtmaint-sig/virt-preview/
[2] https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/th...
-- /kashyap
On Thu, May 04, 2023 at 08:49:23AM -0700, Clark Boylan wrote:
On Thu, May 4, 2023, at 5:52 AM, Kashyap Chamarthy wrote:
On Wed, May 03, 2023 at 03:28:00PM +0000, Jeremy Stanley wrote:
[...]
FWIW, I think it is still valuable to have at least the latest Fedora-only image available. Often times kernel and virtualization fixes are first fixed on Fedora -- many upstream kernel developers happen to use Fedora.
Once upon a time we had OpenSUSE Tumbleweed images too. These distros are in theory great for catching problems early, but in practice it seems very few people (if any) have the time and ability to address every problem that comes up in them.
I agree; it's hard to keep up with that grind.
Another reason I wanted to mention is the 'virt-preview' repo[1]. But I see that 'virt-preview' is also available for CentOS stream. So that's a less strong reason.
Right, it seems like CentOS Stream is filling a very similar role these days and importantly with a much longer runway of support. This means there is less work required to keep CentOS Stream images up compared to Fedora images.
Yeah, resorting to CentOS Stream images is a perfectly reasonable compromise.
That said, if dropping Fedora is the compromise we need to make to reduce the workload of "mirroring and image management, then I don't insist.
Currently there isn't anyone volunteering/able to add Fedora 38 images (which is latest). I think if someone wanted to do that we could proceed, but likely without locally hosted package mirrors. That would reduce a lot of the overhead and limit the work required to spinning up images with diskimage-builder. Based on other feedback [2], I suspect that we just don't have any interested users (and consequently no interested volunteers)? That said we'll wait a bit for more feedback and discussion before making a concrete decision.
Fair; Fedora _is_ a "fast moving target" (as noted in that thread) and can be difficult to keep up in terms of testing it. And I can't offer the time to maintain Fedora images myself, afraid. So I'm personally fine to rely on CentOS Stream images.
Thanks for the considered response, as always.
On Fri, May 5, 2023 at 8:52 AM Kashyap Chamarthy kchamart@redhat.com wrote:
On Thu, May 04, 2023 at 08:49:23AM -0700, Clark Boylan wrote:
On Thu, May 4, 2023, at 5:52 AM, Kashyap Chamarthy wrote:
On Wed, May 03, 2023 at 03:28:00PM +0000, Jeremy Stanley wrote:
[...]
FWIW, I think it is still valuable to have at least the latest Fedora-only image available. Often times kernel and virtualization fixes are first fixed on Fedora -- many upstream kernel developers happen to use Fedora.
Once upon a time we had OpenSUSE Tumbleweed images too. These distros are in theory great for catching problems early, but in practice it seems very few people (if any) have the time and ability to address every problem that comes up in them.
I agree; it's hard to keep up with that grind.
Another reason I wanted to mention is the 'virt-preview' repo[1]. But I see that 'virt-preview' is also available for CentOS stream. So that's a less strong reason.
Right, it seems like CentOS Stream is filling a very similar role these days and importantly with a much longer runway of support. This means there is less work required to keep CentOS Stream images up compared to Fedora images.
Yeah, resorting to CentOS Stream images is a perfectly reasonable compromise.
That said, if dropping Fedora is the compromise we need to make to reduce the workload of "mirroring and image management, then I don't insist.
Currently there isn't anyone volunteering/able to add Fedora 38 images (which is latest). I think if someone wanted to do that we could proceed, but likely without locally hosted package mirrors. That would reduce a lot of the overhead and limit the work required to spinning up images with diskimage-builder. Based on other feedback [2], I suspect that we just don't have any interested users (and consequently no interested volunteers)? That said we'll wait a bit for more feedback and discussion before making a concrete decision.
Fair; Fedora _is_ a "fast moving target" (as noted in that thread) and can be difficult to keep up in terms of testing it. And I can't offer the time to maintain Fedora images myself, afraid. So I'm personally fine to rely on CentOS Stream images.
Thanks for the considered response, as always.
I'm confused why you're mirroring Fedora repos and building custom Fedora images. Is there something we could do on the Fedora Cloud side to make this workload easier? We already produce OpenStack-targeted Fedora Cloud images, as an example.
As a member of Fedora Cloud, I would like to see OpenStack support Fedora fully, I just don't know what's going on here...
-- 真実はいつも一つ！/ Always, there's only one truth!
On Fri, May 05, 2023 at 09:01:01AM -0400, Neal Gompa wrote:
On Fri, May 5, 2023 at 8:52 AM Kashyap Chamarthy kchamart@redhat.com wrote:
[...]
That said, if dropping Fedora is the compromise we need to make to reduce the workload of "mirroring and image management, then I don't insist.
Currently there isn't anyone volunteering/able to add Fedora 38 images (which is latest). I think if someone wanted to do that we could proceed, but likely without locally hosted package mirrors. That would reduce a lot of the overhead and limit the work required to spinning up images with diskimage-builder. Based on other feedback [2], I suspect that we just don't have any interested users (and consequently no interested volunteers)? That said we'll wait a bit for more feedback and discussion before making a concrete decision.
Fair; Fedora _is_ a "fast moving target" (as noted in that thread) and can be difficult to keep up in terms of testing it. And I can't offer the time to maintain Fedora images myself, afraid. So I'm personally fine to rely on CentOS Stream images.
Thanks for the considered response, as always.
I'm confused why you're mirroring Fedora repos and building custom Fedora images. Is there something we could do on the Fedora Cloud side to make this workload easier? We already produce OpenStack-targeted Fedora Cloud images, as an example.
I myself don't remember _why_ the OpenDev CI infrastructure tooling (in this case, `diskimage-builder`, "DIB") have to build distro images thesmselves, instead of relying on [1]. But if you see the Fedora README, the DIB tool does use the official Fedora mirror:
https://docs.openstack.org/diskimage-builder/latest/elements/fedora/README.h... https://docs.openstack.org/diskimage-builder/latest/elements/fedora-minimal/...
I'll let Clark, et al from the infra team elaborate further.
As a member of Fedora Cloud, I would like to see OpenStack support Fedora fully, I just don't know what's going on here...
In the distant past, I vaguely recall having discussions about using "official" Fedora cloud images[1], but I completely lost memory of why we didn't end up going that route. IIUC, same for other distro images too. Again, I'll defer that to our fine OpenDev infra team on the "why".
[1] https://alt.fedoraproject.org/cloud/
On 2023-05-05 09:01:01 -0400 (-0400), Neal Gompa wrote: [...]
I'm confused why you're mirroring Fedora repos and building custom Fedora images. Is there something we could do on the Fedora Cloud side to make this workload easier? We already produce OpenStack-targeted Fedora Cloud images, as an example.
[...]
There are two main reasons:
First, we want the images to be as minimal as possible. Take DevStack-based test jobs for example, they use pip to install Python libraries into the system context, and so any preinstalled distribution packages of Python libraries can cause conflicts which are hard to mitigate. This is, for example, a big part of why we developed our own minimal alternative to cloud-init (glean). If we used pre-built images we'd still need to maintain a cross-distro toolchain capable of altering images in order to cleanly uninstall packages which could cause conflicts.
Second, in order to speed up jobs, we pre-cache a lot of content onto the images we build (Git repositories for projects, frequently downloaded files like CirrOS images, et cetera). This means that, at a minimum, we need to maintain a cross-distro toolchain capable of altering images in order to embed the cache, so building the images from scratch isn't a major leap past there.
Also, bear in mind that building the images isn't all of the problem with fast-moving/short-support distros like Fedora (or SUSE Tumbleweed, or Debian Sid, or non-LTS Ubuntu versions, or...). It's that projects want to freeze their testing for stable branches at release time, and continue to use versions contemporary to those releases for the support timeframe of the stable branch, which in OpenStack's case is measured in years. In theory we could continue testing OpenStack stable branches on old Fedora versions, but there comes a point in time where Fedora mirrors drop the packages for those versions, or they cease to be viable in our donated cloud environments for other reasons.
The crux of the matter is that we've built our CI platform (and OpenStack's support model for that matter) around the idea of testing software for deployment on LTS "server" distributions, and have made choices which optimize for those cases at the expense of not being able to as easily track the "bleeding edge" distributions.
On Fri, May 5, 2023, at 6:57 AM, Jeremy Stanley wrote:
On 2023-05-05 09:01:01 -0400 (-0400), Neal Gompa wrote: [...]
I'm confused why you're mirroring Fedora repos and building custom Fedora images. Is there something we could do on the Fedora Cloud side to make this workload easier? We already produce OpenStack-targeted Fedora Cloud images, as an example.
[...]
There are two main reasons:
First, we want the images to be as minimal as possible. Take DevStack-based test jobs for example, they use pip to install Python libraries into the system context, and so any preinstalled distribution packages of Python libraries can cause conflicts which are hard to mitigate. This is, for example, a big part of why we developed our own minimal alternative to cloud-init (glean). If we used pre-built images we'd still need to maintain a cross-distro toolchain capable of altering images in order to cleanly uninstall packages which could cause conflicts.
Second, in order to speed up jobs, we pre-cache a lot of content onto the images we build (Git repositories for projects, frequently downloaded files like CirrOS images, et cetera). This means that, at a minimum, we need to maintain a cross-distro toolchain capable of altering images in order to embed the cache, so building the images from scratch isn't a major leap past there.
There is a third reason: automated consistency and updates. We have Debian, Ubuntu, Fedora, CentOS, Rocky, OpenEuler, and OpenSUSE images. For many of these we also build several versions of the distro. Alongside that we've got a number of clouds we operate in. Some of these clouds need vhd images, others qcow2 images, and others raw images. Some clouds are running on x86_64 hardware and others are arm64.
Building our own images ensures that we can build images for any one of these distros, upload it to any one of these clouds, and have it operate in a consistent manner without being impacted by external choices made by the various upstream distros. Before we started building our own images in this way, we spent significant amounts of time debugging why one image was different to another despite having only two cloud and a single CPU architecture to worry about then.
This also enables a number quality of life improvements. We build the images with a consistent Zuul user and don't have to worry about every distro's insistence for a special cloud image user that changes over time. Our images are rebuilt daily ensuring they are kept up to date. We can build our images before the distros do. This was the case when CentOS Stream started, and makes it possible to test new distros before they are released.
participants (4)
-
Clark Boylan
-
Jeremy Stanley
-
Kashyap Chamarthy
-
Neal Gompa