Future of Fedora and CentOS Stream Test Images
Hello everyone, Over the last couple of years we've grown support for a number of RHEL like or RHEL adjacent distros in our CI system. We have added OpenEuler, Rocky Linux, and CentOS Stream (it replaced CentOS 8). At the same time we've fallen behind producing and updating images for Fedora. Currently we have disk images and test nodes for Fedora 35 and 36. We mirror packages for Fedora 36 and 37. In the past our intention was always to primarily have a single up to date Fedora image and mirror setup for our CI system which would be Fedora 38 as of April 18, 2023. One option we have is to drop Fedora 35 and 37 from Nodepool and our mirrors, add Fedora 38, and then drop Fedora 36. This would basically maintain the status quo of the last few years. That said I think this is probably a good opportunity to rethink our approach, and we are looking for your feedback. Currently the RHEL like distros are overlapping with each other a bit and for this reason we added Rocky Linux without adding extra mirroring for it. So far this seems to be stable enough as we don't run significant numbers of jobs on Rocky Linux. I think one option available to us would be to drop Fedora mirroring entirely, then add Fedora 38 images/nodes, and then drop Fedora 36 images/nodes. One upside to this approach is it will free up disk space on our OpenAFS fileservers which are under a space crunch currently. I also wonder if we can drop Fedora entirely. It seems like CentOS Stream now occupies a similar role within our CI system. It gives us forward looking updates that will eventually be in RHEL. This allows us to test against a future state in preparation for RHEL changes. The upside to this is that Fedora's support lifetime is about 13 months but CentOS Stream releases appear to have several years of support. This should reduce the amount of mirror and image work we need to perform. Are there specific use cases that Fedora serves that CentOS Stream does not within our CI system? If so are they important enough to continue to try and run on the Fedora release treadmill? If we do want to try and keep up with Fedora do we have any concerns dropping our local mirroring of Fedora packages? Please let us know, Clark
On Wed, 2023-05-03 at 08:12 -0700, Clark Boylan wrote:
. This allows us to test against a future state in preparation for RHEL changes. The upside to this is that Fedora's support lifetime is about 13 months but CentOS Stream releases appear to have several years of support. This should reduce the amount of mirror and image work we need to perform.
Are there specific use cases that Fedora serves that CentOS Stream does not within our CI system? If so are they important enough to continue to try and run on the Fedora release treadmill? If we do want to try and keep up with Fedora do we have any concerns dropping our local mirroring of Fedora packages?
this is my personal opipion but i think we shoudl move to rocky for RHEL like disto testing and avoid Centos Stream or fedora for that usecase. We have discused having voting Centos stream and fedor jobs in the past and decided that we did nto want to add etiehr to the nova ci in the futur due to the instablity we have seen in the past with centos stream based ci and the time it took for issues to get adressed. the only usecase i have had in the past for fedora is testing very very very new libvirt/qemu via the virt-preview repo. you can enable the fedora virt preview repo on CentOS Stream too and the libvirt it ships is almost as up to date as fedora even without it. so i dont think we woudl loose much if we were to remvoe fedora in that respect if its becoming a mantance burden. so i would propsoe using Rocky for RHEL like testing and stream for rhel.next type testing.
Please let us know, Clark
On Thu, May 4, 2023, at 12:34 AM, Slawek Kaplonski wrote:
Thank you for the feedback. We'll hold off on making any changes for another week, but then starting May 16th we'll begin taking steps based on the feedback we have received. If you have anything to share on this topic please do so soon. Clark
On Tue, May 9, 2023 at 6:09 PM Clark Boylan <cboylan@sapwetik.org> wrote:
Is there a reason we can't use RHEL itself? RHEL for CI seems to be a thing they allow at no cost[1]. In terms of mirroring RHEL packages, I know of at least one maintained method[2] for doing it easily enough (disclaimer, I helped write it long ago...). Given that it's a thing nowadays, I would much rather see us use RHEL + CentOS Stream as a combo. [1]: https://www.redhat.com/en/blog/extending-no-cost-red-hat-enterprise-linux-op... [2]: https://github.com/datto/rhel-reposync-playbook -- 真実はいつも一つ！/ Always, there's only one truth!
On Wed, May 10, 2023 at 9:53 AM Neal Gompa <ngompa13@gmail.com> wrote:
Is there a reason we can't use RHEL itself? RHEL for CI seems to be a thing they allow at no cost[1].
Philosophically, we've avoided anything that required a subscription or special access, as the whole idea is to build things that do not rely on any such things. Though practicalities do win out, for example the FIPS work uses a key, and some of our production servers use ESM keys. Practically, we give developers full root access to a complete VM, which precludes anything we use in CI being truly "secret" (note I'm talking about the untrusted check jobs. Post commit jobs are different). So we have to consider that, modulo some level of obfuscation, any subscriptions should be considered public. Also practically, the diskimage-builder path would need quite a bit of work to spit out useful images; I can't currently see that there would be resources to do any of this. -i
On 2023-05-09 19:52:54 -0400 (-0400), Neal Gompa wrote: [...]
That announcement two years ago definitely flew under my radar, so this is certainly worth revisiting. As Ian said though, we do generally prefer to test on platforms which are entirely open access, not merely free for personal or free for open source project use. -- Jeremy Stanley
On Wed, 2023-05-10 at 01:15 +0000, Jeremy Stanley wrote:
ya we did disucss this at the time that annouchment was made. im going to email the team that runs that program and ask if there has been any change in the policy but our usage of public cloud for the zuul workers porhibited the first party ci form using this program in the past. so even if we wanted too, as written this program does not cover our senario it only allow openstouce comunities to use rhel on premise on infra they own we cant use third party cloud providers under that program.
On 2023-05-10 11:36:57 +0100 (+0100), Sean Mooney wrote: [...]
ya we did disucss this at the time that annouchment was made. [...]
Aha, I had conflated that discussion with an earlier one where people were asking why we couldn't just use the free developer licenses to do it.
Thanks. I misinterpreted that as saying we could use it in a CI system we run, just not in a CI system run by someone else (e.g. Travis CI, GitHub Actions, or whatever). If the CI system also has to be on entirely internal infrastructure and can't run in a cloud operated by someone else, then yes that does exclude our use case. -- Jeremy Stanley
On 2023-05-10 07:58:59 -0400 (-0400), Neal Gompa wrote: [...]
Well, asking again anyway, since Sean already asked once and they never answered. But also, the implementation details will matter. If this relies on us having a sensitive registration key which must be present on test nodes so that they can install packages at job run time, we have no effective means of securing that from exposure or exfiltration by users since random members of the public have the ability to run arbitrary code as root on those systems. In the case of the Ubuntu Advantage FIPS support license we're comped, we got a written statement from Canonical staff that said they were okay with the risk of someone extracting the activation key from a test node, and that they would work with us to rotate the key if that ever became a problem for them. -- Jeremy Stanley
On Wed, 2023-05-10 at 12:10 +0000, Jeremy Stanley wrote:
honestly i prefer avoiding all that complexity and useing distos that dont require it too so im just reaching out to that team for complete ness. i still think using CentOS Stream for a RHEL.Next proxy and Rocky for a Rhel current proxy is a simpler approach but that is a vailid point the images that are uploaded to the ci providres are also publicly avaiable its been a while but i ahve actully downloaded them to try and repoduce a issue we only saw in ci in the past. so unless the subsction key was injected in the job from zuul secret it would be in the nodepool image which is publicly hosted.
On 2023-05-10 13:21:48 +0100 (+0100), Sean Mooney wrote: [...]
The approach we took with the UA FIPS token was to add it at job run time from a Zuul secret supplied by a trusted config repo abstract job, which avoided baking it into our images, but it's still added to the job node in ways that could potentially be exposed to access by workloads in untrusted jobs inheriting from that. And yes, it *is* a complicated implementation which took months of back and forth and several false-starts to get just right. We would have preferred to avoid that entirely and do FIPS compliance testing on CentOS Stream, but that proved unstable for other reasons. -- Jeremy Stanley
On Tue, 2023-05-09 at 19:52 -0400, Neal Gompa wrote: liesnceing. when https://www.redhat.com/en/blog/extending-no-cost-red-hat-enterprise-linux-op... was annouched i reached out to the team that manages it and they never reponded. the internal FAQ and some initall question i asked indicated that we could not use this for upstream rhel ci. i could try emailing rosi-program@redhat.com again and perhapse we will get a responce this time but that was the main blocker. we were prvisuly told we coudl not use that program to buiild rhel based ci image and upload them to our cloud providers to do ci with rhel. i agree that if the intent is to test rhel like operating systems then usign rhel woudl be betere then a proxy but until the issue of sybscriptions can be resolved rhel is not an option. the relevent paragraph that mean we cant use this program are """ Under the program’s terms, eligible organizations will be granted access to no-cost RHEL subscriptions for any use within the confines of their infrastructure. This includes build systems, continuous integration (CI) testing and general project requirements (i.e. web servers, mail servers, etc.). """ """ We realize this program doesn’t cover situations where open source projects are using Public CI infrastructure provided by third-parties. This and other programs are still being worked on. So, we’re definitely not yet done expanding RHEL programs to meet community needs and want to hear from you. Send your questions and requests to us at centos-questions@redhat.com. """ the opendev/openstack foundation could use this program to have rhel for the service it uses to run its website or zuul or anything that is self hosted on its own servers but not for the vms we run on the public clouds that are used for the ci system.
participants (7)
-
Clark Boylan
-
Ian Wienand
-
Jeremy Stanley
-
Michael Johnson
-
Neal Gompa
-
Sean Mooney
-
Slawek Kaplonski