Feedback on Changes to Supported Python Version in Zuul
Hello everyone,
Eventually (hopefully soon), Zuul will add Ansible 7 (Ansible Core 2.14 support). According to the Ansible docs [0] this new version of Ansible drops support for Python 3.8. Currently Zuul is tested on Python 3.8 and 3.11 with the intent of supporting Zuul across Python 3.8, 3.9, 3.10, and 3.11. Unfortunately, Ansible's dropping of Python 3.8 support means that Zuul will also need to drop support for Python 3.8 when Ansible 7 support is added to Zuul.
On top of that, Zuul test suite runs in CI are approximately 30 minutes quicker running on python 3.10+ than on older Pythons. Unfortunately, I don't have the best data to illustrate this since we currently only run jobs for 3.8 and 3.11, but the major performance improvements appear to have shown up with 3.10. You can get a sense for this difference comparing the 3.8 runtimes [1] to the 3.11 runtimes [2]. As a developer I selfishly think that this improvement is worth considering, and we might also take this opportunity to drop Python 3.9 support (and testing).
For anyone running on Python 3.8 today now is probably a good time to start looking at updated platforms. Is there anyone out there running Zuul on Python 3.9 today? If you are running Zuul on Python < 3.10 we'd love to hear from you to ensure we understand the potential impact of changes like this.
[0] https://docs.ansible.com/ansible/latest/installation_guide/intro_installatio... [1] https://zuul.opendev.org/t/zuul/builds?job_name=zuul-nox-py38&pipeline=g... [2] https://zuul.opendev.org/t/zuul/builds?job_name=zuul-nox-py311&pipeline=...
Clark
Hello,
We run Zuul and Nodepool in containers based on the UBI8 image [0] for various reasons (dog fooding, compliance, etc). We use python 3.8 with these containers, but could potentially bump to python 3.9 - which is also supported in UBI9. None of these base images have official support for python 3.10+ however, so we will probably be stuck with 3.9 for a while in production, until the next UBI release [1]. Another solution would be for us to move to Fedora-based images or even just use the opendev containers, if we can lower our compliance expectations. Are the upstream containers already using python 3.10+?
I see there are enough significant new features and accompanying syntax introduced in 3.10 that would make maintaining backward compatibility with 3.9 a pain ... what would be a tentative timeline to drop 3.9 support?
Cheers,
[0] build definition in dhall for our Zuul base image: https://softwarefactory-project.io/cgit/containers/tree/images-sf/master/zuu... [1] CentOS Stream will package python 3.11 but seemingly no plans for 3.10 https://gitlab.com/redhat/centos-stream/rpms/python3.11 - and 3.11 isn't available for installation yet via rpm on the latest container image.
On Fri, Jan 20, 2023 at 8:34 PM Clark Boylan cboylan@sapwetik.org wrote:
Hello everyone,
Eventually (hopefully soon), Zuul will add Ansible 7 (Ansible Core 2.14 support). According to the Ansible docs [0] this new version of Ansible drops support for Python 3.8. Currently Zuul is tested on Python 3.8 and 3.11 with the intent of supporting Zuul across Python 3.8, 3.9, 3.10, and 3.11. Unfortunately, Ansible's dropping of Python 3.8 support means that Zuul will also need to drop support for Python 3.8 when Ansible 7 support is added to Zuul.
On top of that, Zuul test suite runs in CI are approximately 30 minutes quicker running on python 3.10+ than on older Pythons. Unfortunately, I don't have the best data to illustrate this since we currently only run jobs for 3.8 and 3.11, but the major performance improvements appear to have shown up with 3.10. You can get a sense for this difference comparing the 3.8 runtimes [1] to the 3.11 runtimes [2]. As a developer I selfishly think that this improvement is worth considering, and we might also take this opportunity to drop Python 3.9 support (and testing).
For anyone running on Python 3.8 today now is probably a good time to start looking at updated platforms. Is there anyone out there running Zuul on Python 3.9 today? If you are running Zuul on Python < 3.10 we'd love to hear from you to ensure we understand the potential impact of changes like this.
[0] https://docs.ansible.com/ansible/latest/installation_guide/intro_installatio... [1] https://zuul.opendev.org/t/zuul/builds?job_name=zuul-nox-py38&pipeline=g... [2] https://zuul.opendev.org/t/zuul/builds?job_name=zuul-nox-py311&pipeline=...
Clark
Are the upstream containers already using python 3.10+?
self-answering that one: yes, it seems to be running on 3.11 -> https://hub.docker.com/layers/zuul/zuul-scheduler/latest/images/sha256-da003...
On Mon, Jan 23, 2023 at 12:46 PM Matthieu Huin mhuin@redhat.com wrote:
Hello,
We run Zuul and Nodepool in containers based on the UBI8 image [0] for various reasons (dog fooding, compliance, etc). We use python 3.8 with these containers, but could potentially bump to python 3.9 - which is also supported in UBI9. None of these base images have official support for python 3.10+ however, so we will probably be stuck with 3.9 for a while in production, until the next UBI release [1]. Another solution would be for us to move to Fedora-based images or even just use the opendev containers, if we can lower our compliance expectations. Are the upstream containers already using python 3.10+?
I see there are enough significant new features and accompanying syntax introduced in 3.10 that would make maintaining backward compatibility with 3.9 a pain ... what would be a tentative timeline to drop 3.9 support?
Cheers,
[0] build definition in dhall for our Zuul base image: https://softwarefactory-project.io/cgit/containers/tree/images-sf/master/zuu... [1] CentOS Stream will package python 3.11 but seemingly no plans for 3.10 https://gitlab.com/redhat/centos-stream/rpms/python3.11 - and 3.11 isn't available for installation yet via rpm on the latest container image.
On Fri, Jan 20, 2023 at 8:34 PM Clark Boylan cboylan@sapwetik.org wrote:
Hello everyone,
Eventually (hopefully soon), Zuul will add Ansible 7 (Ansible Core 2.14 support). According to the Ansible docs [0] this new version of Ansible drops support for Python 3.8. Currently Zuul is tested on Python 3.8 and 3.11 with the intent of supporting Zuul across Python 3.8, 3.9, 3.10, and 3.11. Unfortunately, Ansible's dropping of Python 3.8 support means that Zuul will also need to drop support for Python 3.8 when Ansible 7 support is added to Zuul.
On top of that, Zuul test suite runs in CI are approximately 30 minutes quicker running on python 3.10+ than on older Pythons. Unfortunately, I don't have the best data to illustrate this since we currently only run jobs for 3.8 and 3.11, but the major performance improvements appear to have shown up with 3.10. You can get a sense for this difference comparing the 3.8 runtimes [1] to the 3.11 runtimes [2]. As a developer I selfishly think that this improvement is worth considering, and we might also take this opportunity to drop Python 3.9 support (and testing).
For anyone running on Python 3.8 today now is probably a good time to start looking at updated platforms. Is there anyone out there running Zuul on Python 3.9 today? If you are running Zuul on Python < 3.10 we'd love to hear from you to ensure we understand the potential impact of changes like this.
[0] https://docs.ansible.com/ansible/latest/installation_guide/intro_installatio... [1] https://zuul.opendev.org/t/zuul/builds?job_name=zuul-nox-py38&pipeline=g... [2] https://zuul.opendev.org/t/zuul/builds?job_name=zuul-nox-py311&pipeline=...
Clark
--
Matthieu Huin (He/Him/His)
Senior Software Developer
Red Hat https://www.redhat.com
On 2023-01-23 12:46:49 +0100 (+0100), Matthieu Huin wrote: [...]
what would be a tentative timeline to drop 3.9 support?
[...]
I think that's what we're trying to determine. When we start testing with/supporting Ansible 7 we're not going to want to maintain separate Ansible 6 Python 3.8 jobs, so that will presumably be the point at which we stop supporting 3.8. Since 3.10 has significantly better performance, it would be nice to be able to drop 3.9 support as well and get faster turn-around on changes since we wouldn't have to wait for results from the slower jobs.
Currently, the latest Ubuntu LTS (Jammy) has Python 3.10 as its default, and the impending Debian Bullseye is slated have 3.11 as default barring any last-minute issues (but will be at least 3.10).
On Mon, 23 Jan 2023 at 14:20, Jeremy Stanley fungi@yuggoth.org wrote:
Currently, the latest Ubuntu LTS (Jammy) has Python 3.10 as its default, and the impending Debian Bullseye is slated have 3.11 as default barring any last-minute issues (but will be at least 3.10).
Obviously, you mean Debian Bookworm. Bullseye already exists and runs on 3.9.
On 2023-01-23 14:25:32 +0100 (+0100), Radosław Piliszek wrote: [...]
Obviously, you mean Debian Bookworm. Bullseye already exists and runs on 3.9.
Yes, indeed. I'm going to blame a seemingly endless series of release codenames starting with the same letter. At least Trixie will finally break the streak.
I have done a bit of digging and it seems support for python 3.11 is planned in RHEL 9.2 [0] which should be released within 4-8 weeks from now.
If the next LTS releases/updates of most major distros are within this timeframe, it'd be nice to drop zuul's support of python 3.9 after that time.
[0] https://bugzilla.redhat.com/show_bug.cgi?id=2127923
On Mon, Jan 23, 2023 at 2:34 PM Jeremy Stanley fungi@yuggoth.org wrote:
On 2023-01-23 14:25:32 +0100 (+0100), Radosław Piliszek wrote: [...]
Obviously, you mean Debian Bookworm. Bullseye already exists and runs on 3.9.
Yes, indeed. I'm going to blame a seemingly endless series of release codenames starting with the same letter. At least Trixie will finally break the streak. -- Jeremy Stanley
On Mon, Jan 23, 2023, at 3:46 AM, Matthieu Huin wrote:
Hello,
Snip.
I see there are enough significant new features and accompanying syntax introduced in 3.10 that would make maintaining backward compatibility with 3.9 a pain ... what would be a tentative timeline to drop 3.9 support?
To be clear Zuul is currently tested against python 3.8 and 3.11 with the intent that testing the upper and lower bound ensures we support the entire range of 3.8, 3.9, 3.10, and 3.11. There is no issue maintaining backward compatibility due to features and syntax changes in python 3.10.
Instead one of our dependencies is dropping support for python 3.8 which makes it difficult for us to continue supporting 3.8. Separately, we have noticed some very large performance gains that we can take advantage of if we drop python 3.9 as well.
On 1/23/23 09:05, Clark Boylan wrote:
On Mon, Jan 23, 2023, at 3:46 AM, Matthieu Huin wrote:
Hello,
Snip.
I see there are enough significant new features and accompanying syntax introduced in 3.10 that would make maintaining backward compatibility with 3.9 a pain ... what would be a tentative timeline to drop 3.9 support?
To be clear Zuul is currently tested against python 3.8 and 3.11 with the intent that testing the upper and lower bound ensures we support the entire range of 3.8, 3.9, 3.10, and 3.11. There is no issue maintaining backward compatibility due to features and syntax changes in python 3.10.
Instead one of our dependencies is dropping support for python 3.8 which makes it difficult for us to continue supporting 3.8. Separately, we have noticed some very large performance gains that we can take advantage of if we drop python 3.9 as well.
To summarize the discussion:
* We will need to drop 3.8 when we add support for Ansible 7 (Ansible forces our hand on this), and that is imminent. * If we increase our minimum to 3.9 at that time, UBI users should still be able to use a supported Python. * In 4-8 weeks from now, once newer RHEL is out, we should be able to drop 3.9 support (raising the minimum to either 3.10 or 3.11) without causing issues.
That sound about right?
-Jim
On 2023-01-24 07:29:22 -0800 (-0800), James E. Blair wrote: [...]
In 4-8 weeks from now, once newer RHEL is out, we should be able to drop 3.9 support (raising the minimum to either 3.10 or 3.11) without causing issues.
[...]
Additionally Debian's bookworm release (I got the name right this time!) reaches hard freeze around the same time, with full freeze and hopefully release with Python 3.10 or 3.11 a month or so after that:
https://release.debian.org/testing/freeze_policy.html
The current Debian stable (bullseye) is still on Python 3.9:
https://packages.debian.org/python3
to clarify: keep in mind that the "4-8 weeks" estimate is really just an estimate and there's no guarantee that we'll see a release in this time frame. Maybe I should also rephrase to be less RHEL-centric: zuul should drop support for python 3.9 once enough major distros' LTS versions have a way to support 3.10 or 3.11, or fully upgraded to any of these versions or above.
On Tue, Jan 24, 2023 at 5:34 PM Jeremy Stanley fungi@yuggoth.org wrote:
On 2023-01-24 07:29:22 -0800 (-0800), James E. Blair wrote: [...]
In 4-8 weeks from now, once newer RHEL is out, we should be able to drop 3.9 support (raising the minimum to either 3.10 or 3.11) without causing issues.
[...]
Additionally Debian's bookworm release (I got the name right this time!) reaches hard freeze around the same time, with full freeze and hopefully release with Python 3.10 or 3.11 a month or so after that:
https://release.debian.org/testing/freeze_policy.html
The current Debian stable (bullseye) is still on Python 3.9:
https://packages.debian.org/python3
-- Jeremy Stanley
participants (5)
-
Clark Boylan
-
James E. Blair
-
Jeremy Stanley
-
Matthieu Huin
-
Radosław Piliszek