- service-announce - lists.opendev.org

review.opendev.org Gerrit 3.13 Upgrade on June 5, 2026 at 1600 UTC
by Clark Boylan 27 May '26

27 May '26

The OpenDev team will be upgrading Gerrit from version 3.12 to 3.13 on Friday June 5, 2026 at 1600 UTC. We are allocating an hour for this process. During the upgrade you will notice a short Gerrit outage. There are a number of changes in 3.13. The most visible is likely to be the removal of robot comments and the support for time scoped authentication tokens instead of the old HTTP password system. Removal of robot comments will force Zuul to leave normal inline comments on changes. It may also cause your web browser client to error if it attempts to fetch robot comments. You can safely dismiss these error messages then perform a hard refresh of your browser client to fetch the new 3.13 client implementation which should prevent it from happening again. Authentication tokens are the future though 3.13 supports both the old HTTP passwords and the authentication tokens. There are automated tools to convert the old passwords to authentication tokens which we will likely employ at a date further into the future. In the meantime, it should be safe for users to convert to authentication tokens if they wish. Eventually the old HTTP passwords will go away entirely, but our automated conversion should make this transparent to users. If you would like to learn more you can read the Gerrit 3.13 release notes here: https://www.gerritcodereview.com/3.13.html Please let us know if you have any questions or concerns, Clark on behalf of the OpenDev Team

1 0

ubuntu-resolute (Ubuntu 26.04 LTS) amd64 in Zuul
by Jeremy Stanley 14 May '26

14 May '26

We now have nodes and package mirroring for the amd64 (x86_64) architecture of Ubuntu 26.04 LTS codenamed "Resolute Raccoon" available for use in all our standard Zuul providers. If you encounter any issues with them, please let us know on the service-discuss(a)lists.opendev.org mailing list or in the #opendev:opendev.org Matrix room. Note that this does not include the arm64 (AArch64) architecture, we have not added nodes or mirroring for ubuntu-resolute-arm64 yet and will announce that separately once available. -- Jeremy Stanley on behalf of the OpenDev Collaboratory Root Sysadmins

1 0

Upgrading Gerrit on review.opendev.org to 3.12 on April 12, 2026
by Clark Boylan 30 Mar '26

30 Mar '26

The OpenDev team plans to upgrade Gerrit on review.opendev.org from version 3.11 to 3.12 on April 12, 2026 at 2000 UTC. We are allocating an hour for the upgrade, but should be done quicker if all goes according to plan. This upgrade will ensure that we continue to run a supported Gerrit version after the expected 3.14 release in May. This release brings a number of changes, but many of them are backend details. For example the persistent cache implementation is updated from H2 v1 to H2 v2. One consequence of this change, is that everyone will be logged out of the server as our persistent session cache will be recreated on the new cache version. The release notes, https://www.gerritcodereview.com/3.12.html, provide many more details if you are curious to see what else is changing. You may also see shorter Gerrit outages after the OpenStack release on April 1. We have a few preparation steps that we would like to apply before we get to the upgrade. In particular there are is a new bugfix release for 3.11 and there are some features we want to disable or enable prior to upgrading to 3.12. Let us know if you have any questions or concerns, Clark

1 0

Dropping Ubuntu Bionic Images from Zuul and our Mirrors Next Week
by Clark Boylan 26 Mar '26

26 Mar '26

Ubuntu Bionic has been unsupported for some time. We have kept it around within our CI system and mirrors due to the number of systems relying on it; however, there are now two good reasons to drop it: 1. Ubuntu Resolute is going to release next month and we would like to free space on our mirror volume to mirror Resolute packages 2. Zuul's default Ansible versions does not support the version of Python on Bionic. At some point Zuul will drop support for the older Ansible version that does work with Bionic. Essentially, dropping Bionic allows us to move forward and there is value in being able to do that without also continuing to maintain old unsupported systems. Expect the Bionic cleanup to begin after the OpenStack Gazpacho release on April 1. Most likely happening either April 2nd or 3rd. Let us know if you have any questions or concerns, Clark

1 0

Moving OpenDev Synchronous Communications to Matrix
by Clark Boylan 20 Jan '26

20 Jan '26

On January 20, 2026 at 20:00 UTC the OpenDev team will move synchronous communications from #opendev on the OFTC IRC network to #opendev:opendev.org [0] on Matrix. Given the number of people joining #opendev on OFTC via Matrix bridges and the uncertain future of those bridges it seemed like our users would be best served using Matrix instead of IRC. Matrix also comes with a number of benefits including the use of web protocols, browser driven signups and user management, and persistent scrollback for clients without running special processes. The specification written to plan this move [1] covers the motivation in much more detail. This new Matrix room already exists, and you can join it today in preparation for the move. We will also continue to lurk in the old IRC channel to redirect conversations after the move. Feel free to reach out to us if you have any questions about this change. We're looking forward to seeing you on Matrix. [0] https://matrix.to/#/#opendev:opendev.org [1] https://docs.opendev.org/opendev/infra-specs/latest/specs/matrix_for_opende… Clark

1 2

Rekeying Zuul's Encryption Keys for Secrets and SSH Keys
by Clark Boylan 08 Jan '26

08 Jan '26

Today we became aware of a path traversal bug in Zuul's Web component. We have since mitigated the problem, and Zuul is fixing it properly. Despite this, we cannot be certain that the data used to encrypt Zuul secrets has remained private. For this reason we will be rekeying Zuul's encryption system. This process may take some time, and you will probably notice a Zuul outage while we prepare to start Zuul up again. The effect this will have is that all existing secrets within Zuul configuration will no longer be valid. Secrets will need to be reencrypted after we rekey the system. On top of that Zuul users should rotate these secrets and replace their values with new ones as part of this rotation. Do not simply reencrypt the old secret using the new Zuul encryption keys. Additionally, all project specific SSH keys will be regenerated. The old public keys should be removed from any authorized_keys files that may have included them. This is likely to be quite disruptive to OpenDev CI users. We're working through it ourselves, and will do our best to help you work through it as well. It is likely that secret rotations will need to occur in order as we bootstrap more fundamental aspects of CI jobs. Feel free to reach out to us if you run into any unexpected problems or have additional questions or concerns. Clark on behalf of the OpenDev team

1 0

Upgrading OpenDev's Gerrit Server on December 7, 2025 at 21:00 UTC
by Clark Boylan 09 Dec '25

09 Dec '25

The OpenDev team will be upgrading the Gerrit server on review.opendev.org on December 7, 2025 at 21:00 UTC. This will require a short downtime of the service if everything goes well and a longer downtime if we need to revert. We are allocating an hour (21:00 - 22:00 UTC) for this work. We will upgrade from Gerrit 3.10 to 3.11. Gerrit 3.11 brings a number of changes including performance improvements, new features, and bug fixes. The upstream release notes [0] cover these updates in much more detail if you are interested. Let us know if you have any questions or concerns. [0] https://www.gerritcodereview.com/3.11.html Clark

2 1

Mailing lists maintenance 2025-09-19 at 20:00-21:00 UTC
by Jeremy Stanley 19 Sep '25

19 Sep '25

Mailing list services will be offline briefly for all hosted domains (lists.airshipit.org, lists.katacontainers.io, lists.opendev.org, lists.openinfra.org, lists.openstack.org, lists.starlingx.io, lists.zuul-ci.org) between 20:00 and 21:00 UTC on Friday, 2025-09-19 in order to address recent performance degradation related to the rise in automated crawler traffic. Inbound E-mail deliveries should queue temporarily in the MTA, which is not being taken offline, so at worst posts to the lists will be delayed until shortly after maintenance concludes. Web access to archives and subscription management will be unavailable while this work is in progress. -- Jeremy Stanley on behalf of the OpenDev Collaboratory Sysadmins

1 1

OpenDev python-builder, python-base, and uwsgi-base container images moved to quay.io/opendevorg
by Clark Boylan 11 Sep '25

11 Sep '25

OpenDev has been moving the container images it builds to quay.io/opendevorg. The time to move the three python base images we build (python-builder, python-base, and uwsgi-base) to quay.io/opendevorg is now. These changes [0][1] just merged and you can see the image updates on quay.io [2][3][4]. Users of these images should start pulling the images from their new home at this time. You can see examples of what that involves in these related OpenDev changes [5][6][7]. You may remember that we attempted to do this in the past and had to revert. Since then we've identified and fixed the problems with speculative image building in zuul using both podman and docker. We have also addressed the use of speculative container images at runtime by moving OpenDev's python based deployments to podman. Fixing speculative testing was the last major blocker for hosting the images on quay.io. Doing so should help us reduce overall reliance on docker hub and as a result reduce chances of hitting API limits there. We will not be deleting the existing images on docker hub at this time, but those images will not receive updates. Our goal is to followup in approximately 3 months and delete the images from docker hub at that time. This should give everyone plenty of time to switch to the new location. Then deletion will provide signal to anyone who is using the images and missed this email that they need to update before the images become very stale. Note this means that quay.io/opendevmirror/python-builder, quay.io/opendevmirror/python-base, and quay.io/opendevmirror/uwsgi-base will also stop getting updates. Any use of these images should pull from the new quay.io/opendevorg/ locations. [0] https://review.opendev.org/c/opendev/system-config/+/957277 [1] https://review.opendev.org/c/opendev/system-config/+/960644 [2] https://quay.io/repository/opendevorg/python-builder?tab=tags [3] https://quay.io/repository/opendevorg/python-base?tab=tags [4] https://quay.io/repository/opendevorg/uwsgi-base?tab=tags [5] https://review.opendev.org/c/opendev/lodgeit/+/958602 [6] https://review.opendev.org/c/opendev/system-config/+/958593 [7] https://review.opendev.org/c/opendev/system-config/+/958597 Clark

1 0

Updating OpenDev Zuul's default Ansible version to Ansible 11
by Clark Boylan 12 Aug '25

12 Aug '25

Currently OpenDev Zuul's default Ansible version is Ansible 9. Zuul has recently added support for Ansible 11. The Zuul and OpenDev Zuul tenants switched to Ansible 11 by default last week. So far we have seen no problems with jobs run in these two tenants related to the Ansible version change. We intend on switching the default Ansible version for all Zuul tenants in OpenDev's Zuul installation to Ansible 11 on August 18, 2025. The reason for this is Zuul will update the default Ansible version to Ansible 11 in Zuul itself. Not long after that change we can expect Zuul to drop support for Ansible 9. Changing things in OpenDev first allows us to do these changes in a controlled manner and get ahead of any potential problems. If you would like to test your jobs before the tenant configuration updates go into effect you can do so by setting `ansible-version: '11'` in the configuration of a job [0], push the resulting change to Gerrit, and then wait for Zuul to run the job and provide you with results. It is not necessary to merge the change; Zuul will speculatively test this setting. If you discover any problems through this process please let us know. Similarly, after the tenant defaults roll out next week, you can roll back to Ansible 9 using the same job level override if problems arise. You will need to correct these problems to ensure Ansible 11 compatibility before Ansible 9 is dropped from Zuul. [0] https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.ansible-versi… Clark

1 0