I had meant to send one of these every couple of months, but it has been about six months since I sent the last update email. Apologies for the delay. It has been a busy six months; let's dive in.
We bootstrapped the ability to deploy new control plane nodes on Ubuntu Jammy. All newly created servers are built on Jammy unless we have a very good reason to use something older. Additionally, we have updated our base Python Docker images to include support for Python 3.11. Python 3.11 is only a few weeks old, but we are able to deploy software on top of it taking advantage of the performance benefits it provides.
Gerrit has been upgraded to 3.6.3. This release is just behind 3.7.0 which was released about a month ago. I believe this is the closest we've been to the latest Gerrit release in about a decade. We do plan to upgrade to 3.7.x, but this upgrade appears to be a bit more involved than the last few. In particular, the upgrade itself needs offline re-indexing, we need to port our label configs to the new submit requirements system, and the web UI is now based on Lit instead of polymer.
The Gitea cluster is now running 1.17. This update didn't bring any major user facing changes. The next update to 1.18.x should correct a problem with Gitea misidentifying certain files as being "vendored".
Prior to the October PTG event our Meetpad deployment was updated to use the latest stable releases of Jitsi Meet, and we applied a modern configuration to our installation. One major feature this brought is a landing page for meetings where users can configure their name and settings. Importantly, this landing page should prevent browsers' auto mute behavior for auto playing video+audio from breaking calls for our users. We believe this was a major source of users not being able to hear others in calls.
Mailing lists for lists.opendev.org and lists.zuul-ci.org have been migrated to a new Mailman 3 server. We are using these two list domains to work out issues while we prepare for migrating the rest of our mailing lists to the new server. This should give us a much more friendly web interface for browsing archives with built in search capability as well as easier to use moderation tools for list moderators.
Our Zuul deployment has seen a number of improvements. The ZooKeeper cluster backing Zuul has been upgraded to ZooKeeper 3.7 which brings performance improvements. Zuul supports Ansible 6 now, and all of our tenants are running jobs with this version of Ansible. This version of Ansible fixes pipelining which should provide some improvements to job runtime. Finally, we are upgrading Zuul weekly using automated zero downtime rolling restarts. This ensures we're getting the latest and greatest Zuul features and can provide quick and early feedback to Zuul.
Looking ahead to 2023, I expect to see a lot more of what we've done over the last six months. Gerrit will be upgraded to 3.7, Gitea will be upgraded to 1.18, more mailing lists will be migrated to Mailman 3, and more services will be deployed on Jammy running with Python 3.11. See you next year!
We are planning upgrade review.opendev.org to the Gerrit 3.6 release
on 12th December, 2022 at around 20:00UTC.
This will involve an outage of around 1 hour.
There should not be much user-visible change after the upgrade; the
major items do not affect our usage. The full release notes are at
On Monday, December 5 at 20:00 UTC, the OpenDev Collaboratory
systems administrators will be migrating the lists.opendev.org and
lists.zuul-ci.org mailing list sites from their existing Mailman
2.1.29 server to a shiny new Mailman 3.3.7 deployment. This
maintenance window is expected to last no more than one hour. The
key takeaways are as follows:
* There will be a brief outage for the affected sites while DNS is
updated and configuration, subscriber lists and message archives
are imported, but incoming messages should correctly queue at
the sender's end and arrive at the conclusion of our work
* Because this is on a new server, there are new IP addresses from
which list mail will be originating, 22.214.171.124 and
* Moderation queues will not be copied to the new server, so
moderators are encouraged to process any held messages prior to
the migration time in order to avoid losing them
* Anyone wishing to adjust their list subscriptions, or handle
moderation or list configuration after the migration needs to
use the Sign Up button on the Web page to create a new account;
it will be linked to your imported roles as soon as you confirm
by following a URL from an E-mail message the server sends you,
and is global for all sites on the server so you only need to do
* The software providing Web front-ends for list management and
archive browsing is entirely new in Mailman v3 and therefore has
a much different appearance, though we've added appropriate
redirects and frozen copies of old archives in order to
accommodate existing hyperlinks
Once we smooth out any unanticipated rough patches with the new
service, we'll schedule additional maintenance to migrate mailing
list sites for other domains (Airship, Kata, OpenInfra, OpenStack,
StarlingX). If you have any questions or concerns, feel free to
follow up on the service-discuss mailing list or find us in the
#opendev channel on the OFTC IRC network.
I wanted to make sure that everyone subscribed to this list is aware that we are discussing the Storyboard service, its maintenance needs, and user needs on a thread on service-discuss(a)lists.opendev.org . I have been told that not everyone is aware of the service-discuss list and may have missed that thread. Please follow up on that thread if you have any feedback for (not) wanting to use storyboard or desire to help maintain the service.
This is a good opportunity to remind everyone that OpenDev runs three mailing lists for user interaction:
1. service-announce(a)lists.opendev.org  This mailing list. Intended to be kept low volume and limited to the most important email notices.
2. service-discuss(a)lists.opendev.org  Where we discuss the service needs, meeting agendas, and other topics with more volume. The bulk of our mailing list discussion will be here. All are welcome to join and participate. -discuss and -announce are separate to give people a lower volume option with -announce.
3. service-incident(a)lists.opendev.org This is a private mailing list that OpenDev admins and other trusted individuals may be subscribed to. Users can email this list with sensitive messages like vulnerability disclosures.
As mentioned in the previous email  about updating our default Ansible version to Ansible 6, Zuul will be removing support for Ansible 5 soon. Based on Zuul's currently estimated release schedule we expect this to happen on or near October 18. This change won't take effect until OpenDev restarts Zuul. We will try to make that happen quickly after Zuul lands this change in order to catch any issues quickly.
Additionally, it has been called out that we are still defaulting our base job nodeset to ubuntu-focal. Ubuntu Jammy has been released for about 6 months now and has a .1 release as well. To catch up we will update the default nodeset in our base jobs to ubuntu-jammy on October 25, 2022.
If you would like to test that your jobs run against Ubuntu Jammy you can modify your jobs to set `nodeset: ubuntu-jammy` today and test them. Note, any jobs that expect a specific platform should already be explicitly setting the nodeset they rely on. There are a number of jobs already running on Jammy which gives us confidence in the newer images.
The OpenDev team is planning to update the default-ansible-version to 6 in the near future. Zuul is planning to drop support for Ansible 5, and switching OpenDev to Ansible 6 early allows us to provide valuable feedback if there are problems. There are other reasons for the switch as well. Ansible 5 broke pipelining, but we expect Ansible 6 to correct this problem. Having pipelining should improve the runtime of jobs.
We had been holding off on making this change as StarlingX and OpenStack have both recently worked through some fairly major releases. Those releases exist now making this a good time for a change like this.
As far as testing goes, Zuul's Zuul tenant has been running with an Ansible 6 default for several weeks now. I've also tested this change with Devstack specifically  as Devstack seems to have a good representative set of complicated jobs.
The change to update our defaults is up for review . Feel free to leave any comments there or on this email thread. Note that you can override this change if necessary using the same mechanism I used to test Devstack . However, if you do this please communicate that change to us as Zuul does plan to remove Ansible 5 support which means any problems with Ansible 6 need to be communicated back to Zuul.
Following the earlier 2.8/2.9 deprecation in June and change of our
default to Ansible 5 in July, we will soon be restarting onto a
version of Zuul which no longer supports 2.x versions at all. While
we've looked for any lingering overrides to old Ansible versions in
master branch jobs and given projects a heads up individually, we
lack insight into whether the same may be occurring on non-default
branches of projects.
If you are aware of any jobs on any of your branches which pin back
to an earlier Ansible version, we strongly recommend you remove
those overrides at your earliest opportunity, otherwise those jobs
will break in the coming days.
Here is the change to remove support for Ansible 2.8 and 2.9 in Zuul
(we generally restart our Zuul services every weekend, so that's
when it's most likely to take effect):
For some example changes which clean up old Ansible version
overrides, see this review topic list:
This is just a reminder to users of our Zuul service that if you
find changes aren't getting any jobs run, or seem to not be running
some jobs you expect, you should double-check your configuration to
make sure you don't have a lingering "queue" setting in your project
Over the weekend, we updated to a version of Zuul which removed the
long-deprecated support for pipeline level queue declarations, and
it now expects queues to be set at the project level instead. We
directly notified some projects which we knew would be impacted, but
did not send a broader announcement to this list, so I just wanted
to be sure anyone taken by surprise with this change doesn't have to
waste additional time troubleshooting it.
Earlier today, the OpenDev sysadmins corrected numerous cases of
this error in the central openstack/project-config repository used
by the openstack tenant, but there are still many projects which
have similar errors in their in-repository configuration as well.
For the reminder from the Zuul maintainers, see here:
Also these posts to the OpenStack discussion lists may include
helpful information about the transition:
The OpenDev team will be updating the default Ansible version in our Zuul tenants from Ansible 2.9 to Ansible 5 on June 30, 2022. Zuul itself will eventually update its default, but making the change in our tenant configs allows us to control exactly when this happens.
There are some known issues with this transition; however, we've already worked through and addressed a number of them. If you'd like to double check your jobs work with Ansible 5 today you can do that by pushing a change(s) that updates job.ansible-version  to the value '5'. If you do this and find problems please let us know.
The Zuul zuul tenant is already running with this set tenant wide which means we expect most of the base job pre-run and post-run content to work with Ansible 5. We also did some spot checking of things like devstack and tempest and expect most of those jobs to function too.
Finally, we can eventually expect Zuul to drop support for Ansible 2.8 and 2.9 in the future. What this means is we shouldn't just pin to those old versions without a plan forward to addressing any remaining v5 incompatibilities.
If you have any questions let us know.
It has been a couple months since I sent out our last activity update, and now seems like a good time to share what is new.
Ubuntu Jammy test nodes are now available in both x86_64 and ARM64 flavors. This version of Ubuntu ships with python3.10 and is ready for you to start testing your python projects on this newer runtime.
Our Gerrit server running at https://review.opendev.org has been upgraded from Gerrit 3.4 to 3.5. This newer version of Gerrit brings features like new change search predicates, updates to the UI, and performance updates. On the whole this is a bit more of an incremental update compared to previous upgrades. We expect many users haven't even noticed that the upgrade happened. If you'd like to learn more you can read the Gerrit 3.5 release notes: https://www.gerritcodereview.com/3.5.html.
Glean 1.23.0 has been released and finally adds proper support for static ipv6 configuration on RPM based distros.
Our Ethercalc server previously running at https://ethercalc.openstack.org has been shutdown and removed.
On the Zuul side of things there are two changes you should be aware of and planning for.
The first is the planned removal of pipeline specific queues in your Zuul config. This configuration was deprecated a number of releases ago and is slated for removal in the next major Zuul release. The OpenStack tenant is the only Zuul tenant that has this old style of configuration. More information on which projects are affected and how to correct it can be found in my email about this topic: https://lists.openstack.org/pipermail/openstack-discuss/2022-May/028603.html. Note that projects with this configuration in place that do not update will have broken Zuul configurations after we upgrade to the next major Zuul release.
Next, is the plan for changing the default Zuul Ansible version from Ansible 2.9 to Ansible 5 on June 30, 2022. We've already done some testing of this and the base jobs as well as devstack and tempest seem to work with newer Ansible. If you'd like to learn more (including how you can test your specific projects early) please check my previous email on this subject: https://lists.opendev.org/pipermail/service-announce/2022-June/000041.html.
Looking ahead we should expect our Gitea farm to be upgraded to v1.17. Work will begin on upgrading Gerrit to version 3.6, but it is still too early to know when we will perform that upgrade.
There are a few larger tasks we'd like to get moving on including updating our Keycloak server to provide SSO capabilities for our services, upgrading our list servers to Mailman 3, modernizing our StoryBoard deployment, and deploying a Prometheus service to collect server and service metrics. If any of these sound interesting to you, and you would like to help, please reach out.