- service-announce - lists.opendev.org

Upgrading OpenDev's Gerrit Server on December 7, 2025 at 21:00 UTC
by Clark Boylan 26 Nov '25

26 Nov '25

The OpenDev team will be upgrading the Gerrit server on review.opendev.org on December 7, 2025 at 21:00 UTC. This will require a short downtime of the service if everything goes well and a longer downtime if we need to revert. We are allocating an hour (21:00 - 22:00 UTC) for this work. We will upgrade from Gerrit 3.10 to 3.11. Gerrit 3.11 brings a number of changes including performance improvements, new features, and bug fixes. The upstream release notes [0] cover these updates in much more detail if you are interested. Let us know if you have any questions or concerns. [0] https://www.gerritcodereview.com/3.11.html Clark

1 0

Mailing lists maintenance 2025-09-19 at 20:00-21:00 UTC
by Jeremy Stanley 19 Sep '25

19 Sep '25

Mailing list services will be offline briefly for all hosted domains (lists.airshipit.org, lists.katacontainers.io, lists.opendev.org, lists.openinfra.org, lists.openstack.org, lists.starlingx.io, lists.zuul-ci.org) between 20:00 and 21:00 UTC on Friday, 2025-09-19 in order to address recent performance degradation related to the rise in automated crawler traffic. Inbound E-mail deliveries should queue temporarily in the MTA, which is not being taken offline, so at worst posts to the lists will be delayed until shortly after maintenance concludes. Web access to archives and subscription management will be unavailable while this work is in progress. -- Jeremy Stanley on behalf of the OpenDev Collaboratory Sysadmins

1 1

OpenDev python-builder, python-base, and uwsgi-base container images moved to quay.io/opendevorg
by Clark Boylan 11 Sep '25

11 Sep '25

OpenDev has been moving the container images it builds to quay.io/opendevorg. The time to move the three python base images we build (python-builder, python-base, and uwsgi-base) to quay.io/opendevorg is now. These changes [0][1] just merged and you can see the image updates on quay.io [2][3][4]. Users of these images should start pulling the images from their new home at this time. You can see examples of what that involves in these related OpenDev changes [5][6][7]. You may remember that we attempted to do this in the past and had to revert. Since then we've identified and fixed the problems with speculative image building in zuul using both podman and docker. We have also addressed the use of speculative container images at runtime by moving OpenDev's python based deployments to podman. Fixing speculative testing was the last major blocker for hosting the images on quay.io. Doing so should help us reduce overall reliance on docker hub and as a result reduce chances of hitting API limits there. We will not be deleting the existing images on docker hub at this time, but those images will not receive updates. Our goal is to followup in approximately 3 months and delete the images from docker hub at that time. This should give everyone plenty of time to switch to the new location. Then deletion will provide signal to anyone who is using the images and missed this email that they need to update before the images become very stale. Note this means that quay.io/opendevmirror/python-builder, quay.io/opendevmirror/python-base, and quay.io/opendevmirror/uwsgi-base will also stop getting updates. Any use of these images should pull from the new quay.io/opendevorg/ locations. [0] https://review.opendev.org/c/opendev/system-config/+/957277 [1] https://review.opendev.org/c/opendev/system-config/+/960644 [2] https://quay.io/repository/opendevorg/python-builder?tab=tags [3] https://quay.io/repository/opendevorg/python-base?tab=tags [4] https://quay.io/repository/opendevorg/uwsgi-base?tab=tags [5] https://review.opendev.org/c/opendev/lodgeit/+/958602 [6] https://review.opendev.org/c/opendev/system-config/+/958593 [7] https://review.opendev.org/c/opendev/system-config/+/958597 Clark

1 0

Updating OpenDev Zuul's default Ansible version to Ansible 11
by Clark Boylan 13 Aug '25

13 Aug '25

Currently OpenDev Zuul's default Ansible version is Ansible 9. Zuul has recently added support for Ansible 11. The Zuul and OpenDev Zuul tenants switched to Ansible 11 by default last week. So far we have seen no problems with jobs run in these two tenants related to the Ansible version change. We intend on switching the default Ansible version for all Zuul tenants in OpenDev's Zuul installation to Ansible 11 on August 18, 2025. The reason for this is Zuul will update the default Ansible version to Ansible 11 in Zuul itself. Not long after that change we can expect Zuul to drop support for Ansible 9. Changing things in OpenDev first allows us to do these changes in a controlled manner and get ahead of any potential problems. If you would like to test your jobs before the tenant configuration updates go into effect you can do so by setting `ansible-version: '11'` in the configuration of a job [0], push the resulting change to Gerrit, and then wait for Zuul to run the job and provide you with results. It is not necessary to merge the change; Zuul will speculatively test this setting. If you discover any problems through this process please let us know. Similarly, after the tenant defaults roll out next week, you can roll back to Ansible 9 using the same job level override if problems arise. You will need to correct these problems to ensure Ansible 11 compatibility before Ansible 9 is dropped from Zuul. [0] https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.ansible-versi… Clark

1 0

Removal of CLA enforcement and configuration in Gerrit
by Jeremy Stanley 10 Jul '25

10 Jul '25

The configured contributor license agreements in our Gerrit refer to the old Open Infrastructure Foundation, which is in the process of being dissolved after its operations joined the Linux Foundation. As such, they are no longer valid for future contributions to projects (and it's questionable whether they were ever valid for non-OpenInfra projects anyway). OpenInfra projects which were previously relying on these have since switched to the DCO with a different enforcement mechanism. On July 18 we're planning to merge a change removing CLA enforcement from all remaining repositories in the OpenDev Collaboratory Gerrit. If you want to make adjustments to your project's ACLs in the meantime, please propose changes into Gerrit for their files under https://opendev.org/openstack/project-config/src/branch/master/gerrit/acls so there won't be any gap in enforcement (for example, if you prefer to switch them to DCO enforcement by swapping the requireContributorAgreement control over to requireSignedOffBy instead). Feel free to reply on the service-discuss mailing list or find us in the #opendev channel on the OFTC IRC network if you have any questions about this. -- Jeremy Stanley on behalf of the OpenDev Collaboratory Sysadmins

1 0

Enabling Gerrit Hashtags For All Registered Users
by Clark Boylan 28 May '25

28 May '25

For some time now Gerrit has supported HashTags as an alternative to Change Topics. Over time Change Topics have evolved for most Gerrit installations to indicate common sets of work that should be merged together when submitWholeTopic is enabled. In these situations the expectation is that you would use HashTags to annotate Changes with information that we have often used Change Topics for within OpenDev. To catch up with the rest of the Gerrit community OpenDev will enable HashTags by default for all Registered Users on June 3, 2025. This will allow you to edit HashTags on any change as long as you are logged in. Some projects have already enabled this feature for specific groups of users. This change will override those existing rules enabling HashTags for all Registered Users on those projects too. The number of projects this affects is quite small, but let us know if you have any questions or concerns. Clark

1 0

review.opendev.org Gerrit Outage April 21, 2025
by Clark Boylan 11 Apr '25

11 Apr '25

The OpenDev team will shut down Gerrit services on review.opendev.org at 16:00 UTC April 21, 2025 in order to migrate and upgrade the underlying server instance. The work is expected to take about an hour and should largely be transparent to users other than the required outage. One thing users may notice is the IP Addresses for review.opendev.org will change. If you need to update your firewall rules you can do so by allowing TCP port 443 and TCP port 29418 to: 162.253.55.233 2604:e100:1:0:f816:3eff:fe31:926a These IPs belong to the new review03.opendev.org server which will replace review02.opendev.org. Let us know if you have any questions or concerns. Clark

1 0

bindep 2.12.0
by Jeremy Stanley 25 Jan '25

25 Jan '25

I've released bindep 2.12.0 just now. These are the highlights: New Features ------------ * Basic support for PopOS is now included. Upgrade Notes ------------- * Python 2.7 and 3.5 are no longer supported. * Bindep now depends on the distro python library to determine details about the current platform. This library looks at both /etc/os-release and lsb_release to find platform info. The os-release file data is preferred and at times has slightly different data than lsb_release. Every effort has been made to make this transition backward compatible but some things may have been missed. The motivation for this change is that not all distros have lsb_release available and we can let the distro library sort that out for us. -- Jeremy Stanley

1 0

Planned outage for mailing lists on Monday 2024-12-02 15:00-17:00 UTC
by Jeremy Stanley 27 Nov '24

27 Nov '24

The OpenDev Sysadmins will be performing maintenance on our mailing list server between 15:00 and 17:00 UTC on Monday, 2024-12-02. During this time, messages to and from the lists for all our sites will be held temporarily while we make adjustments in the backing database for renaming one of the domains in use. Web archives may be unavailable for part of the time as well. The outage should start some time around 15:00 UTC and is not expected to last more than two hours (but will hopefully conclude far more quickly). Please follow up (either to the service-discuss(a)lists.opendev.org mailing list or to me personally) with any questions you may have on this activity. -- Jeremy Stanley, OpenDev Systems Administrator

1 0

Upgrading OpenDev's Gerrit Server on December 6, 2024 at 16:00 UTC
by Clark Boylan 13 Nov '24

13 Nov '24

The OpenDev team will be upgrading the Gerrit server on review.opendev.org on December 6, 2024 at 16:00 UTC. This will require a short downtime of the service if everything goes well and a longer downtime if we need to revert. We are allocating an hour (16:00 - 17:00 UTC) for this work. We will upgrade from Gerrit 3.9 to 3.10 remaining approximately one release behind the latest Gerrit release as long as the Gerrit 3.11 release happens on time this month. Gerrit 3.10 brings a number of changes including many performance improvements and fewer email notifications. The upstream release notes [0] cover these updates in much more detail if you are interested. Let us know if you have any questions or concerns. [0] https://www.gerritcodereview.com/3.10.html Clark

1 0

Pruning OpenDev's Intermediate Registry on insecure-ci-registry.opendev.org November 15, 2024
by Clark Boylan 06 Nov '24

06 Nov '24

The OpenDev team will begin some long overdue pruning of the storage system behind the intermediate container registry running on insecure-ci-registry.opendev.org on November 15, 2024. The last time we tried this we believe container image client errors were due to the pruning. Unfortunately, we don't have enough information to know if this will happen again. For this reason we will be prepared to replace the entire Swift container on the backend and start over fresh if necessary. We ask that you report any unexpected client behaviors with the intermediate registry on or after November 15, 2024. Additionally, if we do end up replacing the entire backend storage container we will lose all intermediate container image state. If this happens, jobs relying on state in the intermediate registry may fail. In those cases you may need to recheck jobs that build the images or merge additional changes to refresh the content in the intermediate registry. Clark

1 0

Switching OpenDev's Default Zuul Nodeset to ubuntu-noble
by Clark Boylan 15 Aug '24

15 Aug '24

The OpenDev team plans to switch the default nodeset in our Zuul base jobs from ubuntu-jammy to ubuntu-noble on August 21, 2024. The new release has been available for several months now and we have jobs relying on it. One of the last steps in the Noble rollout is to switch the default over. This change will impact any jobs that don't explicitly override the nodeset as they will start running on the newer Ubuntu release after this change. If this creates problems for your jobs you can either override the nodeset to ubuntu-jammy forcing the job to run on the nodes used before this change, or you can update your job to accommodate whatever changes the newer version of Ubuntu requires. If you have any questions or concerns feel free to respond to this email or reach out via IRC. For OpenStack in particular, we've been told that they expect most jobs to already override the nodeset and this change will largely be a noop for them. Clark

1 0

Removal of Inactive Projects from Zuul's Tenant Configuration
by Clark Boylan 10 Jul '24

10 Jul '24

The OpenDev team is going to start removing inactive x/* namespace projects from Zuul's tenant configuration. The two primary motivations behind this are to remove projects that already have broken configurations within Zuul (see the Zuul config errors page [0]), and to avoid projects entering this state in the future as we clean up test nodes like Ubuntu Xenial or CentOS 8 Stream or otherwise modernize the CI system. Removal from Zuul's tenant configuration is a lightweight method of ensuring we're addressing problems with active projects that can be fixed without the added noise from projects that are unlikely to fix things on their own. A first pass at this sort of cleanup is being done with this change [1]. Feel free to chime in if you know there is already work underway to correct x/* projects' config errors, or if you know of any additional projects that should be removed from Zuul. We aim to merge this change on July 16, 2024. It should be noted that if projects do become active again and have volunteers willing to correct the CI configuration state, it is straightforward to add the project back into Zuul enabling that effort to progress. Projects will simply add themselves back to the tenant configuration and then push changes to fix the CI config. [0] https://zuul.opendev.org/t/openstack/config-errors [1] https://review.opendev.org/c/openstack/project-config/+/923509 Clark

1 0

CentOS 8 Stream End of Life and Removal from OpenDev CI
by Clark Boylan 20 Jun '24

20 Jun '24

Hello everyone, I wanted to give you a heads up that CentOS 8 Stream has reached its end of life and the upstream package repos have been removed. Our mirrors have faithfully replicated the empty mirror content (there is only a readme file now [0]). The end result of which is failing CI jobs if the jobs run on CentOS 8 Stream. The OpenDev team intends on removing these nodes from Zuul and Nodepool in the near future as jobs can no longer successfully run on CentOS 8 Stream. Projects should check their CI jobs (on older branches too) and either remove jobs that rely on CentOS 8 Stream or update those jobs to run on a different platform. [0] https://mirror.dfw.rax.opendev.org/centos/8-stream/readme Clark

1 0

Ubuntu Noble Test Nodes Now Available in Zuul
by Clark Boylan 24 May '24

24 May '24

The OpenDev team has added Ubuntu Noble test nodes to Zuul and Nodepool. This means you can start running jobs on the latest Ubuntu release using `nodeset: ubuntu-noble` or `nodeset: ubuntu-noble-arm64`. If you build your own custom nodesets the labels you will want to use are `ubuntu-noble` and `ubuntu-noble-arm64`. We expect these to generally work and have already successfully run a number of builds on the new nodes. That said they are new, and you may need updates to bindep files and/or Ansible roles and playbooks. Let us know if you have any questions or find problems. Clark

1 0

Gerrit Downtime on May 31, 2024 at 16:00 UTC
by Clark Boylan 22 May '24

22 May '24

Hello, The OpenDev team is planning to upgrade Gerrit on review.opendev.org to version 3.9 on May 31, 2024 beginning at 16:00 UTC. In order to perform this upgrade we require a short service outage. We are allocating an hour for this outage, from 16:00 to 17:00 UTC, as that should give us plenty of time to implement the rollback procedure if that becomes necessary. The Gerrit 3.9 release notes can be found here: https://www.gerritcodereview.com/3.9.html. One major feature our users may be interested in is the ability to suggest and apply updates to files via review comments. If you have any questions about the upgrade or the new version of Gerrit, feel free to reach out to the OpenDev team via this email thread or on IRC in #opendev on OFTC. Clark

1 0

Gerrit Outage for Project Renames Monday April 22, 2024 at 20:00 UTC
by Clark Boylan 17 Apr '24

17 Apr '24

The OpenDev team is planning a scheduled Gerrit service outage on review.opendev.org in order to rename projects. This outage will occur on Monday April 22, 2024 at 20:00 UTC until 21:00 UTC. If you have a project you would like to rename please get in touch with the OpenDev team soon (you can respond to this email or find us in #opendev on the OFTC IRC network). Please reach out if you have any questions or concerns. The same methods described above work for that too. Clark

1 0

CVE-2024-31497 and SSH key statistics
by Jeremy Stanley 16 Apr '24

16 Apr '24

For those of you who haven't seen, CVE-2024-31497 in the PuTTY tool was announced in the past day. If you're using a NIST P521 ECDSA key and have ever used it with affected versions of PuTTY, you should remove its access everywhere and consider it potentially disclosed. Details here: https://www.chiark.greenend.org.uk/~sgtatham/putty/wishlist/vuln-p521-bias.… The key type is generally listed as the first field in the corresponding public key file, or in each line of an authorized_keys entry, including those you may have set for your account in our Gerrit server (review.opendev.org) If the type is listed as ecdsa-sha2-nistp521 or sk-ecdsa-sha2-nistp521(a)openssh.com then it's been put at risk if ever used with PuTTY versions 0.68 through 0.80. On a related note, since I wanted to see how popular that key type is among our Gerrit users, I ran some quick statistics. Of the 36976 accounts created there to date, 19888 of them have SSH keys authorized. There are 35057 total authorized keys, which means accounts with SSH keys average 1.76 keys per account. Out of those keys, 44 claim to be NIST P521 type (0.13%) used by 41 different accounts (0.20% of accounts with keys, 0.11% of all accounts). Here's a breakdown of the most popular types for keys authorized in our Gerrit: 95% ssh-rsa 3.2% ssh-ed25519 1.4% ssh-dss 0.38% ecdsa-sha2-nistp256 0.25% ssh-dsa 0.13% ecdsa-sha2-nistp521 0.023% ecdsa-sha2-nistp384 So as you can see, the key type potentially impacted by CVE-2024-31497 isn't all that popular in our circles, but it's definitely nonzero. -- Jeremy Stanley, OpenDev SysAdmin

1 0

git-review 2.4.0
by Jeremy Stanley 18 Mar '24

18 Mar '24

It's my pleasure to announce the release of git-review 2.4.0. These are the corresponding release notes since the previous release: New Features ------------ * By default, git rebase --abort is run automatically following a failed test rebase, letting the user rebase again on their own if that's the state they want to be in. A -K/--keep-rebase option is added to get the old behavior. * For convenience, --wip is now synonymous with the longer --work-in-progress option. * If HEAD already exists on a remote branch, git review will now only warn and prompt for confirmation when pushing, in order to let the Gerrit server decide whether to accept or reject it. * A copy of the Gerrit upstream project's commit-msg hook is now included directly and written to disk by default rather than pulling a remote copy. This approach is safer and more robust, but may break workflows for sites which rely on users retrieving a customized commit-msg hook. For this reason, a command-line override is provided so that users can, for example, execute git review -s --remote-hook to get the old behavior when setting up a new clone. * The GIT_SSH environment variable will now be consulted, if set, in order to determine which ssh executable to call outside of git invocations, for consistency. * A new --hashtags option allows specifying a list of one or more Gerrit hashtag strings, separated by commas. Note that spaces are not allowed either within a hashtag or between separators. * New --message option can be used to include a custom patchset message when pushing. As always, the latest version of git-review is available from PyPI: https://pypi.org/project/git-review Current documentation can be found here: https://docs.opendev.org/opendev/git-review Its source code can be obtained from the OpenDev Collaboratory: https://opendev.org/opendev/git-review Please don't hesitate to follow up through the service-discuss(a)lists.opendev.org mailing list or in the #opendev channel on the OFTC IRC network if you have any questions. -- Jeremy Stanley

1 0

Removal of Ansible 6 support from OpenDev's Zuul Service
by Clark Boylan 14 Feb '24

14 Feb '24

The Zuul project will be removing Ansible 6 support in preparation for the upcoming Zuul 10.0.0 release. The exact timing of this change isn't known yet, but it may happen as early as this week. OpenDev has automated Zuul upgrades that should pull this change into our Zuul deployment not long after upstream merges the change. We have defaulted to Ansible 8 for many months now, and we don't expect this to create many issues. If you do see issues you will need to update to Ansible 8. Zuul will also be adding support for Ansible 9 in preparation for the upcoming Zuul 10.0.0 release. Once these changes are deployed in OpenDev you may want to start testing your jobs with Ansible 9 to ensure there are no problems with that update. At some point further in the future we will be changing the default Ansible version for OpenDev's Zuul to Ansible 9. Let us know if you have any questions or concerns, Clark

1 0

Upgrading mailing lists on 2023-11-02
by Jeremy Stanley 31 Oct '23

31 Oct '23

On Thursday, November 2, 2023 beginning at 15:30 UTC the OpenDev Collaboratory is upgrading the version of Mailman used to host the Airship, Kata, OpenDev, OpenInfra, OpenStack, StarlingX and Zuul mailing list sites from 3.3.8 to 3.3.9 along with related updates to its various component services (django-mailman3 1.3.9->1.3.11, hyperkitty 1.3.7->1.3.8, postorius 1.3.8->1.3.10, mailman-web 0.0.6->0.0.8). There will be brief downtime for the web interface while services are restarted, and some message deliveries may be briefly delayed but will still eventually be processed. One of the updates includes a database migration that implicitly clears the auth sessions table, so users who were logged into the web interface will be effectively logged out and need to log in again before performing authenticated operations. Web interfaces may have a slightly different look and feel after the update, and some new options too. Changelogs can be found here for anyone interested in seeing a list of what's new: https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/NEWS.… https://gitlab.com/mailman/django-mailman3/-/blob/v1.3.11/README.rst https://gitlab.com/mailman/hyperkitty/-/blob/v1.3.8/doc/news.rst https://gitlab.com/mailman/postorius/-/blob/v1.3.10/src/postorius/doc/news.… https://gitlab.com/mailman/mailman-web/-/blob/0.0.8/README.rst -- Jeremy Stanley

1 0

Upgrading review.opendev.org to Gerrit 3.8 on November 17, 2023
by Clark Boylan 30 Oct '23

30 Oct '23

On November 17, 2023 beginning at 15:30 UTC the OpenDev team will upgrade Gerrit at https://review.opendev.org from Gerrit 3.7 to 3.8. This upgrade shouldn't take more than an hour. We have managed to to keep our Gerrit installation about one major release behind upstream's release cadence, and Gerrit 3.9's impending release means it is time for us to upgrade to 3.8. There are a number of new features, bug fixes, and performance improvements in the new version. The Gerrit project publishes detailed release information [0] which you can review if you are interested in what sorts of changes to expect. If you are interested in the upgrade process you can follow that in our planning document [1]. It is worth noting that if all goes well we expect the total Gerrit downtime to be much shorter than an hour (should only take a few minutes). If we need to revert to Gerrit 3.7, we will need to do offline reindexing of our changes which takes about half an hour. We are using an hour long upgrade window to ensure we have time for downgrades should that become necessary. [0] https://www.gerritcodereview.com/3.8.html [1] https://etherpad.opendev.org/p/gerrit-upgrade-3.8 Clark

1 0

Zuul regex deprecation [action required]
by James E. Blair 21 Sep '23

21 Sep '23

Hi, Zuul is making a change that may require action from developers on projects hosted in OpenDev. Here is the release note regarding the change: > Regular expressions using Python syntax are deprecated in favor of RE2 > syntax instead, for additional speed and safety. Negative lookahead > assertions may be replaced using the negate keyword. See Regular > Expressions[1] for more information. > > If Zuul detects a regular expression using Python syntax that is not > supported by RE2, it will register a configuration syntax > warning. These may be viewed on the configuration errors page in the > web interface. > > A future version of Zuul will remove Python regex support completely > and these warnings will become errors. [1] https://zuul-ci.org/docs/zuul/latest/project-config.html#regex It's easy to see if your project on OpenDev is affected. Each tenant has a page where the current configuration errors and warnings are displayed. Visit any page in Zuul for your tenant, and if there is a bell icon in the top right, click on that. For example, the configuration error page for the OpenStack tenant is here: https://zuul.opendev.org/t/openstack/config-errors On that page, look for the "Regex Deprecation" warnings. You can expand the warning to see exactly the file and line number where the warning is found. Almost all of the issues identified so far have been due to the fact that RE2 does not support negative lookahead assertions. In the past, we used these a lot to say things like "run this job on any branch except master" with regexes like "(?!master)". Because this is such a common pattern, we added a feature in Zuul to make this easy. A configuration like: branches: "^(?!master).*$" can now be written as: branches: regex: "^master$" negate: true This simply negates the specified regex. It's not an exact translation of the original regex, but is slightly improved and to the point: anything but exactly "master". Please take a moment and check to see if your project has instances of this warning, and go ahead and fix it. Even if there are a lot of instances, they are likely to be very similar and correcting them may take only a few minutes. Here are two changes where we update the central configuration that applies to all projects: https://review.opendev.org/c/openstack/project-config/+/893702 https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/893792 These provide examples of how to make changes for several common cases; you may be able to simply copy what was done in those changes. If you have any trouble adapting existing regexes to the new format, please contact us in #opendev to discuss the issue. We may be able to find a simple solution, or if there is another special case that is difficult to handle, we may be able to add support to Zuul to mitigate the issue. If you use "update-regexes" as a topic name, it will help people keep track of progress and possible issues. The Zuul project has not yet set a date for when support for the old syntax will be removed. I expect it to be on the order of months, which should give folks plenty of time to make the necessary changes. But please do so early so that we have time to deal with any issues. Thanks, Jim (on behalf of the OpenDev team)

1 0

Removing OpenDev's Fedora Test Images on September 11, 2023
by Clark Boylan 06 Sep '23

06 Sep '23

OpenDev has been unable to update Fedora images in a timely manner leaving us with Fedora 35 and 36 images in our CI system. Fedora 35 and 36 are long past their EOL date, and we have been planning to remove them [0]. Somewhat recently the Fedora project stopped mirroring packages for these releases which resulted in empty mirrors on our end [1]. The end result is that Fedora nodes haven't been very functional within OpenDev since. We have slowly been cleaning up usage of Fedora nodes and recently removed the fedora-latest nodeset from Zuul [2]. The next step is to remove these images from Nodepool entirely. We will do so on Monday September 11, 2023. Chances are any Fedora jobs are already broken, but this is a good reminder to look into correcting any of those jobs by moving them to another platform. [0] https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… [1] http://mirror.dfw.rax.opendev.org/fedora/releases/36/ [2] https://review.opendev.org/c/opendev/base-jobs/+/892380 Clark

1 0

Updating OpenDev Zuul's default Ansible version to Ansible 8
by Clark Boylan 23 Aug '23

23 Aug '23

Currently OpenDev Zuul's default Ansible version is Ansible 6. Zuul has recently added support for Ansible 8, and is dogfooding Ansible 8 in the Zuul tenant for the Zuul project. Yesterday, the OpenDev team updated the Zuul tenant for the OpenDev project to default to Ansible 8 as well. So far we have seen no problems with jobs run in these two tenants related to the Ansible version change. We would like to update the default Ansible version for all of the Zuul tenants in OpenDev's Zuul installation to Ansible 8 in the near future. The reason for this is Zuul will update the default Ansible version to Ansible 8 in Zuul itself. Not long after that change, support for Ansible 6 in Zuul will be removed. Changing things in OpenDev first allows us to do these changes in a controlled manner and get ahead of any potential problems. Assuming the Zuul and OpenDev tenants continue to run without trouble our plan is to switch the nebulous, pyca, pypa, and vexxhost tenants to Ansible 8 by default on August 28, 2023. Then if things continue to run without trouble, we will complete this change by updating the openstack tenant on September 4, 2023. I had initially thought that we would do a more conservative rollout, but initial data indicates this change may be straightforward so we'd prefer a quicker move. Note that if there are major issues we can revert the setting in our Ansible config for any of these tenants and return to a default to Ansible 6 while the issues get addressed. Issues will need to be addressed as Ansible 6 is EOL and Zuul intends to drop support for this version of Ansible. If you would like to test your jobs before the tenant configuration updates go into effect you can do so by setting `ansible-version: '8'` in the configuration of a job [0], push the resulting change to Gerrit, and then wait for Zuul to run the job and provide you with results. It is not necessary to merge the change; Zuul will speculatively test this setting. If you discover any problems through this process please let us know. [0] https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.ansible-versi… Clark

1 0

opendevorg docker hub images moving to quay.io
by Clark Boylan 24 May '23

24 May '23

The OpenDev team has decided to move hosting for its container images from docker hub to quay.io. Over the weekend we landed changes to update the publication location for our python base images (opendevorg/python-builder, opendevorg/python-base, opendevorg/uwsgi-base). We will continue to move additional images as quickly as possible. While we publish these images primarily for our own consumption, they are entirely open source and others may be using them as well. If you are using these images, you should update the image source locations to quay.io/opendevorg/$imagename. Once we have updated publishing for our last image, we will begin a 30 day countdown, after which we will clean up images on docker hub to remove any confusion over which images are maintained and up to date. We will send notice for a more specific time frame once it is known. Let us know if you have an questions. Clark

1 1

Etherpad outage April 19, 2023 at 22:00 UTC
by Clark Boylan 17 Apr '23

17 Apr '23

The OpenDev team will be taking etherpad services at https://etherpad.opendev.org offline for approximately 90 minutes beginning at 22:00 UTC on Wednesday April 19, 2023. This is necessary for us to replace the server hosting etherpad and perform necessary data migrations. If you have any questions or concerns please reach out to us and let us know. Clark

1 0

review.opendev.org Downtime for Gerrit Upgrade and Project Renames April 6, 2023
by Clark Boylan 15 Mar '23

15 Mar '23

Hello, The OpenDev team intends to have a Gerrit outage of approximately two hours on April 6, 2023 beginning at 22:00 UTC. The purpose of this outage is to upgrade Gerrit from version 3.6 to 3.7. We will also take advantage of the downtime to perform project renames. For more information on the new version of Gerrit you can refer to the upstream release notes [0]. If you would like to rename a project please get it onto our TODO list on the wiki [1] and push changes according to the documented process [2]. We've chosen this time and date because we expect it to be a quiet period, and it occurs after known releases and events like the PTG. We are happy for you to raise and concerns, questions, comments or feedback with this plan; the earlier the better. [0] https://www.gerritcodereview.com/3.7.html [1] https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Upcoming_Project_… [2] https://docs.opendev.org/opendev/infra-manual/latest/creators.html#project-…

1 0

Intent to grant control of x/virtualpdu to OpenStack community
by Jeremy Stanley 02 Mar '23

02 Mar '23

The OpenStack Ironic project relies on VirtualPDU, which is no longer actively developed. Ironic contributors reached out to the VirtualPDU maintainers with an offer to officially assume maintenance responsibilities in order to avoid forking it, and received a response from one (Mathieu Mitchell) who indicated support for that plan. Unfortunately, communication died off shortly thereafter, and the Ironic team has been unable to raise any of the current maintainers to actually add their access to the x/virtualpdu project in OpenDev's Gerrit. We don't have a documented process since this is the first time it's really come up, but I'm officially announcing that I intend to use my administrative permissions as an OpenDev sysadmin to add membership of an OpenStack Ironic team representative to the following Gerrit groups if no objections are raised before Wednesday, March 8: * virtualpdu-core * virtualpdu-release This will effectively grant full control of the x/virtualpdu repository to OpenStack Ironic contributors. Please follow up to the service-discuss(a)lists.opendev.org or openstack-discuss(a)lists.openstack.org mailing list with any concerns as soon as possible, or you can feel free to reach out to me directly by email or in IRC (fungi in the #opendev channel on the OFTC network). -- Jeremy Stanley

1 0

December 2022 OpenDev Activity Update
by Clark Boylan 14 Dec '22

14 Dec '22

I had meant to send one of these every couple of months, but it has been about six months since I sent the last update email. Apologies for the delay. It has been a busy six months; let's dive in. We bootstrapped the ability to deploy new control plane nodes on Ubuntu Jammy. All newly created servers are built on Jammy unless we have a very good reason to use something older. Additionally, we have updated our base Python Docker images to include support for Python 3.11. Python 3.11 is only a few weeks old, but we are able to deploy software on top of it taking advantage of the performance benefits it provides. Gerrit has been upgraded to 3.6.3. This release is just behind 3.7.0 which was released about a month ago. I believe this is the closest we've been to the latest Gerrit release in about a decade. We do plan to upgrade to 3.7.x, but this upgrade appears to be a bit more involved than the last few. In particular, the upgrade itself needs offline re-indexing, we need to port our label configs to the new submit requirements system, and the web UI is now based on Lit instead of polymer. The Gitea cluster is now running 1.17. This update didn't bring any major user facing changes. The next update to 1.18.x should correct a problem with Gitea misidentifying certain files as being "vendored". Prior to the October PTG event our Meetpad deployment was updated to use the latest stable releases of Jitsi Meet, and we applied a modern configuration to our installation. One major feature this brought is a landing page for meetings where users can configure their name and settings. Importantly, this landing page should prevent browsers' auto mute behavior for auto playing video+audio from breaking calls for our users. We believe this was a major source of users not being able to hear others in calls. Mailing lists for lists.opendev.org and lists.zuul-ci.org have been migrated to a new Mailman 3 server. We are using these two list domains to work out issues while we prepare for migrating the rest of our mailing lists to the new server. This should give us a much more friendly web interface for browsing archives with built in search capability as well as easier to use moderation tools for list moderators. Our Zuul deployment has seen a number of improvements. The ZooKeeper cluster backing Zuul has been upgraded to ZooKeeper 3.7 which brings performance improvements. Zuul supports Ansible 6 now, and all of our tenants are running jobs with this version of Ansible. This version of Ansible fixes pipelining which should provide some improvements to job runtime. Finally, we are upgrading Zuul weekly using automated zero downtime rolling restarts. This ensures we're getting the latest and greatest Zuul features and can provide quick and early feedback to Zuul. Looking ahead to 2023, I expect to see a lot more of what we've done over the last six months. Gerrit will be upgraded to 3.7, Gitea will be upgraded to 1.18, more mailing lists will be migrated to Mailman 3, and more services will be deployed on Jammy running with Python 3.11. See you next year! Clark

1 0

review.opendev.org Gerrit 3.6 upgrade December 12th, 20:00UTC
by Ian Wienand 07 Dec '22

07 Dec '22

Hello, We are planning upgrade review.opendev.org to the Gerrit 3.6 release on 12th December, 2022 at around 20:00UTC. This will involve an outage of around 1 hour. There should not be much user-visible change after the upgrade; the major items do not affect our usage. The full release notes are at [1] Thank you, -i [1] https://www.gerritcodereview.com/3.6.html

1 0

Mailman v3 migration for OpenDev and Zuul mailing lists 2022-12-05
by Jeremy Stanley 06 Dec '22

06 Dec '22

On Monday, December 5 at 20:00 UTC, the OpenDev Collaboratory systems administrators will be migrating the lists.opendev.org and lists.zuul-ci.org mailing list sites from their existing Mailman 2.1.29 server to a shiny new Mailman 3.3.7 deployment. This maintenance window is expected to last no more than one hour. The key takeaways are as follows: * There will be a brief outage for the affected sites while DNS is updated and configuration, subscriber lists and message archives are imported, but incoming messages should correctly queue at the sender's end and arrive at the conclusion of our work * Because this is on a new server, there are new IP addresses from which list mail will be originating, 162.209.78.70 and 2001:4800:7813:516:be76:4eff:fe04:5423 * Moderation queues will not be copied to the new server, so moderators are encouraged to process any held messages prior to the migration time in order to avoid losing them * Anyone wishing to adjust their list subscriptions, or handle moderation or list configuration after the migration needs to use the Sign Up button on the Web page to create a new account; it will be linked to your imported roles as soon as you confirm by following a URL from an E-mail message the server sends you, and is global for all sites on the server so you only need to do this once * The software providing Web front-ends for list management and archive browsing is entirely new in Mailman v3 and therefore has a much different appearance, though we've added appropriate redirects and frozen copies of old archives in order to accommodate existing hyperlinks Once we smooth out any unanticipated rough patches with the new service, we'll schedule additional maintenance to migrate mailing list sites for other domains (Airship, Kata, OpenInfra, OpenStack, StarlingX). If you have any questions or concerns, feel free to follow up on the service-discuss mailing list or find us in the #opendev channel on the OFTC IRC network. -- Jeremy Stanley

1 1

Discussing Storyboard Needs on service-discuss
by Clark Boylan 07 Nov '22

07 Nov '22

I wanted to make sure that everyone subscribed to this list is aware that we are discussing the Storyboard service, its maintenance needs, and user needs on a thread on service-discuss(a)lists.opendev.org [0]. I have been told that not everyone is aware of the service-discuss list and may have missed that thread. Please follow up on that thread if you have any feedback for (not) wanting to use storyboard or desire to help maintain the service. This is a good opportunity to remind everyone that OpenDev runs three mailing lists for user interaction: 1. service-announce(a)lists.opendev.org [1] This mailing list. Intended to be kept low volume and limited to the most important email notices. 2. service-discuss(a)lists.opendev.org [2] Where we discuss the service needs, meeting agendas, and other topics with more volume. The bulk of our mailing list discussion will be here. All are welcome to join and participate. -discuss and -announce are separate to give people a lower volume option with -announce. 3. service-incident(a)lists.opendev.org This is a private mailing list that OpenDev admins and other trusted individuals may be subscribed to. Users can email this list with sensitive messages like vulnerability disclosures. [0] https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html [1] https://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce [2] https://lists.opendev.org/cgi-bin/mailman/listinfo/service-discuss

1 0

Changes to our base job nodeset and Ansible 5 removal
by Clark Boylan 12 Oct '22

12 Oct '22

As mentioned in the previous email [0] about updating our default Ansible version to Ansible 6, Zuul will be removing support for Ansible 5 soon. Based on Zuul's currently estimated release schedule we expect this to happen on or near October 18. This change won't take effect until OpenDev restarts Zuul. We will try to make that happen quickly after Zuul lands this change in order to catch any issues quickly. Additionally, it has been called out that we are still defaulting our base job nodeset to ubuntu-focal. Ubuntu Jammy has been released for about 6 months now and has a .1 release as well. To catch up we will update the default nodeset in our base jobs to ubuntu-jammy on October 25, 2022. If you would like to test that your jobs run against Ubuntu Jammy you can modify your jobs to set `nodeset: ubuntu-jammy` today and test them. Note, any jobs that expect a specific platform should already be explicitly setting the nodeset they rely on. There are a number of jobs already running on Jammy which gives us confidence in the newer images. [0] https://lists.opendev.org/pipermail/service-announce/2022-October/000046.ht… Clark

1 0

Updating Zuul tenants' default-ansible-version to 6
by Clark Boylan 05 Oct '22

05 Oct '22

Hello, The OpenDev team is planning to update the default-ansible-version to 6 in the near future. Zuul is planning to drop support for Ansible 5, and switching OpenDev to Ansible 6 early allows us to provide valuable feedback if there are problems. There are other reasons for the switch as well. Ansible 5 broke pipelining, but we expect Ansible 6 to correct this problem. Having pipelining should improve the runtime of jobs. We had been holding off on making this change as StarlingX and OpenStack have both recently worked through some fairly major releases. Those releases exist now making this a good time for a change like this. As far as testing goes, Zuul's Zuul tenant has been running with an Ansible 6 default for several weeks now. I've also tested this change with Devstack specifically [0] as Devstack seems to have a good representative set of complicated jobs. The change to update our defaults is up for review [1]. Feel free to leave any comments there or on this email thread. Note that you can override this change if necessary using the same mechanism I used to test Devstack [0]. However, if you do this please communicate that change to us as Zuul does plan to remove Ansible 5 support which means any problems with Ansible 6 need to be communicated back to Zuul. [0] https://review.opendev.org/c/openstack/devstack/+/858436 [1] https://review.opendev.org/c/openstack/project-config/+/860474

1 0

Dropping support for Ansible 2.x in our Zuul
by Jeremy Stanley 21 Sep '22

21 Sep '22

Following the earlier 2.8/2.9 deprecation in June and change of our default to Ansible 5 in July, we will soon be restarting onto a version of Zuul which no longer supports 2.x versions at all. While we've looked for any lingering overrides to old Ansible versions in master branch jobs and given projects a heads up individually, we lack insight into whether the same may be occurring on non-default branches of projects. If you are aware of any jobs on any of your branches which pin back to an earlier Ansible version, we strongly recommend you remove those overrides at your earliest opportunity, otherwise those jobs will break in the coming days. Here is the change to remove support for Ansible 2.8 and 2.9 in Zuul (we generally restart our Zuul services every weekend, so that's when it's most likely to take effect): https://review.opendev.org/857796 For some example changes which clean up old Ansible version overrides, see this review topic list: https://review.opendev.org/q/topic:cleanup-old-ansible -- Jeremy Stanley

1 0

Move Zuul queue declarations to the project level
by Jeremy Stanley 19 Sep '22

19 Sep '22

This is just a reminder to users of our Zuul service that if you find changes aren't getting any jobs run, or seem to not be running some jobs you expect, you should double-check your configuration to make sure you don't have a lingering "queue" setting in your project pipelines. Over the weekend, we updated to a version of Zuul which removed the long-deprecated support for pipeline level queue declarations, and it now expects queues to be set at the project level instead. We directly notified some projects which we knew would be impacted, but did not send a broader announcement to this list, so I just wanted to be sure anyone taken by surprise with this change doesn't have to waste additional time troubleshooting it. Earlier today, the OpenDev sysadmins corrected numerous cases of this error in the central openstack/project-config repository used by the openstack tenant, but there are still many projects which have similar errors in their in-repository configuration as well. For the reminder from the Zuul maintainers, see here: https://lists.zuul-ci.org/pipermail/zuul-discuss/2022-May/001801.html Also these posts to the OpenStack discussion lists may include helpful information about the transition: https://lists.openstack.org/pipermail/openstack-discuss/2022-May/028603.html https://lists.openstack.org/pipermail/openstack-discuss/2022-September/0305… -- Jeremy Stanley

1 0

Updating Zuul's Default Ansible Version to Ansible v5
by Clark Boylan 11 Jul '22

11 Jul '22

Hello, The OpenDev team will be updating the default Ansible version in our Zuul tenants from Ansible 2.9 to Ansible 5 on June 30, 2022. Zuul itself will eventually update its default, but making the change in our tenant configs allows us to control exactly when this happens. There are some known issues with this transition; however, we've already worked through and addressed a number of them. If you'd like to double check your jobs work with Ansible 5 today you can do that by pushing a change(s) that updates job.ansible-version [0] to the value '5'. If you do this and find problems please let us know. The Zuul zuul tenant is already running with this set tenant wide which means we expect most of the base job pre-run and post-run content to work with Ansible 5. We also did some spot checking of things like devstack and tempest and expect most of those jobs to function too. Finally, we can eventually expect Zuul to drop support for Ansible 2.8 and 2.9 in the future. What this means is we shouldn't just pin to those old versions without a plan forward to addressing any remaining v5 incompatibilities. [0] https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.ansible-versi… If you have any questions let us know. Clark

2 1

A Quick OpenDev Activity Update
by Clark Boylan 23 Jun '22

23 Jun '22

It has been a couple months since I sent out our last activity update, and now seems like a good time to share what is new. Ubuntu Jammy test nodes are now available in both x86_64 and ARM64 flavors. This version of Ubuntu ships with python3.10 and is ready for you to start testing your python projects on this newer runtime. Our Gerrit server running at https://review.opendev.org has been upgraded from Gerrit 3.4 to 3.5. This newer version of Gerrit brings features like new change search predicates, updates to the UI, and performance updates. On the whole this is a bit more of an incremental update compared to previous upgrades. We expect many users haven't even noticed that the upgrade happened. If you'd like to learn more you can read the Gerrit 3.5 release notes: https://www.gerritcodereview.com/3.5.html. Glean 1.23.0 has been released and finally adds proper support for static ipv6 configuration on RPM based distros. Our Ethercalc server previously running at https://ethercalc.openstack.org has been shutdown and removed. On the Zuul side of things there are two changes you should be aware of and planning for. The first is the planned removal of pipeline specific queues in your Zuul config. This configuration was deprecated a number of releases ago and is slated for removal in the next major Zuul release. The OpenStack tenant is the only Zuul tenant that has this old style of configuration. More information on which projects are affected and how to correct it can be found in my email about this topic: https://lists.openstack.org/pipermail/openstack-discuss/2022-May/028603.html. Note that projects with this configuration in place that do not update will have broken Zuul configurations after we upgrade to the next major Zuul release. Next, is the plan for changing the default Zuul Ansible version from Ansible 2.9 to Ansible 5 on June 30, 2022. We've already done some testing of this and the base jobs as well as devstack and tempest seem to work with newer Ansible. If you'd like to learn more (including how you can test your specific projects early) please check my previous email on this subject: https://lists.opendev.org/pipermail/service-announce/2022-June/000041.html. Looking ahead we should expect our Gitea farm to be upgraded to v1.17. Work will begin on upgrading Gerrit to version 3.6, but it is still too early to know when we will perform that upgrade. There are a few larger tasks we'd like to get moving on including updating our Keycloak server to provide SSO capabilities for our services, upgrading our list servers to Mailman 3, modernizing our StoryBoard deployment, and deploying a Prometheus service to collect server and service metrics. If any of these sound interesting to you, and you would like to help, please reach out. Clark

1 0

CentOS/Fedora IPv6 on gate nodes
by Ian Wienand 02 Jun '22

02 Jun '22

Hello, Consistent IPv6 setup will soon be coming to CentOS and Fedora nodes pending a release of glean [1], the tool we use in the OpenDev gate to configure networking on test nodes. After recent work we believe we now handle the heterogeneous IPv6 setup methods in all our donated clouds consistently. Debian-based distros have had better support for this, so the IPv6 path is generally well tested. However, if this reveals any issues, please reach out (IRC #opendev or service-discuss(a)lists.opendev.org) and we can look into it. Thank you, -i [1] https://opendev.org/opendev/glean/

1 0

review.opendev.org Gerrit 3.5 upgrade June 19th, 20:00UTC
by Ian Wienand 18 May '22

18 May '22

Hello, We are planning upgrade review.opendev.org to the Gerrit 3.5 release on 19th June, 2022 at around 20:00UTC. This will involve an outage of around 1 hour. There should not be much user-visible change after the upgrade. There are some enhanced search predicates [1] which you may find useful. Thank you, -i [1] https://www.gerritcodereview.com/3.5.html#new-change-predicates

1 0

Retiring and shutting down the ethercalc.openstack.org service
by Clark Boylan 04 May '22

04 May '22

Hello, The https://ethercalc.openstack.org service has become problematic for several reasons: * Our configuration management for the service is still based on Puppet and needs to be modernized to continue running the service. * The service itself isn't well maintained. Its build system is apparently broken [0]. * The service does not capture a history of changes like etherpad, confusing users. * The service crashes semi frequently. The major known user of this service was the PTG planning process. We've talked to the folks that help organize and plan the PTG, and they think other tools (like PTGBot) can be used instead without major trouble. Considering all of this, the OpenDev team would like to retire and shutdown this service. The current plan is to keep the service up and running until the end of the month. Then on May 31, 2022 shut down the server hosting the service, snapshot this server, and finally delete it. Please back up any data you would like to preserve before May 31, 2022. [0] https://github.com/audreyt/ethercalc/pull/781 If you have any questions or concerns feel free to respond to this thread or reach out to the OpenDev team. Clark

1 0

A not so Short OpenDev Activity Update
by Clark Boylan 23 Apr '22

23 Apr '22

It has been a busy few weeks for the OpenDev team. We thought it would be a good idea to capture some of the more user facing aspects of what we have been doing lately. Git-review and Bindep both have new releases. Git-review 2.3.0 and 2.3.1 [0] bring better support for newer OpenSSH and signed Git commits, as well as drop compatibility with Python 3.5. Bindep 2.11.0 [1] adds support for AlmaLinux. Git fixed a security issue [2] that broke DevStack and other test jobs. The OpenDev team helped to quickly diagnose the problem and proposed fixes to projects once the issue was understood. If you have jobs with unexpected git operation failures or Python package install issues there is a good chance it is related to this update. You will want to ensure that the user running git commands in a repo is the same user that owns the files in the repo (typically "zuul"). Otherwise you will need to mark the repository as a "safe" repository. Our Twitter status account [3] has been resurrected to provide another way for our users to follow along as we make changes. Whenever we make `#status` messages in IRC that content gets mirrored to this account which you can follow on Twitter. We have also been doing quite a bit of Spring Cleaning (it is spring for some of us anyway). We no longer build Debian Buster or Python 3.7 versions of opendevorg/python-builder and opendevorg/python-base. We did our best to check that no one was using these images, but if you are, the images will remain on Docker Hub until Docker Hub times them out. You should update to a Bullseye Python 3.8 or newer image. Bullseye Python 3.10 images are newly available too. The OpenStack Health service was shut down some time ago, and its configuration management and source repositories have been retired or cleaned up as appropriate. The ELK system that powered elastic-recheck is in the process of being shutdown and retired in favor of a new OpenSearch hosted system managed by OpenStack more directly. Please read the documentation [4] for using the new system if you would like to continue to query job logs using Kibana. Our distro mirrors are getting a haircut to remove packages for unused architectures in our CI system, ISOs/images, and old distro releases. This cleanup will ensure we have plenty of room to add a Jammy Jellyfish mirror to our collection. We'd like to remind our users that while these mirrors are publicly available they are not intended for public consumption. We can and do modify them as our CI system needs change. Finally, looking ahead we'll begin to work on rolling out Jammy Jellyfish images for the CI system now that this latest Ubuntu LTS is available. Our Gerrit 3.5 upgrade should also pick up steam. Most of the changes we expect Gerrit users to see will be in the UI itself. You can read the release notes [5] to find out more. We aren't ready to set a date for this, but you should expect a short downtime which we will announce via service-announce(a)lists.opendev.org. Unfortunately, the Gerrit problems with SSH RSA keys won't be resolved until Gerrit 3.6. [0] https://lists.opendev.org/pipermail/service-announce/2022-April/000036.html [1] https://lists.opendev.org/pipermail/service-announce/2022-April/000035.html [2] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-24765 [3] https://twitter.com/opendevinfra/ [4] https://governance.openstack.org/sigs/tact-sig.html#opensearch [5] https://www.gerritcodereview.com/3.5.html

1 0

git-review 2.3.0 and 2.3.1
by Jeremy Stanley 19 Apr '22

19 Apr '22

It's my pleasure to announce the release of git-review 2.3.0 and follow-up fix version 2.3.1. These are the corresponding release notes since the previous release: ======= 2.3.1 ======= Bug Fixes --------- * Improved version string parsing for Git in order to no longer break on Mac OSX/Darwin systems (story 2010002). ======= 2.3.0 ======= Upgrade Notes ------------- * Python 3.5 is no longer supported. * Git 2.10.0 or later is required as git-review now needs the ``--no-show-signature`` option for git-log. Bug Fixes --------- * Git's log.ShowSignature option is now supported. Previously, setting it would confuse git-review into thinking there were twice as many changes to push. * OpenSSH has deprecated its use of scp/rcp protocol in favor of SFTP, which the embedded Apache mina-sshd in widely-deployed Gerrit versions does not yet support. The default officially changed in OpenSSH 9.0 (some distributions, such as Fedora and CentOS, switched their default behavior to this as early as OpenSSH 8.7 or 8.8), leading to a "subsystem request failed on channel 0" error during commit-msg hook retrieval. Now git-review will attempt to detect whether scp's -O option is available to force use of the legacy scp/rcp protocol, and apply it if so. Future git-review versions may change the fallback behavior once an SFTP subsystem is implemented and available for a new Gerrit release. As always, the latest version of git-review is available from PyPI: https://pypi.org/project/git-review Current documentation can be found here: https://docs.opendev.org/opendev/git-review Its source code can be obtained from the OpenDev Collaboratory: https://opendev.org/opendev/git-review Please don't hesitate to follow up through the service-discuss(a)lists.opendev.org mailing list or in the #opendev channel on the OFTC IRC network if you have any questions. -- Jeremy Stanley

1 0

bindep 2.11.0
by Jeremy Stanley 19 Apr '22

19 Apr '22

It's my pleasure to announce the release of bindep 2.11.0. These are the corresponding release notes since the previous release: New Features ------------ * Add support for AlmaLinux to Bindep. As always, the latest version of bindep is available from PyPI: https://pypi.org/project/bindep Current documentation can be found here: https://docs.opendev.org/opendev/bindep Its source code can be obtained from the OpenDev Collaboratory: https://opendev.org/opendev/bindep Please don't hesitate to follow up through the service-discuss(a)lists.opendev.org mailing list or in the #opendev channel on the OFTC IRC network if you have any questions. -- Jeremy Stanley

1 0

review.opendev.org Gerrit downtime for project renames April 15, 2022
by Clark Boylan 07 Apr '22

07 Apr '22

Hello, The OpenDev team will be renaming Gerrit projects on April 15, 2022 beginning at 15:00 UTC. This process requires a Gerrit downtime for the service running on review.opendev.org and is expected to last no more than an hour. If you would like to rename a project, now is the time to begin planning for that work ensuring the necessary changes are proposed. If you have any questions please feel free to respond to this email or reach out to the team. Clark

1 0

Zuul behavior change with Depends-On across queues
by Jeremy Stanley 05 Apr '22

05 Apr '22

A recent improvement to more accurately report Zuul's enqueuing has resulted in an unanticipated difference in behavior for changes declaring a Depends-On relationship between projects which don't share a dependent queue. If you approve the depending change before the dependency has merged, Zuul will now report a Verified -2 vote on the depending change (where previously it just ignored your approval entirely). As a result, because the openstack tenant is configured to require a positive Verified vote before enqueuing into the gate pipeline, this means the depending change will need to be rechecked in order to replace that -2 vote with a +1 in that tenant. For tenants without such a "clean check" rule, the change in behavior is entirely cosmetic. One thing you can do to avoid this is make sure any projects which regularly utilize Depends-On between their changes are configured to belong to the same named queue, which will result in Zuul enqueuing those approved changes into the gate pipeline together without needing to wait for one to merge before the other can be tested. Zuul expects related projects to share a queue and, on approval, will only look up parent and child changes to enqueue within the related projects list, as determined by the queue to which they belong. An alternative workaround is to only approve a change once you've confirmed that any Depends-On has merged, if that dependency is in another project which does not share the same queue. Feel free to follow up on the service-discuss ML with any comments/concerns, or find us in the #opendev channel on OFTC as usual. -- Jeremy Stanley

1 0

Temporary addition of unsigned tag permissions
by Jeremy Stanley 02 Feb '22

02 Feb '22

In the wake of yesterday's Gerrit 3.4 upgrade maintenance, we've observed a regression in behavior where signed Git tags are being misidentified as unsigned[*]. Since our ACLs normally disallow pushing of unsigned (annotated) tags, the result is tag pushes being rejected with a permissions error. As an emergency measure, and in an attempt to not roll back our Gerrit upgrade while working with their developers on a proper patch so solve this, we have updated the ACLs for all projects with a workaround so that they will be able to continue pushing tags. Unfortunately, this workaround means that projects will temporarily be able to push unsigned tags as well as signed ones, so we ask anyone manually tagging their projects be extra careful to remember to pass the -s option to git's tag subcommand. Once we have a proper fix in place, the temporary addition of unsigned tag permissions will be rolled back, and a follow-up announcement sent to this mailing list. [*] https://bugs.chromium.org/p/gerrit/issues/detail?id=15616 -- Jeremy Stanley

1 1

Gerrit Maintenance and Downtime January 24, 2022
by Clark Boylan 19 Jan '22

19 Jan '22

Hello, The OpenDev team will be taking the early year quiet as an opportunity to rename a few projects in Gerrit and upgrade Gerrit to 3.4 starting January 24, 2022 at 22:00 UTC. We expect to be done by 00:00 UTC January 25, 2022. If you would like to rename a project now is the time to get that preparation done. Please link any rename proposals in our list here: https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Upcoming_Project_… The Gerrit 3.4 release notes can be found at https://www.gerritcodereview.com/3.4.html. This upgrade does not require offline reindexing which means it should happen quickly. We are also automatically testing the 3.3 to 3.4 upgrade in our CI system. Additionally there is a downgrade process that we can use to fall back to 3.3 if necessary. Let us know if you have any questions or concerns, Clark

1 0

Changes to OpenDev's stable of CI images
by Clark Boylan 07 Dec '21

07 Dec '21

Hello, CentOS 8 reaches end of life on December 31, 2021. This means now is the time for you to convert jobs to CentOS 8|9 Stream or some other appropriate platform. The OpenDev team will plan to remove CentOS 8 images and mirrors from our infrastructure in early January, hopefully by January 14, 2022. Again, please update your jobs now so that you are prepared. These changes are necessary as the distro will no longer be supported, and there is no way for us to ensure that the images will be secure or that we'll even be able to build them. We've also got Fedora 35 lined up to replace Fedora 34. Due to Fedora's short support window we typically swap out one Fedora release with the next when the next is ready. If you are using the fedora-34 label please update it to fedora-35. If you are using fedora-latest we'll be updating that to use fedora-35 and the swap will happen automatically for you. We expect to have Fedora 34 cleaned up by the end of the year. Additionally both OpenSUSE 15 and Tumbleweed could use some help. For OpenSUSE 15 we need to update it to 15.3 as well as updating the associated mirrors if we plan to keep it around. I'm happy to keep this image around if we can get volunteers in the community that can help with that. If there isn't interest in modernizing the OpenSUSE 15 images I'd like to propose we clean them up in January along with the CentOS 8 images. For Tumbleweed, the images haven't received much use or attention. I'd like to propose that we simply retire this image and clean it up. We had thought it might be a good platform for monitoring upcoming changes in the Linux world, but the reality is that only works if you've got people active caring for it and adjusting to those changes. The Tumbleweed cleanup will likely begin before the end of the year. Finally, our Gentoo images bounce between being buildable and broken with more time spent being broken than not. I'd like to propose that this image also get cleaned up in the near future. Again, if there is interest in the community for keeping this alive I think we can make that happen. But as is this image is not in a sustainable place and it would be better for us to stop running jobs on it. Let us know if you have any questions or comments, Clark

1 0