- service-announce - lists.opendev.org

review.opendev.org Gerrit Outage April 21, 2025
by Clark Boylan 11 Apr '25

11 Apr '25

The OpenDev team will shut down Gerrit services on review.opendev.org at 16:00 UTC April 21, 2025 in order to migrate and upgrade the underlying server instance. The work is expected to take about an hour and should largely be transparent to users other than the required outage. One thing users may notice is the IP Addresses for review.opendev.org will change. If you need to update your firewall rules you can do so by allowing TCP port 443 and TCP port 29418 to: 162.253.55.233 2604:e100:1:0:f816:3eff:fe31:926a These IPs belong to the new review03.opendev.org server which will replace review02.opendev.org. Let us know if you have any questions or concerns. Clark

1 0

bindep 2.12.0
by Jeremy Stanley 24 Jan '25

24 Jan '25

I've released bindep 2.12.0 just now. These are the highlights: New Features ------------ * Basic support for PopOS is now included. Upgrade Notes ------------- * Python 2.7 and 3.5 are no longer supported. * Bindep now depends on the distro python library to determine details about the current platform. This library looks at both /etc/os-release and lsb_release to find platform info. The os-release file data is preferred and at times has slightly different data than lsb_release. Every effort has been made to make this transition backward compatible but some things may have been missed. The motivation for this change is that not all distros have lsb_release available and we can let the distro library sort that out for us. -- Jeremy Stanley

1 0

Planned outage for mailing lists on Monday 2024-12-02 15:00-17:00 UTC
by Jeremy Stanley 27 Nov '24

27 Nov '24

The OpenDev Sysadmins will be performing maintenance on our mailing list server between 15:00 and 17:00 UTC on Monday, 2024-12-02. During this time, messages to and from the lists for all our sites will be held temporarily while we make adjustments in the backing database for renaming one of the domains in use. Web archives may be unavailable for part of the time as well. The outage should start some time around 15:00 UTC and is not expected to last more than two hours (but will hopefully conclude far more quickly). Please follow up (either to the service-discuss(a)lists.opendev.org mailing list or to me personally) with any questions you may have on this activity. -- Jeremy Stanley, OpenDev Systems Administrator

1 0

Upgrading OpenDev's Gerrit Server on December 6, 2024 at 16:00 UTC
by Clark Boylan 13 Nov '24

13 Nov '24

The OpenDev team will be upgrading the Gerrit server on review.opendev.org on December 6, 2024 at 16:00 UTC. This will require a short downtime of the service if everything goes well and a longer downtime if we need to revert. We are allocating an hour (16:00 - 17:00 UTC) for this work. We will upgrade from Gerrit 3.9 to 3.10 remaining approximately one release behind the latest Gerrit release as long as the Gerrit 3.11 release happens on time this month. Gerrit 3.10 brings a number of changes including many performance improvements and fewer email notifications. The upstream release notes [0] cover these updates in much more detail if you are interested. Let us know if you have any questions or concerns. [0] https://www.gerritcodereview.com/3.10.html Clark

1 0

Pruning OpenDev's Intermediate Registry on insecure-ci-registry.opendev.org November 15, 2024
by Clark Boylan 06 Nov '24

06 Nov '24

The OpenDev team will begin some long overdue pruning of the storage system behind the intermediate container registry running on insecure-ci-registry.opendev.org on November 15, 2024. The last time we tried this we believe container image client errors were due to the pruning. Unfortunately, we don't have enough information to know if this will happen again. For this reason we will be prepared to replace the entire Swift container on the backend and start over fresh if necessary. We ask that you report any unexpected client behaviors with the intermediate registry on or after November 15, 2024. Additionally, if we do end up replacing the entire backend storage container we will lose all intermediate container image state. If this happens, jobs relying on state in the intermediate registry may fail. In those cases you may need to recheck jobs that build the images or merge additional changes to refresh the content in the intermediate registry. Clark

1 0

Switching OpenDev's Default Zuul Nodeset to ubuntu-noble
by Clark Boylan 15 Aug '24

15 Aug '24

The OpenDev team plans to switch the default nodeset in our Zuul base jobs from ubuntu-jammy to ubuntu-noble on August 21, 2024. The new release has been available for several months now and we have jobs relying on it. One of the last steps in the Noble rollout is to switch the default over. This change will impact any jobs that don't explicitly override the nodeset as they will start running on the newer Ubuntu release after this change. If this creates problems for your jobs you can either override the nodeset to ubuntu-jammy forcing the job to run on the nodes used before this change, or you can update your job to accommodate whatever changes the newer version of Ubuntu requires. If you have any questions or concerns feel free to respond to this email or reach out via IRC. For OpenStack in particular, we've been told that they expect most jobs to already override the nodeset and this change will largely be a noop for them. Clark

1 0

Removal of Inactive Projects from Zuul's Tenant Configuration
by Clark Boylan 10 Jul '24

10 Jul '24

The OpenDev team is going to start removing inactive x/* namespace projects from Zuul's tenant configuration. The two primary motivations behind this are to remove projects that already have broken configurations within Zuul (see the Zuul config errors page [0]), and to avoid projects entering this state in the future as we clean up test nodes like Ubuntu Xenial or CentOS 8 Stream or otherwise modernize the CI system. Removal from Zuul's tenant configuration is a lightweight method of ensuring we're addressing problems with active projects that can be fixed without the added noise from projects that are unlikely to fix things on their own. A first pass at this sort of cleanup is being done with this change [1]. Feel free to chime in if you know there is already work underway to correct x/* projects' config errors, or if you know of any additional projects that should be removed from Zuul. We aim to merge this change on July 16, 2024. It should be noted that if projects do become active again and have volunteers willing to correct the CI configuration state, it is straightforward to add the project back into Zuul enabling that effort to progress. Projects will simply add themselves back to the tenant configuration and then push changes to fix the CI config. [0] https://zuul.opendev.org/t/openstack/config-errors [1] https://review.opendev.org/c/openstack/project-config/+/923509 Clark

1 0

CentOS 8 Stream End of Life and Removal from OpenDev CI
by Clark Boylan 20 Jun '24

20 Jun '24

Hello everyone, I wanted to give you a heads up that CentOS 8 Stream has reached its end of life and the upstream package repos have been removed. Our mirrors have faithfully replicated the empty mirror content (there is only a readme file now [0]). The end result of which is failing CI jobs if the jobs run on CentOS 8 Stream. The OpenDev team intends on removing these nodes from Zuul and Nodepool in the near future as jobs can no longer successfully run on CentOS 8 Stream. Projects should check their CI jobs (on older branches too) and either remove jobs that rely on CentOS 8 Stream or update those jobs to run on a different platform. [0] https://mirror.dfw.rax.opendev.org/centos/8-stream/readme Clark

1 0

Ubuntu Noble Test Nodes Now Available in Zuul
by Clark Boylan 24 May '24

24 May '24

The OpenDev team has added Ubuntu Noble test nodes to Zuul and Nodepool. This means you can start running jobs on the latest Ubuntu release using `nodeset: ubuntu-noble` or `nodeset: ubuntu-noble-arm64`. If you build your own custom nodesets the labels you will want to use are `ubuntu-noble` and `ubuntu-noble-arm64`. We expect these to generally work and have already successfully run a number of builds on the new nodes. That said they are new, and you may need updates to bindep files and/or Ansible roles and playbooks. Let us know if you have any questions or find problems. Clark

1 0

Gerrit Downtime on May 31, 2024 at 16:00 UTC
by Clark Boylan 22 May '24

22 May '24

Hello, The OpenDev team is planning to upgrade Gerrit on review.opendev.org to version 3.9 on May 31, 2024 beginning at 16:00 UTC. In order to perform this upgrade we require a short service outage. We are allocating an hour for this outage, from 16:00 to 17:00 UTC, as that should give us plenty of time to implement the rollback procedure if that becomes necessary. The Gerrit 3.9 release notes can be found here: https://www.gerritcodereview.com/3.9.html. One major feature our users may be interested in is the ability to suggest and apply updates to files via review comments. If you have any questions about the upgrade or the new version of Gerrit, feel free to reach out to the OpenDev team via this email thread or on IRC in #opendev on OFTC. Clark

1 0

Gerrit Outage for Project Renames Monday April 22, 2024 at 20:00 UTC
by Clark Boylan 17 Apr '24

17 Apr '24

The OpenDev team is planning a scheduled Gerrit service outage on review.opendev.org in order to rename projects. This outage will occur on Monday April 22, 2024 at 20:00 UTC until 21:00 UTC. If you have a project you would like to rename please get in touch with the OpenDev team soon (you can respond to this email or find us in #opendev on the OFTC IRC network). Please reach out if you have any questions or concerns. The same methods described above work for that too. Clark

1 0

CVE-2024-31497 and SSH key statistics
by Jeremy Stanley 16 Apr '24

16 Apr '24

For those of you who haven't seen, CVE-2024-31497 in the PuTTY tool was announced in the past day. If you're using a NIST P521 ECDSA key and have ever used it with affected versions of PuTTY, you should remove its access everywhere and consider it potentially disclosed. Details here: https://www.chiark.greenend.org.uk/~sgtatham/putty/wishlist/vuln-p521-bias.… The key type is generally listed as the first field in the corresponding public key file, or in each line of an authorized_keys entry, including those you may have set for your account in our Gerrit server (review.opendev.org) If the type is listed as ecdsa-sha2-nistp521 or sk-ecdsa-sha2-nistp521(a)openssh.com then it's been put at risk if ever used with PuTTY versions 0.68 through 0.80. On a related note, since I wanted to see how popular that key type is among our Gerrit users, I ran some quick statistics. Of the 36976 accounts created there to date, 19888 of them have SSH keys authorized. There are 35057 total authorized keys, which means accounts with SSH keys average 1.76 keys per account. Out of those keys, 44 claim to be NIST P521 type (0.13%) used by 41 different accounts (0.20% of accounts with keys, 0.11% of all accounts). Here's a breakdown of the most popular types for keys authorized in our Gerrit: 95% ssh-rsa 3.2% ssh-ed25519 1.4% ssh-dss 0.38% ecdsa-sha2-nistp256 0.25% ssh-dsa 0.13% ecdsa-sha2-nistp521 0.023% ecdsa-sha2-nistp384 So as you can see, the key type potentially impacted by CVE-2024-31497 isn't all that popular in our circles, but it's definitely nonzero. -- Jeremy Stanley, OpenDev SysAdmin

1 0

git-review 2.4.0
by Jeremy Stanley 18 Mar '24

18 Mar '24

It's my pleasure to announce the release of git-review 2.4.0. These are the corresponding release notes since the previous release: New Features ------------ * By default, git rebase --abort is run automatically following a failed test rebase, letting the user rebase again on their own if that's the state they want to be in. A -K/--keep-rebase option is added to get the old behavior. * For convenience, --wip is now synonymous with the longer --work-in-progress option. * If HEAD already exists on a remote branch, git review will now only warn and prompt for confirmation when pushing, in order to let the Gerrit server decide whether to accept or reject it. * A copy of the Gerrit upstream project's commit-msg hook is now included directly and written to disk by default rather than pulling a remote copy. This approach is safer and more robust, but may break workflows for sites which rely on users retrieving a customized commit-msg hook. For this reason, a command-line override is provided so that users can, for example, execute git review -s --remote-hook to get the old behavior when setting up a new clone. * The GIT_SSH environment variable will now be consulted, if set, in order to determine which ssh executable to call outside of git invocations, for consistency. * A new --hashtags option allows specifying a list of one or more Gerrit hashtag strings, separated by commas. Note that spaces are not allowed either within a hashtag or between separators. * New --message option can be used to include a custom patchset message when pushing. As always, the latest version of git-review is available from PyPI: https://pypi.org/project/git-review Current documentation can be found here: https://docs.opendev.org/opendev/git-review Its source code can be obtained from the OpenDev Collaboratory: https://opendev.org/opendev/git-review Please don't hesitate to follow up through the service-discuss(a)lists.opendev.org mailing list or in the #opendev channel on the OFTC IRC network if you have any questions. -- Jeremy Stanley

1 0

Removal of Ansible 6 support from OpenDev's Zuul Service
by Clark Boylan 14 Feb '24

14 Feb '24

The Zuul project will be removing Ansible 6 support in preparation for the upcoming Zuul 10.0.0 release. The exact timing of this change isn't known yet, but it may happen as early as this week. OpenDev has automated Zuul upgrades that should pull this change into our Zuul deployment not long after upstream merges the change. We have defaulted to Ansible 8 for many months now, and we don't expect this to create many issues. If you do see issues you will need to update to Ansible 8. Zuul will also be adding support for Ansible 9 in preparation for the upcoming Zuul 10.0.0 release. Once these changes are deployed in OpenDev you may want to start testing your jobs with Ansible 9 to ensure there are no problems with that update. At some point further in the future we will be changing the default Ansible version for OpenDev's Zuul to Ansible 9. Let us know if you have any questions or concerns, Clark

1 0

Upgrading mailing lists on 2023-11-02
by Jeremy Stanley 31 Oct '23

31 Oct '23

On Thursday, November 2, 2023 beginning at 15:30 UTC the OpenDev Collaboratory is upgrading the version of Mailman used to host the Airship, Kata, OpenDev, OpenInfra, OpenStack, StarlingX and Zuul mailing list sites from 3.3.8 to 3.3.9 along with related updates to its various component services (django-mailman3 1.3.9->1.3.11, hyperkitty 1.3.7->1.3.8, postorius 1.3.8->1.3.10, mailman-web 0.0.6->0.0.8). There will be brief downtime for the web interface while services are restarted, and some message deliveries may be briefly delayed but will still eventually be processed. One of the updates includes a database migration that implicitly clears the auth sessions table, so users who were logged into the web interface will be effectively logged out and need to log in again before performing authenticated operations. Web interfaces may have a slightly different look and feel after the update, and some new options too. Changelogs can be found here for anyone interested in seeing a list of what's new: https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/NEWS.… https://gitlab.com/mailman/django-mailman3/-/blob/v1.3.11/README.rst https://gitlab.com/mailman/hyperkitty/-/blob/v1.3.8/doc/news.rst https://gitlab.com/mailman/postorius/-/blob/v1.3.10/src/postorius/doc/news.… https://gitlab.com/mailman/mailman-web/-/blob/0.0.8/README.rst -- Jeremy Stanley

1 0

Upgrading review.opendev.org to Gerrit 3.8 on November 17, 2023
by Clark Boylan 30 Oct '23

30 Oct '23

On November 17, 2023 beginning at 15:30 UTC the OpenDev team will upgrade Gerrit at https://review.opendev.org from Gerrit 3.7 to 3.8. This upgrade shouldn't take more than an hour. We have managed to to keep our Gerrit installation about one major release behind upstream's release cadence, and Gerrit 3.9's impending release means it is time for us to upgrade to 3.8. There are a number of new features, bug fixes, and performance improvements in the new version. The Gerrit project publishes detailed release information [0] which you can review if you are interested in what sorts of changes to expect. If you are interested in the upgrade process you can follow that in our planning document [1]. It is worth noting that if all goes well we expect the total Gerrit downtime to be much shorter than an hour (should only take a few minutes). If we need to revert to Gerrit 3.7, we will need to do offline reindexing of our changes which takes about half an hour. We are using an hour long upgrade window to ensure we have time for downgrades should that become necessary. [0] https://www.gerritcodereview.com/3.8.html [1] https://etherpad.opendev.org/p/gerrit-upgrade-3.8 Clark

1 0

Zuul regex deprecation [action required]
by James E. Blair 21 Sep '23

21 Sep '23

Hi, Zuul is making a change that may require action from developers on projects hosted in OpenDev. Here is the release note regarding the change: > Regular expressions using Python syntax are deprecated in favor of RE2 > syntax instead, for additional speed and safety. Negative lookahead > assertions may be replaced using the negate keyword. See Regular > Expressions[1] for more information. > > If Zuul detects a regular expression using Python syntax that is not > supported by RE2, it will register a configuration syntax > warning. These may be viewed on the configuration errors page in the > web interface. > > A future version of Zuul will remove Python regex support completely > and these warnings will become errors. [1] https://zuul-ci.org/docs/zuul/latest/project-config.html#regex It's easy to see if your project on OpenDev is affected. Each tenant has a page where the current configuration errors and warnings are displayed. Visit any page in Zuul for your tenant, and if there is a bell icon in the top right, click on that. For example, the configuration error page for the OpenStack tenant is here: https://zuul.opendev.org/t/openstack/config-errors On that page, look for the "Regex Deprecation" warnings. You can expand the warning to see exactly the file and line number where the warning is found. Almost all of the issues identified so far have been due to the fact that RE2 does not support negative lookahead assertions. In the past, we used these a lot to say things like "run this job on any branch except master" with regexes like "(?!master)". Because this is such a common pattern, we added a feature in Zuul to make this easy. A configuration like: branches: "^(?!master).*$" can now be written as: branches: regex: "^master$" negate: true This simply negates the specified regex. It's not an exact translation of the original regex, but is slightly improved and to the point: anything but exactly "master". Please take a moment and check to see if your project has instances of this warning, and go ahead and fix it. Even if there are a lot of instances, they are likely to be very similar and correcting them may take only a few minutes. Here are two changes where we update the central configuration that applies to all projects: https://review.opendev.org/c/openstack/project-config/+/893702 https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/893792 These provide examples of how to make changes for several common cases; you may be able to simply copy what was done in those changes. If you have any trouble adapting existing regexes to the new format, please contact us in #opendev to discuss the issue. We may be able to find a simple solution, or if there is another special case that is difficult to handle, we may be able to add support to Zuul to mitigate the issue. If you use "update-regexes" as a topic name, it will help people keep track of progress and possible issues. The Zuul project has not yet set a date for when support for the old syntax will be removed. I expect it to be on the order of months, which should give folks plenty of time to make the necessary changes. But please do so early so that we have time to deal with any issues. Thanks, Jim (on behalf of the OpenDev team)

1 0

Removing OpenDev's Fedora Test Images on September 11, 2023
by Clark Boylan 06 Sep '23

06 Sep '23

OpenDev has been unable to update Fedora images in a timely manner leaving us with Fedora 35 and 36 images in our CI system. Fedora 35 and 36 are long past their EOL date, and we have been planning to remove them [0]. Somewhat recently the Fedora project stopped mirroring packages for these releases which resulted in empty mirrors on our end [1]. The end result is that Fedora nodes haven't been very functional within OpenDev since. We have slowly been cleaning up usage of Fedora nodes and recently removed the fedora-latest nodeset from Zuul [2]. The next step is to remove these images from Nodepool entirely. We will do so on Monday September 11, 2023. Chances are any Fedora jobs are already broken, but this is a good reminder to look into correcting any of those jobs by moving them to another platform. [0] https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… [1] http://mirror.dfw.rax.opendev.org/fedora/releases/36/ [2] https://review.opendev.org/c/opendev/base-jobs/+/892380 Clark

1 0

Updating OpenDev Zuul's default Ansible version to Ansible 8
by Clark Boylan 23 Aug '23

23 Aug '23

Currently OpenDev Zuul's default Ansible version is Ansible 6. Zuul has recently added support for Ansible 8, and is dogfooding Ansible 8 in the Zuul tenant for the Zuul project. Yesterday, the OpenDev team updated the Zuul tenant for the OpenDev project to default to Ansible 8 as well. So far we have seen no problems with jobs run in these two tenants related to the Ansible version change. We would like to update the default Ansible version for all of the Zuul tenants in OpenDev's Zuul installation to Ansible 8 in the near future. The reason for this is Zuul will update the default Ansible version to Ansible 8 in Zuul itself. Not long after that change, support for Ansible 6 in Zuul will be removed. Changing things in OpenDev first allows us to do these changes in a controlled manner and get ahead of any potential problems. Assuming the Zuul and OpenDev tenants continue to run without trouble our plan is to switch the nebulous, pyca, pypa, and vexxhost tenants to Ansible 8 by default on August 28, 2023. Then if things continue to run without trouble, we will complete this change by updating the openstack tenant on September 4, 2023. I had initially thought that we would do a more conservative rollout, but initial data indicates this change may be straightforward so we'd prefer a quicker move. Note that if there are major issues we can revert the setting in our Ansible config for any of these tenants and return to a default to Ansible 6 while the issues get addressed. Issues will need to be addressed as Ansible 6 is EOL and Zuul intends to drop support for this version of Ansible. If you would like to test your jobs before the tenant configuration updates go into effect you can do so by setting `ansible-version: '8'` in the configuration of a job [0], push the resulting change to Gerrit, and then wait for Zuul to run the job and provide you with results. It is not necessary to merge the change; Zuul will speculatively test this setting. If you discover any problems through this process please let us know. [0] https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.ansible-versi… Clark

1 0

opendevorg docker hub images moving to quay.io
by Clark Boylan 24 May '23

24 May '23

The OpenDev team has decided to move hosting for its container images from docker hub to quay.io. Over the weekend we landed changes to update the publication location for our python base images (opendevorg/python-builder, opendevorg/python-base, opendevorg/uwsgi-base). We will continue to move additional images as quickly as possible. While we publish these images primarily for our own consumption, they are entirely open source and others may be using them as well. If you are using these images, you should update the image source locations to quay.io/opendevorg/$imagename. Once we have updated publishing for our last image, we will begin a 30 day countdown, after which we will clean up images on docker hub to remove any confusion over which images are maintained and up to date. We will send notice for a more specific time frame once it is known. Let us know if you have an questions. Clark

1 1

Etherpad outage April 19, 2023 at 22:00 UTC
by Clark Boylan 17 Apr '23

17 Apr '23

The OpenDev team will be taking etherpad services at https://etherpad.opendev.org offline for approximately 90 minutes beginning at 22:00 UTC on Wednesday April 19, 2023. This is necessary for us to replace the server hosting etherpad and perform necessary data migrations. If you have any questions or concerns please reach out to us and let us know. Clark

1 0

review.opendev.org Downtime for Gerrit Upgrade and Project Renames April 6, 2023
by Clark Boylan 15 Mar '23

15 Mar '23

Hello, The OpenDev team intends to have a Gerrit outage of approximately two hours on April 6, 2023 beginning at 22:00 UTC. The purpose of this outage is to upgrade Gerrit from version 3.6 to 3.7. We will also take advantage of the downtime to perform project renames. For more information on the new version of Gerrit you can refer to the upstream release notes [0]. If you would like to rename a project please get it onto our TODO list on the wiki [1] and push changes according to the documented process [2]. We've chosen this time and date because we expect it to be a quiet period, and it occurs after known releases and events like the PTG. We are happy for you to raise and concerns, questions, comments or feedback with this plan; the earlier the better. [0] https://www.gerritcodereview.com/3.7.html [1] https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Upcoming_Project_… [2] https://docs.opendev.org/opendev/infra-manual/latest/creators.html#project-…

1 0

Intent to grant control of x/virtualpdu to OpenStack community
by Jeremy Stanley 01 Mar '23

01 Mar '23

The OpenStack Ironic project relies on VirtualPDU, which is no longer actively developed. Ironic contributors reached out to the VirtualPDU maintainers with an offer to officially assume maintenance responsibilities in order to avoid forking it, and received a response from one (Mathieu Mitchell) who indicated support for that plan. Unfortunately, communication died off shortly thereafter, and the Ironic team has been unable to raise any of the current maintainers to actually add their access to the x/virtualpdu project in OpenDev's Gerrit. We don't have a documented process since this is the first time it's really come up, but I'm officially announcing that I intend to use my administrative permissions as an OpenDev sysadmin to add membership of an OpenStack Ironic team representative to the following Gerrit groups if no objections are raised before Wednesday, March 8: * virtualpdu-core * virtualpdu-release This will effectively grant full control of the x/virtualpdu repository to OpenStack Ironic contributors. Please follow up to the service-discuss(a)lists.opendev.org or openstack-discuss(a)lists.openstack.org mailing list with any concerns as soon as possible, or you can feel free to reach out to me directly by email or in IRC (fungi in the #opendev channel on the OFTC network). -- Jeremy Stanley

1 0

December 2022 OpenDev Activity Update
by Clark Boylan 14 Dec '22

14 Dec '22

I had meant to send one of these every couple of months, but it has been about six months since I sent the last update email. Apologies for the delay. It has been a busy six months; let's dive in. We bootstrapped the ability to deploy new control plane nodes on Ubuntu Jammy. All newly created servers are built on Jammy unless we have a very good reason to use something older. Additionally, we have updated our base Python Docker images to include support for Python 3.11. Python 3.11 is only a few weeks old, but we are able to deploy software on top of it taking advantage of the performance benefits it provides. Gerrit has been upgraded to 3.6.3. This release is just behind 3.7.0 which was released about a month ago. I believe this is the closest we've been to the latest Gerrit release in about a decade. We do plan to upgrade to 3.7.x, but this upgrade appears to be a bit more involved than the last few. In particular, the upgrade itself needs offline re-indexing, we need to port our label configs to the new submit requirements system, and the web UI is now based on Lit instead of polymer. The Gitea cluster is now running 1.17. This update didn't bring any major user facing changes. The next update to 1.18.x should correct a problem with Gitea misidentifying certain files as being "vendored". Prior to the October PTG event our Meetpad deployment was updated to use the latest stable releases of Jitsi Meet, and we applied a modern configuration to our installation. One major feature this brought is a landing page for meetings where users can configure their name and settings. Importantly, this landing page should prevent browsers' auto mute behavior for auto playing video+audio from breaking calls for our users. We believe this was a major source of users not being able to hear others in calls. Mailing lists for lists.opendev.org and lists.zuul-ci.org have been migrated to a new Mailman 3 server. We are using these two list domains to work out issues while we prepare for migrating the rest of our mailing lists to the new server. This should give us a much more friendly web interface for browsing archives with built in search capability as well as easier to use moderation tools for list moderators. Our Zuul deployment has seen a number of improvements. The ZooKeeper cluster backing Zuul has been upgraded to ZooKeeper 3.7 which brings performance improvements. Zuul supports Ansible 6 now, and all of our tenants are running jobs with this version of Ansible. This version of Ansible fixes pipelining which should provide some improvements to job runtime. Finally, we are upgrading Zuul weekly using automated zero downtime rolling restarts. This ensures we're getting the latest and greatest Zuul features and can provide quick and early feedback to Zuul. Looking ahead to 2023, I expect to see a lot more of what we've done over the last six months. Gerrit will be upgraded to 3.7, Gitea will be upgraded to 1.18, more mailing lists will be migrated to Mailman 3, and more services will be deployed on Jammy running with Python 3.11. See you next year! Clark

1 0

review.opendev.org Gerrit 3.6 upgrade December 12th, 20:00UTC
by Ian Wienand 07 Dec '22

07 Dec '22

Hello, We are planning upgrade review.opendev.org to the Gerrit 3.6 release on 12th December, 2022 at around 20:00UTC. This will involve an outage of around 1 hour. There should not be much user-visible change after the upgrade; the major items do not affect our usage. The full release notes are at [1] Thank you, -i [1] https://www.gerritcodereview.com/3.6.html

1 0

Mailman v3 migration for OpenDev and Zuul mailing lists 2022-12-05
by Jeremy Stanley 06 Dec '22

06 Dec '22

On Monday, December 5 at 20:00 UTC, the OpenDev Collaboratory systems administrators will be migrating the lists.opendev.org and lists.zuul-ci.org mailing list sites from their existing Mailman 2.1.29 server to a shiny new Mailman 3.3.7 deployment. This maintenance window is expected to last no more than one hour. The key takeaways are as follows: * There will be a brief outage for the affected sites while DNS is updated and configuration, subscriber lists and message archives are imported, but incoming messages should correctly queue at the sender's end and arrive at the conclusion of our work * Because this is on a new server, there are new IP addresses from which list mail will be originating, 162.209.78.70 and 2001:4800:7813:516:be76:4eff:fe04:5423 * Moderation queues will not be copied to the new server, so moderators are encouraged to process any held messages prior to the migration time in order to avoid losing them * Anyone wishing to adjust their list subscriptions, or handle moderation or list configuration after the migration needs to use the Sign Up button on the Web page to create a new account; it will be linked to your imported roles as soon as you confirm by following a URL from an E-mail message the server sends you, and is global for all sites on the server so you only need to do this once * The software providing Web front-ends for list management and archive browsing is entirely new in Mailman v3 and therefore has a much different appearance, though we've added appropriate redirects and frozen copies of old archives in order to accommodate existing hyperlinks Once we smooth out any unanticipated rough patches with the new service, we'll schedule additional maintenance to migrate mailing list sites for other domains (Airship, Kata, OpenInfra, OpenStack, StarlingX). If you have any questions or concerns, feel free to follow up on the service-discuss mailing list or find us in the #opendev channel on the OFTC IRC network. -- Jeremy Stanley

1 1

Discussing Storyboard Needs on service-discuss
by Clark Boylan 07 Nov '22

07 Nov '22

I wanted to make sure that everyone subscribed to this list is aware that we are discussing the Storyboard service, its maintenance needs, and user needs on a thread on service-discuss(a)lists.opendev.org [0]. I have been told that not everyone is aware of the service-discuss list and may have missed that thread. Please follow up on that thread if you have any feedback for (not) wanting to use storyboard or desire to help maintain the service. This is a good opportunity to remind everyone that OpenDev runs three mailing lists for user interaction: 1. service-announce(a)lists.opendev.org [1] This mailing list. Intended to be kept low volume and limited to the most important email notices. 2. service-discuss(a)lists.opendev.org [2] Where we discuss the service needs, meeting agendas, and other topics with more volume. The bulk of our mailing list discussion will be here. All are welcome to join and participate. -discuss and -announce are separate to give people a lower volume option with -announce. 3. service-incident(a)lists.opendev.org This is a private mailing list that OpenDev admins and other trusted individuals may be subscribed to. Users can email this list with sensitive messages like vulnerability disclosures. [0] https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html [1] https://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce [2] https://lists.opendev.org/cgi-bin/mailman/listinfo/service-discuss

1 0

Changes to our base job nodeset and Ansible 5 removal
by Clark Boylan 12 Oct '22

12 Oct '22

As mentioned in the previous email [0] about updating our default Ansible version to Ansible 6, Zuul will be removing support for Ansible 5 soon. Based on Zuul's currently estimated release schedule we expect this to happen on or near October 18. This change won't take effect until OpenDev restarts Zuul. We will try to make that happen quickly after Zuul lands this change in order to catch any issues quickly. Additionally, it has been called out that we are still defaulting our base job nodeset to ubuntu-focal. Ubuntu Jammy has been released for about 6 months now and has a .1 release as well. To catch up we will update the default nodeset in our base jobs to ubuntu-jammy on October 25, 2022. If you would like to test that your jobs run against Ubuntu Jammy you can modify your jobs to set `nodeset: ubuntu-jammy` today and test them. Note, any jobs that expect a specific platform should already be explicitly setting the nodeset they rely on. There are a number of jobs already running on Jammy which gives us confidence in the newer images. [0] https://lists.opendev.org/pipermail/service-announce/2022-October/000046.ht… Clark

1 0

Updating Zuul tenants' default-ansible-version to 6
by Clark Boylan 05 Oct '22

05 Oct '22

Hello, The OpenDev team is planning to update the default-ansible-version to 6 in the near future. Zuul is planning to drop support for Ansible 5, and switching OpenDev to Ansible 6 early allows us to provide valuable feedback if there are problems. There are other reasons for the switch as well. Ansible 5 broke pipelining, but we expect Ansible 6 to correct this problem. Having pipelining should improve the runtime of jobs. We had been holding off on making this change as StarlingX and OpenStack have both recently worked through some fairly major releases. Those releases exist now making this a good time for a change like this. As far as testing goes, Zuul's Zuul tenant has been running with an Ansible 6 default for several weeks now. I've also tested this change with Devstack specifically [0] as Devstack seems to have a good representative set of complicated jobs. The change to update our defaults is up for review [1]. Feel free to leave any comments there or on this email thread. Note that you can override this change if necessary using the same mechanism I used to test Devstack [0]. However, if you do this please communicate that change to us as Zuul does plan to remove Ansible 5 support which means any problems with Ansible 6 need to be communicated back to Zuul. [0] https://review.opendev.org/c/openstack/devstack/+/858436 [1] https://review.opendev.org/c/openstack/project-config/+/860474

1 0

Dropping support for Ansible 2.x in our Zuul
by Jeremy Stanley 21 Sep '22

21 Sep '22

Following the earlier 2.8/2.9 deprecation in June and change of our default to Ansible 5 in July, we will soon be restarting onto a version of Zuul which no longer supports 2.x versions at all. While we've looked for any lingering overrides to old Ansible versions in master branch jobs and given projects a heads up individually, we lack insight into whether the same may be occurring on non-default branches of projects. If you are aware of any jobs on any of your branches which pin back to an earlier Ansible version, we strongly recommend you remove those overrides at your earliest opportunity, otherwise those jobs will break in the coming days. Here is the change to remove support for Ansible 2.8 and 2.9 in Zuul (we generally restart our Zuul services every weekend, so that's when it's most likely to take effect): https://review.opendev.org/857796 For some example changes which clean up old Ansible version overrides, see this review topic list: https://review.opendev.org/q/topic:cleanup-old-ansible -- Jeremy Stanley

1 0

Move Zuul queue declarations to the project level
by Jeremy Stanley 19 Sep '22

19 Sep '22

This is just a reminder to users of our Zuul service that if you find changes aren't getting any jobs run, or seem to not be running some jobs you expect, you should double-check your configuration to make sure you don't have a lingering "queue" setting in your project pipelines. Over the weekend, we updated to a version of Zuul which removed the long-deprecated support for pipeline level queue declarations, and it now expects queues to be set at the project level instead. We directly notified some projects which we knew would be impacted, but did not send a broader announcement to this list, so I just wanted to be sure anyone taken by surprise with this change doesn't have to waste additional time troubleshooting it. Earlier today, the OpenDev sysadmins corrected numerous cases of this error in the central openstack/project-config repository used by the openstack tenant, but there are still many projects which have similar errors in their in-repository configuration as well. For the reminder from the Zuul maintainers, see here: https://lists.zuul-ci.org/pipermail/zuul-discuss/2022-May/001801.html Also these posts to the OpenStack discussion lists may include helpful information about the transition: https://lists.openstack.org/pipermail/openstack-discuss/2022-May/028603.html https://lists.openstack.org/pipermail/openstack-discuss/2022-September/0305… -- Jeremy Stanley

1 0

Updating Zuul's Default Ansible Version to Ansible v5
by Clark Boylan 11 Jul '22

11 Jul '22

Hello, The OpenDev team will be updating the default Ansible version in our Zuul tenants from Ansible 2.9 to Ansible 5 on June 30, 2022. Zuul itself will eventually update its default, but making the change in our tenant configs allows us to control exactly when this happens. There are some known issues with this transition; however, we've already worked through and addressed a number of them. If you'd like to double check your jobs work with Ansible 5 today you can do that by pushing a change(s) that updates job.ansible-version [0] to the value '5'. If you do this and find problems please let us know. The Zuul zuul tenant is already running with this set tenant wide which means we expect most of the base job pre-run and post-run content to work with Ansible 5. We also did some spot checking of things like devstack and tempest and expect most of those jobs to function too. Finally, we can eventually expect Zuul to drop support for Ansible 2.8 and 2.9 in the future. What this means is we shouldn't just pin to those old versions without a plan forward to addressing any remaining v5 incompatibilities. [0] https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.ansible-versi… If you have any questions let us know. Clark

2 1

A Quick OpenDev Activity Update
by Clark Boylan 23 Jun '22

23 Jun '22

It has been a couple months since I sent out our last activity update, and now seems like a good time to share what is new. Ubuntu Jammy test nodes are now available in both x86_64 and ARM64 flavors. This version of Ubuntu ships with python3.10 and is ready for you to start testing your python projects on this newer runtime. Our Gerrit server running at https://review.opendev.org has been upgraded from Gerrit 3.4 to 3.5. This newer version of Gerrit brings features like new change search predicates, updates to the UI, and performance updates. On the whole this is a bit more of an incremental update compared to previous upgrades. We expect many users haven't even noticed that the upgrade happened. If you'd like to learn more you can read the Gerrit 3.5 release notes: https://www.gerritcodereview.com/3.5.html. Glean 1.23.0 has been released and finally adds proper support for static ipv6 configuration on RPM based distros. Our Ethercalc server previously running at https://ethercalc.openstack.org has been shutdown and removed. On the Zuul side of things there are two changes you should be aware of and planning for. The first is the planned removal of pipeline specific queues in your Zuul config. This configuration was deprecated a number of releases ago and is slated for removal in the next major Zuul release. The OpenStack tenant is the only Zuul tenant that has this old style of configuration. More information on which projects are affected and how to correct it can be found in my email about this topic: https://lists.openstack.org/pipermail/openstack-discuss/2022-May/028603.html. Note that projects with this configuration in place that do not update will have broken Zuul configurations after we upgrade to the next major Zuul release. Next, is the plan for changing the default Zuul Ansible version from Ansible 2.9 to Ansible 5 on June 30, 2022. We've already done some testing of this and the base jobs as well as devstack and tempest seem to work with newer Ansible. If you'd like to learn more (including how you can test your specific projects early) please check my previous email on this subject: https://lists.opendev.org/pipermail/service-announce/2022-June/000041.html. Looking ahead we should expect our Gitea farm to be upgraded to v1.17. Work will begin on upgrading Gerrit to version 3.6, but it is still too early to know when we will perform that upgrade. There are a few larger tasks we'd like to get moving on including updating our Keycloak server to provide SSO capabilities for our services, upgrading our list servers to Mailman 3, modernizing our StoryBoard deployment, and deploying a Prometheus service to collect server and service metrics. If any of these sound interesting to you, and you would like to help, please reach out. Clark

1 0

CentOS/Fedora IPv6 on gate nodes
by Ian Wienand 02 Jun '22

02 Jun '22

Hello, Consistent IPv6 setup will soon be coming to CentOS and Fedora nodes pending a release of glean [1], the tool we use in the OpenDev gate to configure networking on test nodes. After recent work we believe we now handle the heterogeneous IPv6 setup methods in all our donated clouds consistently. Debian-based distros have had better support for this, so the IPv6 path is generally well tested. However, if this reveals any issues, please reach out (IRC #opendev or service-discuss(a)lists.opendev.org) and we can look into it. Thank you, -i [1] https://opendev.org/opendev/glean/

1 0

review.opendev.org Gerrit 3.5 upgrade June 19th, 20:00UTC
by Ian Wienand 18 May '22

18 May '22

Hello, We are planning upgrade review.opendev.org to the Gerrit 3.5 release on 19th June, 2022 at around 20:00UTC. This will involve an outage of around 1 hour. There should not be much user-visible change after the upgrade. There are some enhanced search predicates [1] which you may find useful. Thank you, -i [1] https://www.gerritcodereview.com/3.5.html#new-change-predicates

1 0

Retiring and shutting down the ethercalc.openstack.org service
by Clark Boylan 04 May '22

04 May '22

Hello, The https://ethercalc.openstack.org service has become problematic for several reasons: * Our configuration management for the service is still based on Puppet and needs to be modernized to continue running the service. * The service itself isn't well maintained. Its build system is apparently broken [0]. * The service does not capture a history of changes like etherpad, confusing users. * The service crashes semi frequently. The major known user of this service was the PTG planning process. We've talked to the folks that help organize and plan the PTG, and they think other tools (like PTGBot) can be used instead without major trouble. Considering all of this, the OpenDev team would like to retire and shutdown this service. The current plan is to keep the service up and running until the end of the month. Then on May 31, 2022 shut down the server hosting the service, snapshot this server, and finally delete it. Please back up any data you would like to preserve before May 31, 2022. [0] https://github.com/audreyt/ethercalc/pull/781 If you have any questions or concerns feel free to respond to this thread or reach out to the OpenDev team. Clark

1 0

A not so Short OpenDev Activity Update
by Clark Boylan 22 Apr '22

22 Apr '22

It has been a busy few weeks for the OpenDev team. We thought it would be a good idea to capture some of the more user facing aspects of what we have been doing lately. Git-review and Bindep both have new releases. Git-review 2.3.0 and 2.3.1 [0] bring better support for newer OpenSSH and signed Git commits, as well as drop compatibility with Python 3.5. Bindep 2.11.0 [1] adds support for AlmaLinux. Git fixed a security issue [2] that broke DevStack and other test jobs. The OpenDev team helped to quickly diagnose the problem and proposed fixes to projects once the issue was understood. If you have jobs with unexpected git operation failures or Python package install issues there is a good chance it is related to this update. You will want to ensure that the user running git commands in a repo is the same user that owns the files in the repo (typically "zuul"). Otherwise you will need to mark the repository as a "safe" repository. Our Twitter status account [3] has been resurrected to provide another way for our users to follow along as we make changes. Whenever we make `#status` messages in IRC that content gets mirrored to this account which you can follow on Twitter. We have also been doing quite a bit of Spring Cleaning (it is spring for some of us anyway). We no longer build Debian Buster or Python 3.7 versions of opendevorg/python-builder and opendevorg/python-base. We did our best to check that no one was using these images, but if you are, the images will remain on Docker Hub until Docker Hub times them out. You should update to a Bullseye Python 3.8 or newer image. Bullseye Python 3.10 images are newly available too. The OpenStack Health service was shut down some time ago, and its configuration management and source repositories have been retired or cleaned up as appropriate. The ELK system that powered elastic-recheck is in the process of being shutdown and retired in favor of a new OpenSearch hosted system managed by OpenStack more directly. Please read the documentation [4] for using the new system if you would like to continue to query job logs using Kibana. Our distro mirrors are getting a haircut to remove packages for unused architectures in our CI system, ISOs/images, and old distro releases. This cleanup will ensure we have plenty of room to add a Jammy Jellyfish mirror to our collection. We'd like to remind our users that while these mirrors are publicly available they are not intended for public consumption. We can and do modify them as our CI system needs change. Finally, looking ahead we'll begin to work on rolling out Jammy Jellyfish images for the CI system now that this latest Ubuntu LTS is available. Our Gerrit 3.5 upgrade should also pick up steam. Most of the changes we expect Gerrit users to see will be in the UI itself. You can read the release notes [5] to find out more. We aren't ready to set a date for this, but you should expect a short downtime which we will announce via service-announce(a)lists.opendev.org. Unfortunately, the Gerrit problems with SSH RSA keys won't be resolved until Gerrit 3.6. [0] https://lists.opendev.org/pipermail/service-announce/2022-April/000036.html [1] https://lists.opendev.org/pipermail/service-announce/2022-April/000035.html [2] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-24765 [3] https://twitter.com/opendevinfra/ [4] https://governance.openstack.org/sigs/tact-sig.html#opensearch [5] https://www.gerritcodereview.com/3.5.html

1 0

git-review 2.3.0 and 2.3.1
by Jeremy Stanley 19 Apr '22

19 Apr '22

It's my pleasure to announce the release of git-review 2.3.0 and follow-up fix version 2.3.1. These are the corresponding release notes since the previous release: ======= 2.3.1 ======= Bug Fixes --------- * Improved version string parsing for Git in order to no longer break on Mac OSX/Darwin systems (story 2010002). ======= 2.3.0 ======= Upgrade Notes ------------- * Python 3.5 is no longer supported. * Git 2.10.0 or later is required as git-review now needs the ``--no-show-signature`` option for git-log. Bug Fixes --------- * Git's log.ShowSignature option is now supported. Previously, setting it would confuse git-review into thinking there were twice as many changes to push. * OpenSSH has deprecated its use of scp/rcp protocol in favor of SFTP, which the embedded Apache mina-sshd in widely-deployed Gerrit versions does not yet support. The default officially changed in OpenSSH 9.0 (some distributions, such as Fedora and CentOS, switched their default behavior to this as early as OpenSSH 8.7 or 8.8), leading to a "subsystem request failed on channel 0" error during commit-msg hook retrieval. Now git-review will attempt to detect whether scp's -O option is available to force use of the legacy scp/rcp protocol, and apply it if so. Future git-review versions may change the fallback behavior once an SFTP subsystem is implemented and available for a new Gerrit release. As always, the latest version of git-review is available from PyPI: https://pypi.org/project/git-review Current documentation can be found here: https://docs.opendev.org/opendev/git-review Its source code can be obtained from the OpenDev Collaboratory: https://opendev.org/opendev/git-review Please don't hesitate to follow up through the service-discuss(a)lists.opendev.org mailing list or in the #opendev channel on the OFTC IRC network if you have any questions. -- Jeremy Stanley

1 0

bindep 2.11.0
by Jeremy Stanley 19 Apr '22

19 Apr '22

It's my pleasure to announce the release of bindep 2.11.0. These are the corresponding release notes since the previous release: New Features ------------ * Add support for AlmaLinux to Bindep. As always, the latest version of bindep is available from PyPI: https://pypi.org/project/bindep Current documentation can be found here: https://docs.opendev.org/opendev/bindep Its source code can be obtained from the OpenDev Collaboratory: https://opendev.org/opendev/bindep Please don't hesitate to follow up through the service-discuss(a)lists.opendev.org mailing list or in the #opendev channel on the OFTC IRC network if you have any questions. -- Jeremy Stanley

1 0

review.opendev.org Gerrit downtime for project renames April 15, 2022
by Clark Boylan 07 Apr '22

07 Apr '22

Hello, The OpenDev team will be renaming Gerrit projects on April 15, 2022 beginning at 15:00 UTC. This process requires a Gerrit downtime for the service running on review.opendev.org and is expected to last no more than an hour. If you would like to rename a project, now is the time to begin planning for that work ensuring the necessary changes are proposed. If you have any questions please feel free to respond to this email or reach out to the team. Clark

1 0

Zuul behavior change with Depends-On across queues
by Jeremy Stanley 05 Apr '22

05 Apr '22

A recent improvement to more accurately report Zuul's enqueuing has resulted in an unanticipated difference in behavior for changes declaring a Depends-On relationship between projects which don't share a dependent queue. If you approve the depending change before the dependency has merged, Zuul will now report a Verified -2 vote on the depending change (where previously it just ignored your approval entirely). As a result, because the openstack tenant is configured to require a positive Verified vote before enqueuing into the gate pipeline, this means the depending change will need to be rechecked in order to replace that -2 vote with a +1 in that tenant. For tenants without such a "clean check" rule, the change in behavior is entirely cosmetic. One thing you can do to avoid this is make sure any projects which regularly utilize Depends-On between their changes are configured to belong to the same named queue, which will result in Zuul enqueuing those approved changes into the gate pipeline together without needing to wait for one to merge before the other can be tested. Zuul expects related projects to share a queue and, on approval, will only look up parent and child changes to enqueue within the related projects list, as determined by the queue to which they belong. An alternative workaround is to only approve a change once you've confirmed that any Depends-On has merged, if that dependency is in another project which does not share the same queue. Feel free to follow up on the service-discuss ML with any comments/concerns, or find us in the #opendev channel on OFTC as usual. -- Jeremy Stanley

1 0

Temporary addition of unsigned tag permissions
by Jeremy Stanley 01 Feb '22

01 Feb '22

In the wake of yesterday's Gerrit 3.4 upgrade maintenance, we've observed a regression in behavior where signed Git tags are being misidentified as unsigned[*]. Since our ACLs normally disallow pushing of unsigned (annotated) tags, the result is tag pushes being rejected with a permissions error. As an emergency measure, and in an attempt to not roll back our Gerrit upgrade while working with their developers on a proper patch so solve this, we have updated the ACLs for all projects with a workaround so that they will be able to continue pushing tags. Unfortunately, this workaround means that projects will temporarily be able to push unsigned tags as well as signed ones, so we ask anyone manually tagging their projects be extra careful to remember to pass the -s option to git's tag subcommand. Once we have a proper fix in place, the temporary addition of unsigned tag permissions will be rolled back, and a follow-up announcement sent to this mailing list. [*] https://bugs.chromium.org/p/gerrit/issues/detail?id=15616 -- Jeremy Stanley

1 1

Gerrit Maintenance and Downtime January 24, 2022
by Clark Boylan 19 Jan '22

19 Jan '22

Hello, The OpenDev team will be taking the early year quiet as an opportunity to rename a few projects in Gerrit and upgrade Gerrit to 3.4 starting January 24, 2022 at 22:00 UTC. We expect to be done by 00:00 UTC January 25, 2022. If you would like to rename a project now is the time to get that preparation done. Please link any rename proposals in our list here: https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Upcoming_Project_… The Gerrit 3.4 release notes can be found at https://www.gerritcodereview.com/3.4.html. This upgrade does not require offline reindexing which means it should happen quickly. We are also automatically testing the 3.3 to 3.4 upgrade in our CI system. Additionally there is a downgrade process that we can use to fall back to 3.3 if necessary. Let us know if you have any questions or concerns, Clark

1 0

Changes to OpenDev's stable of CI images
by Clark Boylan 07 Dec '21

07 Dec '21

Hello, CentOS 8 reaches end of life on December 31, 2021. This means now is the time for you to convert jobs to CentOS 8|9 Stream or some other appropriate platform. The OpenDev team will plan to remove CentOS 8 images and mirrors from our infrastructure in early January, hopefully by January 14, 2022. Again, please update your jobs now so that you are prepared. These changes are necessary as the distro will no longer be supported, and there is no way for us to ensure that the images will be secure or that we'll even be able to build them. We've also got Fedora 35 lined up to replace Fedora 34. Due to Fedora's short support window we typically swap out one Fedora release with the next when the next is ready. If you are using the fedora-34 label please update it to fedora-35. If you are using fedora-latest we'll be updating that to use fedora-35 and the swap will happen automatically for you. We expect to have Fedora 34 cleaned up by the end of the year. Additionally both OpenSUSE 15 and Tumbleweed could use some help. For OpenSUSE 15 we need to update it to 15.3 as well as updating the associated mirrors if we plan to keep it around. I'm happy to keep this image around if we can get volunteers in the community that can help with that. If there isn't interest in modernizing the OpenSUSE 15 images I'd like to propose we clean them up in January along with the CentOS 8 images. For Tumbleweed, the images haven't received much use or attention. I'd like to propose that we simply retire this image and clean it up. We had thought it might be a good platform for monitoring upcoming changes in the Linux world, but the reality is that only works if you've got people active caring for it and adjusting to those changes. The Tumbleweed cleanup will likely begin before the end of the year. Finally, our Gentoo images bounce between being buildable and broken with more time spent being broken than not. I'd like to propose that this image also get cleaned up in the near future. Again, if there is interest in the community for keeping this alive I think we can make that happen. But as is this image is not in a sustainable place and it would be better for us to stop running jobs on it. Let us know if you have any questions or comments, Clark

1 0

git-review 2.2.0
by Jeremy Stanley 24 Nov '21

24 Nov '21

It's my pleasure to announce the release of git-review 2.2.0. These are the corresponding release notes since the previous release: Bug Fixes --------- * When checking for unstaged or uncommitted changes to avoid performing a test rebase, unstaged and uncommitted changes in Git submodules are now ignored since those won’t be rebased anyway. * git-review now handles the Git core.hooksPath configuration option correctly. Thus, it installs the commit-msg hook into the core.hooksPath directory, if that option is set. Otherwise, it continues to install the hook into .git/hooks, relative to the root of the checkout. * With Git 2.18.0 and later, the --rebase-merges (-r) option will be used in place of the deprecated --preserve-merges (-p) option. This fixes a critical error when git-review is used with Git versions 2.34.0 onward, where the old option is no longer available. As always, the latest version of git-review is available from PyPI: https://pypi.org/project/git-review Current documentation can be found here: https://docs.opendev.org/opendev/git-review Its source code can be obtained from the OpenDev Collaboratory: https://opendev.org/opendev/git-review Please don't hesitate to follow up through the service-discuss(a)lists.opendev.org mailing list or in the #opendev channel on the OFTC IRC network if you have any questions. -- Jeremy Stanley

1 0

Bindep 2.10.1 has been released
by Jeremy Stanley 20 Oct '21

20 Oct '21

Please take notice that version 2.10.1 of Bindep has been released. Notable Changes --------------- * The Packaging module dependency from 2.10.0 was adjusted to accommodate versions of Pip prior to 9.0.0 (when PEP 503 data-requires-python support was added) on Python 3.5 and earlier. As always, the latest version of Bindep is available from PyPI: https://pypi.org/project/bindep Read the latest documentation: https://docs.opendev.org/opendev/bindep/latest/ And source code can be obtained through the OpenDev Collaboratory: https://opendev.org/opendev/bindep Please don't hesitate to follow up through the service-discuss at lists.opendev.org mailing list or in the #opendev channel on the OFTC IRC network if you have any questions. -- Jeremy Stanley

1 0

Bindep 2.10.0 has been released
by Clark Boylan 18 Oct '21

18 Oct '21

I am happy to announce version 2.10.0 of Bindep has been released. Notable Changes --------------- * Packaging (https://pypi.org/project/packaging/) is now a required dependency. * Support for Rocky Linux is added. * Support for Manjaro is added. As always, the latest version of Bindep is available from PyPI: https://pypi.org/project/bindep Read the latest documentation: https://docs.opendev.org/opendev/bindep/latest/ And source code can be obtained through the OpenDev Collaboratory: https://opendev.org/opendev/bindep Please don't hesitate to follow up through the service-discuss at lists.opendev.org mailing list or in the #opendev channel on the OFTC IRC network if you have any questions. Clark

1 0

Gear 0.16.0 has been released
by Clark Boylan 18 Oct '21

18 Oct '21

I am happy to announce version 0.16.0 of Gear has been released. Notable Changes --------------- * The testsuite has been updated to support newer OpenSSL fixing issues like https://storyboard.openstack.org/#!/story/2009250. * TLS version negotiation is improved to prefer the newest version of TLS supported by both client and server excluding TLS 1.3. * Connections to gearman servers are now randomly distributed which should more evenly distribute load. * Timeout processing is improved to avoid a situation where a connection timeout could deadlock the client. As always, the latest version of gear is available from PyPI: https://pypi.org/project/gear Read the latest documentation: https://docs.opendev.org/opendev/gear/latest/ And source code can be obtained through the OpenDev Collaboratory: https://opendev.org/opendev/gear Please don't hesitate to follow up through the service-discuss at lists.opendev.org mailing list or in the #opendev channel on the OFTC IRC network if you have any questions. Clark

1 0

review.opendev.org Gerrit Outages in the Next Couple of Weeks
by Clark Boylan 06 Oct '21

06 Oct '21

Hello, The OpenDev team will be performing a Gerrit 3.3 upgrade on Sunday October 10, 2021 from 21:00UTC to 23:00UTC. Then on Friday October 15, 2021 from 18:00UTC to 19:00UTC we will be renaming a number of projects in Gerrit. Both of these actions require Gerrit service downtime on review.opendev.org. We expect the outages will be shorter than the allotted time but this gives us some flexibility. If you would like to rename your project, now is the time to do the work to make sure your renames are on the list. Please reach out to the OpenDev team and we'll help you get the necessary changes proposed and reviewed. If you are curious, Gerrit 3.3 release notes can be found at: https://www.gerritcodereview.com/3.3.html. Notably, we are able to revert to 3.2 should that become necessary. Feel free to bring up any questions or concerns with the team and we'll do our best to answer, Clark

1 0

Renaming Projects on July 30, 2021 at 15:00UTC
by Clark Boylan 26 Jul '21

26 Jul '21

Hello everyone, The OpenDev team will be renaming projects in Gerrit and Gitea on Friday July 30, 2021 starting at 15:00 UTC. This requires a short Gerrit downtime. If you need to rename a project now is the time to get it on the list at https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Upcoming_Project_…. Let us know if you have any questions. Clark

1 0

Take care when deleting email addresses in Gerrit
by Clark Boylan 14 Jul '21

14 Jul '21

Hello everyone, We have discovered a bug in Gerrit [0] that allows you to delete your account's Ubuntu One OpenID association if you delete the email from your account associated with that OpenID. When you do this the next time you log in Gerrit will create a new account for you. Fixing this after the fact is currently difficult. We are working to correct this but the required corrections take time. We would like to avoid these issues in the first place. Please avoid deleting email addresses from your accounts unless you are sure they are not associated with your OpenID. If you would like help double checking that for you please let us know and one of our Gerrit admins can double check. We are working to make this fixable after the fact, and will try to work with the Gerrit upstream to prevent you from deleting Ubuntu One OpenID associations from your account in the first place. Until then we apologize for the inconvenience. Finally, this message and others like it are sent to service-announce(a)lists.opendev.org. I am sending this email to a much broader set of mailing lists because the service-announce list membership is quite small. All of our users should be subscribed to that list to get important notices like this one. We keep the traffic low and only send mail there when we feel it is important for you to see it. [0] https://bugs.chromium.org/p/gerrit/issues/detail?id=14776 Clark

1 0

OpenDev IRC services are moving to OFTC this weekend
by Jeremy Stanley 26 May '21

26 May '21

As a majority of our constituent projects have voiced a preference for enacting our long-standing evacuation plan, the OpenDev Collaboratory's IRC service bots will be switching from Freenode to the OFTC network this weekend (May 29-30, 2021). We understand this is short notice, but multiple projects have requested that we act quickly. Please expect some gaps in channel logging and notifications from our various bots over the course of the weekend. I have provided a much more detailed writeup to the service-discuss mailing list, and encourage anyone with questions to read it and follow up there if needed. Subsequent updates will be sent only to service-discuss, in order to limit noise for individual project lists and keep further discussion focused in one place as much as possible: http://lists.opendev.org/pipermail/service-discuss/2021-May/000249.html -- Jeremy Stanley

1 0

Switching default nodeset in base job to ubuntu-focal
by Jeremy Stanley 18 May '21

18 May '21

The OpenDev Collaboratory is normally somewhat conservative about when it changes its Zuul node "default" (that is, the nodeset in its base job). It's been ubuntu-bionic since at least July of 2019 when we first moved our base job into its own repository, and ubuntu-focal nodes have been available in our deployment since before Ubuntu's 20.04 LTS was officially released more than a year ago. All that is to say, updating our default is overdue. On Tuesday, 2021-05-18 (two weeks from today) we'll approve https://review.opendev.org/789098 to update the nodeset to ubuntu-focal for any jobs which don't specify their own or inherit from another job which does. We've already merged a change which updates our base-test job to reflect this, so you can test either by explicitly setting your jobs to use an ubuntu-focal nodeset or to inherit from base-test (though we don't recommend merging changes for the latter, just rely on speculative execution to identify problems which you may encounter when the default changes). Please reply to the service-discuss mailing list with any questions or concerns. You should also feel free to leave comments on the change linked above. -- Jeremy Stanley

1 1

git-review 2.1.0
by Jeremy Stanley 24 Apr '21

24 Apr '21

It's my pleasure to announce the release of git-review 2.1.0. These are the corresponding release notes since the previous release: New Features ------------ * Add support for --no-thin which is passed to git push. This is useful to work around situations where Gerrit’s JGit and git-review’s C git implementations cannot agree on the pack file contents used to transfer a push. When they disagree you see this as unpack failures due to missing trees. Using --no-thin avoids complicated negotiations and works around this problem. As always, the latest version of git-review is available from PyPI: https://pypi.org/project/git-review Find the updated documentation at its new home: https://docs.opendev.org/opendev/git-review And source code can be obtained through the OpenDev Collaboratory: https://opendev.org/opendev/git-review Please don't hesitate to follow up through the service-discuss(a)lists.opendev.org mailing list or in the #opendev channel on the Freenode IRC network if you have any questions. -- Jeremy Stanley

1 0

git-review 2.0.0
by Jeremy Stanley 03 Mar '21

03 Mar '21

It's my pleasure to announce the release of git-review 2.0.0. These are the corresponding release notes since the previous release: New Features ------------ * A new gitreview.branchauthor option allows configuring the change author (owner) name part of the autocreated branch name when downloading changes. * A new gitreview.notopic configuration setting makes git-review always behave as if the --notopic command line option was supplied. Upgrade Notes ------------- * Not really a deprecation so much as a removal, Gerrit 2.15 (released in 2017) ceased providing a draft state for changes, and many large sites were disabling support for it far earlier than that. As of now, git-review has removed the -D and --draft options entirely, and so can no longer push draft changes on old Gerrit deployments which still allowed them. * As of this release, git-review no longer supports running under the Python 2.7 interpreter. The major version component has been increased accordingly. Users wishing to use git-review with Python 2.7 can cap their installations like pip install "git-review<2", though pip version 9 and later will use python_requires package metadata to avoid installing git-review on an unsupported interpreter. Future removal of support for Python 3 minor versions will only result in a minor version increase for corresponding git-review releases. Bug Fixes --------- * The Change-Id generating commit hook is now installed into any detected Git submodules, so that it will take effect when creating changes within a submodule and not just in the parent repository. * Encoding was corrected for Git credentials, which was previously causing HTTP authentication to fail under some Python 3 versions. * Change download functions now preserve any unapplied local edits on the branch when attempting to reset the worktree. * The --no-follow-tags option to git push is now used for all pushes, so that git-review won’t error when encountering a local tag which is absent from Gerrit. * For safety, attempts to push a commit with unstaged or uncommitted changes in the worktree will be caught and an error reported, rather than leaving it up to git rebase to spot them. This addresses a situation where users enabling rebase.autostash would otherwise experience data loss when the test rebase is subsequently reset. As always, the latest version of git-review is available from PyPI: https://pypi.org/project/git-review Find the updated documentation at its new home: https://docs.opendev.org/opendev/git-review And source code can be obtained through the OpenDev Collaboratory: https://opendev.org/opendev/git-review Please don't hesitate to follow up through the service-discuss(a)lists.opendev.org mailing list or in the #opendev channel on the Freenode IRC network if you have any questions. -- Jeremy Stanley

1 0

Recommended change to Zuul pipeline config for all tenants
by jim＠acmegating.com 16 Feb '21

16 Feb '21

Hi, If you manage the Zuul pipeline definitions for a tenant in OpenDev's Zuul, please make the following change to the pipeline definitions as soon as possible: * Remove any reference to the "mysql" reporter. The current version of Zuul now automatically adds the SQL reporter to every pipeline definition, so the explicit configuration is unnecessary. It is now marked as deprecated and will produce an error in a future release of Zuul, and it produces an warning message now. Here is an example change which I performed for the OpenStack tenant: https://review.opendev.org/775716 Thanks, Jim

1 0

bindep 2.9.0
by Jeremy Stanley 06 Feb '21

06 Feb '21

It's my pleasure to announce the release of bindep 2.9.0. These are the corresponding release notes since the previous release: New Features ------------ * Bindep can now be imported as a Python module, useful when making plugins for other Python-based tools or experimenting in the REPL. * Basic support for Oracle Linux is now included. * A new set of virtual platform profiles, platform:base-py2 and platform:base-py3 can be used to differentiate packages on Debian derivatives (including Ubuntu), Red Hat derivatives (including CentOS and Fedora), and MacOS X/Darwin. Bug Fixes --------- * Improved parsing of warning and error output from the apk tool in alpine. * Now handles parsing pacman 1.5.2 and later output so –brief works on newer releases of Archlinux. * If a local file has the same name as a package, this no longer confuses parsing of pacman output on Arch Linux. Find the updated documentation at its new home: https://docs.opendev.org/opendev/bindep/ Please don't hesitate to follow up through the service-discuss(a)lists.opendev.org mailing list or in the #opendev channel on the Freenode IRC network if you have any questions. -- Jeremy Stanley

1 0

review.opendev.org Gerrit outage and upgrade 15:00UTC November 20 to 01:00UTC November 23, 2020
by Clark Boylan 23 Nov '20

23 Nov '20

The OpenDev team is planning a long weekend Gerrit outage on review.opendev.org starting 15:00UTC November 20 and running to 01:00UTC November 23, 2020 in order to upgrade to Gerrit 3.2. The upgrade has two major portions. First we will incrementally move from our current version 2.13 to 2.16. Each point release requires a database migration and git indexing operations that take the majority of the time. The second major part, once at version 2.16, is to convert Gerrit to use the new NoteDB backend. This stores reviews together with code in the git trees, rather than in a separate database. There are known problems converting to NoteDB from any version prior to 2.16, which is why we need the initial upgrade steps. This is this slowest portion of the upgrade process, and the one most prone to unforeseen issues despite extensive pre-testing. We will be creating snapshots to facilitate quick fallback if required. Once this is complete, we can move to the 3.x series and incrementally upgrade from 3.0 to 3.2. Timing wise we expect the first 2.13 -> 2.16 phase to be completed in about 8 hours. Next we'll start the notedb migration and let that run overnight. When we return the next morning the NoteDB migration should be complete and we can finish with the last set of upgrades to 3.2. Hopefully that means we're done in the first half of the outage window, but if we have problems, things are slower than we expect, or if we have to work through any details after finishing the upgrade we'd like to have the extra buffer in place. Some important things to know: * Gerrit's web UI will be changing. We don't have a lot of control over this. As a result we'll lose some existing CI result rendering niceness that we have on 2.13 (in particular the summary table and CI results toggle will go away). We hope that once we've upgraded we can investigate solving this through Gerrit plugins. * All of our existing integrations with launchpad and storyboard may not be fully functional. The changes to the database system impact aspects of how we lookup user details in order to update bugs. * Gerrit 3.2 requires git 2.2.0 or newer to use the Change ID commit hook. This may be a problem for RHEL/CentOS 7 users. * This upgrade will fix the openssh problems with host keys that Fedora 33 users have experienced. * Q&A * If this upgrade isn't perfect why are we doing it anyway? We've come to the realization that if we don't make imperfect progress we'll never make any progress. We have decided that the benefits outweigh the known drawbacks and we'll do our best to work on those issues after the upgrade. Why now? This is long overdue and the quiet time at the end of the year tends to be a good time for these big upgrades Can I install my favorite Gerrit plugin? We're hoping to complete the upgrade then start looking at things like this. One big reason for it is finishing the upgrade simplifies the number of Gerrit docker images we need to care for. Will my third party CI system continue to work? We expect they will. We know Zuul is tested against newer Gerrit already. How can I help? Once the upgrade is complete you'll want to confirm the basic functionality you rely on is there. We know there will be differences or missing features. Patience as we figure out how to address those on a new Gerrit installation is much appreciated. If you're interested in hacking on Java and Javascript we'd love help with the plugins necessary to address the known problems. You should be able to build this out locally without any special access. Please let us know if you are interested and we can help you bootstrap. Finally, we're continuing to test the upgrade and trying to sort out if we can get this upgrade completed in a shorter period of time. We'll keep you up to date if further testing forces us to postpone or if the outage window timeframe changes at all. As always feel free to send your questions our way and we'll do our best to answer them. On behalf of the OpenDev team, Clark

2 2

October 20 Gerrit Outage Update
by Ian Wienand 21 Oct '20

21 Oct '20

As of this mail, Gerrit access has been restored. Please read on for important information, especially around change verification. Background ----------- On 2020-10-20 at 01:30 a user unexpectedly added a workflow approval to a change that they were not expected to have access to. At 02:06 UTC an alert was raised via IRC. Administrators found the account had added themselves to a core group and made the +W vote. The account was disabled, and removed from the groups it had added itself to by 02:55 UTC. Administrators began to analyse the situation and Gerrit was taken offline at 04:02 UTC to preserve state and allow for analysis. >From this time, administrators were working on log collection and analysis, along with restoring backups for comparison purposes. By around 08:45 UTC it was clear that the privilege escalation had been achieved by gaining control of a Launchpad SSO account with Gerrit administrator privileges. By this time, we had ruled out software vulnerabilities. Logs showed the first unauthorized access of the administrator account in Gerrit on 2020-10-06. Communication with Launchpad admins agrees with this analysis. We saw one session opened as the administrator user to StoryBoard on this same day, but logs show no data was modified or hidden stories viewed. Analysis has been performed on the Gerrit database and git trees from October 1st, pre-dating any known unauthorized access. Access was restored at around 2020-10-21 00:00 UTC Outcomes ----------- The following has been verified: The administrator account used has been disabled and credentials updated We have verified that all group and user addition/removals since Oct 1 are valid. The only invalid additions were made by the compromised administrator account to add a single user account to the Administrators group; and then that account added itself to another known group. The account given administrator privilege has been removed from the groups it added itself to and is disabled. There is no evidence of any unauthorized access via methods other than Gerrit HTTP and Gerrit SSH access. No commits have been pushed to git trees bypassing code review. Every git tree has been compared to the Oct 1 version and all commits have been correctly inserted via Gerrit changes. The version of Gerrit we use stores HTTP API passwords in plain-text. We know that a limited number of passwords were gathered via the HTTP API and it is possible passwords were gathered via the database. We thus have assumed that all HTTP API passwords have been disclosed. This password needs to be explicitly enabled by users, and many users do not have it enabled. Remediation ----------- This leaves us with the following remediation actions: Users should double-check their Launchpad recent activity at https://login.launchpad.net/activity for any suspicious logins. If found, please notify the OpenDev admins in Freenode #opendev and Launchpad admins in #launchpad immediately. All HTTP API passwords have been cleared. If you push changes via HTTPS (instead of typical SSH), are a gertty user, or run a CI system or something else that communicates with the Gerrit HTTP API, you will need to regenerate a password. Any SSH keys added to accounts since 2020-10-01 have been removed. This affects only a limited number of accounts. This is done in an abundance of caution, and we do not believe any accounts had unauthorized SSH keys added We should audit all changes for projects since 2020-10-01. We have no evidence that any account had its ssh keys compromised, thus we can rule out any unauthorized changes being uploaded via SSH. However we can not conclusively rule out that compromised HTTP API passwords were used to push a change through Gerrit. For example, a change could be uploaded that looks like it came from a user, or the API key of a core team member may have been used to approve a change without authorization. Given our extensive analysis we consider it exceedingly unlikely that this vector was used. We have had no notifications of users seeing unexpected changes either uploaded by them, or approved by them in projects they work on. This said, we believe it is important to inform the community of this very unlikely, but still possible, vulnerability of the source code. To this end, we have prepared a list of all changes from the known affected period which should be audited for correctness. These are available at https://static.opendev.org/project/opendev.org/gerrit-diffs/ Team members should browse these changes and make sure they were correctly approved in Gerrit. If any change looks suspicious you should notify OpenDev administrators in Freenode #opendev immediately. Further actions ---------------- We are planning the following for the short term future: The Opendev administrators will be looking at alternative models for Gerrit admin account management. We are already well into planning and testing a coming upgrade to a version of Gerrit which does not store plain-text API keys. Longer term, we've written a spec for replacing Launchpad SSO as our authentication provider. We thank you for your patience during this trying time, and we look forward to returning to supporting the community doing what it does best -- working together to create great things.

1 0

October 20 Gerrit Outage
by Clark Boylan 20 Oct '20

20 Oct '20

Hello everyone, By now most of you have probably noticed that we took Gerrit offline recently. The reason for that is we believe an admin account in Gerrit was compromised allowing an attacker to escalate privileges within Gerrit. Around 02:00 UTC October 20 suspicious review activity was noticed, and we were made aware of it shortly afterwards. The involved account was disabled and removed from privileged Gerrit groups. After further investigation we decided that we needed to stop the service, this happened at about 04:00 UTC. After the service was stopped we shifted focus to identifying the source of the issue as well as investigating impact. We believe this originated on October 6th with at least two compromised Ubuntu One accounts. One of which was a Gerrit admin account. These accounts, like the one that initially tipped us off, have been dealt with at this point. In order to evaluate impact we are using backups from October 1 to find configuration, database, and git repo changes that have been made. We have identified 97 accounts that updated ssh keys after that point in time. These ssh keys are being removed as we can't be sure the changes were valid changes made by the user. If you are one of these users you will need to add your key(s) back in. We will also attempt to reach out to the affected users directly by email soon. We will be checking openid urls and group membership changes as well. We will determine what actions make sense for these items once we have evaluated the impact to them. All Gerrit HTTP API tokens will be deleted. You will need to generate new ones if you are an API user. Sorry, gertty fans. On the git repo side of things there are a few things that we will need to check. Using our October 1 state we will generate lists of commits that have landed since then for each branch on each repo. We will verify that the latest commit can reach the last known good commit in the git DAG. For non merge commits we will also correlate these to Gerrit changes. We will then ask that you help us by verifying the commits on your projects are as reviewed and not malicious. We will also need to check git tags which should all be signed and can be verified that way. This is a good reminder to check activity on your online accounts and identities for anything unexpected. We understand that an inaccessible Gerrit is not fun. We are trying to go as quickly as we can while also not sacrificing caution and care. Clark

1 0

Removing fedora-30 nodes from Nodepool and Zuul
by Clark Boylan 18 Sep '20

18 Sep '20

The OpenDev team is planning to remove the fedora-30 node label from Nodepool and Zuul in the near future. Fedora 30 has been EOL for a couple months now and we have Fedora 32 nodes we want to get people onto. Before this can happen one of the things we need to do is bump the fedora-latest nodeset from fedora-30 to fedora-32. We plan to make this change to fedora-latest on September 22, 2020. If you use the fedora-30 label direction (not via fedora-latest) it would be great if you can start switching to fedora-32 or fedora-latest instead. Once we've removed these uses of the label we'll remove the image entirely from Nodepool and Zuul. Clark

1 0

Short review.opendev.org Outage for Project Renames Friday, July 24
by Clark Boylan 22 Jul '20

22 Jul '20

We will be performing project renames on the Gerrit instance at review.opendev.org July 24, 2020 around 15:00 UTC. This requires a short outage of the service so that the database can be updated safely. Thank you for your patience and let us know if you have any questions, Clark

1 0

Deprecating and Removing https://review.opendev.org/p/ git repo mirrors
by Clark Boylan 20 Jul '20

20 Jul '20

The review.opendev.org Gerrit server's /p/ git mirrors are being deprecated and will be removed in the future. The reason for this is newer versions of Gerrit repurpose the /p/ path for PolyGerrit UI project dashboards. We have historically hosted a set of git repo mirrors at this path and CI jobs (and others) have used it. In order to avoid any trouble post Gerrit upgrades we would like to deprecate then disable this mirror as soon as possible. If you are using https://review.opendev.org/p/ or https://review.openstack.org/p/ urls you can replace them with https://opendev.org/ instead. This is our current canonical set of Gerrit git repo mirrors. If you do this you shouldn't notice any change when we disable the /p/ mirrors or when we upgrade Gerrit. In particular you will want to check your CI configurations for both server configs (in the case of Zuul, Jenkins, etc) and job content (devstack-gate GIT_BASE and other places that clone repositories). On July 31 we will disable the /p/ path to help shake out any remaining users to help them update in advance of any Gerrit upgrades. Thank you, Clark

1 0

Temporarily blocking AS4837 (China Unicom) from Git
by Jeremy Stanley 10 Jul '20

10 Jul '20

Due to a flood of connections from random addresses within AS4837 (China Unicom), we have temporarily blocked access to the opendev.org Git service from all prefixes they're announcing in BGP. This is not a decision we took lightly, but whatever distributed system these requests are coming from is not limited to any easily classified network set. We're also seeing similar activity from AS4134 (ChinaNet), and so may have to add filtering for their prefixes too in the near term. Hopefully we can lift this block once the problematic traffic subsides, or as soon as we implement better alternative mitigations. -- Jeremy Stanley

1 1

Short review.opendev.org Outage for Project Renames Friday, June 12
by Clark Boylan 10 Jun '20

10 Jun '20

We will be performing project renames on the Gerrit instance at review.opendev.org June 12, 2020 around 21:00 UTC. This requires a short outage of the service so that the database can be updated safely. Thank you for your patience and let us know if you have any questions, Clark

1 0

Etherpad and Gerrit maintenance on 2020-04-10
by corvus＠inaugust.com 10 Apr '20

10 Apr '20

Hi, Tomorrow, April 10, 2020, the OpenDev team will be performing minor updates to two services. * Gerrit restart for https://review.opendev.org/ at 16:00 UTC. This should be a momentary outage to restart the service in a container. We will maintain the same version in the same configuration; this is a preparatory step in order to perform future version upgrades more easily. Other than a short outage of about a minute, there should be no service impact or visible changes, but since the environment is changing, we wanted to announce this in case any problems are detected. * Etherpad migration and domain change at 17:00 UTC. We will migrate the data currently hosted on etherpad.openstack.org to a new server, also running in a container, on etherpad.opendev.org. We will set up redirects so that on completion of the migration, all etherpad.openstack.org URLs will redirect to the same pages on etherpad.opendev.org. https://etherpad.opendev.org/ will become the new canonical service URL. This also includes an upgrade to the latest release of Etherpad Lite (1.8.0). This involves a database migration as well. The service will be unavailable for the duration of this migration (expected to be about 30 minutes). Please report any issues in #opendev on Freenode or to the service-discuss(a)lists.opendev.org mailing list. Note that this message is being sent to service-announce and several other lists. Future announcements of this type will only be sent to service-announce, so please sign up[1] for that list if you haven't already. [1] http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce -Jim

1 0

Using #opendev on Freenode for OpenDev Infrastructure Discussions
by Clark Boylan 18 Mar '20

18 Mar '20

Hello all, Due to OpenDev's origin in OpenStack, the team has been using the existing #openstack-infra on Freenode for most IRC communication. Recently we decided it was time to transition to a more neutral and appropriate #opendev on Freenode. The old #openstack-infra channel will remain for OpenStack specific issues, but going forward we expect #opendev to be used for OpenDev services. Feel free to join us on #opendev to discuss the tools we use to develop our software. We understand this is a time of transition and will try to direct questions and comments to the appropriate venue as necessary. Clark

1 0

review.opendev.org Downtime March 20, 2020 at 1400UTC
by Clark Boylan 18 Mar '20

18 Mar '20

Hello all, The OpenDev team is planning to take a short downtime March 20, 2020 from 14:00UTC to 15:00UTC. This outage will enable us to rename a few projects and redeploy Gerrit using containers. The container based deployment is an important first step in our longer term plan for upgrading Gerrit to a modern version. We recognize that this will be a disruption during an already disrupted week for many. We feel this is worthwhile as it fits into project release schedules and sets us up well for the future. Thank you for your patience. Feel free to ask us any questions you may have on this thread or in #opendev on Freenode. Clark

1 0

Long overdue cleanups of Zuulv2 compatibility base configs
by Clark Boylan 05 Jun '19

05 Jun '19

As part of our transition to Zuulv3 a year and a half ago, we carried over some compatibility tooling that we would now like to clean up. Specifically, we add a zuul-cloner (which went away in zuulv3) shim and set a global bindep fallback file value in all jobs. Zuulv3 native jobs are expected to use the repos zuul has precloned for you (no zuul-cloner required) as well as supply an in repo bindep.txt (or specify a bindep.txt path or install packages via some other method). This means that we should be able to remove both of these items from the non legacy base job in OpenDev's zuul. The legacy base job will continue to carry these for you so that you can write new native jobs over time. We have two changes [0][1] ready to go for this; however, due to the potential for disruption we would like to give everyone some time to test and prepare for this change. Fungi has a change to base-test [2] which will remove the zuul-cloner shim. Once this is in you can push "Do Not Merge" changes to your zuul config that reparent your tests from "base" to "base-test" and that will run the jobs without the zuul-cloner shim. Testing the bindep fallback removal is a bit more difficult as we set that in zuul's server config globally. What you can do is check your jobs' job-output.txt log files for usage of "bindep-fallback.txt". Our current plan is to merge these changes on June 24, 2019. We will be around to help debug any unexpected issues that come up. Jobs can be updated to use the "legacy-base" base job instead of the "base" base job if they need to be reverted to the old behavior quickly. Finally, Fungi did some excellent spelunking through history to understand how we got here. If you are curious you can find more details at: http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006881.html. [0] https://review.opendev.org/656195 [1] https://review.opendev.org/663151 [2] https://review.opendev.org/663135 Clark

1 0