- service-announce - lists.opendev.org

Upgrading review.opendev.org to Gerrit 3.8 on November 17, 2023
by Clark Boylan 30 Oct '23

30 Oct '23

On November 17, 2023 beginning at 15:30 UTC the OpenDev team will upgrade Gerrit at https://review.opendev.org from Gerrit 3.7 to 3.8. This upgrade shouldn't take more than an hour. We have managed to to keep our Gerrit installation about one major release behind upstream's release cadence, and Gerrit 3.9's impending release means it is time for us to upgrade to 3.8. There are a number of new features, bug fixes, and performance improvements in the new version. The Gerrit project publishes detailed release information [0] which you can review if you are interested in what sorts of changes to expect. If you are interested in the upgrade process you can follow that in our planning document [1]. It is worth noting that if all goes well we expect the total Gerrit downtime to be much shorter than an hour (should only take a few minutes). If we need to revert to Gerrit 3.7, we will need to do offline reindexing of our changes which takes about half an hour. We are using an hour long upgrade window to ensure we have time for downgrades should that become necessary. [0] https://www.gerritcodereview.com/3.8.html [1] https://etherpad.opendev.org/p/gerrit-upgrade-3.8 Clark

1 0

Zuul regex deprecation [action required]
by James E. Blair 21 Sep '23

21 Sep '23

Hi, Zuul is making a change that may require action from developers on projects hosted in OpenDev. Here is the release note regarding the change: > Regular expressions using Python syntax are deprecated in favor of RE2 > syntax instead, for additional speed and safety. Negative lookahead > assertions may be replaced using the negate keyword. See Regular > Expressions[1] for more information. > > If Zuul detects a regular expression using Python syntax that is not > supported by RE2, it will register a configuration syntax > warning. These may be viewed on the configuration errors page in the > web interface. > > A future version of Zuul will remove Python regex support completely > and these warnings will become errors. [1] https://zuul-ci.org/docs/zuul/latest/project-config.html#regex It's easy to see if your project on OpenDev is affected. Each tenant has a page where the current configuration errors and warnings are displayed. Visit any page in Zuul for your tenant, and if there is a bell icon in the top right, click on that. For example, the configuration error page for the OpenStack tenant is here: https://zuul.opendev.org/t/openstack/config-errors On that page, look for the "Regex Deprecation" warnings. You can expand the warning to see exactly the file and line number where the warning is found. Almost all of the issues identified so far have been due to the fact that RE2 does not support negative lookahead assertions. In the past, we used these a lot to say things like "run this job on any branch except master" with regexes like "(?!master)". Because this is such a common pattern, we added a feature in Zuul to make this easy. A configuration like: branches: "^(?!master).*$" can now be written as: branches: regex: "^master$" negate: true This simply negates the specified regex. It's not an exact translation of the original regex, but is slightly improved and to the point: anything but exactly "master". Please take a moment and check to see if your project has instances of this warning, and go ahead and fix it. Even if there are a lot of instances, they are likely to be very similar and correcting them may take only a few minutes. Here are two changes where we update the central configuration that applies to all projects: https://review.opendev.org/c/openstack/project-config/+/893702 https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/893792 These provide examples of how to make changes for several common cases; you may be able to simply copy what was done in those changes. If you have any trouble adapting existing regexes to the new format, please contact us in #opendev to discuss the issue. We may be able to find a simple solution, or if there is another special case that is difficult to handle, we may be able to add support to Zuul to mitigate the issue. If you use "update-regexes" as a topic name, it will help people keep track of progress and possible issues. The Zuul project has not yet set a date for when support for the old syntax will be removed. I expect it to be on the order of months, which should give folks plenty of time to make the necessary changes. But please do so early so that we have time to deal with any issues. Thanks, Jim (on behalf of the OpenDev team)

1 0

Removing OpenDev's Fedora Test Images on September 11, 2023
by Clark Boylan 05 Sep '23

05 Sep '23

OpenDev has been unable to update Fedora images in a timely manner leaving us with Fedora 35 and 36 images in our CI system. Fedora 35 and 36 are long past their EOL date, and we have been planning to remove them [0]. Somewhat recently the Fedora project stopped mirroring packages for these releases which resulted in empty mirrors on our end [1]. The end result is that Fedora nodes haven't been very functional within OpenDev since. We have slowly been cleaning up usage of Fedora nodes and recently removed the fedora-latest nodeset from Zuul [2]. The next step is to remove these images from Nodepool entirely. We will do so on Monday September 11, 2023. Chances are any Fedora jobs are already broken, but this is a good reminder to look into correcting any of those jobs by moving them to another platform. [0] https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… [1] http://mirror.dfw.rax.opendev.org/fedora/releases/36/ [2] https://review.opendev.org/c/opendev/base-jobs/+/892380 Clark

1 0

Updating OpenDev Zuul's default Ansible version to Ansible 8
by Clark Boylan 23 Aug '23

23 Aug '23

Currently OpenDev Zuul's default Ansible version is Ansible 6. Zuul has recently added support for Ansible 8, and is dogfooding Ansible 8 in the Zuul tenant for the Zuul project. Yesterday, the OpenDev team updated the Zuul tenant for the OpenDev project to default to Ansible 8 as well. So far we have seen no problems with jobs run in these two tenants related to the Ansible version change. We would like to update the default Ansible version for all of the Zuul tenants in OpenDev's Zuul installation to Ansible 8 in the near future. The reason for this is Zuul will update the default Ansible version to Ansible 8 in Zuul itself. Not long after that change, support for Ansible 6 in Zuul will be removed. Changing things in OpenDev first allows us to do these changes in a controlled manner and get ahead of any potential problems. Assuming the Zuul and OpenDev tenants continue to run without trouble our plan is to switch the nebulous, pyca, pypa, and vexxhost tenants to Ansible 8 by default on August 28, 2023. Then if things continue to run without trouble, we will complete this change by updating the openstack tenant on September 4, 2023. I had initially thought that we would do a more conservative rollout, but initial data indicates this change may be straightforward so we'd prefer a quicker move. Note that if there are major issues we can revert the setting in our Ansible config for any of these tenants and return to a default to Ansible 6 while the issues get addressed. Issues will need to be addressed as Ansible 6 is EOL and Zuul intends to drop support for this version of Ansible. If you would like to test your jobs before the tenant configuration updates go into effect you can do so by setting `ansible-version: '8'` in the configuration of a job [0], push the resulting change to Gerrit, and then wait for Zuul to run the job and provide you with results. It is not necessary to merge the change; Zuul will speculatively test this setting. If you discover any problems through this process please let us know. [0] https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.ansible-versi… Clark

1 0

opendevorg docker hub images moving to quay.io
by Clark Boylan 24 May '23

24 May '23

The OpenDev team has decided to move hosting for its container images from docker hub to quay.io. Over the weekend we landed changes to update the publication location for our python base images (opendevorg/python-builder, opendevorg/python-base, opendevorg/uwsgi-base). We will continue to move additional images as quickly as possible. While we publish these images primarily for our own consumption, they are entirely open source and others may be using them as well. If you are using these images, you should update the image source locations to quay.io/opendevorg/$imagename. Once we have updated publishing for our last image, we will begin a 30 day countdown, after which we will clean up images on docker hub to remove any confusion over which images are maintained and up to date. We will send notice for a more specific time frame once it is known. Let us know if you have an questions. Clark

1 1

Etherpad outage April 19, 2023 at 22:00 UTC
by Clark Boylan 17 Apr '23

17 Apr '23

The OpenDev team will be taking etherpad services at https://etherpad.opendev.org offline for approximately 90 minutes beginning at 22:00 UTC on Wednesday April 19, 2023. This is necessary for us to replace the server hosting etherpad and perform necessary data migrations. If you have any questions or concerns please reach out to us and let us know. Clark

1 0

review.opendev.org Downtime for Gerrit Upgrade and Project Renames April 6, 2023
by Clark Boylan 14 Mar '23

14 Mar '23

Hello, The OpenDev team intends to have a Gerrit outage of approximately two hours on April 6, 2023 beginning at 22:00 UTC. The purpose of this outage is to upgrade Gerrit from version 3.6 to 3.7. We will also take advantage of the downtime to perform project renames. For more information on the new version of Gerrit you can refer to the upstream release notes [0]. If you would like to rename a project please get it onto our TODO list on the wiki [1] and push changes according to the documented process [2]. We've chosen this time and date because we expect it to be a quiet period, and it occurs after known releases and events like the PTG. We are happy for you to raise and concerns, questions, comments or feedback with this plan; the earlier the better. [0] https://www.gerritcodereview.com/3.7.html [1] https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Upcoming_Project_… [2] https://docs.opendev.org/opendev/infra-manual/latest/creators.html#project-…

1 0

Intent to grant control of x/virtualpdu to OpenStack community
by Jeremy Stanley 01 Mar '23

01 Mar '23

The OpenStack Ironic project relies on VirtualPDU, which is no longer actively developed. Ironic contributors reached out to the VirtualPDU maintainers with an offer to officially assume maintenance responsibilities in order to avoid forking it, and received a response from one (Mathieu Mitchell) who indicated support for that plan. Unfortunately, communication died off shortly thereafter, and the Ironic team has been unable to raise any of the current maintainers to actually add their access to the x/virtualpdu project in OpenDev's Gerrit. We don't have a documented process since this is the first time it's really come up, but I'm officially announcing that I intend to use my administrative permissions as an OpenDev sysadmin to add membership of an OpenStack Ironic team representative to the following Gerrit groups if no objections are raised before Wednesday, March 8: * virtualpdu-core * virtualpdu-release This will effectively grant full control of the x/virtualpdu repository to OpenStack Ironic contributors. Please follow up to the service-discuss(a)lists.opendev.org or openstack-discuss(a)lists.openstack.org mailing list with any concerns as soon as possible, or you can feel free to reach out to me directly by email or in IRC (fungi in the #opendev channel on the OFTC network). -- Jeremy Stanley

1 0

December 2022 OpenDev Activity Update
by Clark Boylan 13 Dec '22

13 Dec '22

I had meant to send one of these every couple of months, but it has been about six months since I sent the last update email. Apologies for the delay. It has been a busy six months; let's dive in. We bootstrapped the ability to deploy new control plane nodes on Ubuntu Jammy. All newly created servers are built on Jammy unless we have a very good reason to use something older. Additionally, we have updated our base Python Docker images to include support for Python 3.11. Python 3.11 is only a few weeks old, but we are able to deploy software on top of it taking advantage of the performance benefits it provides. Gerrit has been upgraded to 3.6.3. This release is just behind 3.7.0 which was released about a month ago. I believe this is the closest we've been to the latest Gerrit release in about a decade. We do plan to upgrade to 3.7.x, but this upgrade appears to be a bit more involved than the last few. In particular, the upgrade itself needs offline re-indexing, we need to port our label configs to the new submit requirements system, and the web UI is now based on Lit instead of polymer. The Gitea cluster is now running 1.17. This update didn't bring any major user facing changes. The next update to 1.18.x should correct a problem with Gitea misidentifying certain files as being "vendored". Prior to the October PTG event our Meetpad deployment was updated to use the latest stable releases of Jitsi Meet, and we applied a modern configuration to our installation. One major feature this brought is a landing page for meetings where users can configure their name and settings. Importantly, this landing page should prevent browsers' auto mute behavior for auto playing video+audio from breaking calls for our users. We believe this was a major source of users not being able to hear others in calls. Mailing lists for lists.opendev.org and lists.zuul-ci.org have been migrated to a new Mailman 3 server. We are using these two list domains to work out issues while we prepare for migrating the rest of our mailing lists to the new server. This should give us a much more friendly web interface for browsing archives with built in search capability as well as easier to use moderation tools for list moderators. Our Zuul deployment has seen a number of improvements. The ZooKeeper cluster backing Zuul has been upgraded to ZooKeeper 3.7 which brings performance improvements. Zuul supports Ansible 6 now, and all of our tenants are running jobs with this version of Ansible. This version of Ansible fixes pipelining which should provide some improvements to job runtime. Finally, we are upgrading Zuul weekly using automated zero downtime rolling restarts. This ensures we're getting the latest and greatest Zuul features and can provide quick and early feedback to Zuul. Looking ahead to 2023, I expect to see a lot more of what we've done over the last six months. Gerrit will be upgraded to 3.7, Gitea will be upgraded to 1.18, more mailing lists will be migrated to Mailman 3, and more services will be deployed on Jammy running with Python 3.11. See you next year! Clark

1 0

review.opendev.org Gerrit 3.6 upgrade December 12th, 20:00UTC
by Ian Wienand 06 Dec '22

06 Dec '22

Hello, We are planning upgrade review.opendev.org to the Gerrit 3.6 release on 12th December, 2022 at around 20:00UTC. This will involve an outage of around 1 hour. There should not be much user-visible change after the upgrade; the major items do not affect our usage. The full release notes are at [1] Thank you, -i [1] https://www.gerritcodereview.com/3.6.html

1 0