- service-announce - lists.opendev.org

October 20 Gerrit Outage
by Clark Boylan 20 Oct '20

20 Oct '20

Hello everyone, By now most of you have probably noticed that we took Gerrit offline recently. The reason for that is we believe an admin account in Gerrit was compromised allowing an attacker to escalate privileges within Gerrit. Around 02:00 UTC October 20 suspicious review activity was noticed, and we were made aware of it shortly afterwards. The involved account was disabled and removed from privileged Gerrit groups. After further investigation we decided that we needed to stop the service, this happened at about 04:00 UTC. After the service was stopped we shifted focus to identifying the source of the issue as well as investigating impact. We believe this originated on October 6th with at least two compromised Ubuntu One accounts. One of which was a Gerrit admin account. These accounts, like the one that initially tipped us off, have been dealt with at this point. In order to evaluate impact we are using backups from October 1 to find configuration, database, and git repo changes that have been made. We have identified 97 accounts that updated ssh keys after that point in time. These ssh keys are being removed as we can't be sure the changes were valid changes made by the user. If you are one of these users you will need to add your key(s) back in. We will also attempt to reach out to the affected users directly by email soon. We will be checking openid urls and group membership changes as well. We will determine what actions make sense for these items once we have evaluated the impact to them. All Gerrit HTTP API tokens will be deleted. You will need to generate new ones if you are an API user. Sorry, gertty fans. On the git repo side of things there are a few things that we will need to check. Using our October 1 state we will generate lists of commits that have landed since then for each branch on each repo. We will verify that the latest commit can reach the last known good commit in the git DAG. For non merge commits we will also correlate these to Gerrit changes. We will then ask that you help us by verifying the commits on your projects are as reviewed and not malicious. We will also need to check git tags which should all be signed and can be verified that way. This is a good reminder to check activity on your online accounts and identities for anything unexpected. We understand that an inaccessible Gerrit is not fun. We are trying to go as quickly as we can while also not sacrificing caution and care. Clark

1 0

Removing fedora-30 nodes from Nodepool and Zuul
by Clark Boylan 18 Sep '20

18 Sep '20

The OpenDev team is planning to remove the fedora-30 node label from Nodepool and Zuul in the near future. Fedora 30 has been EOL for a couple months now and we have Fedora 32 nodes we want to get people onto. Before this can happen one of the things we need to do is bump the fedora-latest nodeset from fedora-30 to fedora-32. We plan to make this change to fedora-latest on September 22, 2020. If you use the fedora-30 label direction (not via fedora-latest) it would be great if you can start switching to fedora-32 or fedora-latest instead. Once we've removed these uses of the label we'll remove the image entirely from Nodepool and Zuul. Clark

1 0

Short review.opendev.org Outage for Project Renames Friday, July 24
by Clark Boylan 21 Jul '20

21 Jul '20

We will be performing project renames on the Gerrit instance at review.opendev.org July 24, 2020 around 15:00 UTC. This requires a short outage of the service so that the database can be updated safely. Thank you for your patience and let us know if you have any questions, Clark

1 0

Deprecating and Removing https://review.opendev.org/p/ git repo mirrors
by Clark Boylan 20 Jul '20

20 Jul '20

The review.opendev.org Gerrit server's /p/ git mirrors are being deprecated and will be removed in the future. The reason for this is newer versions of Gerrit repurpose the /p/ path for PolyGerrit UI project dashboards. We have historically hosted a set of git repo mirrors at this path and CI jobs (and others) have used it. In order to avoid any trouble post Gerrit upgrades we would like to deprecate then disable this mirror as soon as possible. If you are using https://review.opendev.org/p/ or https://review.openstack.org/p/ urls you can replace them with https://opendev.org/ instead. This is our current canonical set of Gerrit git repo mirrors. If you do this you shouldn't notice any change when we disable the /p/ mirrors or when we upgrade Gerrit. In particular you will want to check your CI configurations for both server configs (in the case of Zuul, Jenkins, etc) and job content (devstack-gate GIT_BASE and other places that clone repositories). On July 31 we will disable the /p/ path to help shake out any remaining users to help them update in advance of any Gerrit upgrades. Thank you, Clark

1 0

Temporarily blocking AS4837 (China Unicom) from Git
by Jeremy Stanley 10 Jul '20

10 Jul '20

Due to a flood of connections from random addresses within AS4837 (China Unicom), we have temporarily blocked access to the opendev.org Git service from all prefixes they're announcing in BGP. This is not a decision we took lightly, but whatever distributed system these requests are coming from is not limited to any easily classified network set. We're also seeing similar activity from AS4134 (ChinaNet), and so may have to add filtering for their prefixes too in the near term. Hopefully we can lift this block once the problematic traffic subsides, or as soon as we implement better alternative mitigations. -- Jeremy Stanley

1 1

Short review.opendev.org Outage for Project Renames Friday, June 12
by Clark Boylan 10 Jun '20

10 Jun '20

We will be performing project renames on the Gerrit instance at review.opendev.org June 12, 2020 around 21:00 UTC. This requires a short outage of the service so that the database can be updated safely. Thank you for your patience and let us know if you have any questions, Clark

1 0

Etherpad and Gerrit maintenance on 2020-04-10
by corvus＠inaugust.com 10 Apr '20

10 Apr '20

Hi, Tomorrow, April 10, 2020, the OpenDev team will be performing minor updates to two services. * Gerrit restart for https://review.opendev.org/ at 16:00 UTC. This should be a momentary outage to restart the service in a container. We will maintain the same version in the same configuration; this is a preparatory step in order to perform future version upgrades more easily. Other than a short outage of about a minute, there should be no service impact or visible changes, but since the environment is changing, we wanted to announce this in case any problems are detected. * Etherpad migration and domain change at 17:00 UTC. We will migrate the data currently hosted on etherpad.openstack.org to a new server, also running in a container, on etherpad.opendev.org. We will set up redirects so that on completion of the migration, all etherpad.openstack.org URLs will redirect to the same pages on etherpad.opendev.org. https://etherpad.opendev.org/ will become the new canonical service URL. This also includes an upgrade to the latest release of Etherpad Lite (1.8.0). This involves a database migration as well. The service will be unavailable for the duration of this migration (expected to be about 30 minutes). Please report any issues in #opendev on Freenode or to the service-discuss(a)lists.opendev.org mailing list. Note that this message is being sent to service-announce and several other lists. Future announcements of this type will only be sent to service-announce, so please sign up[1] for that list if you haven't already. [1] http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce -Jim

1 0

Using #opendev on Freenode for OpenDev Infrastructure Discussions
by Clark Boylan 18 Mar '20

18 Mar '20

Hello all, Due to OpenDev's origin in OpenStack, the team has been using the existing #openstack-infra on Freenode for most IRC communication. Recently we decided it was time to transition to a more neutral and appropriate #opendev on Freenode. The old #openstack-infra channel will remain for OpenStack specific issues, but going forward we expect #opendev to be used for OpenDev services. Feel free to join us on #opendev to discuss the tools we use to develop our software. We understand this is a time of transition and will try to direct questions and comments to the appropriate venue as necessary. Clark

1 0

review.opendev.org Downtime March 20, 2020 at 1400UTC
by Clark Boylan 18 Mar '20

18 Mar '20

Hello all, The OpenDev team is planning to take a short downtime March 20, 2020 from 14:00UTC to 15:00UTC. This outage will enable us to rename a few projects and redeploy Gerrit using containers. The container based deployment is an important first step in our longer term plan for upgrading Gerrit to a modern version. We recognize that this will be a disruption during an already disrupted week for many. We feel this is worthwhile as it fits into project release schedules and sets us up well for the future. Thank you for your patience. Feel free to ask us any questions you may have on this thread or in #opendev on Freenode. Clark

1 0

Long overdue cleanups of Zuulv2 compatibility base configs
by Clark Boylan 05 Jun '19

05 Jun '19

As part of our transition to Zuulv3 a year and a half ago, we carried over some compatibility tooling that we would now like to clean up. Specifically, we add a zuul-cloner (which went away in zuulv3) shim and set a global bindep fallback file value in all jobs. Zuulv3 native jobs are expected to use the repos zuul has precloned for you (no zuul-cloner required) as well as supply an in repo bindep.txt (or specify a bindep.txt path or install packages via some other method). This means that we should be able to remove both of these items from the non legacy base job in OpenDev's zuul. The legacy base job will continue to carry these for you so that you can write new native jobs over time. We have two changes [0][1] ready to go for this; however, due to the potential for disruption we would like to give everyone some time to test and prepare for this change. Fungi has a change to base-test [2] which will remove the zuul-cloner shim. Once this is in you can push "Do Not Merge" changes to your zuul config that reparent your tests from "base" to "base-test" and that will run the jobs without the zuul-cloner shim. Testing the bindep fallback removal is a bit more difficult as we set that in zuul's server config globally. What you can do is check your jobs' job-output.txt log files for usage of "bindep-fallback.txt". Our current plan is to merge these changes on June 24, 2019. We will be around to help debug any unexpected issues that come up. Jobs can be updated to use the "legacy-base" base job instead of the "base" base job if they need to be reverted to the old behavior quickly. Finally, Fungi did some excellent spelunking through history to understand how we got here. If you are curious you can find more details at: http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006881.html. [0] https://review.opendev.org/656195 [1] https://review.opendev.org/663151 [2] https://review.opendev.org/663135 Clark

1 0