- service-discuss - lists.opendev.org

Storyboard Needs and Alternatives
by Clark Boylan 01 Dec '22

01 Dec '22

Hello, There have apparently been disparate discussions among different projects on whether or not they want to continue using Storyboard and what alternatives they should consider. Having these discussion is healthy, but I think we should try to have them in the open with a broad audience to ensure we've considered all the various moving parts involved. I'll do my best to kick start that discussion here in this thread. The Problem(s) * Storyboard is slow. * Storyboard's deployment is aging and needs to be updated. * Storyboard is largely unmaintained at this point. * Web crawlers/indexers struggle to index Storyboard content because it is rendered client side via API calls. These first two problems are related. There are apparently fixes for known performance issues and other long-standing bugs that have yet to be deployed as we need to update the deployment platform to a newer version of python. Options * Update the Storyboard deployment and see if that improves things enough to make people happy. * Migrate projects onto one or more different issue trackers. The first option would almost certainly need someone to step up and help out. But before we do that it is probably worthwhile to solicit feedback indicating people would want to continue to use storyboard if its performance and other usability issues are addressed. The second option opens us to a tremendous number of different possibilities. Launchpad, Gitea, GitHub, Bugzilla, Trac, Redmine, etc. I think for this discussion to be productive we should avoid digging too deeply into these options now. And instead determine if Storyboard is something that we as a community want to try and keep alive. Once we've answered that question we can shift gears to this option if necessary. We would love to hear your feedback. Additional user issues with storyboard, interest in updating/maintaining Storyboard (or the opposite), and thoughts on whether or not addressing known problems with the service would be sufficient to make people want to keep using it are all helpful. The goal here isn't to try and convince anyone they should use Storyboard. Instead we're trying to understand if there are those that would like to keep using Storyboard, and if so, what effort would be necessary to make that possible. Once we've sorted this out we can have a followup discussion on what would be necessary to support projects that don't want to use Storyboard any longer. Clark

6 9

Team Meeting Agenda for November 29, 2022
by Clark Boylan 29 Nov '22

29 Nov '22

Hello, We will meet November 29, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221129) *** https://review.opendev.org/q/topic:prod-bastion-group *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221129) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20221129) *** https://etherpad.opendev.org/p/mm3migration *** Ready to add a new jammy node to our inventory and let Ansible do its thing. **** https://review.opendev.org/c/opendev/system-config/+/865361 ** Quo vadis storyboard (frickler 20221129) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Followup sent to hash out options for OpenDev. ** Nova server rescue behavior in vexxhost (clarkb 20221129) *** Should we consider building our own rescue image for non BFV instances? ** Gerrit 3.6 (ianw 20221129) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.6 ** acme.sh failures (clarkb 20221129) *** Recently we've had a number of certs fail to renew and our checker is complaining about them. *** acme.sh did do a recent release and has since updated their default from rsa to ecdsa. * Open discussion

1 0

Team Meeting Agenda for November 22, 2022
by Clark Boylan 22 Nov '22

22 Nov '22

We will meet with this agenda on November 22, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221115) *** https://review.opendev.org/q/topic:prod-bastion-group *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221115) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20221108) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** Testing updated images with updated packages is underway. ** Updating base python docker images to use `pip wheel` (clarkb 20221115) *** The change to modify this has landed and new images have been promoted. We don't expect problems but please call them out if seen. ** Quo vadis storyboard (frickler 20221115) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** It has been two weeks since we asked for additional feedback and haven't received any. We should begin making plans with the feedback that we were given. ** Nova server rescue behavior in vexxhost (clarkb 20221115) *** Should we consider building our own rescue image for non BFV instances? * Open discussion

1 0

Team Meeting Agenda for November 15, 2022
by Clark Boylan 16 Nov '22

16 Nov '22

We will meet with this agenda on Tuesday, November 15, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221115) *** https://review.opendev.org/q/topic:prod-bastion-group *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221115) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20221108) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** Currently debugging why updated images aren't working as expected. Appears to be some problem with django. ** Updating base python docker images to use `pip wheel` (clarkb 20221115) *** Will aim to land this change soon, then double check images built with new assemble process are happy. ** Etherpad docker container logs growth (clarkb 20221115) *** https://review.opendev.org/c/opendev/system-config/+/864060 ** Quo vadis storyboard (frickler 20221115) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** No new movement on the mailing list. Maybe give it another week then we can consider feedback collected? ** Nova server rescue behavior in vexxhost (clarkb 20221115) *** There are image settings that need to be set for rescue images. A dedicated rescue image would also be able to set a non colliding disk label. ** Goodbye birdsite, hello tooters? (frickler 20221115) *** Twitter seems to be on a downward spiral and we should consider stopping using it *** OIF already has an account on fosstodon.org, also ianw and frickler *** Do we want to move the opendevinfra acc there, too? *** How much work is needed in terms of tooling? Any volunteers? * Open discussion

1 0

Team Meeting Agenda for November 8, 2022
by Clark Boylan 08 Nov '22

08 Nov '22

Hello, We will meet with this agenda on November 8, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221108) *** https://review.opendev.org/q/topic:prod-bastion-group *** https://review.opendev.org/q/topic:bridge-ansible-venv *** https://review.opendev.org/c/opendev/system-config/+/863564 *** https://review.opendev.org/c/opendev/system-config/+/863568 ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221101) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20221108) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** https://review.opendev.org/c/opendev/system-config/+/860157 Forking the upstream images due to lack of attention on issues and PRs we've filed ** Updating base python docker images to use `pip wheel` (clarkb 20221108) *** Pip changed how it addresses wheels in its wheel cache which broke our assemble script's ability to build packages on our image builds. *** I've filed a bug against pip for this and pushed a PR to fix it. However, upstream pip says we shouldn't rely on the wheel cache like this and should use `pip wheel` instead. *** https://github.com/pypa/pip/issues/11527 *** https://github.com/pypa/pip/pull/11538 *** https://review.opendev.org/c/opendev/system-config/+/862152 ** Etherpad docker container logs growth (clarkb 20221101) *** Need change to log to syslog instead similar to other services. ** Quo vadis storyboard (frickler 20221107) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Many people interested in moving away from storyboard *** No volunteers to update our installation *** How long can we/do we want to commit to running an outdated installation on an EOL OS? ** Nova server rescue behavior in vexxhost (clarkb 20221108) *** Clarkb has tested this and the behaviors are surprising *** Rescuing a normal disk instance results in the instance boot with the rescue image kernel and the rescued instance's root device mounted on / instead of the rescue image mounted to /. *** Rescuing a BFV instance fails. First with the default microversion because old Nova API doesn't support this. When using microversion 2.88 it also fails, but only after attempting to rescue. **** The instance goes into an error state due to "cannot be rescued: Driver Error: Cannot access storage file" **** Attempting to unrescue the instance also fails because you cannot unrescue an instance that is in an error state * Open discussion

1 0

Team Meeting Agenda for November 1, 2022
by Clark Boylan 01 Nov '22

01 Nov '22

== Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221101) *** https://review.opendev.org/q/topic:prod-bastion-group *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221101) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/c/opendev/system-config/+/862835/ Disable phased package updates ** Removing snapd (clarkb 20221101) *** https://review.opendev.org/c/opendev/system-config/+/862834 ** Mailman 3 (clarkb 20221101) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** https://review.opendev.org/c/opendev/system-config/+/860157 Forking the upstream images due to lack of attention on issues and PRs we've filed ** Updating base python docker images to use `pip wheel` (clarkb 20221101) *** Pip changed how it addresses wheels in its wheel cache which broke our assemble script's ability to build packages on our image builds. *** I've filed a bug against pip for this and pushed a PR to fix it. However, upstream pip says we shouldn't rely on the wheel cache like this and should use `pip wheel` instead. *** https://github.com/pypa/pip/issues/11527 *** https://github.com/pypa/pip/pull/11538 *** https://review.opendev.org/c/opendev/system-config/+/862152 ** Etherpad docker container logs growth (clarkb 20221101) *** Need change to log to syslog instead similar to other services. * Open discussion

1 0

RE: Storyboard Needs and Alternatives
by Michael Johnson 27 Oct '22

27 Oct '22

Sorry this post may be out of the existing thread, I wasn't aware of this list (just the service-announce) until now, so I just signed up. Here are my issues with Storyboard: 1. It is unmaintained and even contributed patches get ignored (6+ months with no review). 2. It is slow (frankly it seems like it is getting slower over time). 3. The search is horrible. 4. It is not flexible enough to work with teams processes. a. For example, we can't link duplicate stories, so it's a pain to piece all of the stories together. b. There is no lifecycle status that can be used for search, etc. Triaged, confirmed, etc. c. Object storage never became a thing, so no screenshots, no logs, etc. d. There are no "My bugs" quick links, etc. so stuck with the search capability. e. Tags are free-form, so you get "RFE" "rfe" "feature", etc. that are all different searches. f. No way to search by "authored by" g. No "importance" fields, again making search and sorting hard without creating ordered lists. 5. Not all projects are on Storyboard, so it is a huge pain to move a story from one project to the other. 6. The Storyboard community was very opinionated on "how" it should be used and was not open to enabling alternate workflows/processes. a. This causes problems when community members have downstream tools/processes they need to align stories with. b. This made maintaining stories/bugs very difficult and people stopped triaging stories. These are the top of head issues I have with Storyboard. I know from the Octavia team perspective, we have tried[1] to work with and help Storyboard become a thing, but it just isn't happening. At this point I think the Octavia team is favoring going back to launchpad. I think given how Storyboard has gone, I would not be in favor of an alternate tool unless the community was 100% onboard to all move inside a cycle. Michael [1] https://etherpad.opendev.org/p/storyboard-issues

1 0

Team Meeting Agenda for October 25, 2022
by Clark Boylan 25 Oct '22

25 Oct '22

We will meet with this agenda on October 25, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221025) *** Move ansible to a venv. *** Replace bridge host once existing host is running ansible out of a venv *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? *** Are console log files no longer being written? Did base-jobs playbook get updated too? *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221025) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** gitea-lb02 was launched using a Jammy image and is now in prod **** Used upstream image published by Ubuntu uploaded to vexxhost and converted to raw. This allows for max compatibility with bfv on vexxhost (gitea-lb02 is not bfv). **** Needed to use an up to date paramiko out of a virtualenv to run launch node as Jammy doesn't do SSH RSA + SHA1. ** Removing snapd (clarkb 20221025) *** In the past we removed snapd as we didn't use it for anything. *** Stopped removing snapd in order to install kubectl. *** Should we go back to removing snapd globally and only add it back in where needed? ** Mailman 3 (clarkb 20221025) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** https://review.opendev.org/c/opendev/system-config/+/860157 Forking the upstream images due to lack of attention on issues and PRs we've filed ** Switch base job nodeset to ubuntu-jammy (frickler 20221007) *** Switching the default base job's nodeset to Ubuntu Jammy on October 25 (day of the meeting). ** Updating base python docker images to use `pip wheel` (clarkb 20221025) *** Pip changed how it addresses wheels in its wheel cache which broke our assemble script's ability to build packages on our image builds. *** I've filed a bug against pip for this and pushed a PR to fix it. However, upstream pip says we shouldn't rely on the wheel cache like this and should use `pip wheel` instead. *** https://github.com/pypa/pip/issues/11527 *** https://github.com/pypa/pip/pull/11538 *** https://review.opendev.org/c/opendev/system-config/+/862152 ** Dropping python 3.8 base images (clarkb 20221025) *** To make room for python 3.11 it would be good to drop our python3.8 images. *** https://review.opendev.org/q/status:open+(topic:use-new-python+OR+topic:doc… ** iweb cloud going away at the end of the year (clarkb 20221025) *** We've been told this will need to be shutdown by the end of the year. We are good to use it until then. *** Possibility for using cloudstack resources as backfill, but would require a nodepool driver. ** Etherpad docker container logs growth (clarkb 20221025) *** The etherpad docker container's logs grow over time when we don't update the container. *** Can/should we add the syslogging for containers to this service? * Open discussion

1 0

Cancelling the Team Meeting on October 18, 2022
by Clark Boylan 18 Oct '22

18 Oct '22

Hello, We've decided to cancel the team meeting tomorrow (October 18, 2022) as a number of us are participating in the PTG and that is consuming a fair bit of time. Feel free to bring up items of concern either here on the mailing list or in IRC in #opendev on OFTC. We'll meet at our usual time and location on October 25, 2022. Clark

1 0

OpenSSH 8.8, RSA keys, and Gerrit
by Clark Boylan 14 Oct '22

14 Oct '22

Hello, About a year ago Fedora 33 released and gave us a preview of OpenSSH's sha1 + RSA key deprecation fallout. Fedora 33 users noticed they could no longer use SSH RSA keys to connect to our Gerrit at review.opendev.org. This happens because Fedora 33's OpenSSH packaging has deprecated sha1 hashes for RSA, and despite both the client and server supporting rsa-sha2-* variants they couldn't negotiate their use between them. OpenSSH 8.8 released recently and did similar in the upstream software which means users with up to date OpenSSH installations are noticing similar problems (Arch Linux for example). There are a couple of workarounds that you can use. Probably the best option is to use an ed25519 or ecdsa key with our Gerrit. Modern clients and our Gerrit SSHD negotiate these keys without issue. Less optimal is to manually re-enable the use of the ssh-rsa hash, but we recommend against this as your software providers have decided this is no longer secure enough. On our end we've brought this up with the MINA SSHD devs [0] with the hope that the SSH implementation that Gerrit uses can be updated to negotiate the sha2 hashes properly. Also, the rsa-sha2 RFC indicates [1] clients may fallback to a sha2 variant instead of the sha1 variant which would workaround MINA's lack of support for negotiation in the protocol. If you are an OpenSSH>=8.8 or Fedora>=33 user you might consider filing bugs against your ssh clients to change the default fallback to a sha2 variant on your platforms. [0] https://issues.apache.org/jira/browse/SSHD-1141 [1] https://datatracker.ietf.org/doc/html/rfc8332#section-3.3 Hopefully I've put enough keywords in this email that the various search engines will index it, and the next time someone runs into these problems they'll find this explanation. Clark

1 1