October 2022 - service-discuss

Storyboard Needs and Alternatives
by Clark Boylan 30 Nov '22

30 Nov '22

Hello, There have apparently been disparate discussions among different projects on whether or not they want to continue using Storyboard and what alternatives they should consider. Having these discussion is healthy, but I think we should try to have them in the open with a broad audience to ensure we've considered all the various moving parts involved. I'll do my best to kick start that discussion here in this thread. The Problem(s) * Storyboard is slow. * Storyboard's deployment is aging and needs to be updated. * Storyboard is largely unmaintained at this point. * Web crawlers/indexers struggle to index Storyboard content because it is rendered client side via API calls. These first two problems are related. There are apparently fixes for known performance issues and other long-standing bugs that have yet to be deployed as we need to update the deployment platform to a newer version of python. Options * Update the Storyboard deployment and see if that improves things enough to make people happy. * Migrate projects onto one or more different issue trackers. The first option would almost certainly need someone to step up and help out. But before we do that it is probably worthwhile to solicit feedback indicating people would want to continue to use storyboard if its performance and other usability issues are addressed. The second option opens us to a tremendous number of different possibilities. Launchpad, Gitea, GitHub, Bugzilla, Trac, Redmine, etc. I think for this discussion to be productive we should avoid digging too deeply into these options now. And instead determine if Storyboard is something that we as a community want to try and keep alive. Once we've answered that question we can shift gears to this option if necessary. We would love to hear your feedback. Additional user issues with storyboard, interest in updating/maintaining Storyboard (or the opposite), and thoughts on whether or not addressing known problems with the service would be sufficient to make people want to keep using it are all helpful. The goal here isn't to try and convince anyone they should use Storyboard. Instead we're trying to understand if there are those that would like to keep using Storyboard, and if so, what effort would be necessary to make that possible. Once we've sorted this out we can have a followup discussion on what would be necessary to support projects that don't want to use Storyboard any longer. Clark

6 9

Team Meeting Agenda for November 1, 2022
by Clark Boylan 31 Oct '22

31 Oct '22

== Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221101) *** https://review.opendev.org/q/topic:prod-bastion-group *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221101) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/c/opendev/system-config/+/862835/ Disable phased package updates ** Removing snapd (clarkb 20221101) *** https://review.opendev.org/c/opendev/system-config/+/862834 ** Mailman 3 (clarkb 20221101) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** https://review.opendev.org/c/opendev/system-config/+/860157 Forking the upstream images due to lack of attention on issues and PRs we've filed ** Updating base python docker images to use `pip wheel` (clarkb 20221101) *** Pip changed how it addresses wheels in its wheel cache which broke our assemble script's ability to build packages on our image builds. *** I've filed a bug against pip for this and pushed a PR to fix it. However, upstream pip says we shouldn't rely on the wheel cache like this and should use `pip wheel` instead. *** https://github.com/pypa/pip/issues/11527 *** https://github.com/pypa/pip/pull/11538 *** https://review.opendev.org/c/opendev/system-config/+/862152 ** Etherpad docker container logs growth (clarkb 20221101) *** Need change to log to syslog instead similar to other services. * Open discussion

1 0

RE: Storyboard Needs and Alternatives
by Michael Johnson 26 Oct '22

26 Oct '22

Sorry this post may be out of the existing thread, I wasn't aware of this list (just the service-announce) until now, so I just signed up. Here are my issues with Storyboard: 1. It is unmaintained and even contributed patches get ignored (6+ months with no review). 2. It is slow (frankly it seems like it is getting slower over time). 3. The search is horrible. 4. It is not flexible enough to work with teams processes. a. For example, we can't link duplicate stories, so it's a pain to piece all of the stories together. b. There is no lifecycle status that can be used for search, etc. Triaged, confirmed, etc. c. Object storage never became a thing, so no screenshots, no logs, etc. d. There are no "My bugs" quick links, etc. so stuck with the search capability. e. Tags are free-form, so you get "RFE" "rfe" "feature", etc. that are all different searches. f. No way to search by "authored by" g. No "importance" fields, again making search and sorting hard without creating ordered lists. 5. Not all projects are on Storyboard, so it is a huge pain to move a story from one project to the other. 6. The Storyboard community was very opinionated on "how" it should be used and was not open to enabling alternate workflows/processes. a. This causes problems when community members have downstream tools/processes they need to align stories with. b. This made maintaining stories/bugs very difficult and people stopped triaging stories. These are the top of head issues I have with Storyboard. I know from the Octavia team perspective, we have tried[1] to work with and help Storyboard become a thing, but it just isn't happening. At this point I think the Octavia team is favoring going back to launchpad. I think given how Storyboard has gone, I would not be in favor of an alternate tool unless the community was 100% onboard to all move inside a cycle. Michael [1] https://etherpad.opendev.org/p/storyboard-issues

1 0

Team Meeting Agenda for October 25, 2022
by Clark Boylan 24 Oct '22

24 Oct '22

We will meet with this agenda on October 25, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221025) *** Move ansible to a venv. *** Replace bridge host once existing host is running ansible out of a venv *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? *** Are console log files no longer being written? Did base-jobs playbook get updated too? *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221025) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** gitea-lb02 was launched using a Jammy image and is now in prod **** Used upstream image published by Ubuntu uploaded to vexxhost and converted to raw. This allows for max compatibility with bfv on vexxhost (gitea-lb02 is not bfv). **** Needed to use an up to date paramiko out of a virtualenv to run launch node as Jammy doesn't do SSH RSA + SHA1. ** Removing snapd (clarkb 20221025) *** In the past we removed snapd as we didn't use it for anything. *** Stopped removing snapd in order to install kubectl. *** Should we go back to removing snapd globally and only add it back in where needed? ** Mailman 3 (clarkb 20221025) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** https://review.opendev.org/c/opendev/system-config/+/860157 Forking the upstream images due to lack of attention on issues and PRs we've filed ** Switch base job nodeset to ubuntu-jammy (frickler 20221007) *** Switching the default base job's nodeset to Ubuntu Jammy on October 25 (day of the meeting). ** Updating base python docker images to use `pip wheel` (clarkb 20221025) *** Pip changed how it addresses wheels in its wheel cache which broke our assemble script's ability to build packages on our image builds. *** I've filed a bug against pip for this and pushed a PR to fix it. However, upstream pip says we shouldn't rely on the wheel cache like this and should use `pip wheel` instead. *** https://github.com/pypa/pip/issues/11527 *** https://github.com/pypa/pip/pull/11538 *** https://review.opendev.org/c/opendev/system-config/+/862152 ** Dropping python 3.8 base images (clarkb 20221025) *** To make room for python 3.11 it would be good to drop our python3.8 images. *** https://review.opendev.org/q/status:open+(topic:use-new-python+OR+topic:doc… ** iweb cloud going away at the end of the year (clarkb 20221025) *** We've been told this will need to be shutdown by the end of the year. We are good to use it until then. *** Possibility for using cloudstack resources as backfill, but would require a nodepool driver. ** Etherpad docker container logs growth (clarkb 20221025) *** The etherpad docker container's logs grow over time when we don't update the container. *** Can/should we add the syslogging for containers to this service? * Open discussion

1 0

Cancelling the Team Meeting on October 18, 2022
by Clark Boylan 17 Oct '22

17 Oct '22

Hello, We've decided to cancel the team meeting tomorrow (October 18, 2022) as a number of us are participating in the PTG and that is consuming a fair bit of time. Feel free to bring up items of concern either here on the mailing list or in IRC in #opendev on OFTC. We'll meet at our usual time and location on October 25, 2022. Clark

1 0

OpenSSH 8.8, RSA keys, and Gerrit
by Clark Boylan 13 Oct '22

13 Oct '22

Hello, About a year ago Fedora 33 released and gave us a preview of OpenSSH's sha1 + RSA key deprecation fallout. Fedora 33 users noticed they could no longer use SSH RSA keys to connect to our Gerrit at review.opendev.org. This happens because Fedora 33's OpenSSH packaging has deprecated sha1 hashes for RSA, and despite both the client and server supporting rsa-sha2-* variants they couldn't negotiate their use between them. OpenSSH 8.8 released recently and did similar in the upstream software which means users with up to date OpenSSH installations are noticing similar problems (Arch Linux for example). There are a couple of workarounds that you can use. Probably the best option is to use an ed25519 or ecdsa key with our Gerrit. Modern clients and our Gerrit SSHD negotiate these keys without issue. Less optimal is to manually re-enable the use of the ssh-rsa hash, but we recommend against this as your software providers have decided this is no longer secure enough. On our end we've brought this up with the MINA SSHD devs [0] with the hope that the SSH implementation that Gerrit uses can be updated to negotiate the sha2 hashes properly. Also, the rsa-sha2 RFC indicates [1] clients may fallback to a sha2 variant instead of the sha1 variant which would workaround MINA's lack of support for negotiation in the protocol. If you are an OpenSSH>=8.8 or Fedora>=33 user you might consider filing bugs against your ssh clients to change the default fallback to a sha2 variant on your platforms. [0] https://issues.apache.org/jira/browse/SSHD-1141 [1] https://datatracker.ietf.org/doc/html/rfc8332#section-3.3 Hopefully I've put enough keywords in this email that the various search engines will index it, and the next time someone runs into these problems they'll find this explanation. Clark

1 1

Testing OpenStackSDK/OSC on real clouds
by Artem Goncharov 13 Oct '22

13 Oct '22

Hi all, Last years of my work on OpenStackSDK and CLI on one hand and being representative of a public cloud on the other hand made clear to me that not a single OpenStack cloud behaves like another one, doesn’t matter what you do and how hard you try. And sadly Interop is not helping here at all. The API calls you use to talk with one public cloud are guaranteed to work slightly differently on another one. This is clearly visible through the Storyboard of SDK, IRC and other offline discussions. Lot of customers are capable to work without issues with SDK/CLI on their cloud, while others report the same things are not working for them. This is a relatively complex discussion and Clark reserved a slot in the TC meeting during PTG https://etherpad.opendev.org/p/tc-2023-1-ptg#L38 <https://etherpad.opendev.org/p/tc-2023-1-ptg#L38> to describe his stake in the problem. For a quite some time I am trying to follow an approach of having possibility to run functional tests of both SDK and CLI on real clouds and not only on devstack (on devstack things are working fine but it means as much as nothing at all in real life). On multiple occurrences I was asking for credentials and I got at the moment credentials for 2 public clouds. This already helped both to find issues in SDK (i.e. RADOSGW != Swift) as well as to point clouds on real issues on their side. It should be actually in the interest of those clouds to have some useful checks that give guarantee to their users SDK and CLI (ideally also Ansible/TF) are working fine. This is even a better form of interoperability check than anything else what we have now. Since people in this ML are involved in maintaining resource donations to OpenDev I would like to ask you to consider allocating cloud credentials for the SDK/CLI team to ensure things are running smoothly for your users (pls do not mix with allocating real resources - we need only cloud credentials to test APIs and there are no static resources required). At the moment I maintain those credentials manually until we finally get post-review pipeline created in OpenStack tenant to fully automate those tests. If you have questions or suggestions please feel free to post here, join SDK/CLI team during PTG on Friday 27.10 13:00-15:00 or contact me directly for clarifications. Thanks a lot for support, Artem

2 1

Team Meeting Agenda for October 11, 2022
by Clark Boylan 10 Oct '22

10 Oct '22

We will meet with this agenda on October 11, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** PTG next week * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221011) *** Move ansible to a venv. *** Replace bridge host once existing host is running ansible out of a venv *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? *** https://review.opendev.org/c/opendev/system-config/+/855472 Disable writing console log files on bridge and static **** Needs a second change pushed to opendev/base-jobs/playbooks/infra-prod/setup-keys.yaml *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221011) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Need to retest launching a Jammy node now that user management is updated. ** Mailman 3 (clarkb 20221011) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** https://review.opendev.org/c/opendev/system-config/+/860157 Forking the upstream images due to lack of attention on issues and PRs we've filed ** Switch base job nodeset to ubuntu-jammy (frickler 20221007) *** Ubuntu 22.04.1 is out for some months now and thus can be considered stable, no need to wait a full year like we did with 20.04 *** Patch to switch base-test job: https://review.opendev.org/c/opendev/base-jobs/+/860686/2 *** Plan for further steps? Timetable, which repos to test, announcements *** Devstack currently inherits the base nodeset (via multinode) without pinning * Open discussion

1 0

Team Meeting Agenda for October 4, 2022
by Clark Boylan 03 Oct '22

03 Oct '22

We will meet on October 4, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenStack Release this week. * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221004) *** Move ansible to a venv. *** Replace bridge host once existing host is running ansible out of a venv *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? *** https://review.opendev.org/c/opendev/system-config/+/855472 *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221004) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/c/opendev/system-config/+/860112 Update to launch node to handle jammy hosts ** Mailman 3 (clarkb 20221004) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** https://review.opendev.org/c/opendev/system-config/+/860157 Forking the upstream images due to lack of attention on issues and PRs we've filed ** Gitea connectivity issues (clarkb 20221004) *** Users in certain geographies have reported trouble with Gitea access *** Gitea changed http libraries which broke our ability to trace requests from haproxy -> apache -> gitea **** Partially worked around that by removing X-Forwarded-For headers. * Open discussion

1 0