From cboylan at sapwetik.org Mon Oct 3 23:09:01 2022 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 03 Oct 2022 16:09:01 -0700 Subject: Team Meeting Agenda for October 4, 2022 Message-ID: <2d7c0fc9-2aca-45fa-881d-94dd1c67d6ea@app.fastmail.com> We will meet on October 4, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenStack Release this week. * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221004) *** Move ansible to a venv. *** Replace bridge host once existing host is running ansible out of a venv *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? *** https://review.opendev.org/c/opendev/system-config/+/855472 *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221004) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/c/opendev/system-config/+/860112 Update to launch node to handle jammy hosts ** Mailman 3 (clarkb 20221004) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** https://review.opendev.org/c/opendev/system-config/+/860157 Forking the upstream images due to lack of attention on issues and PRs we've filed ** Gitea connectivity issues (clarkb 20221004) *** Users in certain geographies have reported trouble with Gitea access *** Gitea changed http libraries which broke our ability to trace requests from haproxy -> apache -> gitea **** Partially worked around that by removing X-Forwarded-For headers. * Open discussion From cboylan at sapwetik.org Tue Oct 11 00:09:07 2022 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 10 Oct 2022 17:09:07 -0700 Subject: Team Meeting Agenda for October 11, 2022 Message-ID: We will meet with this agenda on October 11, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** PTG next week * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221011) *** Move ansible to a venv. *** Replace bridge host once existing host is running ansible out of a venv *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? *** https://review.opendev.org/c/opendev/system-config/+/855472 Disable writing console log files on bridge and static **** Needs a second change pushed to opendev/base-jobs/playbooks/infra-prod/setup-keys.yaml *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221011) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Need to retest launching a Jammy node now that user management is updated. ** Mailman 3 (clarkb 20221011) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** https://review.opendev.org/c/opendev/system-config/+/860157 Forking the upstream images due to lack of attention on issues and PRs we've filed ** Switch base job nodeset to ubuntu-jammy (frickler 20221007) *** Ubuntu 22.04.1 is out for some months now and thus can be considered stable, no need to wait a full year like we did with 20.04 *** Patch to switch base-test job: https://review.opendev.org/c/opendev/base-jobs/+/860686/2 *** Plan for further steps? Timetable, which repos to test, announcements *** Devstack currently inherits the base nodeset (via multinode) without pinning * Open discussion From artem.goncharov at gmail.com Thu Oct 13 11:23:32 2022 From: artem.goncharov at gmail.com (Artem Goncharov) Date: Thu, 13 Oct 2022 13:23:32 +0200 Subject: Testing OpenStackSDK/OSC on real clouds Message-ID: Hi all, Last years of my work on OpenStackSDK and CLI on one hand and being representative of a public cloud on the other hand made clear to me that not a single OpenStack cloud behaves like another one, doesn?t matter what you do and how hard you try. And sadly Interop is not helping here at all. The API calls you use to talk with one public cloud are guaranteed to work slightly differently on another one. This is clearly visible through the Storyboard of SDK, IRC and other offline discussions. Lot of customers are capable to work without issues with SDK/CLI on their cloud, while others report the same things are not working for them. This is a relatively complex discussion and Clark reserved a slot in the TC meeting during PTG https://etherpad.opendev.org/p/tc-2023-1-ptg#L38 to describe his stake in the problem. For a quite some time I am trying to follow an approach of having possibility to run functional tests of both SDK and CLI on real clouds and not only on devstack (on devstack things are working fine but it means as much as nothing at all in real life). On multiple occurrences I was asking for credentials and I got at the moment credentials for 2 public clouds. This already helped both to find issues in SDK (i.e. RADOSGW != Swift) as well as to point clouds on real issues on their side. It should be actually in the interest of those clouds to have some useful checks that give guarantee to their users SDK and CLI (ideally also Ansible/TF) are working fine. This is even a better form of interoperability check than anything else what we have now. Since people in this ML are involved in maintaining resource donations to OpenDev I would like to ask you to consider allocating cloud credentials for the SDK/CLI team to ensure things are running smoothly for your users (pls do not mix with allocating real resources - we need only cloud credentials to test APIs and there are no static resources required). At the moment I maintain those credentials manually until we finally get post-review pipeline created in OpenStack tenant to fully automate those tests. If you have questions or suggestions please feel free to post here, join SDK/CLI team during PTG on Friday 27.10 13:00-15:00 or contact me directly for clarifications. Thanks a lot for support, Artem -------------- next part -------------- An HTML attachment was scrubbed... URL: From mnaser at vexxhost.com Thu Oct 13 12:40:17 2022 From: mnaser at vexxhost.com (Mohammed Naser) Date: Thu, 13 Oct 2022 08:40:17 -0400 Subject: Testing OpenStackSDK/OSC on real clouds In-Reply-To: References: Message-ID: Hi Artem, We?re more than happy to provide an account assuming you don?t already have one! Thanks, Mohammed On Thu, Oct 13, 2022 at 7:23 AM Artem Goncharov wrote: > Hi all, > > Last years of my work on OpenStackSDK and CLI on one hand and being > representative of a public cloud on the other hand made clear to me that > not a single OpenStack cloud behaves like another one, doesn?t matter what > you do and how hard you try. And sadly Interop is not helping here at all. > The API calls you use to talk with one public cloud are guaranteed to work > slightly differently on another one. This is clearly visible through the > Storyboard of SDK, IRC and other offline discussions. Lot of customers are > capable to work without issues with SDK/CLI on their cloud, while others > report the same things are not working for them. > > This is a relatively complex discussion and Clark reserved a slot in the > TC meeting during PTG https://etherpad.opendev.org/p/tc-2023-1-ptg#L38 to > describe his stake in the problem. > > For a quite some time I am trying to follow an approach of having > possibility to run functional tests of both SDK and CLI on real clouds and > not only on devstack (on devstack things are working fine but it means as > much as nothing at all in real life). On multiple occurrences I was asking > for credentials and I got at the moment credentials for 2 public clouds. > This already helped both to find issues in SDK (i.e. RADOSGW != Swift) as > well as to point clouds on real issues on their side. It should be actually > in the interest of those clouds to have some useful checks that give > guarantee to their users SDK and CLI (ideally also Ansible/TF) are working > fine. This is even a better form of interoperability check than anything > else what we have now. > > Since people in this ML are involved in maintaining resource donations to > OpenDev I would like to ask you to consider allocating cloud credentials > for the SDK/CLI team to ensure things are running smoothly for your users > (pls do not mix with allocating real resources - we need only cloud > credentials to test APIs and there are no static resources required). At > the moment I maintain those credentials manually until we finally get > post-review pipeline created in OpenStack tenant to fully automate those > tests. > > If you have questions or suggestions please feel free to post here, join > SDK/CLI team during PTG on Friday 27.10 13:00-15:00 or contact me directly > for clarifications. > > Thanks a lot for support, > Artem > -- Mohammed Naser VEXXHOST, Inc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cboylan at sapwetik.org Fri Oct 14 00:09:01 2022 From: cboylan at sapwetik.org (Clark Boylan) Date: Thu, 13 Oct 2022 17:09:01 -0700 Subject: OpenSSH 8.8, RSA keys, and Gerrit In-Reply-To: References: Message-ID: <0643be35-951c-48a2-84a6-1e101d7c2445@app.fastmail.com> On Thu, Oct 14, 2021, at 9:04 AM, Clark Boylan wrote: > Hello, > > About a year ago Fedora 33 released and gave us a preview of OpenSSH's > sha1 + RSA key deprecation fallout. Fedora 33 users noticed they could > no longer use SSH RSA keys to connect to our Gerrit at > review.opendev.org. This happens because Fedora 33's OpenSSH packaging > has deprecated sha1 hashes for RSA, and despite both the client and > server supporting rsa-sha2-* variants they couldn't negotiate their use > between them. OpenSSH 8.8 released recently and did similar in the > upstream software which means users with up to date OpenSSH > installations are noticing similar problems (Arch Linux for example). > > There are a couple of workarounds that you can use. Probably the best > option is to use an ed25519 or ecdsa key with our Gerrit. Modern > clients and our Gerrit SSHD negotiate these keys without issue. Less > optimal is to manually re-enable the use of the ssh-rsa hash, but we > recommend against this as your software providers have decided this is > no longer secure enough. > > On our end we've brought this up with the MINA SSHD devs [0] with the > hope that the SSH implementation that Gerrit uses can be updated to > negotiate the sha2 hashes properly. Also, the rsa-sha2 RFC indicates > [1] clients may fallback to a sha2 variant instead of the sha1 variant > which would workaround MINA's lack of support for negotiation in the > protocol. If you are an OpenSSH>=8.8 or Fedora>=33 user you might > consider filing bugs against your ssh clients to change the default > fallback to a sha2 variant on your platforms. And now exactly a year after I wrote this email I get to respond to myself saying we've deployed a corrected version of Gerrit. We had actually managed to correct Gerrit 3.6 earlier this year, but the fix required MINA 2.8.0 which was much newer than what was found in Gerrit 3.5. Recently, MINA 2.8.0 was backported into Gerrit 3.5 allowing us to backport the RSA key fix into Gerrit 3.5 as well. And now we've deployed updated images based on those updates. What this means is that you should be able to use a modern openssh client and an RSA key to communicate to review.opendev.org without any config overrides. If you've already converted to a different key type there is no reason to convert back to RSA. The main impact here is those that have config overrides can remove them and use more secure RSA + SHA2 key negotiation. > > [0] https://issues.apache.org/jira/browse/SSHD-1141 > [1] https://datatracker.ietf.org/doc/html/rfc8332#section-3.3 > > Hopefully I've put enough keywords in this email that the various > search engines will index it, and the next time someone runs into these > problems they'll find this explanation. > > Clark From cboylan at sapwetik.org Mon Oct 17 22:26:11 2022 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 17 Oct 2022 15:26:11 -0700 Subject: Cancelling the Team Meeting on October 18, 2022 Message-ID: <239c54ad-c1c4-4620-bf04-89ebc9df13b7@app.fastmail.com> Hello, We've decided to cancel the team meeting tomorrow (October 18, 2022) as a number of us are participating in the PTG and that is consuming a fair bit of time. Feel free to bring up items of concern either here on the mailing list or in IRC in #opendev on OFTC. We'll meet at our usual time and location on October 25, 2022. Clark From cboylan at sapwetik.org Mon Oct 24 23:13:53 2022 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 24 Oct 2022 16:13:53 -0700 Subject: Team Meeting Agenda for October 25, 2022 Message-ID: <6a7fca13-4ef8-429f-ad70-0908054482ce@app.fastmail.com> We will meet with this agenda on October 25, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221025) *** Move ansible to a venv. *** Replace bridge host once existing host is running ansible out of a venv *** bastion host OS upgrade. prioin-place? new host? wait until have time to return to some of the bootstrapping/parallel job work? *** Are console log files no longer being written? Did base-jobs playbook get updated too? *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221025) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** gitea-lb02 was launched using a Jammy image and is now in prod **** Used upstream image published by Ubuntu uploaded to vexxhost and converted to raw. This allows for max compatibility with bfv on vexxhost (gitea-lb02 is not bfv). **** Needed to use an up to date paramiko out of a virtualenv to run launch node as Jammy doesn't do SSH RSA + SHA1. ** Removing snapd (clarkb 20221025) *** In the past we removed snapd as we didn't use it for anything. *** Stopped removing snapd in order to install kubectl. *** Should we go back to removing snapd globally and only add it back in where needed? ** Mailman 3 (clarkb 20221025) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** https://review.opendev.org/c/opendev/system-config/+/860157 Forking the upstream images due to lack of attention on issues and PRs we've filed ** Switch base job nodeset to ubuntu-jammy (frickler 20221007) *** Switching the default base job's nodeset to Ubuntu Jammy on October 25 (day of the meeting). ** Updating base python docker images to use `pip wheel` (clarkb 20221025) *** Pip changed how it addresses wheels in its wheel cache which broke our assemble script's ability to build packages on our image builds. *** I've filed a bug against pip for this and pushed a PR to fix it. However, upstream pip says we shouldn't rely on the wheel cache like this and should use `pip wheel` instead. *** https://github.com/pypa/pip/issues/11527 *** https://github.com/pypa/pip/pull/11538 *** https://review.opendev.org/c/opendev/system-config/+/862152 ** Dropping python 3.8 base images (clarkb 20221025) *** To make room for python 3.11 it would be good to drop our python3.8 images. *** https://review.opendev.org/q/status:open+(topic:use-new-python+OR+topic:docker-cleanups) ** iweb cloud going away at the end of the year (clarkb 20221025) *** We've been told this will need to be shutdown by the end of the year. We are good to use it until then. *** Possibility for using cloudstack resources as backfill, but would require a nodepool driver. ** Etherpad docker container logs growth (clarkb 20221025) *** The etherpad docker container's logs grow over time when we don't update the container. *** Can/should we add the syslogging for containers to this service? * Open discussion From cboylan at sapwetik.org Tue Oct 25 17:19:12 2022 From: cboylan at sapwetik.org (Clark Boylan) Date: Tue, 25 Oct 2022 10:19:12 -0700 Subject: Storyboard Needs and Alternatives Message-ID: Hello, There have apparently been disparate discussions among different projects on whether or not they want to continue using Storyboard and what alternatives they should consider. Having these discussion is healthy, but I think we should try to have them in the open with a broad audience to ensure we've considered all the various moving parts involved. I'll do my best to kick start that discussion here in this thread. The Problem(s) ?* Storyboard is slow. ?* Storyboard's deployment is aging and needs to be updated. ?* Storyboard is largely unmaintained at this point. ?* Web crawlers/indexers struggle to index Storyboard content because it is rendered client side via API calls. These first two problems are related. There are apparently fixes for known performance issues and other long-standing bugs that have yet to be deployed as we need to update the deployment platform to a newer version of python. Options ?* Update the Storyboard deployment and see if that improves things enough to make people happy. ?* Migrate projects onto one or more different issue trackers. The first option would almost certainly need someone to step up and help out. But before we do that it is probably worthwhile to solicit feedback indicating people would want to continue to use storyboard if its performance and other usability issues are addressed. The second option opens us to a tremendous number of different possibilities. Launchpad, Gitea, GitHub, Bugzilla, Trac, Redmine, etc. I think for this discussion to be productive we should avoid digging too deeply into these options now. And instead determine if Storyboard is something that we as a community want to try and keep alive. Once we've answered that question we can shift gears to this option if necessary. We would love to hear your feedback. Additional user issues with storyboard, interest in updating/maintaining Storyboard (or the opposite), and thoughts on whether or not addressing known problems with the service would be sufficient to make people want to keep using it are all helpful. The goal here isn't to try and convince anyone they should use Storyboard. Instead we're trying to understand if there are those that would like to keep using Storyboard, and if so, what effort would be necessary to make that possible. Once we've sorted this out we can have a followup discussion on what would be necessary to support projects that don't want to use Storyboard any longer. Clark From smooney at redhat.com Tue Oct 25 17:37:56 2022 From: smooney at redhat.com (Sean Mooney) Date: Tue, 25 Oct 2022 18:37:56 +0100 Subject: Storyboard Needs and Alternatives In-Reply-To: References: Message-ID: <7c8f2259f499200fc93f4f11b6afa37f64452291.camel@redhat.com> On Tue, 2022-10-25 at 10:19 -0700, Clark Boylan wrote: > Hello, > > There have apparently been disparate discussions among different projects on whether or not they want to continue using Storyboard and what > alternatives they should consider. Having these discussion is healthy, but I think we should try to have them in the open with a broad audience to > ensure we've considered all the various moving parts involved. I'll do my best to kick start that discussion here in this thread. > > The Problem(s) > ?* Storyboard is slow. > ?* Storyboard's deployment is aging and needs to be updated. > ?* Storyboard is largely unmaintained at this point. > ?* Web crawlers/indexers struggle to index Storyboard content because it is rendered client side via API calls. > > These first two problems are related. There are apparently fixes for known performance issues and other long-standing bugs that have yet to be > deployed as we need to update the deployment platform to a newer version of python. > > Options > ?* Update the Storyboard deployment and see if that improves things enough to make people happy. > ?* Migrate projects onto one or more different issue trackers. > > The first option would almost certainly need someone to step up and help out. But before we do that it is probably worthwhile to solicit feedback > indicating people would want to continue to use storyboard if its performance and other usability issues are addressed. > > The second option opens us to a tremendous number of different possibilities. Launchpad, Gitea, GitHub, Bugzilla, Trac, Redmine, etc. I think for > this discussion to be productive we should avoid digging too deeply into these options now. And instead determine if Storyboard is something that we > as a community want to try and keep alive. Once we've answered that question we can shift gears to this option if necessary. at least in the nova room several of use but not all express intreest in movign away form storyborad entirly. we were going to default back to lanuchpad to use a singel tool for all compute team deliverables and since nova is in lanuchpad and we dont have pland to ever change that we wanted to move placment to launchpad too. even if the performance, ux and indexing issues were adress i woudl still prefer to consolidate on tooling. > > We would love to hear your feedback. Additional user issues with storyboard, interest in updating/maintaining Storyboard (or the opposite), and > thoughts on whether or not addressing known problems with the service would be sufficient to make people want to keep using it are all helpful. > > The goal here isn't to try and convince anyone they should use Storyboard. Instead we're trying to understand if there are those that would like to > keep using Storyboard, and if so, what effort would be necessary to make that possible. Once we've sorted this out we can have a followup discussion > on what would be necessary to support projects that don't want to use Storyboard any longer. ack. personlly i was proposing just closing the storybord project for placment and not porting anything since we largly have not touch it in 4 cycles. as such i think resetting our backlog by not porting anything was more useful since clearly we are unlike to fix any of them if we have not worked on tehm in 2+ years. im not tied to useing launchpad if there was a better alternitive i woudl not be agaisnt suing it but storyboard in its currnt form would be a regresstion based on our currnt usage so i dont see us adopting sotryboard for the other compute team deliverable. if other optiosn like gitea ectra are expored we woudl evaualate those at that point for now launchpad works for our needs in general. > > Clark > From melwittt at gmail.com Tue Oct 25 21:29:04 2022 From: melwittt at gmail.com (melanie witt) Date: Tue, 25 Oct 2022 14:29:04 -0700 Subject: Storyboard Needs and Alternatives In-Reply-To: References: Message-ID: <02261e49-683b-017e-77fb-ac75dedfc6fa@gmail.com> On 10/25/22 10:19, Clark Boylan wrote: > Hello, > > There have apparently been disparate discussions among different projects on whether or not they want to continue using Storyboard and what alternatives they should consider. Having these discussion is healthy, but I think we should try to have them in the open with a broad audience to ensure we've considered all the various moving parts involved. I'll do my best to kick start that discussion here in this thread. > > The Problem(s) > ?* Storyboard is slow. > ?* Storyboard's deployment is aging and needs to be updated. > ?* Storyboard is largely unmaintained at this point. > ?* Web crawlers/indexers struggle to index Storyboard content because it is rendered client side via API calls. > > These first two problems are related. There are apparently fixes for known performance issues and other long-standing bugs that have yet to be deployed as we need to update the deployment platform to a newer version of python. > > Options > ?* Update the Storyboard deployment and see if that improves things enough to make people happy. > ?* Migrate projects onto one or more different issue trackers. > > The first option would almost certainly need someone to step up and help out. But before we do that it is probably worthwhile to solicit feedback indicating people would want to continue to use storyboard if its performance and other usability issues are addressed. > > The second option opens us to a tremendous number of different possibilities. Launchpad, Gitea, GitHub, Bugzilla, Trac, Redmine, etc. I think for this discussion to be productive we should avoid digging too deeply into these options now. And instead determine if Storyboard is something that we as a community want to try and keep alive. Once we've answered that question we can shift gears to this option if necessary. > > We would love to hear your feedback. Additional user issues with storyboard, interest in updating/maintaining Storyboard (or the opposite), and thoughts on whether or not addressing known problems with the service would be sufficient to make people want to keep using it are all helpful. As Sean mentioned, as a team (nova) we are going to discontinue use of storyboard for the placement project and start using launchpad again to make things consistent between nova and placement. Launchpad also works well for tracking bug fix backports because you can "target to series" a single bug to add older releases to it. Backport patches get automatically posted on launchpad by the infra tooling without the need to create a new Task for each backport and adjusting the commit message of the backport to reference that Task. That aside, I can share a little feedback about storyboard based on my experience using it for placement and openstackclient. It might boil down to the first problem you listed "storyboard is slow" but I have found the search function to be unusable. A frequent workflow for me is to search for existing bugs in a project based on a keyword in the title, summary, or comments. With storyboard I haven't been able to succeed in doing that. From what I understand, you are supposed to be able to scope the search by project: but the search will not accept it unless you choose it from the dropdown box that is supposed to appear. The dropdown populates extremely slowly or sometimes not at all. I realize I can just go to a project and choose to show 1000 items per page and use the browser search on the titles instead, but that won't help for keywords in the summary or comments. Storyboard also isn't _as_ easy for the aforementioned patch backporting that happens constantly in nova. We could add a Task per older release to a Story and adjust the backport patch commit messages to reference the matching Tasks (this is essentially what I have to do downstream and it can be tedious ?). Not a huge deal but in comparison not as easy as launchpad. Other than that, I don't have any issues with storyboard. It's just because I use search functions so often in bug trackers in general, it happens to be not so usable for me overall. > The goal here isn't to try and convince anyone they should use Storyboard. Instead we're trying to understand if there are those that would like to keep using Storyboard, and if so, what effort would be necessary to make that possible. Once we've sorted this out we can have a followup discussion on what would be necessary to support projects that don't want to use Storyboard any longer. > > Clark > From smooney at redhat.com Wed Oct 26 09:56:42 2022 From: smooney at redhat.com (Sean Mooney) Date: Wed, 26 Oct 2022 10:56:42 +0100 Subject: Storyboard Needs and Alternatives In-Reply-To: <02261e49-683b-017e-77fb-ac75dedfc6fa@gmail.com> References: <02261e49-683b-017e-77fb-ac75dedfc6fa@gmail.com> Message-ID: <1f11e2b8cd774d77648c3f1705fd6e343b6d48f9.camel@redhat.com> On Tue, 2022-10-25 at 14:29 -0700, melanie witt wrote: > On 10/25/22 10:19, Clark Boylan wrote: > > Hello, > > > > There have apparently been disparate discussions among different projects on whether or not they want to continue using Storyboard and what alternatives they should consider. Having these discussion is healthy, but I think we should try to have them in the open with a broad audience to ensure we've considered all the various moving parts involved. I'll do my best to kick start that discussion here in this thread. > > > > The Problem(s) > > ?* Storyboard is slow. > > ?* Storyboard's deployment is aging and needs to be updated. > > ?* Storyboard is largely unmaintained at this point. > > ?* Web crawlers/indexers struggle to index Storyboard content because it is rendered client side via API calls. > > > > These first two problems are related. There are apparently fixes for known performance issues and other long-standing bugs that have yet to be deployed as we need to update the deployment platform to a newer version of python. > > > > Options > > ?* Update the Storyboard deployment and see if that improves things enough to make people happy. > > ?* Migrate projects onto one or more different issue trackers. > > > > The first option would almost certainly need someone to step up and help out. But before we do that it is probably worthwhile to solicit feedback indicating people would want to continue to use storyboard if its performance and other usability issues are addressed. > > > > The second option opens us to a tremendous number of different possibilities. Launchpad, Gitea, GitHub, Bugzilla, Trac, Redmine, etc. I think for this discussion to be productive we should avoid digging too deeply into these options now. And instead determine if Storyboard is something that we as a community want to try and keep alive. Once we've answered that question we can shift gears to this option if necessary. > > > > We would love to hear your feedback. Additional user issues with storyboard, interest in updating/maintaining Storyboard (or the opposite), and thoughts on whether or not addressing known problems with the service would be sufficient to make people want to keep using it are all helpful. > > As Sean mentioned, as a team (nova) we are going to discontinue use of > storyboard for the placement project and start using launchpad again to > make things consistent between nova and placement. > > Launchpad also works well for tracking bug fix backports because you can > "target to series" a single bug to add older releases to it. Backport > patches get automatically posted on launchpad by the infra tooling > without the need to create a new Task for each backport and adjusting > the commit message of the backport to reference that Task. > > That aside, I can share a little feedback about storyboard based on my > experience using it for placement and openstackclient. > > It might boil down to the first problem you listed "storyboard is slow" > but I have found the search function to be unusable. A frequent workflow > for me is to search for existing bugs in a project based on a keyword in > the title, summary, or comments. With storyboard I haven't been able to > succeed in doing that. From what I understand, you are supposed to be > able to scope the search by project: but the search will not > accept it unless you choose it from the dropdown box that is supposed to > appear. The dropdown populates extremely slowly or sometimes not at all. > I realize I can just go to a project and choose to show 1000 items per > page and use the browser search on the titles instead, but that won't > help for keywords in the summary or comments. > > Storyboard also isn't _as_ easy for the aforementioned patch backporting > that happens constantly in nova. We could add a Task per older release > to a Story and adjust the backport patch commit messages to reference > the matching Tasks (this is essentially what I have to do downstream and > it can be tedious ?). Not a huge deal but in comparison not as easy as > launchpad. > > Other than that, I don't have any issues with storyboard. It's just > because I use search functions so often in bug trackers in general, it > happens to be not so usable for me overall. as a concreate example and to refersh my memory of using story board i tried to navagate to the placment project yesterday. to do that i searched for it in the projects tab as openstack/placement https://storyboard.openstack.org/#!/project/list?q=openstack%2Fplacement as you can porbaly see its not in the list of results if i just seach for placement https://storyboard.openstack.org/#!/project/list?q=placement its the last result with the title openstack/placement ... if we click on that https://storyboard.openstack.org/#!/project/openstack/placement and say take the 3rd bug "Bug in placement conf file for Apache HTTPD" and seach for placement and HTTPD https://storyboard.openstack.org/#!/search?q=placement%20HTTPD&title=placement%20HTTPD its not in the lsit of results if i use the full title https://storyboard.openstack.org/#!/search?q=Bug%20in%20placement%20conf%20file%20for%20Apache%20HTTPD&title=Bug%20in%20placement%20conf%20file%20for%20Apache%20HTTPD also not there however the popup that you get if you wait actully finds the story. this UX makes story board feel like perl. you write it once and never read it again because you cant trivially find things using either the seach fucntionality or paging though the bugs. ile go to https://storyboard.openstack.org/#!/project/openstack/placement and set the limit to 100 or 1000 and using your broser to search. i know it has some nice feature liek being ablt to create boards and worklist ectra to tag and track things bug the out of box or new user expeince is pretty bad if you dont actully go build better views in it as a tool. for example and not picking on anyoen in particalar this board created by the monasca team https://storyboard.openstack.org/#!/board/190 looks pretty nice. its kanban inspired, clear to see and a arguablly niceer way visualise things the an etherpad for cordinating reviews but that isnt a day to day pain point for us in the same way fileing tasks for backports and updatign the commit message woudl be or searching for exising bugs. there defintly are some nice design element to what storyboard was trying to enable but in its current hosted incarnation it makes our life harder not simpler. if we as a comunity were to move to somethign other then storyboard and there was an objection to using launchpad then the 3 i would evauluate are github issues, gitea issues if we want to self host it or gihttps://storyboard.openstack.org/#!/story/2009315t-bug (https://github.com/MichaelMure/git-bug) the main blocker for gitia beside our current installtion not being designed to do it is the lack of Confidential bugs for security issues https://github.com/go-gitea/gitea/issues/3217 i assume sotryborad can do that but gitia and likely git-bug cant. we dont have to deal with a lot of secuity bug but its imporant that when we do we can use the same tool as regular bugs to track it so that its simple for people to correctly file secuiryt bugs. the other issue si have with story boad that if address would make me more accpting with usign it is how you view/manage stories at the project level in your dashboard https://storyboard.openstack.org/#!/dashboard/stories you have this nice filter stories dropdown with check boxes however you cant filter on the subtask states of todo progres review invalid and merged. or on lables/tags. since everythign starts with a story that means both bugs and features are stories and since you cant filter on lables you cant filter in that view. https://storyboard.openstack.org/#!/project/openstack/placement that is also true of the project view althought there its worse since the stories are displayed in tab base on active, meged and invalid state. those tabsl reload when you click on them so you also cant swithc beteen them simply. without the abilty to filter stories on tag/labes i cant get a bug or feature view without createing a borad myself. its extra fustrating since the tags are displayed on the page so i can clearlly see what ones are bugs or rfes but cant filter. i also cant find all stories that are "untriaged" since i cant display only the stories without tags. i can almost get there with https://storyboard.openstack.org/#!/story/list?status=active&project_id=1113&tags=bug that allows be to quickly filter to a proeject and look for a spcific tag, in this case placement and bug but i cant do "!= bug | != rfe" or find all stories without the "triaged" tag ectra so wee what are the new issues that as a bug triager i while writing this reply i spend a littel tiem trying to create a boad to adress these issues and you can mostly hide some of them https://storyboard.openstack.org/#!/board/254 its simple to get all the untiraged issues, rfes adn bugs and group them. where you start hitting rough edges is when you want to dig deeper. storyboard allow syou to filter on storay status (merged invalid active) but that does not actully work properly. going back to the httpd bugs https://storyboard.openstack.org/#!/story/2009315 it has one task in todo and has the bug tag its in the all bug list bug does not show up in the bug backlog. the fixed bugs has 5 listed which is nice. what is not nice is if i look for stories in status merged for placment with the bug key its empty. i has to also seach for bugs with the fixed tag which has to be manulaly added. so while at first glance it looks liek its easy to adress the isseus i raised jsut creating boards to provide the view you need when you try you hit roadblocks pretty quickly. the view also does not show tags or allow you to filter within the view storyboad had a lot of potential and some nice features planned but the current install feels like a beta and not something that is ready for production use. it was mentioned this is an old verions. if the isntnace coudl be updated and some/all of these issues have been adress already then im not against trying it for another release but if noti really think its time to move on. its not form a lack of trying to use story borad that i dislike it. its form using it to try and find things.? i know this may come across overly negitive but i wanted to give feedback on the frustratiosn we have had so that if there are people that can work on adressing it we could continue to use it. in my day job i currently have to use downstream:(jira, bugzilla, trello) and upstream:(github issues, storyboard, launchpad) of those tools whiel storyboad is arguable better the bugzilla in every way excpte searching/grouping issues that alone makes it the hardest tool for me to use out of all of them. none of them are perferct and i can see why the story board team built it but it does not align well to projects that need to maintain and backport our code across multiple releases. > > The goal here isn't to try and convince anyone they should use Storyboard. Instead we're trying to understand if there are those that would like to keep using Storyboard, and if so, what effort would be necessary to make that possible. Once we've sorted this out we can have a followup discussion on what would be necessary to support projects that don't want to use Storyboard any longer. > > > > Clark > > > > From artem.goncharov at gmail.com Wed Oct 26 11:22:04 2022 From: artem.goncharov at gmail.com (Artem Goncharov) Date: Wed, 26 Oct 2022 13:22:04 +0200 Subject: Storyboard Needs and Alternatives In-Reply-To: <1f11e2b8cd774d77648c3f1705fd6e343b6d48f9.camel@redhat.com> References: <02261e49-683b-017e-77fb-ac75dedfc6fa@gmail.com> <1f11e2b8cd774d77648c3f1705fd6e343b6d48f9.camel@redhat.com> Message-ID: <5258114D-70B9-4364-BD54-36E889167FC6@gmail.com> My few cents In addition to all mentioned already I would like to add that I personally don?t feel UX intuitive. Even trying to work with it multiple times I am never really sure where do I need to click. Example from today - I want to add dependency in one story to another (in the same project) and I can?t find it. Maybe I want to close story as duplicate - I don?t know how. Relatively often I try to have a look and process some issues (i.e. respond to the reporter) from mobile device. With storyboard this is not possible for me, it feels like a disaster. Having tasks inside single story can be mapped to task and subtask from some other issue tracker. It is all cool, but trying to have multiple tasks under story (https://storyboard.openstack.org/#!/story/2008619) also didn?t end up in being a success. It is good that task is automated to move to i.e. merged, but how to figure out later by which change? You can see on example of this story that event history is so long that you can?t do anything useful with it anymore. I don?t want to continue listing issues. To answer the initial question - even assuming current issues of StoryBoard can be fixed I have issues using it (I tried multiple times and every time I am just frustrated). I don?t find this tool fulfilling task of being effective issue tracker. The fact that the software itself is barely maintained it is even more a sign for me this is not going to end good. I think most of us are having already enough of work to do not to waste time on something where better solutions (including open source) exist. After trying to adapt existing tools to my processes and seeing how fast this becomes unusable I personally started with just relying only on most basic functionality (GitHub/Gitea/GitLab issues) without bells and whistles and am just fine - KISS From all user-facing projects I am taking care of I place my ?-1? to Storyboard. It should not only fit our dev processes, but be easy and intuitive to all users trying to report bugs. Artem > On 26. Oct 2022, at 11:56, Sean Mooney wrote: > > On Tue, 2022-10-25 at 14:29 -0700, melanie witt wrote: >> On 10/25/22 10:19, Clark Boylan wrote: >>> Hello, >>> >>> There have apparently been disparate discussions among different projects on whether or not they want to continue using Storyboard and what alternatives they should consider. Having these discussion is healthy, but I think we should try to have them in the open with a broad audience to ensure we've considered all the various moving parts involved. I'll do my best to kick start that discussion here in this thread. >>> >>> The Problem(s) >>> * Storyboard is slow. >>> * Storyboard's deployment is aging and needs to be updated. >>> * Storyboard is largely unmaintained at this point. >>> * Web crawlers/indexers struggle to index Storyboard content because it is rendered client side via API calls. >>> >>> These first two problems are related. There are apparently fixes for known performance issues and other long-standing bugs that have yet to be deployed as we need to update the deployment platform to a newer version of python. >>> >>> Options >>> * Update the Storyboard deployment and see if that improves things enough to make people happy. >>> * Migrate projects onto one or more different issue trackers. >>> >>> The first option would almost certainly need someone to step up and help out. But before we do that it is probably worthwhile to solicit feedback indicating people would want to continue to use storyboard if its performance and other usability issues are addressed. >>> >>> The second option opens us to a tremendous number of different possibilities. Launchpad, Gitea, GitHub, Bugzilla, Trac, Redmine, etc. I think for this discussion to be productive we should avoid digging too deeply into these options now. And instead determine if Storyboard is something that we as a community want to try and keep alive. Once we've answered that question we can shift gears to this option if necessary. >>> >>> We would love to hear your feedback. Additional user issues with storyboard, interest in updating/maintaining Storyboard (or the opposite), and thoughts on whether or not addressing known problems with the service would be sufficient to make people want to keep using it are all helpful. >> >> As Sean mentioned, as a team (nova) we are going to discontinue use of >> storyboard for the placement project and start using launchpad again to >> make things consistent between nova and placement. >> >> Launchpad also works well for tracking bug fix backports because you can >> "target to series" a single bug to add older releases to it. Backport >> patches get automatically posted on launchpad by the infra tooling >> without the need to create a new Task for each backport and adjusting >> the commit message of the backport to reference that Task. >> >> That aside, I can share a little feedback about storyboard based on my >> experience using it for placement and openstackclient. >> >> It might boil down to the first problem you listed "storyboard is slow" >> but I have found the search function to be unusable. A frequent workflow >> for me is to search for existing bugs in a project based on a keyword in >> the title, summary, or comments. With storyboard I haven't been able to >> succeed in doing that. From what I understand, you are supposed to be >> able to scope the search by project: but the search will not >> accept it unless you choose it from the dropdown box that is supposed to >> appear. The dropdown populates extremely slowly or sometimes not at all. >> I realize I can just go to a project and choose to show 1000 items per >> page and use the browser search on the titles instead, but that won't >> help for keywords in the summary or comments. >> >> Storyboard also isn't _as_ easy for the aforementioned patch backporting >> that happens constantly in nova. We could add a Task per older release >> to a Story and adjust the backport patch commit messages to reference >> the matching Tasks (this is essentially what I have to do downstream and >> it can be tedious ?). Not a huge deal but in comparison not as easy as >> launchpad. >> >> Other than that, I don't have any issues with storyboard. It's just >> because I use search functions so often in bug trackers in general, it >> happens to be not so usable for me overall. > as a concreate example and to refersh my memory of using story board i tried to navagate to the placment project yesterday. > to do that i searched for it in the projects tab as openstack/placement > https://storyboard.openstack.org/#!/project/list?q=openstack%2Fplacement > > as you can porbaly see its not in the list of results > > if i just seach for placement > https://storyboard.openstack.org/#!/project/list?q=placement > its the last result with the title openstack/placement ... > > if we click on that https://storyboard.openstack.org/#!/project/openstack/placement > and say take the 3rd bug "Bug in placement conf file for Apache HTTPD" > and seach for placement and HTTPD > > https://storyboard.openstack.org/#!/search?q=placement%20HTTPD&title=placement%20HTTPD > > its not in the lsit of results > > if i use the full title > https://storyboard.openstack.org/#!/search?q=Bug%20in%20placement%20conf%20file%20for%20Apache%20HTTPD&title=Bug%20in%20placement%20conf%20file%20for%20Apache%20HTTPD > also not there > > however the popup that you get if you wait actully finds the story. > > this UX makes story board feel like perl. you write it once and never read it again because you cant trivially find things using either the seach > fucntionality or paging though the bugs. > ile go to https://storyboard.openstack.org/#!/project/openstack/placement and set the limit to 100 or 1000 and using your broser to search. > > i know it has some nice feature liek being ablt to create boards and worklist ectra to tag and track things bug the out of box or new user expeince is > pretty bad if you dont actully go build better views in it as a tool. > > for example and not picking on anyoen in particalar this board created by the monasca team > https://storyboard.openstack.org/#!/board/190 > looks pretty nice. its kanban inspired, clear to see and a arguablly niceer way visualise things the an etherpad for cordinating reviews > but that isnt a day to day pain point for us in the same way fileing tasks for backports and updatign the commit message woudl be or searching for > exising bugs. > > there defintly are some nice design element to what storyboard was trying to enable but in its current hosted incarnation it makes our life harder not > simpler. > > if we as a comunity were to move to somethign other then storyboard and there was an objection to using launchpad then the 3 > i would evauluate are github issues, gitea issues if we want to self host it or gihttps://storyboard.openstack.org/#!/story/2009315t-bug > (https://github.com/MichaelMure/git-bug) > the main blocker for gitia beside our current installtion not being designed to do it is the lack of Confidential bugs for security issues > https://github.com/go-gitea/gitea/issues/3217 i assume sotryborad can do that but gitia and likely git-bug cant. we dont have to deal > with a lot of secuity bug but its imporant that when we do we can use the same tool as regular bugs to track it so that its simple > for people to correctly file secuiryt bugs. > > the other issue si have with story boad that if address would make me more accpting with usign it is how you view/manage stories at the project level > in your dashboard https://storyboard.openstack.org/#!/dashboard/stories you have this nice filter stories dropdown with check boxes > however you cant filter on the subtask states of todo progres review invalid and merged. or on lables/tags. > since everythign starts with a story that means both bugs and features are stories and since you cant filter on lables you cant filter in that view. > > https://storyboard.openstack.org/#!/project/openstack/placement that is also true of the project view althought there its worse since > the stories are displayed in tab base on active, meged and invalid state. those tabsl reload when you click on them so you also cant swithc beteen > them simply. without the abilty to filter stories on tag/labes i cant get a bug or feature view without createing a borad myself. > its extra fustrating since the tags are displayed on the page so i can clearlly see what ones are bugs or rfes but cant filter. > i also cant find all stories that are "untriaged" since i cant display only the stories without tags. > > i can almost get there with https://storyboard.openstack.org/#!/story/list?status=active&project_id=1113&tags=bug > that allows be to quickly filter to a proeject and look for a spcific tag, in this case placement and bug but i cant > do "!= bug | != rfe" or find all stories without the "triaged" tag ectra so wee what are the new issues that as a bug triager i > > while writing this reply i spend a littel tiem trying to create a boad to adress these issues and you can mostly hide some of them > https://storyboard.openstack.org/#!/board/254 > its simple to get all the untiraged issues, rfes adn bugs and group them. > where you start hitting rough edges is when you want to dig deeper. > storyboard allow syou to filter on storay status (merged invalid active) but that does not actully work properly. > going back to the httpd bugs https://storyboard.openstack.org/#!/story/2009315 it has one task in todo and has the bug tag > its in the all bug list bug does not show up in the bug backlog. > the fixed bugs has 5 listed which is nice. what is not nice is if i look for stories in status merged for placment with the bug key its empty. > i has to also seach for bugs with the fixed tag which has to be manulaly added. > > so while at first glance it looks liek its easy to adress the isseus i raised jsut creating boards to provide the view you need when you try you hit > roadblocks pretty quickly. the view also does not show tags or allow you to filter within the view > > storyboad had a lot of potential and some nice features planned but the current install feels like a beta and not something that is ready for > production use. it was mentioned this is an old verions. if the isntnace coudl be updated and some/all of these issues have been adress already then > im not against trying it for another release but if noti really think its time to move on. its not form a lack of trying to use story borad that i > dislike it. its form using it to try and find things. > > i know this may come across overly negitive but i wanted to give feedback on the frustratiosn > we have had so that if there are people that can work on adressing it we could continue to use it. > in my day job i currently have to use downstream:(jira, bugzilla, trello) and upstream:(github issues, storyboard, launchpad) > of those tools whiel storyboad is arguable better the bugzilla in every way excpte searching/grouping issues that alone makes > it the hardest tool for me to use out of all of them. none of them are perferct and i can see why the story board team built > it but it does not align well to projects that need to maintain and backport our code across multiple releases. > > >>> The goal here isn't to try and convince anyone they should use Storyboard. Instead we're trying to understand if there are those that would like to keep using Storyboard, and if so, what effort would be necessary to make that possible. Once we've sorted this out we can have a followup discussion on what would be necessary to support projects that don't want to use Storyboard any longer. >>> >>> Clark -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnsomor at gmail.com Wed Oct 26 15:18:29 2022 From: johnsomor at gmail.com (Michael Johnson) Date: Wed, 26 Oct 2022 08:18:29 -0700 Subject: Storyboard Needs and Alternatives Message-ID: Sorry this post may be out of the existing thread, I wasn't aware of this list (just the service-announce) until now, so I just signed up. Here are my issues with Storyboard: 1. It is unmaintained and even contributed patches get ignored (6+ months with no review). 2. It is slow (frankly it seems like it is getting slower over time). 3. The search is horrible. 4. It is not flexible enough to work with teams processes. a. For example, we can't link duplicate stories, so it's a pain to piece all of the stories together. b. There is no lifecycle status that can be used for search, etc. Triaged, confirmed, etc. c. Object storage never became a thing, so no screenshots, no logs, etc. d. There are no "My bugs" quick links, etc. so stuck with the search capability. e. Tags are free-form, so you get "RFE" "rfe" "feature", etc. that are all different searches. f. No way to search by "authored by" g. No "importance" fields, again making search and sorting hard without creating ordered lists. 5. Not all projects are on Storyboard, so it is a huge pain to move a story from one project to the other. 6. The Storyboard community was very opinionated on "how" it should be used and was not open to enabling alternate workflows/processes. a. This causes problems when community members have downstream tools/processes they need to align stories with. b. This made maintaining stories/bugs very difficult and people stopped triaging stories. These are the top of head issues I have with Storyboard. I know from the Octavia team perspective, we have tried[1] to work with and help Storyboard become a thing, but it just isn't happening. At this point I think the Octavia team is favoring going back to launchpad. I think given how Storyboard has gone, I would not be in favor of an alternate tool unless the community was 100% onboard to all move inside a cycle. Michael [1] https://etherpad.opendev.org/p/storyboard-issues From cboylan at sapwetik.org Mon Oct 31 22:05:51 2022 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 31 Oct 2022 15:05:51 -0700 Subject: Team Meeting Agenda for November 1, 2022 Message-ID: <8fba8802-a523-4620-b211-c4640ddc0ffa@app.fastmail.com> == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221101) *** https://review.opendev.org/q/topic:prod-bastion-group *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221101) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** https://review.opendev.org/c/opendev/system-config/+/862835/ Disable phased package updates ** Removing snapd (clarkb 20221101) *** https://review.opendev.org/c/opendev/system-config/+/862834 ** Mailman 3 (clarkb 20221101) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** https://review.opendev.org/c/opendev/system-config/+/860157 Forking the upstream images due to lack of attention on issues and PRs we've filed ** Updating base python docker images to use `pip wheel` (clarkb 20221101) *** Pip changed how it addresses wheels in its wheel cache which broke our assemble script's ability to build packages on our image builds. *** I've filed a bug against pip for this and pushed a PR to fix it. However, upstream pip says we shouldn't rely on the wheel cache like this and should use `pip wheel` instead. *** https://github.com/pypa/pip/issues/11527 *** https://github.com/pypa/pip/pull/11538 *** https://review.opendev.org/c/opendev/system-config/+/862152 ** Etherpad docker container logs growth (clarkb 20221101) *** Need change to log to syslog instead similar to other services. * Open discussion