From cboylan at sapwetik.org Mon Nov 7 17:28:44 2022 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 07 Nov 2022 09:28:44 -0800 Subject: Storyboard Needs and Alternatives In-Reply-To: <5258114D-70B9-4364-BD54-36E889167FC6@gmail.com> References: <02261e49-683b-017e-77fb-ac75dedfc6fa@gmail.com> <1f11e2b8cd774d77648c3f1705fd6e343b6d48f9.camel@redhat.com> <5258114D-70B9-4364-BD54-36E889167FC6@gmail.com> Message-ID: <302867a9-e299-423d-9717-97c19e66cac0@app.fastmail.com> On Wed, Oct 26, 2022, at 4:22 AM, Artem Goncharov wrote: > My few cents > snip (not to exclude the feedback, but I'm going to try and revive this discussion and move it forward in a way that is actionable). It has been a while since we received any feedback on this thread. All of the feedback we have received so far has been from those looking to migrate to something other than Storyboard. If there is anyone out there that would like to continue using storyboard now would be a great time to provide that feedback. This is valuable even if you don't have time to dedicate towards reviving the project or its deployment. Please let us know if you have this opinion. > > From all user-facing projects I am taking care of I place my ?-1? to > Storyboard. It should not only fit our dev processes, but be easy and > intuitive to all users trying to report bugs. Storyboard exists because the OpenStack project at one time felt that the alternative tools did not properly fit OpenStack's dev process needs. Based on the feedback we've seen so far, I think that OpenStack's needs have changed, but also Storyboard may not have adequately addressed some of those special needs either. We have also seen feedback from OpenStack projects indicating that the lack of consistency between bug tracking tools has been problematic for collaboration. It would probably be prudent for the OpenStack project to reevaluate what its needs are for issue tracking to ensure whatever decisions are made are cohesive enough to avoid painful disjointed tool use while also sufficiently meeting the needs of OpenStack developers. The OpenDev service-discuss list is a poor forum for this discussion, but the openstack-discuss mailing list would probably be a good place to have that discussion then any conclusions from there can be brought back to this thread. On the OpenDev side of things, we should try to get StarylingX and Zuul to chime in. Then once we've got cross project info we can work towards making decisions for OpenDev that best address the needs of our various users. > > Artem From cboylan at sapwetik.org Tue Nov 8 00:24:27 2022 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 07 Nov 2022 16:24:27 -0800 Subject: Team Meeting Agenda for November 8, 2022 Message-ID: Hello, We will meet with this agenda on November 8, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221108) *** https://review.opendev.org/q/topic:prod-bastion-group *** https://review.opendev.org/q/topic:bridge-ansible-venv *** https://review.opendev.org/c/opendev/system-config/+/863564 *** https://review.opendev.org/c/opendev/system-config/+/863568 ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221101) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20221108) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** https://review.opendev.org/c/opendev/system-config/+/860157 Forking the upstream images due to lack of attention on issues and PRs we've filed ** Updating base python docker images to use `pip wheel` (clarkb 20221108) *** Pip changed how it addresses wheels in its wheel cache which broke our assemble script's ability to build packages on our image builds. *** I've filed a bug against pip for this and pushed a PR to fix it. However, upstream pip says we shouldn't rely on the wheel cache like this and should use `pip wheel` instead. *** https://github.com/pypa/pip/issues/11527 *** https://github.com/pypa/pip/pull/11538 *** https://review.opendev.org/c/opendev/system-config/+/862152 ** Etherpad docker container logs growth (clarkb 20221101) *** Need change to log to syslog instead similar to other services. ** Quo vadis storyboard (frickler 20221107) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Many people interested in moving away from storyboard *** No volunteers to update our installation *** How long can we/do we want to commit to running an outdated installation on an EOL OS? ** Nova server rescue behavior in vexxhost (clarkb 20221108) *** Clarkb has tested this and the behaviors are surprising *** Rescuing a normal disk instance results in the instance boot with the rescue image kernel and the rescued instance's root device mounted on / instead of the rescue image mounted to /. *** Rescuing a BFV instance fails. First with the default microversion because old Nova API doesn't support this. When using microversion 2.88 it also fails, but only after attempting to rescue. **** The instance goes into an error state due to "cannot be rescued: Driver Error: Cannot access storage file" **** Attempting to unrescue the instance also fails because you cannot unrescue an instance that is in an error state * Open discussion From cboylan at sapwetik.org Tue Nov 15 18:01:40 2022 From: cboylan at sapwetik.org (Clark Boylan) Date: Tue, 15 Nov 2022 10:01:40 -0800 Subject: Team Meeting Agenda for November 15, 2022 Message-ID: <11e98e9b-bd2d-4749-8e47-f12201d4368b@app.fastmail.com> We will meet with this agenda on Tuesday, November 15, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221115) *** https://review.opendev.org/q/topic:prod-bastion-group *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221115) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20221108) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** Currently debugging why updated images aren't working as expected. Appears to be some problem with django. ** Updating base python docker images to use `pip wheel` (clarkb 20221115) *** Will aim to land this change soon, then double check images built with new assemble process are happy. ** Etherpad docker container logs growth (clarkb 20221115) *** https://review.opendev.org/c/opendev/system-config/+/864060 ** Quo vadis storyboard (frickler 20221115) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** No new movement on the mailing list. Maybe give it another week then we can consider feedback collected? ** Nova server rescue behavior in vexxhost (clarkb 20221115) *** There are image settings that need to be set for rescue images. A dedicated rescue image would also be able to set a non colliding disk label. ** Goodbye birdsite, hello tooters? (frickler 20221115) *** Twitter seems to be on a downward spiral and we should consider stopping using it *** OIF already has an account on fosstodon.org, also ianw and frickler *** Do we want to move the opendevinfra acc there, too? *** How much work is needed in terms of tooling? Any volunteers? * Open discussion From mnasiadka at gmail.com Wed Nov 16 19:54:30 2022 From: mnasiadka at gmail.com (=?utf-8?Q?Micha=C5=82_Nasiadka?=) Date: Wed, 16 Nov 2022 20:54:30 +0100 Subject: Storyboard Needs and Alternatives In-Reply-To: <302867a9-e299-423d-9717-97c19e66cac0@app.fastmail.com> References: <02261e49-683b-017e-77fb-ac75dedfc6fa@gmail.com> <1f11e2b8cd774d77648c3f1705fd6e343b6d48f9.camel@redhat.com> <5258114D-70B9-4364-BD54-36E889167FC6@gmail.com> <302867a9-e299-423d-9717-97c19e66cac0@app.fastmail.com> Message-ID: <19EC8BBB-9878-489C-B375-896BDD9A7539@gmail.com> Hello, From Kolla OpenStack projects perspective - we?ve been using Storyboard for Kayobe and Launchpad for the rest. The topic has emerged multiple times to have one tool for all projects inside Kolla family (Kolla, Kolla-Ansible, Kayobe) - but we?ve never came to an agreement to move to Storyboard (because of issues mentioned previously). Has anybody investigated tooling for moving from Storyboard to Launchpad? Michal > On 7 Nov 2022, at 18:28, Clark Boylan wrote: > > On Wed, Oct 26, 2022, at 4:22 AM, Artem Goncharov wrote: >> My few cents >> > > snip (not to exclude the feedback, but I'm going to try and revive this discussion and move it forward in a way that is actionable). > > It has been a while since we received any feedback on this thread. All of the feedback we have received so far has been from those looking to migrate to something other than Storyboard. > > If there is anyone out there that would like to continue using storyboard now would be a great time to provide that feedback. This is valuable even if you don't have time to dedicate towards reviving the project or its deployment. Please let us know if you have this opinion. > >> >> From all user-facing projects I am taking care of I place my ?-1? to >> Storyboard. It should not only fit our dev processes, but be easy and >> intuitive to all users trying to report bugs. > > Storyboard exists because the OpenStack project at one time felt that the alternative tools did not properly fit OpenStack's dev process needs. Based on the feedback we've seen so far, I think that OpenStack's needs have changed, but also Storyboard may not have adequately addressed some of those special needs either. We have also seen feedback from OpenStack projects indicating that the lack of consistency between bug tracking tools has been problematic for collaboration. > > It would probably be prudent for the OpenStack project to reevaluate what its needs are for issue tracking to ensure whatever decisions are made are cohesive enough to avoid painful disjointed tool use while also sufficiently meeting the needs of OpenStack developers. The OpenDev service-discuss list is a poor forum for this discussion, but the openstack-discuss mailing list would probably be a good place to have that discussion then any conclusions from there can be brought back to this thread. > > On the OpenDev side of things, we should try to get StarylingX and Zuul to chime in. Then once we've got cross project info we can work towards making decisions for OpenDev that best address the needs of our various users. > >> >> Artem > From cboylan at sapwetik.org Tue Nov 22 00:34:01 2022 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 21 Nov 2022 16:34:01 -0800 Subject: Team Meeting Agenda for November 22, 2022 Message-ID: We will meet with this agenda on November 22, 2022 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221115) *** https://review.opendev.org/q/topic:prod-bastion-group *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221115) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20221108) *** https://review.opendev.org/c/opendev/system-config/+/851248 Worthy of review at this point *** https://etherpad.opendev.org/p/mm3migration *** Testing updated images with updated packages is underway. ** Updating base python docker images to use `pip wheel` (clarkb 20221115) *** The change to modify this has landed and new images have been promoted. We don't expect problems but please call them out if seen. ** Quo vadis storyboard (frickler 20221115) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** It has been two weeks since we asked for additional feedback and haven't received any. We should begin making plans with the feedback that we were given. ** Nova server rescue behavior in vexxhost (clarkb 20221115) *** Should we consider building our own rescue image for non BFV instances? * Open discussion From cboylan at sapwetik.org Tue Nov 29 01:14:20 2022 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 28 Nov 2022 17:14:20 -0800 Subject: Storyboard Needs and Alternatives In-Reply-To: References: Message-ID: On Tue, Oct 25, 2022, at 10:19 AM, Clark Boylan wrote: > Hello, > > There have apparently been disparate discussions among different > projects on whether or not they want to continue using Storyboard and > what alternatives they should consider. Having these discussion is > healthy, but I think we should try to have them in the open with a > broad audience to ensure we've considered all the various moving parts > involved. I'll do my best to kick start that discussion here in this > thread. > snip > > The goal here isn't to try and convince anyone they should use > Storyboard. Instead we're trying to understand if there are those that > would like to keep using Storyboard, and if so, what effort would be > necessary to make that possible. Once we've sorted this out we can have > a followup discussion on what would be necessary to support projects > that don't want to use Storyboard any longer. Its been a month since I kicked this discussion off. Since then we've had good feedback on struggles with using Storyboard. We've also heard from some projects that consistency of tooling is useful to them which is part of their motivation for migrating away from Storyboard. What we haven't heard from is anyone motivated to help maintain the deployment of Storyboard or the Storyboard software itself. I think this puts OpenDev in an awkward position. Storyboard was built to try and address OpenStack's specific needs around issue tracking. Unfortunately, it also seems clear that Storyboard isn't currently meeting those needs for various reasons, and OpenStack isn't stepping up to try and address that. Currently that means OpenDev is an adoptive parent for the project and is clearly not doing a good enough job to keep the service happy. Where does that leave OpenDev? I think we have a few potential options for improving the status quo: * Properly adopt storyboard. Make updating the deployment of storyboard a priority and take on maintenance of the software itself to try and address the issues that have been called out that are not already fixed. This option is unlikely to be successful without help, and we haven't seen any new interest in helping. We would also likely need to help those looking to move regardless of this changes of focus. Which creates an overlap with the next option. * Migrate Storyboard users to another tool and shutdown Storyboard. This option is not without costs either. We'll have to determine how much data preservation is necessary either by creating some sort of read only archive or migrating data from Storyboard into whatever new system is chosen. We would also need to determine what to migrate to and do any necessary work to ensure that service is running. * Many of our users continue to use launchpad and a number of the discussions at the PTG were around moving from stroyboard to LP. This may be an easy option, and possibly the desired option for many given the discussion at the PTG. * Some have suggested using Gitea. While this is theoretically possible there needs to be investigation, testing, and planning to update our Gitea deployment properly to support this. In particular our Gitea deployment is actually 8 independent read only Giteas. Because issues would require authentication and writing content to Gitea we would need to sort that out with a Gitea cluster via updating deployment. Also, we can fairly easily get away with disabling Gitea functionality that we do not want enabled (pull requests, packages, etc) when it is installed as a read only system, but that is potentially more difficult once users can write to the system. Theoretically possible but will require someone to do a fair bit of engineering work. * Insert your ideas here. None of these options are going to make everyone happy, and all of them present non zero effort on our part. I don't have a strong opinion yet on what our best option is and would love to hear thoughts from those who may have other ideas or interest in a specific option. Clark From cboylan at sapwetik.org Tue Nov 29 01:16:48 2022 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 28 Nov 2022 17:16:48 -0800 Subject: Team Meeting Agenda for November 29, 2022 Message-ID: Hello, We will meet November 29, 2022 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20221129) *** https://review.opendev.org/q/topic:prod-bastion-group *** https://review.opendev.org/q/topic:bridge-ansible-venv ** Upgrading Bionic servers to Focal/Jammy (clarkb 20221129) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades ** Mailman 3 (clarkb 20221129) *** https://etherpad.opendev.org/p/mm3migration *** Ready to add a new jammy node to our inventory and let Ansible do its thing. **** https://review.opendev.org/c/opendev/system-config/+/865361 ** Quo vadis storyboard (frickler 20221129) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Followup sent to hash out options for OpenDev. ** Nova server rescue behavior in vexxhost (clarkb 20221129) *** Should we consider building our own rescue image for non BFV instances? ** Gerrit 3.6 (ianw 20221129) *** https://etherpad.opendev.org/p/gerrit-upgrade-3.6 ** acme.sh failures (clarkb 20221129) *** Recently we've had a number of certs fail to renew and our checker is complaining about them. *** acme.sh did do a recent release and has since updated their default from rsa to ecdsa. * Open discussion From saper at saper.info Wed Nov 30 15:20:36 2022 From: saper at saper.info (Marcin Cieslak) Date: Wed, 30 Nov 2022 15:20:36 +0000 Subject: Storyboard Needs and Alternatives In-Reply-To: References: Message-ID: On Mon, 28 Nov 2022, Clark Boylan wrote: > * Properly adopt storyboard. Make updating the deployment of storyboard > a priority and take on maintenance of the software itself to try and > address the issues that have been called out that are not already fixed. > This option is unlikely to be successful without help, and we haven't seen > any new interest in helping. We would also likely need to help those > looking to move regardless of this changes of focus. Which creates > an overlap with the next option. I am mostly lurking here but I decided to have a quick look at the Storyboard backend and I think the performance is so bad that it has a priority of "Unbreak Now" if some projects are even thinking about staying there. Some API queries to the backend took 2 to 6 seconds to complete when invoked from Europe. Anything like usability/frontend/missing features is secondary to that. I think it is worth fixing the perfomance even if the projects decide to switch away - the migration can take time. Why is it so bad? Who can have a look at the machine running it and the database? Would that be possible to obtain a possibly redacted copy of a MySQL database to reproduce the issues locally? I could have a quick look at it but I am not sure if some crisis group could be built to address this promptly and without red tape? I can see some unfortunate architectural decisions in the code but even with them it should not perform that badly for the amount of data we have. But it is all guessing until some real performance data can be collected from the live environment. I have joined #storyboard as "saper" to discuss details. Marcin (I made some tiny contributions to git-review and gerrit in the past, that's why I'm here) -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3706 bytes Desc: S/MIME Cryptographic Signature URL: From cboylan at sapwetik.org Wed Nov 30 17:05:24 2022 From: cboylan at sapwetik.org (Clark Boylan) Date: Wed, 30 Nov 2022 09:05:24 -0800 Subject: Storyboard Needs and Alternatives In-Reply-To: References: Message-ID: <29fca31f-95cd-4eed-a23c-9d7bc8475803@app.fastmail.com> On Wed, Nov 30, 2022, at 7:20 AM, Marcin Cieslak wrote: > On Mon, 28 Nov 2022, Clark Boylan wrote: > >> * Properly adopt storyboard. Make updating the deployment of storyboard >> a priority and take on maintenance of the software itself to try and >> address the issues that have been called out that are not already fixed. >> This option is unlikely to be successful without help, and we haven't seen >> any new interest in helping. We would also likely need to help those >> looking to move regardless of this changes of focus. Which creates >> an overlap with the next option. > > I am mostly lurking here but I decided to have a quick look at > the Storyboard backend and I think the performance is so bad > that it has a priority of "Unbreak Now" if some projects are > even thinking about staying there. > > Some API queries to the backend took 2 to 6 seconds to complete > when invoked from Europe. > > Anything like usability/frontend/missing features is secondary > to that. I think it is worth fixing the perfomance even > if the projects decide to switch away - the migration can > take time. > > Why is it so bad? Who can have a look at the machine running it > and the database? Would that be possible to obtain a possibly > redacted copy of a MySQL database to reproduce the issues locally? > > I could have a quick look at it but I am not sure if some crisis > group could be built to address this promptly and without red tape? Half of my emails to this thread have asked if anyone is interested in stepping up to do the necessary work without any response. A group could be organized to help, but no one seems interested. > > I can see some unfortunate architectural decisions in the code but > even with them it should not perform that badly for the amount > of data we have. But it is all guessing until some real performance > data can be collected from the live environment. It is my understanding that a number of these issues have been addressed, but at some point the storyboard code base updated in such a way that our aging configuration management for it cannot deploy the latest version. For this reason the first step that should be taken is to update the server and its configuration management so that accurate information can be collected against the current codebase. We can fix the codebase all we want, but if we aren't deploying it we won't see those improvements. All that to say I don't think a db dump is helpful at this point. What we need are people interested in rewriting the configuration management and upgrading the server if improving storyboard is a goal. Then we can profile, patch, and deploy fixes to improve things from there. > > I have joined #storyboard as "saper" to discuss details. > > Marcin > (I made some tiny contributions to git-review and gerrit in the past, > that's why I'm here) > Attachments: > * smime.p7s