- service-discuss - lists.opendev.org

Re: Followup on https://review.opendev.org/c/openstack/cinder/+/853785
by Jean-Sébastien BEVILACQUA 19 Apr '23

19 Apr '23

Hello, I allow myself to contact you following the email below sent to me by Clark. Indeed, I want to add a new volume driver in Cinder [0] and for that I must have a working CI which validates the tempest tests with the Lustre [1] driver. I tried for 2 weeks to set up this CI with Software Factory but I didn't manage to make it work. However, the Lustre FS being 100% open-source, I was advised to use the upstream CI directly. In gerrit I have already created a service user with username lustreci. Could you tell me how to set up and run the tests directly in the upstream CI ? For information, I already have ansible playbooks allowing to set up a Lustre FS on CentOS 8. Thanks in advance, [0] https://review.opendev.org/c/openstack/cinder/+/853785 [1] https://www.lustre.org/ Jean-Sébastien Le 19/04/2023 à 18:01, Clark Boylan a écrit : > Hello, > > I wanted to followup on your questions in > https://review.opendev.org/c/openstack/cinder/+/853785 as well as your > query submitted to https://openinfra.dev/projects/contact/. > > As mentioned by Sean on the Gerrit change the Cinder team requires > working CI for all in tree Cinder drivers. Many of the systems that > Cinder integrates with are proprietary storage systems which > necessitates the use of external (to OpenDev) third party CI as > specialized hardware and licensing requirements don't allow us to run > these upstream. > > To make this happen you need a CI system that is capable of listening > to Gerrit events, triggering builds, and reporting the results back to > Gerrit (for example Zuul/Jenkins/etc). Both OpenDev [0] and the Cinder > team [1] attempt to provide documentation, but this will always be > incomplete as we won't be aware of your local network policies, > hardware peculiarities and so on. The OpenDev team can help with > connection and account details for Gerrit, and the Cinder team should > be able to help with test specific needs (like appropriate logging, > service behavior, etc). I cannot speak to Software Factory as I have > never personally used it. > > The good news is that Lustre is open source software which can be > deployed without proprietary licensing restrictions, and there don't > appear to be specialized hardware needs either. In this case I would > test Lustre + Cinder in the upstream CI system. It looks like Cinder > is already doing this with Ceph [2]. My recommendation would be that > you shift focus from attempting to running a third party CI system to > adding a new job that runs against Cinder changes to test Cinder + > Lustre integration. > > This is something that the Cinder team should be able to help with as > they have a number of Zuul jobs already including the one that tests > against Ceph. The OpenDev team can help with higher level concerns > like Gerrit accounts, general Zuul behaviors/syntax, and CI system > limitations. You can reach out to the OpenDev team either via > service-discuss(a)lists.opendev.org or in #opendev on the OFTC IRC > network (all of this info can be found in the footer of > https://opendev.org) > > [0] > https://docs.opendev.org/opendev/system-config/latest/third_party.html > [1] https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers > [2] > https://opendev.org/openstack/cinder/src/commit/2e8aff660b18d1e8f961d6de979… > > Clark > -- Jean-Sébastien BEVILACQUA Responsable technique OSSA Tél : 07 85 55 35 11 Courriel :jsbevilacqua@linagora.com www.linagora.com

2 1

Team Meeting Agenda for April 18, 2023
by Clark Boylan 18 Apr '23

18 Apr '23

We will meet with this agenda on April 18, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Migrating to Quay (clarkb 20230418) *** Intermediate registry publication process appears to be working. Now we need to move content and update jobs? ** Bastion host (ianw 20230418) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230418) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Upgrade and Project Renames (ianw 20230418) *** ACL updates to better match upstream expectations **** https://review.opendev.org/c/openstack/project-config/+/880115 **** https://review.opendev.org/c/openstack/project-config/+/879906 *** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk. *** At some point we should stop building Gerrit 3.6 images, add a Gerrit 3.8 image (rc1 just released), and update our upgrade job. ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230418) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Nameserver planning has begun **** https://review.opendev.org/q/topic:jammy-dns **** https://etherpad.opendev.org/p/2023-opendev-dns *** Etherpad replacement is in process https://review.opendev.org/q/topic:add-static02 **** https://paste.opendev.org/show/brRuhPssVLSi4UnF5hcN/ Etherpad data migration process expect db dump and restore to take about half an hour each. **** Planning for Wednesday at 22:00 to start the move. ** AFS volume quotas and utilization (clarkb 20230411) *** Fedora 36 to be cleaned up in the near future. ** Gitea 1.19 Upgrade (clarkb 20230418) *** Gitea 1.19.1 has been released *** https://review.opendev.org/c/opendev/system-config/+/877541 Upgrade opendev.org to 1.19.1 *** Changes to authentication. Do we care? ** Quo vadis storyboard (frickler 20230418) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG * Open discussion

1 0

Stx docs version menu
by Stone, Ronald 11 Apr '23

11 Apr '23

We are having issues with the version/branch menu on the starlingx docs site. From a release branch, e.g docs.starlingx.com/r/stx.6.0 trying to switch to another release branch append the "r/stx.<version>/" to the current URL instead of pointing to https://docs.starlingx.io/r/stx.<version<https://docs.starlingx.io/r/stx.%3cversion>>. i.e. docs.starlingx.com/r/stx.6.0/r/stx.7.0 is constructed instead of the correct docs.starlingx.com/r/stx.7.0. The settings in the corresponding Sphinx conf.py file are: starlingxdocs_plus_this_version = "r/stx.6.0" starlingxdocs_plus_other_versions = [("Version 7.0","r/stx.7.0"),("Version 8.0","r/stx.8.0"),("Latest","/")] Can we confirm if these conf.py settings are correct? If they are, then we will open a bug against the template. Thanks

2 2

Team Meeting Agenda for April 11, 2023
by Clark Boylan 11 Apr '23

11 Apr '23

We will meet on April 11, 2023 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** New Contributor Onboarding Wednesday April 12, 2023 at 14:00 UTC in https://meetpad.opendev.org/opendev-contributor-bootstrap-202304 * Actions from last meeting * Specs Review * Topics ** Migrating to Quay (clarkb 20230411) *** TODO testing of ianw's changes to generic container roles ** Bastion host (ianw 20230411) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230411) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Upgrade and Project Renames (ianw 20230411) *** This was done successfully as far as I can tell *** Potentially need to manually run manage-projects to resync copyConditions for labels *** Should consider applying https://review.opendev.org/c/openstack/project-config/+/879906 or something like it when we do that to normalize indentation of configs to be what gerrit expects. *** At some point we should stop building Gerrit 3.6 images, add a Gerrit 3.8 image (rc1 just released), and update our upgrade job. ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230411) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Nameserver planning has begun **** https://review.opendev.org/q/topic:jammy-dns **** https://etherpad.opendev.org/p/2023-opendev-dns *** static.opendev.org has moved to static02. Still need to cleanup static01 when we are convinced we won't revert. **** static02 runs jammy's openafs-client package not our ppa package. *** Etherpad replacement is in process https://review.opendev.org/q/topic:add-static02 ** AFS volume quotas and utilization (clarkb 20230411) *** Fedora 36 to be cleaned up in the near future. ** Gitea 1.19 Upgrade (clarkb 20230411) *** Gitea 1.19.0 has been released *** https://review.opendev.org/c/opendev/system-config/+/877541 Upgrade opendev.org to 1.19.0 *** I half expect 1.19.1 to be released soon. May make more sense to wait for that update and upgrade directly to 1.19.1 ** Quo vadis storyboard (frickler 20230411) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG * Open discussion

1 0

Wheel cache builds
by Ian Wienand 04 Apr '23

04 Apr '23

Hi there, The wheel cache builds are becoming harder and harder to maintain, so I think we need to re-evaluate what we're doing. To summarise; currently for every platform, every day * job starts with zuul clone of requirements * run bindep (for master only? ... probably wrong) and do some more (now-looking-a-bit-dubious) setup [1] * we iterate over master + stable/* and "pip wheel" build each item in requirements, putting it into a local wheel cache [2] * except for arm64, where we take the latest two branches (the choosing of which was recently broken by the change in sort ordering from the "YYYY.X" release format) * then we grep the build logs to find out which .whl files were downloaded from PyPI, and delete them from the local cache * then we move to the publish step, where we copy the wheels to AFS. This never removes, but it does overwrite (so the .whl is very likely to change every day, as timestamps, etc. mean .whl builds are not reproducable). [3] * then we make a pypi index from the files in afs [4]. * We wait for all the publishing jobs to complete successfully, then we release the AFS volumes. If any fail, we don't publish that day [5] This started a long time ago, when we had a few platforms and a few branches. We now have newton->2023.1 branches in requirements, and we currently do this for 15 different platforms. When you multiply that out, it's not sustainable. Daily build jobs are timing out now, which holds up all publishing (I think the latest release pushed us over the edge). For some years, we were not pruning wheels we downloaded from PyPI [6]. If a .whl is built and on PyPI we should get it directly from upstream -- we have a caching proxy setup for CI jobs. I have written a small tool to help us clean up our caches [7]. It would be good if we could audit this tool, and when we're happy with it's output we can look at clearing out our caches. But that still leaves what is going into them every day. Iterating every branch is fairly useless. Ideally, we'd have a matrix of platforms v branches that gave us an exact mapping of what platforms run jobs on what branches. This does not generally exist; we all have some vague ideas and the extremes are obvious (we are not running Zed jobs on centos-8, and we are not running newton jobs on Ubuntu Jammy) but the middle is fuzzy. I'd like to solicit opinions on what we want this cache to do? One compelling option is to just build master requirements into the cache. The theory being that as branches are made, the requirement must have passed through master; ergo as we have an additive cache we will have wheels built. This seems OK, but it also seems that we need a cut-off point. It doesn't seem useful to build building "master" on centos-8/xenial as the requirements are all pinning things for Python's way in advance of what's there. If we do this, how do we maintain where a platform stops building master? stable/* requirements shouldn't change much; but if they do, we should push new .whls into the cache -- how do we do that in this model? This also makes our cache "precious" in that we are never building old branches -- if we lose AFS for some reason, we have a job ahead of us to restore all the old wheels. I think a perfect solution here might involve making the entire publishing pipeline driven by changes to openstack/requirements. Firstly, we have a non-trivial amount of work to figure out modifying the release process from "everything builds and releases or nothing does" to individual builds. I think we can do this with Zuul semaphores, and there's a decent chance it was written like this because mutexes/sempahores weren't available. This would be a non-trivial amount of work, and also be handing off a significant amount of this from what has traditionally been an infra job to the requirements project. Is anyone interested on working on this? I welcome any and all suggestions on what we want out of the wheel cache and how we can achieve it :) -i [1] https://opendev.org/openstack/openstack-zuul-jobs/src/commit/699e811cb8fd3f… [2] https://opendev.org/openstack/openstack-zuul-jobs/src/commit/699e811cb8fd3f… [3] https://opendev.org/openstack/project-config/src/branch/master/roles/copy-w… [4] https://opendev.org/openstack/project-config/src/commit/6e4748ca35008a4c25e… [5] https://opendev.org/openstack/project-config/src/branch/master/playbooks/wh… [6] https://review.opendev.org/c/openstack/project-config/+/703487 [7] https://review.opendev.org/c/opendev/system-config/+/879239

3 3

Team Meeting Agenda for April 4, 2023
by Clark Boylan 04 Apr '23

04 Apr '23

We will meet with this agenda on April 4, 2023 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Docker Hub Team shutdown (clarkb 20230404) *** Docker has decided not to follow through with their plan afterall **** I think we should continue as planned just with a bit less urgency (which is nice) *** https://review.opendev.org/q/topic:tag-deletion Changes to handle publication to registries generically. *** TODO need to implement promote from intermediate registry path or use tag deletion role. ** Bastion host (ianw 20230404) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230404) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Upgrade and Project Renames (ianw 20230404) *** Projects renames and Gerrit 3.7.2 upgrade April 6 at 22:00 UTC **** If we rename after upgrade then reindex for renames can happen online shortening downtime **** Are we comfortable relying on jeeypb's cache believing old project names are created in Gerrit preventing recreation? **** Has the 3.7 -> 3.6 revert path been tested? ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230404) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Nameserver planning has begun **** https://review.opendev.org/q/topic:jammy-dns **** https://etherpad.opendev.org/p/2023-opendev-dns *** Static and etherpad replacements are in process https://review.opendev.org/q/topic:add-static02 *** launch-env on bridge cannot list rax volumes *** reverse rax dns doesn't get automatically populated ** AFS volume quotas and utilization (clarkb 20230404) *** Fedora 36 to be cleaned up in the near future. ** Gitea 1.19 Upgrade (clarkb 20230404) *** Gitea 1.19.0 has been released *** https://review.opendev.org/c/opendev/system-config/+/877541 Upgrade opendev.org to 1.19.0 ** Quo vadis storyboard (frickler 20230404) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Some projects decided to move back to launchpad at the most recent PTG * Open discussion

1 0

Skipping the team meeting on March 28, 2023
by Clark Boylan 28 Mar '23

28 Mar '23

Heads up, we will skip tomorrow's (March 28, 2023) team meeting. A number of us are participating in the virtual PTG this week and not having another hour of meetings would be nice. As always, if there is something important, urgent, or just interesting feel free to bring it up on this mailing list or on IRC. We can continue to work asynchronously as usual. Clark

1 0

Team Meeting Agenda for March 21, 2023
by Clark Boylan 21 Mar '23

21 Mar '23

We will meet on March 21, 2023 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenStack release happening this week (should happen tomorrow). ** PTG next week * Actions from last meeting * Specs Review * Topics ** Docker Hub Team shutdown (clarkb 20230321) *** https://etherpad.opendev.org/p/MJTzrNTDMFyEUxi1ReSo Notes on the situation and planning *** https://quay.io/organization/opendevorg Will become the new home for opendev images *** https://review.opendev.org/c/zuul/zuul-jobs/+/877834 Role to precreate public images on quay.io *** TODO Update zuul jobs for publication and promotion to quay.io *** TODO Copy historical content form docker hub to quay.io *** TODO publish new up to date images to quay.io ** Bastion host (ianw 20230321) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups *** Fungi needed to boot old bridge to recover gpg key material and generation tools. ** Mailman 3 (clarkb 20230321) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Updates (ianw 20230321) *** https://review.opendev.org/c/opendev/system-config/+/868054 Better testing of submit-requirements and copyConditions *** https://bugs.chromium.org/p/gerrit/issues/detail?id=16765 Gerrit 3.7.1 related changes bug **** https://gerrit-review.googlesource.com/c/gerrit/+/364214 Gerrit 3.7.1 related changes bug fix ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230321) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Gitea09-14 are the six production giteas behind haproxy *** Gitea01-04 are receiving Gerrit replication updates until we are confident we will not fallback to needing them any longer. **** Will cleanup after the openstack release. *** Nameserver planning has begun **** https://review.opendev.org/q/topic:jammy-dns **** https://etherpad.opendev.org/p/2023-opendev-dns ** AFS volume quotas and utilization (clarkb 20230321) *** Fedora 36 to be cleaned up in the near future. ** Scheduling project renames (clarkb 20230321) *** April 6, 2023 at 22:00 UTC *** https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/… ** Quo vadis storyboard (frickler 20230321) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Clarkb needs to resync on this but then we should make some decisions. ** Gitea 1.19 Upgrade (clarkb 20230321) *** Gitea 1.19.0 has been released *** https://review.opendev.org/c/opendev/system-config/+/877541 Upgrade opendev.org to 1.19.0 **** Probably best to wait for the openstack release to be finished before landing this change. * Open discussion

1 0

Team Meeting Agenda for March 14, 2023
by Clark Boylan 14 Mar '23

14 Mar '23

We will meet on March 14, 2023 at 19:00 UTC (note the DST change in some locations) in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Daylight Savings time has taken effect in North America and will in a few weeks for others. This may affect the time you attend our meeting in your local timezone, but we continue to meet at 19:00 UTC. * Actions from last meeting * Specs Review * Topics ** Bastion host (ianw 20230314) *** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge *** https://review.opendev.org/q/topic:bridge-backups ** Mailman 3 (clarkb 20230314) *** https://etherpad.opendev.org/p/mm3migration *** Need to change site domains ** Gerrit Updates (ianw 20230314) *** https://review.opendev.org/q/topic:gerrit-s-r-3.7 Cleaning up deprecated Gerrit ACL stuff *** We discovered that 'and' and 'AND' are not equivalent in Gerrit search queries. *** Conversions to submit requirements and CopyConditions continue. ** Upgrading Bionic servers to Focal/Jammy (clarkb 20230314) *** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades *** Gitea09-14 are the six production giteas behind haproxy *** Gitea05-08 have been deleted and removed from DNS *** Gitea01-04 are receiving Gerrit replication updates until we are confident we will not fallback to needing them any longer. *** Nameserver planning has begun **** https://review.opendev.org/q/topic:jammy-dns **** https://etherpad.opendev.org/p/2023-opendev-dns ** AFS volume quotas and utilization (clarkb 20230314) *** Quotas increased last week on a number of volumes to avoid hitting limits *** Addition of Fedora 37 and removal of Fedora 35 seems to have resulted in a net increase in data storage. *** Do we need to be thinking about increasing the size of backing volumes? I was unable to find any short term deletion wins. ** Scheduling project renames (clarkb 20230314) *** April 7, 2023 are things still looking good for this date? *** We should pick a time of day and announce it. ** Quo vadis storyboard (frickler 20230314) *** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html *** Clarkb needs to resync on this but then we should make some decisions. * Open discussion

1 0

Modernising ACLs for Gerrit 3.7
by Ian Wienand 07 Mar '23

07 Mar '23

Hi, The changes to update copy-conditionals and submit-requirements have multiplied a bit, so I thought it might be good to have a bit of a written-down plan to get us ready for Gerrit 3.7. I think we should probably start with a manual update to All-Projects, as we don't really manage that in Ansible. The two reviews to update the bootstrap documentation should serve as review for the production change: https://review.opendev.org/c/opendev/system-config/+/876236 https://review.opendev.org/c/opendev/system-config/+/876237 I've consolidated those changes an actual diff to our production All-Projects/project.config shown in https://paste.opendev.org/show/bmy36m7TO2b4cpndO5gO/ I have that in a checkout, so if we're happy with that, I can manually push it (or will rework for comments). I think this gives us a good place to start and covers both the copy-conditions and submit-requirements changes for the top-level All-Projects. Updating the copy-conditions for the project ACL's is already reviewed in https://review.opendev.org/c/openstack/project-config/+/867931 I think I'll feel better about that having already done All-Projects and seen that working, but that just needs to be applied. Then we need to move onto submit-requirements for the project ACL's. Firstly https://review.opendev.org/c/openstack/project-config/+/875992/1 is just mechnical to handle the arguments in the normalize tool. The first change https://review.opendev.org/c/openstack/project-config/+/875804/3 updates labels using "function = NoOp" to also have a no-op submit-requirement. We know from testing that labels must still specify a function. Strictly, I don't think the s-r is necessary; but it does seem that we can use this to add a description explaining what the label is doing. I also think this future-proofs us against the removal of label functions (although it is currently unclear how future gerrit will deal with this). The second change is similar https://review.opendev.org/c/openstack/project-config/+/875993/2 except this is a large selection of Backport-Candidate labels using "function = NoBlock", which is alised to NoOp. For consistency, this is all changed to NoOp and, again, given a submit-requirement as well. Following that is https://review.opendev.org/c/openstack/project-config/+/875994/2 which is a mechanical change again for the normalize tool, and then the two final changes https://review.opendev.org/c/openstack/project-config/+/875995 https://review.opendev.org/c/openstack/project-config/+/875996 convert more complex labels that have blocking semantics submit-requirements. I think these just want careful review to ensure I've translated what they're doing correctly. At that point, I think we're fully migrated off the deprecated ACL options, clearing our path for a Gerrit 3.7 upgrade, general notes for which are kept at https://etherpad.opendev.org/p/gerrit-upgrade-3.7 -i

1 1