From fungi at yuggoth.org Wed Dec 1 20:29:42 2021 From: fungi at yuggoth.org (Jeremy Stanley) Date: Wed, 1 Dec 2021 20:29:42 +0000 Subject: UbuntuOne/Launchpad two-factor authentication In-Reply-To: <20211122201504.xqwvpt6t575wsjq2@yuggoth.org> References: <20211122201504.xqwvpt6t575wsjq2@yuggoth.org> Message-ID: <20211201202942.mdeivqoxzvv7x44n@yuggoth.org> On 2021-11-22 20:15:05 +0000 (+0000), Jeremy Stanley wrote: [...] > I'm bringing it up here first for discussion, in order to see if > anyone has any concerns or related suggestions, but barring none > I'd like to move forward with a "soft" (quiet) call for wider > testing the first week of December. I also brought this up in yesterday's OpenDev meeting[*] and there were no objections, nor have any been raised in the past week here. As such, anyone interested in using Ubuntu's 2FA feature feel free to request membership in https://launchpad.net/~opendev-2fa as long as you've read the lengthy disclaimers in the group description there. We've already added two early volunteers. If it continues to work out well for folks, we can make a bit more noise about the availability early next year. In the meantime, if you have any questions or observations you want to share, feel free to follow up here or in the #opendev channel on the OFTC IRC network. [*] https://meetings.opendev.org/meetings/infra/2021/infra.2021-11-30-19.01.html -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From cboylan at sapwetik.org Mon Dec 6 22:48:21 2021 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 06 Dec 2021 14:48:21 -0800 Subject: Team Meeting Agenda for December 7, 2021 Message-ID: Hello, We will meet with this agenda on December 7, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs Review * Topics ** Improving OpenDev's CD throughput (clarkb 20211207) *** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies *** We ended up in a weird spot after relying on the per buildset update source job: **** We weren't syncing updated repo content to bridge but to a single use test node **** We weren't hard depending on the job so if it didn't run we would run the other jobs without updated sources **** We weren't adding the update source job to hourly and daily periodic buildsets *** Next steps happening in https://review.opendev.org/c/opendev/system-config/+/820320 ** User management on our systems (clarkb 20211207) *** Still struggling with matrix-gerritbot. Now openssh on it complains that uid 11000 doesn't exist. *** Eventually convert mariadb container's from uid 999 to something that makes more sense on the system. ** Adding a lists.openinfra.dev mailman site (fungi 20211207) *** Ansible newlist invocations still struggling *** https://review.opendev.org/c/opendev/system-config/+/820397 and parent are a stack to investigate and fix this. ** keycloak.opendev.org exists now (clarkb 20211207) *** This can be used to gain more familiarity with the tool to implement our authentication spec. *** Are we treating this as production ready? Might be good to avoid that for now while we investigate how to apply this to our previous spec? ** Gerrit User Summit (clarkb 20211207) *** Gerrit 3.2 is EOL *** We can integrate with the new Checks UI stuff if we implement a Checks plugin for Zuul. **** This plugin would run within Gerrit and poll an API on the Zuul side for CI results and render them in a consistent manner. *** Composable Submit Requirements **** This is a new system for expressing what is required to merge a change **** Relies on boolean query language using Gerrit's existing query predicates. **** Mostly we should be careful to avoid accidentally allowing too much access when this shows up. *** ChronicleMap libmodule plugin persistent caches **** This replaces H2 caches for a number of things and gives the caches persistence. Can apparently have a big impact on performance **** One suggestion was that Gerrit incorporate this by default. ** Spring cleaning our Nodepool images (clarkb 20211207) *** CentOS 8, Fedora 34, Gentoo, OpenSUSE 15, and OpenSUSE tumbleweed are either EOL, going EOL, or lacking care necessary to keep building up to date images. *** https://etherpad.opendev.org/p/xvu2oKUQVLiHIHsvGRUt draft communications for proposed cleanups. * Open discussion From iwienand at redhat.com Tue Dec 7 05:58:44 2021 From: iwienand at redhat.com (Ian Wienand) Date: Tue, 7 Dec 2021 16:58:44 +1100 Subject: Parallel production jobs changes In-Reply-To: References: Message-ID: Thank you for prior reviews and sticking with this rather complicated fiddling; we ended up with some failures and reverts that I hope have been addressed: Firstly, we have a typo in matching install playbooks for our current roles that has propogated as we copy-paste; a perfunctory fix https://review.opendev.org/c/opendev/system-config/+/820281 I am then proposing we rename the job we currently call infra-prod-install-ansible to infra-prod-bootstrap-bridge. This is a back-to-the-future situation; as it was originally called something like "install-bridge". I think this better reflects where it will end up, as hopefully revealed in the following steps https://review.opendev.org/c/opendev/system-config/+/820282 After this, we need to run infra-prod-bootstrap-bridge as the base job before all other production jobs. To recap -- this will be the synchronisation point that puts Zuul's checkout of system-config on the bastion host (bridge) that is then used to deploy all production systems (eventually, in as parallel way as possible). This revealed what I think is a problem with the original job -- it runs the install-ansible role via playbooks/zuul/run-production-playbook.yaml. This is a chicken-and-egg problem -- run-production-playbook.yaml uses the Ansible installed on bridge to run playbooks in system-config to ... install Ansible on bridge. Addressed with: https://review.opendev.org/c/opendev/system-config/+/820320/ this makes infra-prod-bootstrap bridge a stand-alone job that should * install the required version of Ansible on bridge * setup system-config to Zuul's checkout for the buildset For sanity, we keep the current parent of the infra-prod-* jobs the same -- this means each infra-prod-* job will still be re-checking-out system-config as it runs. We should validate * the install of ansible/openstacksdk/etc. is actually idempotent; (i.e. very run of infra-prod-bootstrap-bridge isn't reinstalling everything). * infra-prod-bootstrap-bridge always runs first in a deployment buildset (i.e. dependencies are correct) * infra-prod-bootstrap-bridge correctly puts the right checkout of system-config on bridge (as mentioned, for now the other infra-prod jobs will continue to overwrite it) Once we have validated infra-prod-bootstrap-bridge is running as we like, we can drop the other jobs checking-out code with: https://review.opendev.org/c/opendev/system-config/+/820651 and also cleanup base-jobs https://review.opendev.org/c/opendev/base-jobs/+/820652 At this point, we should be ready to run in parallel (touch wood...) -i On Wed, Nov 17, 2021 at 04:04:33PM +1100, Ian Wienand wrote: > Hi, > > To recap: currently production deployment jobs run sequentially. Zuul > starts the job on an executor, which is setup to log into the bastion > host. The job sets up the system-config playbooks on the bastion host > and Ansible is run from there against the production server. > > To run in parallel, each job needs to not assume it owns the > system-config playbooks on the bastion host. > > Each Zuul *buildset* can use the same system-config playbook checkout > though. To achieve this we need to rework the dependencies; each > production job needs to depend on a common source-setup job. Once the > source is setup on the bastion host, the actual production jobs can > run in parallel. > > To the changes... > > Firstly, I believe we're doing the setup steps for the executor to log > into bridge twice: > > https://review.opendev.org/c/opendev/system-config/+/818190 > > removes this duplication, and should be safe to merge. > > As pointed out in prior reviews when running in the periodic or hourly > pipelines each job overrides that bastion host checkout to master. > > https://review.opendev.org/c/opendev/base-jobs/+/818189 > > moves this step into base-jobs, in preparation for only being done > once by the separate source-setup job. I believe this will be safe to > merge; system-config will just do it again in an idempotent way, > until: > > https://review.opendev.org/c/opendev/system-config/+/818191 > > merges, which drops this step from system-config. > > We can then merge the system-config job dependency updates in > > https://review.opendev.org/c/opendev/system-config/+/807672 > > This should mean that all jobs not only rely on the correct base jobs, > but jobs that need certificates, etc. will be relying on the > letsencrypt job, etc. This should be safe to merge as nothing should > actually change, we just have stricter dependencies. > > After this, I think we are ready to refactor the base jobs into the > two separate steps -- firstly setup the keys on the executor to log > into the bastion host, then setup the source to use on the bastion > host: > > https://review.opendev.org/c/opendev/base-jobs/+/807807 > > This initial refactor should be safe to merge as it creates two new > jobs, but the existing base job keeps running both steps as-is. > > Then we are ready for the penultimate change: > > https://review.opendev.org/c/opendev/system-config/+/807808 > > This updates the system-config jobs to all depend on > "infra-prod-setup-src" which will be the canonical job that sets up > the source repository on bridge.o.o. All other jobs in the buildset > will depend on this job, ensuring consistency for a run. > > This should also be safe, as it again doesn't actually change > ordering. > > Once all this is in, we need the final change to enable parallel > running (and think about correct semaphores between periodic/hourly > and regular runs). That is yet to be written, but we have enough to > get to that point! > > -i From jack at jento.io Tue Dec 7 20:09:34 2021 From: jack at jento.io (Jack Morgan) Date: Tue, 7 Dec 2021 12:09:34 -0800 Subject: [service-announce] Changes to OpenDev's stable of CI images In-Reply-To: <1b643139-77e2-4518-9a2b-8d6e75f81037@www.fastmail.com> References: <1b643139-77e2-4518-9a2b-8d6e75f81037@www.fastmail.com> Message-ID: <7b662144-16bd-f451-a71b-2046dcfdb7a4@jento.io> Hey Clark, I'd like to help out / volunteer on whatever needs to be done. It doesn't have to be specific to image clean up. I'm not sure how to get started so thought we could meet for coffee but open to other suggestions. On 12/7/21 08:38, Clark Boylan wrote: > Hello, > > CentOS 8 reaches end of life on December 31, 2021. This means now is the time for you to convert jobs to CentOS 8|9 Stream or some other appropriate platform. The OpenDev team will plan to remove CentOS 8 images and mirrors from our infrastructure in early January, hopefully by January 14, 2022. Again, please update your jobs now so that you are prepared. These changes are necessary as the distro will no longer be supported, and there is no way for us to ensure that the images will be secure or that we'll even be able to build them. > > We've also got Fedora 35 lined up to replace Fedora 34. Due to Fedora's short support window we typically swap out one Fedora release with the next when the next is ready. If you are using the fedora-34 label please update it to fedora-35. If you are using fedora-latest we'll be updating that to use fedora-35 and the swap will happen automatically for you. We expect to have Fedora 34 cleaned up by the end of the year. > > Additionally both OpenSUSE 15 and Tumbleweed could use some help. For OpenSUSE 15 we need to update it to 15.3 as well as updating the associated mirrors if we plan to keep it around. I'm happy to keep this image around if we can get volunteers in the community that can help with that. If there isn't interest in modernizing the OpenSUSE 15 images I'd like to propose we clean them up in January along with the CentOS 8 images. For Tumbleweed, the images haven't received much use or attention. I'd like to propose that we simply retire this image and clean it up. We had thought it might be a good platform for monitoring upcoming changes in the Linux world, but the reality is that only works if you've got people active caring for it and adjusting to those changes. The Tumbleweed cleanup will likely begin before the end of the year. > > Finally, our Gentoo images bounce between being buildable and broken with more time spent being broken than not. I'd like to propose that this image also get cleaned up in the near future. Again, if there is interest in the community for keeping this alive I think we can make that happen. But as is this image is not in a sustainable place and it would be better for us to stop running jobs on it. Thanks, -- Jack Morgan From cboylan at sapwetik.org Tue Dec 7 22:27:48 2021 From: cboylan at sapwetik.org (Clark Boylan) Date: Tue, 07 Dec 2021 14:27:48 -0800 Subject: [service-announce] Changes to OpenDev's stable of CI images In-Reply-To: <7b662144-16bd-f451-a71b-2046dcfdb7a4@jento.io> References: <1b643139-77e2-4518-9a2b-8d6e75f81037@www.fastmail.com> <7b662144-16bd-f451-a71b-2046dcfdb7a4@jento.io> Message-ID: <3d187837-ad9f-4160-94c3-32e1e74bae01@www.fastmail.com> On Tue, Dec 7, 2021, at 12:09 PM, Jack Morgan wrote: > Hey Clark, > > I'd like to help out / volunteer on whatever needs to be done. It > doesn't have to be specific to image clean up. I'm not sure how to get > started so thought we could meet for coffee but open to other suggestions. That is great to hear. For this particular set of items we'll need some help removing and turning off old uses of distros like CentOS 8. We're also interested in help fixing and maintaining our opensuse and gentoo images if that is interesting. I think the cleanup bits for CentOS 8 are actually a good task for someone wanting to get involved with our config management, CI, and code review processes. I'd be happy to help walk you through that and we can negotiate in person time off list. > > On 12/7/21 08:38, Clark Boylan wrote: >> Hello, >> >> CentOS 8 reaches end of life on December 31, 2021. This means now is the time for you to convert jobs to CentOS 8|9 Stream or some other appropriate platform. The OpenDev team will plan to remove CentOS 8 images and mirrors from our infrastructure in early January, hopefully by January 14, 2022. Again, please update your jobs now so that you are prepared. These changes are necessary as the distro will no longer be supported, and there is no way for us to ensure that the images will be secure or that we'll even be able to build them. >> >> We've also got Fedora 35 lined up to replace Fedora 34. Due to Fedora's short support window we typically swap out one Fedora release with the next when the next is ready. If you are using the fedora-34 label please update it to fedora-35. If you are using fedora-latest we'll be updating that to use fedora-35 and the swap will happen automatically for you. We expect to have Fedora 34 cleaned up by the end of the year. >> >> Additionally both OpenSUSE 15 and Tumbleweed could use some help. For OpenSUSE 15 we need to update it to 15.3 as well as updating the associated mirrors if we plan to keep it around. I'm happy to keep this image around if we can get volunteers in the community that can help with that. If there isn't interest in modernizing the OpenSUSE 15 images I'd like to propose we clean them up in January along with the CentOS 8 images. For Tumbleweed, the images haven't received much use or attention. I'd like to propose that we simply retire this image and clean it up. We had thought it might be a good platform for monitoring upcoming changes in the Linux world, but the reality is that only works if you've got people active caring for it and adjusting to those changes. The Tumbleweed cleanup will likely begin before the end of the year. >> >> Finally, our Gentoo images bounce between being buildable and broken with more time spent being broken than not. I'd like to propose that this image also get cleaned up in the near future. Again, if there is interest in the community for keeping this alive I think we can make that happen. But as is this image is not in a sustainable place and it would be better for us to stop running jobs on it. > > > Thanks, > > -- > Jack Morgan From cboylan at sapwetik.org Mon Dec 13 23:10:20 2021 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 13 Dec 2021 15:10:20 -0800 Subject: Team Meeting Agenda for December 14, 2021 Message-ID: We will meet with this agenda on December 14, 2021 at 19:00 UTC in #opendev-meeting: == Agenda for next meeting == * Announcements ** Meetings Cancelled December 21, 2021 and January 4, 2022 for Holidays ** We may have a meeting on December 28, 2021 depending on who is around. * Actions from last meeting * Specs Review * Topics ** Log4j vulnerability fun (clarkb 20211214) *** Discuss what we've learned about this bug and its impact. ** Improving OpenDev's CD throughput (clarkb 20211214) *** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this? ** Container maintenance (clarkb 20211214) *** https://etherpad.opendev.org/p/opendev-container-maintenance *** Updating Buster images to Bullseye *** Running container with dedicated users. Next up all the ircbots. *** Upgrading Zookeeper *** Upgrading MariaDB *** Eventually convert MariaDB container's from uid 999 to something that makes more sense on the system. ** Adding a lists.openinfra.dev mailman site (fungi 20211214) *** Ansible newlist invocations still struggling *** Stack at https://review.opendev.org/c/opendev/system-config/+/820900/ should correct this. ** Spring cleaning our Nodepool images (clarkb 20211214) *** CentOS 8, Fedora 34, Gentoo, OpenSUSE 15, and OpenSUSE tumbleweed are either EOL, going EOL, or lacking care necessary to keep building up to date images. *** http://lists.opendev.org/pipermail/service-announce/2021-December/000029.html cleanup announcement *** Will need volunteers for the work. Probably ok if this proceeds a little slower than advertised. * Open discussion From cboylan at sapwetik.org Mon Dec 27 23:06:52 2021 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 27 Dec 2021 15:06:52 -0800 Subject: No Team Meeting December 28, 2021 Message-ID: Just a heads up that things have been quiet (as expected) the last week. I think we can go ahead and cancel tomorrow's meeting. See you all in a couple weeks as we return to normal scheduling in the new year. Clark