From saper at saper.info Sat Aug 1 07:43:34 2020 From: saper at saper.info (Marcin Cieslak) Date: Sat, 1 Aug 2020 07:43:34 +0000 Subject: review.opendev.org SSH key change? Message-ID: I have clned the python-jenkins repo few days ago and today I am getting this: > git remote update Fetching upstream @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the RSA key sent by the remote host is SHA256:RXNl/GKyDaKiIQ93BoDvrNSKUPFvA1PNeAO9QiirYZU. Please contact your system administrator. Update the SSHFP RR in DNS with the new host key to get rid of this message. > git remote -v upstream ssh://saperski at review.opendev.org:29418/jjb/python-jenkins (fetch) upstream ssh://saperski at review.opendev.org:29418/jjb/python-jenkins (push) My cached entries are: [review.opendev.org]:29418 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfsIj/jqpI+2CFdjCL6kOiqdORWvxQ2sQbCzSzzmLXic8yVhCCbwarkvEpfUOHG4eyB0vqVZfMffxf0Yy3qjURrsroBCiuJ8GdiAcGdfYwHNfBI0cR6kydBZL537YDasIk0Z3ILzhwf7474LmkVzS7V2tMTb4ZiBS/jUeiHsVp88FZhIBkyhlb/awAGcUxT5U4QBXCAmerYXeB47FPuz9JFOVyF08LzH9JRe9tfXtqaCNhlSdRe/2pPRvn2EIhn5uHWwATACG9MBdrK8xv8LqPOik2w1JkgLWyBj11vDd5I3IjrmREGw8dqImqp0r6MD8rxqADlc1elfDIXYsy+TVH review.opendev.org ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBjjFqbkqLfVKn5vJnKh/LfGGo0gXp/vWlXRs0H91E0X8ce4r8DHLB8PMScHrX/n7c29/DY3FQqSaYsY6o13dFA= DNS sshfp records: review.opendev.org. 193 IN CNAME review01.opendev.org. review01.opendev.org. 276 IN SSHFP 1 1 4A38DD28E379EA443F8A722D8330EF41754F0D5E review01.opendev.org. 276 IN SSHFP 1 2 4234515B3D59F8BF7E7095BD881F39DB21B97A15511C4B10B7FD0240 25AD93FC review01.opendev.org. 276 IN SSHFP 3 1 382C41E6FFC60CF1939B292FA62879B1918145D4 review01.opendev.org. 276 IN SSHFP 3 2 52A81E8DD662F92D903199DBC5068280D33D21D3A8E5BD023FAADD47 99AC1DDF review01.opendev.org. 276 IN SSHFP 4 1 DE5FAA47C38E616ECB2CCC4C30C7E3F788C0927A review01.opendev.org. 276 IN SSHFP 4 2 4BE301BEEC8DCC06C1084BEF1DB1D136686022B7026F678D958E548E 4B7D2FC7 Is everything ok? Marcin -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3656 bytes Desc: S/MIME Cryptographic Signature URL: From mnaser at vexxhost.com Sat Aug 1 12:29:26 2020 From: mnaser at vexxhost.com (Mohammed Naser) Date: Sat, 1 Aug 2020 08:29:26 -0400 Subject: review.opendev.org SSH key change? In-Reply-To: References: Message-ID: Hi Marin, I did a bit of digging and I'm not sure why you're seeing that message (perhaps the SSHFP records point to the _system_ SSH keys and not Gerrit), anyhow: `ssh-keyscan -p29418 review.opendev.org` matches the key cached in my system (and yours). `ssh-keyscan -p29418 review.opendev.org | ssh-keygen -lf -` matches the fingerprint you're being provided.. have you updated your SSH client recently? Thanks Mohammed On Sat, Aug 1, 2020 at 3:43 AM Marcin Cieslak wrote: > > I have clned the python-jenkins repo few days ago and today I am getting this: > > > git remote update > Fetching upstream > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! > Someone could be eavesdropping on you right now (man-in-the-middle attack)! > It is also possible that a host key has just been changed. > The fingerprint for the RSA key sent by the remote host is > SHA256:RXNl/GKyDaKiIQ93BoDvrNSKUPFvA1PNeAO9QiirYZU. > Please contact your system administrator. > Update the SSHFP RR in DNS with the new host key to get rid of this message. > > git remote -v > upstream ssh://saperski at review.opendev.org:29418/jjb/python-jenkins (fetch) > upstream ssh://saperski at review.opendev.org:29418/jjb/python-jenkins (push) > > My cached entries are: > > [review.opendev.org]:29418 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfsIj/jqpI+2CFdjCL6kOiqdORWvxQ2sQbCzSzzmLXic8yVhCCbwarkvEpfUOHG4eyB0vqVZfMffxf0Yy3qjURrsroBCiuJ8GdiAcGdfYwHNfBI0cR6kydBZL537YDasIk0Z3ILzhwf7474LmkVzS7V2tMTb4ZiBS/jUeiHsVp88FZhIBkyhlb/awAGcUxT5U4QBXCAmerYXeB47FPuz9JFOVyF08LzH9JRe9tfXtqaCNhlSdRe/2pPRvn2EIhn5uHWwATACG9MBdrK8xv8LqPOik2w1JkgLWyBj11vDd5I3IjrmREGw8dqImqp0r6MD8rxqADlc1elfDIXYsy+TVH > review.opendev.org ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBjjFqbkqLfVKn5vJnKh/LfGGo0gXp/vWlXRs0H91E0X8ce4r8DHLB8PMScHrX/n7c29/DY3FQqSaYsY6o13dFA= > > DNS sshfp records: > > review.opendev.org. 193 IN CNAME review01.opendev.org. > review01.opendev.org. 276 IN SSHFP 1 1 4A38DD28E379EA443F8A722D8330EF41754F0D5E > review01.opendev.org. 276 IN SSHFP 1 2 4234515B3D59F8BF7E7095BD881F39DB21B97A15511C4B10B7FD0240 25AD93FC > review01.opendev.org. 276 IN SSHFP 3 1 382C41E6FFC60CF1939B292FA62879B1918145D4 > review01.opendev.org. 276 IN SSHFP 3 2 52A81E8DD662F92D903199DBC5068280D33D21D3A8E5BD023FAADD47 99AC1DDF > review01.opendev.org. 276 IN SSHFP 4 1 DE5FAA47C38E616ECB2CCC4C30C7E3F788C0927A > review01.opendev.org. 276 IN SSHFP 4 2 4BE301BEEC8DCC06C1084BEF1DB1D136686022B7026F678D958E548E 4B7D2FC7 > > Is everything ok? > > Marcin -- Mohammed Naser VEXXHOST, Inc. From j.harbott at x-ion.de Sat Aug 1 13:44:21 2020 From: j.harbott at x-ion.de (Dr. Jens Harbott) Date: Sat, 1 Aug 2020 15:44:21 +0200 Subject: review.opendev.org SSH key change? In-Reply-To: References: Message-ID: Am Sa., 1. Aug. 2020 um 09:43 Uhr schrieb Marcin Cieslak : > > I have clned the python-jenkins repo few days ago and today I am getting this: > > > git remote update > Fetching upstream > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! > Someone could be eavesdropping on you right now (man-in-the-middle attack)! > It is also possible that a host key has just been changed. > The fingerprint for the RSA key sent by the remote host is > SHA256:RXNl/GKyDaKiIQ93BoDvrNSKUPFvA1PNeAO9QiirYZU. > Please contact your system administrator. > Update the SSHFP RR in DNS with the new host key to get rid of this message. > > git remote -v > upstream ssh://saperski at review.opendev.org:29418/jjb/python-jenkins (fetch) > upstream ssh://saperski at review.opendev.org:29418/jjb/python-jenkins (push) > > My cached entries are: > > [review.opendev.org]:29418 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfsIj/jqpI+2CFdjCL6kOiqdORWvxQ2sQbCzSzzmLXic8yVhCCbwarkvEpfUOHG4eyB0vqVZfMffxf0Yy3qjURrsroBCiuJ8GdiAcGdfYwHNfBI0cR6kydBZL537YDasIk0Z3ILzhwf7474LmkVzS7V2tMTb4ZiBS/jUeiHsVp88FZhIBkyhlb/awAGcUxT5U4QBXCAmerYXeB47FPuz9JFOVyF08LzH9JRe9tfXtqaCNhlSdRe/2pPRvn2EIhn5uHWwATACG9MBdrK8xv8LqPOik2w1JkgLWyBj11vDd5I3IjrmREGw8dqImqp0r6MD8rxqADlc1elfDIXYsy+TVH > review.opendev.org ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBjjFqbkqLfVKn5vJnKh/LfGGo0gXp/vWlXRs0H91E0X8ce4r8DHLB8PMScHrX/n7c29/DY3FQqSaYsY6o13dFA= > > DNS sshfp records: > > review.opendev.org. 193 IN CNAME review01.opendev.org. > review01.opendev.org. 276 IN SSHFP 1 1 4A38DD28E379EA443F8A722D8330EF41754F0D5E > review01.opendev.org. 276 IN SSHFP 1 2 4234515B3D59F8BF7E7095BD881F39DB21B97A15511C4B10B7FD0240 25AD93FC > review01.opendev.org. 276 IN SSHFP 3 1 382C41E6FFC60CF1939B292FA62879B1918145D4 > review01.opendev.org. 276 IN SSHFP 3 2 52A81E8DD662F92D903199DBC5068280D33D21D3A8E5BD023FAADD47 99AC1DDF > review01.opendev.org. 276 IN SSHFP 4 1 DE5FAA47C38E616ECB2CCC4C30C7E3F788C0927A > review01.opendev.org. 276 IN SSHFP 4 2 4BE301BEEC8DCC06C1084BEF1DB1D136686022B7026F678D958E548E 4B7D2FC7 > > Is everything ok? The SSHFP records document the keys for the SSH daemon listening on port 22 used for administrative access to the server, not the keys used by gerrit. AFAICT there is no way to specify keys for different ports in DNS, so when accessing gerrit via ssh, you will have to disable DNS verification in order to get rid of this warning. For openssh this would mean to set VerifyHostKeyDNS=no (which is also the default, so likely you must have overridden this somewhere), but I do get a similar error to yours if I set the option to "yes". Side note: Please don't clone from review.opendev.org directly but use our git farm at opendev.org for this, the review site should only be used for gerrit things, like submitting patches. At opendev.org we have multiple gitea servers behind a load balancer, allowing us to scale performance as needed, while we have gerrit is only a single instance with limited resources. Yours Jens -- Dr. Jens Harbott E-Mail: j.harbott at x-ion.de x-ion GmbH Marschnerstrasse 52 22081 Hamburg Vertretungsberechtigter Geschäftsführer: Martin Bosner Registergericht: Amtsgericht Hamburg Registernummer: HRB 125049 Ust-IdNr.: DE 265 898 497 Unsere Informationspflichten gemäß Art. 12 ff. Datenschutz-Grundverordnung finden Sie unter: https://www.x-ion.de/de/datenschutz/informationspflichten From fungi at yuggoth.org Sat Aug 1 14:09:53 2020 From: fungi at yuggoth.org (Jeremy Stanley) Date: Sat, 1 Aug 2020 14:09:53 +0000 Subject: review.opendev.org SSH key change? In-Reply-To: References: Message-ID: <20200801140953.ndttlvyb3pvfwt2e@yuggoth.org> On 2020-08-01 15:44:21 +0200 (+0200), Dr. Jens Harbott wrote: [...] > The SSHFP records document the keys for the SSH daemon listening on > port 22 used for administrative access to the server, not the keys > used by gerrit. AFAICT there is no way to specify keys for different > ports in DNS, so when accessing gerrit via ssh, you will have to > disable DNS verification in order to get rid of this warning. For > openssh this would mean to set VerifyHostKeyDNS=no (which is also the > default, so likely you must have overridden this somewhere), but I do > get a similar error to yours if I set the option to "yes". [...] This is going to be challenging to work around, I think. The cleanest solution is probably going to be separating the review.opendev.org service name from the system's FQDN in DNS. This way we could avoid publishing SSHFP RRs for the service name (or better still, publish different SSHFP RRs), but that means we'll need to separate out the ACME glue for DNS based X.509 cert renewals. That would likely not be too hard if we can just stop putting review01.opendev.org as one of the subject altnames. An alternative would be to sync the Gerrit mina-sshd API and system OpenSSH host keys, though that could present a degradation of security for the base system (maybe effectively not one we care about though?). Another alternative would be to just drop the SSHFP RRs for the Gerrit server, though that makes it inconsistent from the rest of our servers if we do. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From thierry at openstack.org Mon Aug 3 13:37:01 2020 From: thierry at openstack.org (Thierry Carrez) Date: Mon, 3 Aug 2020 15:37:01 +0200 Subject: Retiring the katacontainers tenant Message-ID: <913c53ba-04f9-8dce-7f12-a173163f05a6@openstack.org> Hi! As you may know, the Kata Containers project has no intent to add jobs to its Zuul tenant to drive its GitHub-based development. We originally set up two jobs (a DCO check and a WIP check) there as an experiment to drive more adoption. Since this did not succeed in convincing them to use more Opendev tooling, those jobs were migrated to GitHub actions a couple of weeks ago, and as a result Zuul no longer runs jobs for Kata GitHub repositories. I'm wondering what's the proper way to retire the katacontainers tenant and zuul job repository (kata-containers/zuul-config). I suspect removing the tenant is as simple as clearing out the tenant definition in project-config:zuul/main.yaml. However there might be a catch-22 there for the retirement of kata-containers/zuul-config. I was thinking about these steps: 1- Push a closing commit to kata-containers/zuul-config (that will remove the job definition for zuul-config itself, but IIRC config projects are not self-testing, so this should work ?) 2- Rename kata-containers/zuul-config to x/kata-zuul-config to clear out the namespace from https://opendev.org/explore/organizations (or should it just be left in place?) 3- Remove the katacontainers tenant definition from project-config:zuul/main.yaml (or should that be run before (2)?) Thanks in advance for your guidance. -- Thierry Carrez (ttx) From fungi at yuggoth.org Mon Aug 3 14:12:16 2020 From: fungi at yuggoth.org (Jeremy Stanley) Date: Mon, 3 Aug 2020 14:12:16 +0000 Subject: Retiring the katacontainers tenant In-Reply-To: <913c53ba-04f9-8dce-7f12-a173163f05a6@openstack.org> References: <913c53ba-04f9-8dce-7f12-a173163f05a6@openstack.org> Message-ID: <20200803141215.t6xd4n2x35esckvh@yuggoth.org> On 2020-08-03 15:37:01 +0200 (+0200), Thierry Carrez wrote: [...] > I suspect removing the tenant is as simple as clearing out the tenant > definition in project-config:zuul/main.yaml. However there might be a > catch-22 there for the retirement of kata-containers/zuul-config. Luckily it shouldn't be, since kata-containers/zuul-config is a trusted config project and therefore its retirement change won't be speculatively executed (as you note below). > I was thinking about these steps: > > 1- Push a closing commit to kata-containers/zuul-config (that will remove > the job definition for zuul-config itself, but IIRC config projects are not > self-testing, so this should work ?) > > 2- Rename kata-containers/zuul-config to x/kata-zuul-config to clear out the > namespace from https://opendev.org/explore/organizations (or should it just > be left in place?) I would skip this step, it needs a maintenance window for Gerrit downtime anyway, and could always be performed after removal if it becomes a source of confusion. The retirement readme in step 1 can always link to Kata's org on GitHub to hopefully eliminate that. > 3- Remove the katacontainers tenant definition from > project-config:zuul/main.yaml (or should that be run before (2)?) [...] It's entirely independent of step 2, so doesn't matter either way. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From fungi at yuggoth.org Mon Aug 3 20:48:22 2020 From: fungi at yuggoth.org (Jeremy Stanley) Date: Mon, 3 Aug 2020 20:48:22 +0000 Subject: review.opendev.org SSH key change? In-Reply-To: <20200801140953.ndttlvyb3pvfwt2e@yuggoth.org> References: <20200801140953.ndttlvyb3pvfwt2e@yuggoth.org> Message-ID: <20200803204822.c53mml5xgzgyrrmk@yuggoth.org> On 2020-08-01 14:09:53 +0000 (+0000), Jeremy Stanley wrote: [...] > The cleanest solution is probably going to be separating the > review.opendev.org service name from the system's FQDN in DNS. This > way we could avoid publishing SSHFP RRs for the service name (or > better still, publish different SSHFP RRs), but that means we'll > need to separate out the ACME glue for DNS based X.509 cert > renewals. That would likely not be too hard if we can just stop > putting review01.opendev.org as one of the subject altnames. [...] Clark just reminded me in the #opendev IRC channel that we already serve separate _acme-challenge.review and _acme-challenge.review01 CNAMEs to our acme zone, so nothing actually needs to change with SSL cert renewal verification. We can just replace the review CNAME with A/AAAA, copy the two CAA RRs from review01 to review, and generate the six new SSHFP RRs for the Gerrit API associated with the review hostname. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From fungi at yuggoth.org Mon Aug 3 21:01:52 2020 From: fungi at yuggoth.org (Jeremy Stanley) Date: Mon, 3 Aug 2020 21:01:52 +0000 Subject: review.opendev.org SSH key change? In-Reply-To: <20200803204822.c53mml5xgzgyrrmk@yuggoth.org> References: <20200801140953.ndttlvyb3pvfwt2e@yuggoth.org> <20200803204822.c53mml5xgzgyrrmk@yuggoth.org> Message-ID: <20200803210152.ebptxll6msxfmxci@yuggoth.org> On 2020-08-03 20:48:22 +0000 (+0000), Jeremy Stanley wrote: > On 2020-08-01 14:09:53 +0000 (+0000), Jeremy Stanley wrote: [...] > Clark just reminded me in the #opendev IRC channel that we already > serve separate _acme-challenge.review and _acme-challenge.review01 > CNAMEs to our acme zone, so nothing actually needs to change with > SSL cert renewal verification. We can just replace the review CNAME > with A/AAAA, copy the two CAA RRs from review01 to review, and > generate the six new SSHFP RRs for the Gerrit API associated with > the review hostname. In fact, as a stop gap, we can omit the SSHFP records for review.o.o initially, which will restore the prior situation for folks connecting to the API port. I'll push that now: https://review.opendev.org/744557 -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From cboylan at sapwetik.org Mon Aug 3 22:39:24 2020 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 03 Aug 2020 15:39:24 -0700 Subject: Meeting Agenda for August 4, 2020 Message-ID: <1cede009-1114-40a3-b24f-e32e7cd23f91@www.fastmail.com> We will meet on August 4, 2020 at 19:00UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements ** Third and final Virtual OpenDev event next week: August 10-11 * Actions from last meeting * Specs approval ** Authentication broker service, https://review.opendev.org/#/c/731838/ *** Not yet ready for approval but is worthy of review. * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates **** Gerritbot still outstanding todo item. *** Zuul as CD engine ** OpenDev *** Where is review-test in terms of replicating production? *** Non master repo HEAD support for new projects **** https://review.opendev.org/741277 Needed in Gerritlib first as well as a Gerritlib release with this change. **** https://review.opendev.org/741279 Can land once Gerritlib release is made with above change. * General topics ** Trusty Upgrade Progress (clarkb 20200721) *** Wiki updates ** Bup and Borg Backups (clarkb 20200804) *** review request https://review.opendev.org/741366 ** github 3rd party ci (clarkb 20200804) *** Update on progress *** https://review.opendev.org/#/q/topic:opendev-3pci * Open discussion From thierry at openstack.org Tue Aug 4 12:00:11 2020 From: thierry at openstack.org (Thierry Carrez) Date: Tue, 4 Aug 2020 14:00:11 +0200 Subject: Retiring the katacontainers tenant In-Reply-To: <20200803141215.t6xd4n2x35esckvh@yuggoth.org> References: <913c53ba-04f9-8dce-7f12-a173163f05a6@openstack.org> <20200803141215.t6xd4n2x35esckvh@yuggoth.org> Message-ID: <2886241e-a41e-78fb-e493-241db7b280ad@openstack.org> Jeremy Stanley wrote: > On 2020-08-03 15:37:01 +0200 (+0200), Thierry Carrez wrote: > [...] >> I suspect removing the tenant is as simple as clearing out the tenant >> definition in project-config:zuul/main.yaml. However there might be a >> catch-22 there for the retirement of kata-containers/zuul-config. > > Luckily it shouldn't be, since kata-containers/zuul-config is a > trusted config project and therefore its retirement change won't be > speculatively executed (as you note below). > >> I was thinking about these steps: >> >> 1- Push a closing commit to kata-containers/zuul-config (that will remove >> the job definition for zuul-config itself, but IIRC config projects are not >> self-testing, so this should work ?) >> >> 2- Rename kata-containers/zuul-config to x/kata-zuul-config to clear out the >> namespace from https://opendev.org/explore/organizations (or should it just >> be left in place?) > > I would skip this step, it needs a maintenance window for Gerrit > downtime anyway, and could always be performed after removal if it > becomes a source of confusion. The retirement readme in step 1 can > always link to Kata's org on GitHub to hopefully eliminate that. > >> 3- Remove the katacontainers tenant definition from >> project-config:zuul/main.yaml (or should that be run before (2)?) > [...] > > It's entirely independent of step 2, so doesn't matter either way. Thanks, I proposed https://review.opendev.org/#/q/topic:retire-kata-tenant -- Thierry Carrez (ttx) From cboylan at sapwetik.org Mon Aug 10 22:17:31 2020 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 10 Aug 2020 15:17:31 -0700 Subject: Meeting Agenda for August 11, 2020 Message-ID: We will meet August 11, 2020 in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval ** Authentication broker service, https://review.opendev.org/#/c/731838/ * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates **** Gerritbot in progress at https://review.opendev.org/#/c/745573/1 and parents **** Our bazel built Gerrit images have some broken plugins ***** javamelody is suffering https://groups.google.com/forum/#!topic/repo-discuss/ls6JV2q6b_g ***** Gerrit 2.16 image cannot find codemirror-editor html files which should be part of the codemirror-editor plugin ***** https://review.opendev.org/745595 pushed to trigger builds and logs so that we can investigate. *** Zuul as CD engine ** OpenDev *** Where is review-test in terms of replicating production? *** Non master repo HEAD support for new projects **** https://review.opendev.org/741277 Needed in Gerritlib first as well as a Gerritlib release with this change. **** https://review.opendev.org/741279 Can land once Gerritlib release is made with above change. * General topics ** Trusty Upgrade Progress (clarkb 20200721) *** Wiki updates ** Bup and Borg Backups (clarkb 20200811) *** https://review.opendev.org/741366 is ready to land when we are ready. ** github 3rd party ci (clarkb 20200811) *** Update on progress *** https://review.opendev.org/#/q/topic:opendev-3pci * Open discussion From ssbarnea at redhat.com Tue Aug 11 18:30:20 2020 From: ssbarnea at redhat.com (Sorin Sbarnea) Date: Tue, 11 Aug 2020 19:30:20 +0100 Subject: elastic-reckeck maintenance takeover Message-ID: Hi! I would like to request taking over the maintenance of elastic-recheck project. My team (triple-ci), really valued the tool and has a strong interest not only in keeping it alive but also improving it. While the team considered writing our own tool or forking it and changing it for our own use, adding missing features like multiple gerrit servers or multiple issue trackers, I advocated that we should instead become active maintainers and avoid the not-invented-here approach. I already did some work towards making the tool an easy to deploy standalone container. Starting new instances for testing or production should soon be as easy as a one-line command. I am more than happy to act as a liason between openstack-infra and triplo-ci and assure that no changes are made that could negatively impact openstack main deployment. I propose adding tripleo-ci-core gerrit group as core to the project but if there are concerns that the group is too wide, I can also provide a list of people that I trust to follow the expected review process. Best regards, Sorin Sbarnea Red Hat TripleO-CI From fungi at yuggoth.org Tue Aug 11 18:53:18 2020 From: fungi at yuggoth.org (Jeremy Stanley) Date: Tue, 11 Aug 2020 18:53:18 +0000 Subject: elastic-reckeck maintenance takeover In-Reply-To: References: Message-ID: <20200811185317.h2yuj3adzzzb4i7e@yuggoth.org> On 2020-08-11 19:30:20 +0100 (+0100), Sorin Sbarnea wrote: > I would like to request taking over the maintenance of > elastic-recheck project. > > My team (triple-ci), really valued the tool and has a strong > interest not only in keeping it alive but also improving it. > > While the team considered writing our own tool or forking it and > changing it for our own use, adding missing features like multiple > gerrit servers or multiple issue trackers, I advocated that we > should instead become active maintainers and avoid the > not-invented-here approach. > > I already did some work towards making the tool an easy to deploy > standalone container. Starting new instances for testing or > production should soon be as easy as a one-line command. I wouldn't consider it a takeover. OpenDev exists to promote collaboration on tools by the projects making use of them. That TripleO's CI subteam finds Elastic Recheck a useful tool and wants to take on its maintenance burden is exactly the sort of thing we want, I think. > I am more than happy to act as a liason between openstack-infra > and triplo-ci and assure that no changes are made that could > negatively impact openstack main deployment. Just a reminder, "openstack-infra" is no more, except still being a vestigial name for the OpenStack TaCT SIG's IRC channel. I think we've been considering Elastic Recheck a broader OpenDev service anyway, at least insofar as the basic mechanism by which it operates is easily extended to cover non-OpenStack projects hosted in OpenDev. It is, however, worth noting that the massive 6-node/6TB Elasticsearch cluster and 20 Logstash import workers on which this service relies represent a substantial chunk of our overall "control plane" resource utilization, and lately hasn't seemed to me that it's nearly popular enough to responsibly warrant the cost to our donors. > I propose adding tripleo-ci-core gerrit group as core to the > project but if there are concerns that the group is too wide, I > can also provide a list of people that I trust to follow the > expected review process. This sounds entirely reasonable to me. Thanks to the team for offering to help with it! -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From ssbarnea at redhat.com Tue Aug 11 19:16:59 2020 From: ssbarnea at redhat.com (Sorin Sbarnea) Date: Tue, 11 Aug 2020 20:16:59 +0100 Subject: elastic-recheck maintenance takeover In-Reply-To: <20200811185317.h2yuj3adzzzb4i7e@yuggoth.org> References: <20200811185317.h2yuj3adzzzb4i7e@yuggoth.org> Message-ID: To be clear, the goal is drive maintenance of the tool, not to "control" it. Working in sync with opende- infra is something I assumed. As you well said there is a signifiant deployment / operational part which is currently done with puppet, which is for good reason managed by opendev team. One of my goals was to simplify that part, migrate to ansible, container conversion being part of it. This is highly dependent on opendev and I am confident that I will get the needed help on achieving that goal. Sorin > On 11 Aug 2020, at 19:53, Jeremy Stanley wrote: > > service relies represent a substantial chunk of our overall "control -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Tue Aug 11 21:42:03 2020 From: smooney at redhat.com (Sean Mooney) Date: Tue, 11 Aug 2020 22:42:03 +0100 Subject: elastic-recheck maintenance takeover In-Reply-To: References: <20200811185317.h2yuj3adzzzb4i7e@yuggoth.org> Message-ID: <8c4dc8fec5f9f69659099639bd4f01cd4fccecec.camel@redhat.com> does the tool actully need maitance or do we just need to write more queries and submit them to the tool when we hit a bug. we used to have a strong recommentation that you never recheck a patch without first inspecting why it failed and when you leave a recheck add a reference to a bug. there was then a futuer recommentation that if we see the same bug being reject we add an elastic recheck query refernecing the bug(which you were ment to fined if there was not alredy one) so that if other hit the same error elastic recheck comments letting them know about the know issue. for example if you look a the nueton docs https://docs.openstack.org/neutron/pike/contributor/policies/gerrit-recheck.html "Please, do not recheck without providing the bug number for the failed job. For example, do not just put an empty “recheck” comment but find the related bug number and put a “recheck bug ######” comment instead. If a bug does not exist yet, create one so other team members can have a look. It helps us maintain better visibility of gate failures" nova used to have a similar doc but i cant find it currently. i think we might have removed it and driected people to the centralised contibutors guide at some point. the main contibutor guide docs descibe how to add a new recheck query https://docs.openstack.org/contributors/code-and-documentation/elastic-recheck.html but i dont think it common knoladge that we should add new queries, file bugs or at a minium state the reason for the recheck before rechecking as at least in nove we have not pushed contibutors to do this consitently for 2-3 year. when i first started working on openstack it was common knoladge and the core teams and other explained that you should avoid doing an empty recheck but the last time i brought this up downstream in ternally only about have of my team knew that that convention used to exist. On Tue, 2020-08-11 at 20:16 +0100, Sorin Sbarnea wrote: > To be clear, the goal is drive maintenance of the tool, not to "control" it. Working in sync with opende- infra is > something I assumed. > > As you well said there is a signifiant deployment / operational part which is currently done with puppet, which is for > good reason managed by opendev team. One of my goals was to simplify that part, migrate to ansible, container > conversion being part of it. > > This is highly dependent on opendev and I am confident that I will get the needed help on achieving that goal. > > Sorin > > > On 11 Aug 2020, at 19:53, Jeremy Stanley wrote: > > > > service relies represent a substantial chunk of our overall "control > > From cboylan at sapwetik.org Tue Aug 11 23:18:42 2020 From: cboylan at sapwetik.org (Clark Boylan) Date: Tue, 11 Aug 2020 16:18:42 -0700 Subject: elastic-recheck maintenance takeover In-Reply-To: <8c4dc8fec5f9f69659099639bd4f01cd4fccecec.camel@redhat.com> References: <20200811185317.h2yuj3adzzzb4i7e@yuggoth.org> <8c4dc8fec5f9f69659099639bd4f01cd4fccecec.camel@redhat.com> Message-ID: <70981cd5-f270-43e1-a737-546c5b96e6fe@www.fastmail.com> On Tue, Aug 11, 2020, at 2:42 PM, Sean Mooney wrote: > does the tool actully need maitance or do we just need to write more > queries and submit them to the tool when we hit > a bug. we used to have a strong recommentation that you never recheck a > patch without first inspecting why it > failed and when you leave a recheck add a reference to a bug. there was > then a futuer recommentation that if we > see the same bug being reject we add an elastic recheck query > refernecing the bug(which you were ment to fined if there > was not alredy one) so that if other hit the same error elastic recheck > comments letting them know about the know issue. Its a bit of both. The deployment of the tool could be modernized to python3 as well as related code updates. Ideally we'd also adjust it to be less openstack specific. A big part of that would be loading its queries from configs somewhere and not in the same repo. This way we can host queries for airship and starlingx and zuul too if we want. On the query side of things OpenStack's gate has a >50% unclassified status according to e-r right now. I'm happy if those that find it useful continue to keep it running. Whether that is openstack specific or something different. > > for example if you look a the nueton docs > https://docs.openstack.org/neutron/pike/contributor/policies/gerrit-recheck.html > > "Please, do not recheck without providing the bug number for the failed > job. For example, do not just put an empty > “recheck” comment but find the related bug number and put a “recheck > bug ######” comment instead. If a bug does not > exist yet, create one so other team members can have a look. It helps > us maintain better visibility of gate failures" We stopped this convention because e-r was working better. Humans were constantly identifying the wrong bugs or not attempting to be accurate and that data ended up being incredibly noisy. What we found instead was that the elastic-recheck data was far more accurate and it was better to invest in that. This worked really well as long as people were investing in it. I would not recommend we go back to human recheck tracking as it will just be noisy and lead to bad assumptions. > > nova used to have a similar doc but i cant find it currently. i think > we might have removed it and driected people to > the centralised contibutors guide at some point. the main contibutor > guide docs descibe how to add a new recheck query > https://docs.openstack.org/contributors/code-and-documentation/elastic-recheck.html > > but i dont think it common knoladge that we should add new queries, > file bugs or at a minium state the reason for > the recheck before rechecking as at least in nove we have not pushed > contibutors to do this consitently for 2-3 year. > when i first started working on openstack it was common knoladge and > the core teams and other explained that you should > avoid doing an empty recheck but the last time i brought this up > downstream in ternally only about have of my team knew > that that convention used to exist. > > > On Tue, 2020-08-11 at 20:16 +0100, Sorin Sbarnea wrote: > > To be clear, the goal is drive maintenance of the tool, not to "control" it. Working in sync with opende- infra is > > something I assumed. > > > > As you well said there is a signifiant deployment / operational part which is currently done with puppet, which is for > > good reason managed by opendev team. One of my goals was to simplify that part, migrate to ansible, container > > conversion being part of it. > > > > This is highly dependent on opendev and I am confident that I will get the needed help on achieving that goal. > > > > Sorin > > > > > On 11 Aug 2020, at 19:53, Jeremy Stanley wrote: > > > > > > service relies represent a substantial chunk of our overall "control > > > > > > > From smooney at redhat.com Wed Aug 12 00:05:33 2020 From: smooney at redhat.com (Sean Mooney) Date: Wed, 12 Aug 2020 01:05:33 +0100 Subject: elastic-recheck maintenance takeover In-Reply-To: <70981cd5-f270-43e1-a737-546c5b96e6fe@www.fastmail.com> References: <20200811185317.h2yuj3adzzzb4i7e@yuggoth.org> <8c4dc8fec5f9f69659099639bd4f01cd4fccecec.camel@redhat.com> <70981cd5-f270-43e1-a737-546c5b96e6fe@www.fastmail.com> Message-ID: On Tue, 2020-08-11 at 16:18 -0700, Clark Boylan wrote: > On Tue, Aug 11, 2020, at 2:42 PM, Sean Mooney wrote: > > does the tool actully need maitance or do we just need to write more > > queries and submit them to the tool when we hit > > a bug. we used to have a strong recommentation that you never recheck a > > patch without first inspecting why it > > failed and when you leave a recheck add a reference to a bug. there was > > then a futuer recommentation that if we > > see the same bug being reject we add an elastic recheck query > > refernecing the bug(which you were ment to fined if there > > was not alredy one) so that if other hit the same error elastic recheck > > comments letting them know about the know issue. > > Its a bit of both. The deployment of the tool could be modernized to python3 as well as related code updates. Ideally > we'd also adjust it to be less openstack specific. A big part of that would be loading its queries from configs > somewhere and not in the same repo. This way we can host queries for airship and starlingx and zuul too if we want. On > the query side of things OpenStack's gate has a >50% unclassified status according to e-r right now. > > I'm happy if those that find it useful continue to keep it running. Whether that is openstack specific or something > different. > > > > > for example if you look a the nueton docs > > https://docs.openstack.org/neutron/pike/contributor/policies/gerrit-recheck.html > > > > "Please, do not recheck without providing the bug number for the failed > > job. For example, do not just put an empty > > “recheck” comment but find the related bug number and put a “recheck > > bug ######” comment instead. If a bug does not > > exist yet, create one so other team members can have a look. It helps > > us maintain better visibility of gate failures" > > We stopped this convention because e-r was working better. Humans were constantly identifying the wrong bugs or not > attempting to be accurate and that data ended up being incredibly noisy. What we found instead was that the elastic- > recheck data was far more accurate and it was better to invest in that. This worked really well as long as people were > investing in it. I would not recommend we go back to human recheck tracking as it will just be noisy and lead to bad > assumptions. well the down side that i have found is that now often people dont look at why it failded an just recheck i try to at least do rechcek failed because i dont nessisary reference a bug or file one always but try at least flag why it failed > > > > > nova used to have a similar doc but i cant find it currently. i think > > we might have removed it and driected people to > > the centralised contibutors guide at some point. the main contibutor > > guide docs descibe how to add a new recheck query > > https://docs.openstack.org/contributors/code-and-documentation/elastic-recheck.html > > > > but i dont think it common knoladge that we should add new queries, > > file bugs or at a minium state the reason for > > the recheck before rechecking as at least in nove we have not pushed > > contibutors to do this consitently for 2-3 year. > > when i first started working on openstack it was common knoladge and > > the core teams and other explained that you should > > avoid doing an empty recheck but the last time i brought this up > > downstream in ternally only about have of my team knew > > that that convention used to exist. > > > > > > On Tue, 2020-08-11 at 20:16 +0100, Sorin Sbarnea wrote: > > > To be clear, the goal is drive maintenance of the tool, not to "control" it. Working in sync with opende- infra > > > is > > > something I assumed. > > > > > > As you well said there is a signifiant deployment / operational part which is currently done with puppet, which is > > > for > > > good reason managed by opendev team. One of my goals was to simplify that part, migrate to ansible, container > > > conversion being part of it. > > > > > > This is highly dependent on opendev and I am confident that I will get the needed help on achieving that goal. > > > > > > Sorin > > > > > > > On 11 Aug 2020, at 19:53, Jeremy Stanley wrote: > > > > > > > > service relies represent a substantial chunk of our overall "control > > > > > > > > > > > > > > From cboylan at sapwetik.org Mon Aug 17 22:52:04 2020 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 17 Aug 2020 15:52:04 -0700 Subject: Meeting agenda for August 18, 2020 Message-ID: <52677633-72b8-4071-9070-23cb2fb61c0e@www.fastmail.com> We will meet tomorrow, August 18, 2020 in #opendev-meeting at 19:00 UTC with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval ** Authentication broker service, https://review.opendev.org/#/c/731838/ * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates **** Gerritbot containerized. Needs https://review.opendev.org/746181 as final followup. **** https://review.opendev.org/#/c/745595/ fixes for our Gerrit images **** https://review.opendev.org/#/c/746335/ add missing files to config management *** Zuul as CD engine ** OpenDev *** Where is review-test in terms of replicating production? *** Luca has offered to do a conference call with us. Let me know if interested and I'll include you for scheduling if/when that happens. *** Non master repo HEAD support for new projects **** https://review.opendev.org/741277 Needed in Gerritlib first as well as a Gerritlib release with this change. **** https://review.opendev.org/741279 Can land once Gerritlib release is made with above change. * General topics ** Bup and Borg Backups (clarkb 20200818) *** https://review.opendev.org/741366 is ready to land when we are ready. ** github 3rd party ci (clarkb 20200818) *** Update on progress *** https://review.opendev.org/#/q/topic:opendev-3pci ** Making ask.openstack.org read only (clarkb 20200818) *** https://review.opendev.org/#/c/746497/ ** PTG PLanning (clarkb 20200818) *** October PTG registration is now open: https://www.openstack.org/ptg/ *** OpenDev planning stats here: https://etherpad.opendev.org/opendev-ptg-planning-oct-2020 ** Trusty Upgrade Progress (clarkb 20200721) *** Wiki updates * Open discussion From gmann at ghanshyammann.com Tue Aug 18 18:25:28 2020 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Tue, 18 Aug 2020 13:25:28 -0500 Subject: elastic-reckeck maintenance takeover In-Reply-To: References: Message-ID: <17402d1a679.ca49e76a488759.2512742374205632231@ghanshyammann.com> ---- On Tue, 11 Aug 2020 13:30:20 -0500 Sorin Sbarnea wrote ---- > Hi! > > I would like to request taking over the maintenance of elastic-recheck project. > > My team (triple-ci), really valued the tool and has a strong interest not only in keeping it alive but also improving it. > > While the team considered writing our own tool or forking it and changing it for our own use, adding missing features like multiple gerrit servers or multiple issue trackers, I advocated that we should instead become active maintainers and avoid the not-invented-here approach. > > I already did some work towards making the tool an easy to deploy standalone container. Starting new instances for testing or production should soon be as easy as a one-line command. > > I am more than happy to act as a liason between openstack-infra and triplo-ci and assure that no changes are made that could negatively impact openstack main deployment. > > I propose adding tripleo-ci-core gerrit group as core to the project but if there are concerns that the group is too wide, I can also provide a list of people that I trust to follow the expected review process. >From QA perspective, we are ok with this. Unfortunately, the QA team does not have much bandwidth and also no java expert to take over the maintenance. -gmann > > Best regards, > Sorin Sbarnea > Red Hat TripleO-CI > > From cboylan at sapwetik.org Mon Aug 24 21:24:45 2020 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 24 Aug 2020 14:24:45 -0700 Subject: Meeting Agenda for August 25, 2020 Message-ID: We will meet August 25, 2020 at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates **** https://review.opendev.org/#/c/746335/ add missing files to config management *** Zuul as CD engine ** OpenDev *** Where is review-test in terms of replicating production? **** Has git repos on root disk. Gerrit is not running. *** Luca has offered to do a conference call with us. Let me know if interested and I'll include you for scheduling if/when that happens. *** Non master repo HEAD support for new projects **** https://review.opendev.org/741277 Needed in Gerritlib first as well as a Gerritlib release with this change. **** https://review.opendev.org/741279 Can land once Gerritlib release is made with above change. * General topics ** Bup and Borg Backups (clarkb 20200825) *** https://review.opendev.org/741366 is ready to land when we are ready. ** github 3rd party ci (clarkb 20200825) *** Update on progress ** Making ask.openstack.org read only (clarkb 20200818) *** https://review.opendev.org/#/c/746497/ ** PTG PLanning (clarkb 20200825) *** October PTG registration is now open: https://www.openstack.org/ptg/ *** OpenDev planning stats here: https://etherpad.opendev.org/opendev-ptg-planning-oct-2020 ** Trusty Upgrade Progress (clarkb 20200721) *** Wiki updates * Open discussion From cboylan at sapwetik.org Mon Aug 31 23:13:37 2020 From: cboylan at sapwetik.org (Clark Boylan) Date: Mon, 31 Aug 2020 16:13:37 -0700 Subject: Team Meeting Agenda for September 1, 2020 Message-ID: <6972a361-cc4e-49f5-a732-2a2d07efde8e@www.fastmail.com> We will meet tomorrow, September 1, at 19:00 UTC in #opendev-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Where is review-test in terms of replicating production? **** Has git repos on root disk. Gerrit is not running. **** We should consider spinning up a new host or modifying the existing host to match productions cinder volume usage for git repos. *** Luca has offered to do a conference call with us. Let me know if interested and I'll include you for scheduling if/when that happens. *** Non master repo HEAD support for new projects has landed to gitea, gerritlib, and jeepyb. I think we can offer this to new projects now. **** Keep in mind that this will likely be a learning experience and we should communicate that to our "volunteers" * General topics ** Setuptools 50 release (clarkb 20200901) *** Changes installation path of packages on Debian and Ubuntu *** Seems to have a number of bugs with python3 < 3.8 ** Bup and Borg Backups (clarkb 20200901) *** https://review.opendev.org/741366 is ready to land when we are ready. ** Making ask.openstack.org read only (clarkb 20200901) *** https://review.opendev.org/#/c/746497/ *** Any reason to not land this now? Feedback has been onboard so far. ** PTG PLanning (clarkb 20200901) *** October PTG registration is now open: https://www.openstack.org/ptg/ *** OpenDev planning stats here: https://etherpad.opendev.org/opendev-ptg-planning-oct-2020 ** Trusty Upgrade Progress (clarkb 20200901) *** Wiki updates * Open discussion