From cboylan at sapwetik.org  Mon May  3 20:15:49 2021
From: cboylan at sapwetik.org (Clark Boylan)
Date: Mon, 03 May 2021 13:15:49 -0700
Subject: Team Meeting Agenda for May 4, 2021
Message-ID: <34251a07-37fc-4eb9-8a76-c3aa5363ca92@www.fastmail.com>

We will meet at 19:00 UTC in #opendev-meeting on May 4, 2021 with this agenda:

== Agenda for next meeting ==

* Announcements
** Clark out during May 11 meeting time. Need a volunteer chair or can cancel that meeting.

* Actions from last meeting

* Specs approval

* Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.)
** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management]
*** topic:update-cfg-mgmt
*** Zuul as CD engine
** OpenDev
*** Gerrit account inconsistencies
**** All preferred emails lack external ids issues have been corrected. All group loops have been corrected.
**** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?)
**** Next steps
***** More "dangerous" list has been generated. Should still be safe-ish particularly if we disable the accounts first.
*** Configuration tuning
**** Reduce the number of ssh threads. Possibly create bot/batch user groups and thread counts as part of this.
**** https://groups.google.com/g/repo-discuss/c/BQKxAfXBXuo Upstream conversation with people struggling with similar problems.
*** Update our base job's nodeset
**** https://review.opendev.org/789098
*** Gerrit global config simplification
**** https://review.opendev.org/789383

* General topics
** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210504)
*** Enable Xenial -> Bionic/Focal system upgrades
*** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here
*** Zuul service host updates in progress now. Zuul scheduler is last remaining server that needs an upgrade in the zuul cluster.
** openEuler patches (ianw 20210504)
*** TC level, or being too officious 
** InMotion Cloud reorganization (clarkb 20210504)
*** Currently limited by number of IP addresses available
*** We could deploy an executor in the cloud then set up test nodes without direct external access
*** Any held nodes would need floating IPs attached to them to be accessible
** Removing registration requirement from our IRC channels (clarkb 20210504)
*** TheJulia asks if we think this is still necessary.
*** Looking at the last month or so of eavesdrop logs it seems we had ~1.5 spam attempts in the unregistered channel
** Switching artifact signing keys from RSA to ECC
*** https://review.opendev.org/789062

* Open discussion


From ssbarnea at redhat.com  Wed May  5 05:54:32 2021
From: ssbarnea at redhat.com (Sorin Sbarnea)
Date: Wed, 5 May 2021 06:54:32 +0100
Subject: [service-announce] Switching default nodeset in base job to
 ubuntu-focal
In-Reply-To: <20210504214827.tdiifdu7mepcjlmn@yuggoth.org>
References: <20210504214827.tdiifdu7mepcjlmn@yuggoth.org>
Message-ID: <CAFrDqBztTvf0Dyh8o9vbB-L0fP9mB5Av9Egp2hhvz_nZFCy-wg@mail.gmail.com>

I support the move as I am aware of several issues that are sorted (or
simplified) by using a newer release.

It worth mentioning that GHA has a nice way to announce pending upgrade of
their “nodeset” upgrades: they include a visible warning on each executed
job that use a node that is pending a change, something like: “This job is
using ubuntu-latest which will soon be switched from x to y. Use specific
labels if you want to avoid such a switch.” They do no mention the exact
date of the switch but apparently these warnings to stay on for at least
one month.

On Tue, 4 May 2021 at 22:49, Jeremy Stanley <fungi at yuggoth.org> wrote:

> The OpenDev Collaboratory is normally somewhat conservative about
> when it changes its Zuul node "default" (that is, the nodeset in its
> base job). It's been ubuntu-bionic since at least July of 2019 when
> we first moved our base job into its own repository, and
> ubuntu-focal nodes have been available in our deployment since
> before Ubuntu's 20.04 LTS was officially released more than a year
> ago. All that is to say, updating our default is overdue.
>
> On Tuesday, 2021-05-18 (two weeks from today) we'll approve
> https://review.opendev.org/789098 to update the nodeset to
> ubuntu-focal for any jobs which don't specify their own or inherit
> from another job which does. We've already merged a change which
> updates our base-test job to reflect this, so you can test either by
> explicitly setting your jobs to use an ubuntu-focal nodeset or to
> inherit from base-test (though we don't recommend merging changes
> for the latter, just rely on speculative execution to identify
> problems which you may encounter when the default changes).
>
> Please reply to the service-discuss mailing list with any questions
> or concerns. You should also feel free to leave comments on the
> change linked above.
> --
> Jeremy Stanley
> _______________________________________________
> service-announce mailing list
> service-announce at lists.opendev.org
> http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce
>
-- 
--
/zbr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opendev.org/pipermail/service-discuss/attachments/20210505/a202d0f1/attachment.html>

From mkopec at redhat.com  Fri May  7 11:48:36 2021
From: mkopec at redhat.com (Martin Kopec)
Date: Fri, 7 May 2021 13:48:36 +0200
Subject: [devstack][infra] POST_FAILURE on export-devstack-journal :
 Export journal
In-Reply-To: <7626869f-dab3-41df-a40b-dafa20dcfaf4@www.fastmail.com>
References: <CAKZGdE3zSKA3bsYj3z_fw+ypiriVPtiZNefDBwsz7ChLWiHd4w@mail.gmail.com>
 <20210406160247.gevud2hlvodg7jzt@yuggoth.org>
 <CAKZ_x7_f-x3dqh1C38rbGZDt=nPNg1+9xtuHyOXLhfKVgxLndg@mail.gmail.com>
 <7626869f-dab3-41df-a40b-dafa20dcfaf4@www.fastmail.com>
Message-ID: <CAKZGdE0n2QwcsdwoG=5z9K329dPFNL==uj8fij1ZEc2H20+d9g@mail.gmail.com>

hmm, seems like we have hit the issue again, however in a different job now:
Latest logs:
https://zuul.opendev.org/t/openstack/build/0565c3d252194f9ba67f4af20e8be65d
Link to the review where it occurred:
https://review.opendev.org/c/osf/refstack-client/+/788743


On Tue, 6 Apr 2021 at 18:47, Clark Boylan <cboylan at sapwetik.org> wrote:

> On Tue, Apr 6, 2021, at 9:11 AM, Radosław Piliszek wrote:
> > On Tue, Apr 6, 2021 at 6:02 PM Jeremy Stanley <fungi at yuggoth.org> wrote:
> > > Looking at the error, I strongly suspect memory exhaustion. We could
> > > try tuning xz to use less memory when compressing.
>
> Worth noting that we continue to suspect memory pressure, and in
> particular diving into swap, for random failures that appear timing or
> performance related. I still think it would be a helpful exercise for
> OpenStack to look at its memory consumption (remember end users will
> experience this too) and see if there are any unexpected areas of memory
> use. I think the last time i skimmed logs the privsep daemon was a large
> consumer because we separate instance is run for each service and they all
> add up.
>
> >
> > That was my hunch as well, hence why I test using gzip.
> >
> > On Tue, Apr 6, 2021 at 5:51 PM Clark Boylan <cboylan at sapwetik.org>
> wrote:
> > >
> > > On Tue, Apr 6, 2021, at 8:14 AM, Radosław Piliszek wrote:
> > > > I am testing whether replacing xz with gzip would solve the problem
> [1] [2].
> > >
> > > The reason we used xz is that the files are very large and gz
> compression is very poor compared to xz for these files and these files are
> not really human readable as is (you need to load them into journald
> first). Let's test it and see what the gz file sizes look like but if they
> are still quite large then this is unlikely to be an appropriate fix.
> >
> > Let's see how bad the file sizes are.
> > If they are acceptable, we can keep gzip and be happy.
> > Otherwise we try to tune the params to make xz a better citizen as
> > fungi suggested.
> >
> > -yoctozepto
> >
> >
>
>

-- 
Martin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opendev.org/pipermail/service-discuss/attachments/20210507/66251edd/attachment.html>

From fungi at yuggoth.org  Fri May  7 13:34:56 2021
From: fungi at yuggoth.org (Jeremy Stanley)
Date: Fri, 7 May 2021 13:34:56 +0000
Subject: [devstack][infra] POST_FAILURE on export-devstack-journal :
 Export journal
In-Reply-To: <CAKZGdE0n2QwcsdwoG=5z9K329dPFNL==uj8fij1ZEc2H20+d9g@mail.gmail.com>
References: <CAKZGdE3zSKA3bsYj3z_fw+ypiriVPtiZNefDBwsz7ChLWiHd4w@mail.gmail.com>
 <20210406160247.gevud2hlvodg7jzt@yuggoth.org>
 <CAKZ_x7_f-x3dqh1C38rbGZDt=nPNg1+9xtuHyOXLhfKVgxLndg@mail.gmail.com>
 <7626869f-dab3-41df-a40b-dafa20dcfaf4@www.fastmail.com>
 <CAKZGdE0n2QwcsdwoG=5z9K329dPFNL==uj8fij1ZEc2H20+d9g@mail.gmail.com>
Message-ID: <20210507133456.aaah2b52g35aaat2@yuggoth.org>

On 2021-05-07 13:48:36 +0200 (+0200), Martin Kopec wrote:
> hmm, seems like we have hit the issue again, however in a different job now:
> Latest logs:
> https://zuul.opendev.org/t/openstack/build/0565c3d252194f9ba67f4af20e8be65d
> Link to the review where it occurred:
> https://review.opendev.org/c/osf/refstack-client/+/788743
[...]

It was addressed in the master branch a month ago with
https://review.opendev.org/784964 wasn't backported to any older
branches (or if it was then the backports haven't merged yet).
Looking at the zuul._inheritance_path from the inventory for your
build, it seems to have used stable/wallaby of devstack rather than
master, which explains why you're still seeing xzip used.
-- 
Jeremy Stanley
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 963 bytes
Desc: not available
URL: <http://lists.opendev.org/pipermail/service-discuss/attachments/20210507/ebe7c072/attachment.sig>

From mkopec at redhat.com  Sun May  9 08:54:17 2021
From: mkopec at redhat.com (Martin Kopec)
Date: Sun, 9 May 2021 10:54:17 +0200
Subject: [devstack][infra] POST_FAILURE on export-devstack-journal :
 Export journal
In-Reply-To: <20210507133456.aaah2b52g35aaat2@yuggoth.org>
References: <CAKZGdE3zSKA3bsYj3z_fw+ypiriVPtiZNefDBwsz7ChLWiHd4w@mail.gmail.com>
 <20210406160247.gevud2hlvodg7jzt@yuggoth.org>
 <CAKZ_x7_f-x3dqh1C38rbGZDt=nPNg1+9xtuHyOXLhfKVgxLndg@mail.gmail.com>
 <7626869f-dab3-41df-a40b-dafa20dcfaf4@www.fastmail.com>
 <CAKZGdE0n2QwcsdwoG=5z9K329dPFNL==uj8fij1ZEc2H20+d9g@mail.gmail.com>
 <20210507133456.aaah2b52g35aaat2@yuggoth.org>
Message-ID: <CAKZGdE1OageZMqU-ayi_oSGdT7f9T8tT3St7avh8gXFpWqTC-g@mail.gmail.com>

right, thank you.
I've proposed a backport to wallaby:
https://review.opendev.org/c/openstack/devstack/+/790353
and verifying it solves the problem here:
https://review.opendev.org/c/osf/refstack-client/+/788743

On Fri, 7 May 2021 at 15:35, Jeremy Stanley <fungi at yuggoth.org> wrote:

> On 2021-05-07 13:48:36 +0200 (+0200), Martin Kopec wrote:
> > hmm, seems like we have hit the issue again, however in a different job
> now:
> > Latest logs:
> >
> https://zuul.opendev.org/t/openstack/build/0565c3d252194f9ba67f4af20e8be65d
> > Link to the review where it occurred:
> > https://review.opendev.org/c/osf/refstack-client/+/788743
> [...]
>
> It was addressed in the master branch a month ago with
> https://review.opendev.org/784964 wasn't backported to any older
> branches (or if it was then the backports haven't merged yet).
> Looking at the zuul._inheritance_path from the inventory for your
> build, it seems to have used stable/wallaby of devstack rather than
> master, which explains why you're still seeing xzip used.
> --
> Jeremy Stanley
>


-- 
Martin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opendev.org/pipermail/service-discuss/attachments/20210509/8d6bf9de/attachment.html>

From cboylan at sapwetik.org  Mon May 10 17:37:09 2021
From: cboylan at sapwetik.org (Clark Boylan)
Date: Mon, 10 May 2021 10:37:09 -0700
Subject: Team Meeting on May 11, 2021 Cancelled
Message-ID: <3d5e52e6-873f-4b56-8cbf-6ac6911b3b50@www.fastmail.com>

Hello everyone,

Last week I mentioned that I won't be able to run the meeting tomorrow and others seemed to think skipping it this week would be fine. I have not heard any desire to have a meeting since, so I'll make it official and cancel the meeting now.

We will see you next week on May 18, 2021 at 19:00UTC in #opendev-meeting.

Clark


From cboylan at sapwetik.org  Mon May 17 20:37:10 2021
From: cboylan at sapwetik.org (Clark Boylan)
Date: Mon, 17 May 2021 13:37:10 -0700
Subject: Team meeting agenda for May 18, 2021
Message-ID: <f63f578c-0d87-40b8-b30f-d7f6e02f19ba@www.fastmail.com>

We will meet with this agenda on May 18, 2021 at 19:00 UTC in #opendev-meeting:

== Agenda for next meeting ==

* Announcements

* Actions from last meeting

* Specs approval

* Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.)
** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management]
*** topic:update-cfg-mgmt
*** Zuul as CD engine
** OpenDev
*** Gerrit account inconsistencies
**** All preferred emails lack external ids issues have been corrected. All group loops have been corrected.
**** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?)
**** Next steps
***** More "dangerous" list has been generated. Should still be safe-ish particularly if we disable the accounts first.
*** Configuration tuning
**** Reduce the number of ssh threads. Possibly create bot/batch user groups and thread counts as part of this.
**** https://groups.google.com/g/repo-discuss/c/BQKxAfXBXuo Upstream conversation with people struggling with similar problems.
*** Update our base job's nodeset
**** https://review.opendev.org/789098

* General topics
** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210518)
*** Enable Xenial -> Bionic/Focal system upgrades
*** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here
*** Zuul is done. Mailman next
**** https://review.opendev.org/c/opendev/system-config/+/789622
** Refreshing non LE certs that expire in just under a month (clarkb 20210518)
*** Shutdown/remove: ask.o.o
*** Refresh as yearly cert or try to LE: ethercalc, wiki, translate, storyboard
*** Meeting with foundation to discuss: openstackid and openstackid-dev
** Cleanup of too small swap setups on newer servers (clarkb 20210518)
*** make_swap.sh was updated to accidentally limit swap size to 8MB when we wanted an 8GB limit
** Removing registration requirement from our IRC channels (clarkb 20210518)
*** TheJulia asks if we think this is still necessary.
*** Looking at the last month or so of eavesdrop logs it seems we had ~1.5 spam attempts in the unregistered channel
*** https://review.opendev.org/c/openstack/project-config/+/791818
** "Toggle CI" in Gerrit web UI (rosmaita 20210518)
*** I thought this was on the [https://etherpad.opendev.org/p/gerrit-3.2-post-upgrade-notes 3.2 post-upgrade etherpad] but I can't find it.  We used to have this button in the old web interface
*** The Cinder team is looking for a solution to Change Log pollution from our 3rd party CI systems
**** example: https://review.opendev.org/c/openstack/cinder/+/790796/
*** actual reviewer comments get lost in the Change Log
*** the "Only comments" toggle hides all the "Added to cc:" lines, which is nice.  It also hides the Zuul "build succeeded/failed" comments ... what do we need to do so this will work for other CI responses?
**** The "Only Comments" toggle and the "Zuul Summary" tab are what you should be using. To filter with only comments the CI system needs to set an autogenerated tag on its comments. Zuul does this unconditionally if you set it up to report over http. For the Zuul Summary tab the CI system must have a Full Name in gerrit of "foo name CI" and the result format must match Zuul's.
** Scheduling project renames (clarkb 20210518)
*** Our playbook(s) that do renames likely need updating since the last gerrit upgrade.
**** We can test this with our functional testing of gerrit too.

* Open discussion


From cboylan at sapwetik.org  Wed May 19 17:04:59 2021
From: cboylan at sapwetik.org (Clark Boylan)
Date: Wed, 19 May 2021 10:04:59 -0700
Subject: Updating configuration management for our listservs
Message-ID: <674c34fa-4d9a-4693-98af-8493b50fd648@www.fastmail.com>

Hello everyone,

Just a quick update that we are updating the configuration management for our listservs. This email serves as a quick sanity check that everything is continuing to function as expected. If you notice anything odd please let us know.

Clark


From fungi at yuggoth.org  Fri May 21 19:29:46 2021
From: fungi at yuggoth.org (Jeremy Stanley)
Date: Fri, 21 May 2021 19:29:46 +0000
Subject: Reevaluating choice of IRC network for our service bots
Message-ID: <20210521192945.2p3qsmo2iihh2ckv@yuggoth.org>

As most people reading this are likely aware, there has been a
recent change in management for the Freenode IRC network. According
to public statements[0], the former staff have all left and the
network is in entirely new hands. Setting aside arguments about
which parties are in the wrong, further stability of the service we
provide is once again brought into question. We've seen evidence of
(much) lower levels of abuse mitigation over the past few days, with
a marked increase in registered accounts engaging in harassment over
extended periods of time across our channels without getting banned
from the network.

Due to disagreements between Freenode and other large IRC networks
and frequent related attacks against Freenode's servers, our
communities have experienced many extended periods of instability
there. In order to insure against a potential collapse of the
network, the OpenDev Collaboratory (and the OpenStack Project
Infrastructure before it) has for many years maintained a sort of
evacuation plan: For the multitude of channels overseen by OpenDev,
we also keep equivalent channel registrations on the Open and Free
Technology Community (OFTC) IRC network[1], and have communicated
with its operators on multiple occasions about their willingness and
ability to host our communities there should the need arise.

This week's events are, unfortunately, not an isolated incident. The
systems administrators for OpenDev have encouraged a move to OFTC on
multiple occasions, with the first formal proposal in March 2014[2].
In prior cases, the impact to the community for such a switch
outweighed the perceived benefits, but as we've heard growing
displeasure with Freenode from representatives of many of the
projects we serve, it is time once more to revisit whether we should
enact our longstanding evacuation plans.

Because the OpenDev Collaboratory exists to serve the projects it
hosts, input from project representatives and the Advisory Council
is critical in deciding major changes to services. If projects were
to move their channels to OFTC, the transition would not be
seamless. In particular, differences in authentication (no SASL
support, but you can authenticate your connection with a
certificate[3] if you don't want to identify to the NickServ after
connecting), permissions model (OFTC has coarse-grained RBAC
designed to reduce channel mismanagement), and NickServ and ChanServ
command syntax are among the challenges they're likely to face. The
same IRC nicks may also not be available in some cases, though due
to the generally smaller size of OFTC compared to Freenode this
hopefully won't come up for too many users. On a positive note, we
may be able to go back to not requiring nick registration just to
join channels, easing onboarding for newcomers.

If there is a consensus to move OpenDev's services to OFTC, or any
other IRC network for that matter, this will entail a bit of
development effort in order to accommodate the differences mentioned
above. Please do note, however, that moving to a network other than
OFTC would additionally mean we can't guarantee the availability of
the same IRC channel names either, so that needs to be weighed in
any decision. Some discussion[4][5][6] is already underway within
the OpenStack community as to what they'd prefer, but we haven't
heard much from other projects yet, so please do respond with your
thoughts on the matter.

[0] https://fuchsnet.ch/freenode-resign-letter.txt
[1] https://www.oftc.net/
[2] http://lists.openstack.org/pipermail/openstack-dev/2014-March/028783.html
[3] https://www.oftc.net/NickServ/CertFP/
[4] http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022468.html
[5] http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022539.html
[6] http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23openstack-tc.2021-05-20.log.html#t2021-05-20T15:33:18
-- 
Jeremy Stanley
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 963 bytes
Desc: not available
URL: <http://lists.opendev.org/pipermail/service-discuss/attachments/20210521/06dccfdd/attachment.sig>

From iwienand at redhat.com  Mon May 24 05:46:12 2021
From: iwienand at redhat.com (Ian Wienand)
Date: Mon, 24 May 2021 15:46:12 +1000
Subject: Reevaluating choice of IRC network for our service bots
In-Reply-To: <20210521192945.2p3qsmo2iihh2ckv@yuggoth.org>
References: <20210521192945.2p3qsmo2iihh2ckv@yuggoth.org>
Message-ID: <YKs9pIgjCrXYG978@fedora19.localdomain>

On Fri, May 21, 2021 at 07:29:46PM +0000, Jeremy Stanley wrote:
65;6401;1c> If there is a consensus to move OpenDev's services to OFTC, or any
> other IRC network for that matter, this will entail a bit of
> development effort

Is there a procedure to obtain consensus?

> so please do respond with your thoughts on the matter.

I feel like I've read more than a normal person would/should have, and
I still have no idea what's going on.  Whatever the behind-the-scenes
reality is, the unfortunate result is that being involved with our
projects has acquired the additional step of "overcome existential
crisis of using freenode".

dmsimard's comment [1] is the best one I've seen tracking who is going
where amongst peer projects.  Despite communications like [2] it would
seem to take a phoenix-esque reputation rehabilitation to revert these
already made decisions [3].

To me, moving from freenode is looking less like something we need to
decide, but more like action we need to take for self-preservation
(i.e. not alienating our most important resource; developers).

Per the ~7 years of occasional messages, OFTC seems like a fine
choice.

-i

[1] https://github.com/ansible-community/community-topics/issues/19#issuecomment-844204319
[2] https://freenode.net/news/freenode-is-foss
[3] companies who have had contentious relationships with free
    software over a long period of time have undoubtedly managed to go
    on to later capture a large portion of open source development,
    even when their platforms don't provide for open tools or
    governance; so never say never...

-i



From gmann at ghanshyammann.com  Mon May 24 16:19:12 2021
From: gmann at ghanshyammann.com (Ghanshyam Mann)
Date: Mon, 24 May 2021 11:19:12 -0500
Subject: Reevaluating choice of IRC network for our service bots
In-Reply-To: <20210521192945.2p3qsmo2iihh2ckv@yuggoth.org>
References: <20210521192945.2p3qsmo2iihh2ckv@yuggoth.org>
Message-ID: <1799f2c5355.b70a93d1153288.1617320239230982797@ghanshyammann.com>

 ---- On Fri, 21 May 2021 14:29:46 -0500 Jeremy Stanley <fungi at yuggoth.org> wrote ----
 > As most people reading this are likely aware, there has been a
 > recent change in management for the Freenode IRC network. According
 > to public statements[0], the former staff have all left and the
 > network is in entirely new hands. Setting aside arguments about
 > which parties are in the wrong, further stability of the service we
 > provide is once again brought into question. We've seen evidence of
 > (much) lower levels of abuse mitigation over the past few days, with
 > a marked increase in registered accounts engaging in harassment over
 > extended periods of time across our channels without getting banned
 > from the network.
 > 
 > Due to disagreements between Freenode and other large IRC networks
 > and frequent related attacks against Freenode's servers, our
 > communities have experienced many extended periods of instability
 > there. In order to insure against a potential collapse of the
 > network, the OpenDev Collaboratory (and the OpenStack Project
 > Infrastructure before it) has for many years maintained a sort of
 > evacuation plan: For the multitude of channels overseen by OpenDev,
 > we also keep equivalent channel registrations on the Open and Free
 > Technology Community (OFTC) IRC network[1], and have communicated
 > with its operators on multiple occasions about their willingness and
 > ability to host our communities there should the need arise.
 > 
 > This week's events are, unfortunately, not an isolated incident. The
 > systems administrators for OpenDev have encouraged a move to OFTC on
 > multiple occasions, with the first formal proposal in March 2014[2].
 > In prior cases, the impact to the community for such a switch
 > outweighed the perceived benefits, but as we've heard growing
 > displeasure with Freenode from representatives of many of the
 > projects we serve, it is time once more to revisit whether we should
 > enact our longstanding evacuation plans.
 > 
 > Because the OpenDev Collaboratory exists to serve the projects it
 > hosts, input from project representatives and the Advisory Council
 > is critical in deciding major changes to services. If projects were
 > to move their channels to OFTC, the transition would not be
 > seamless. In particular, differences in authentication (no SASL
 > support, but you can authenticate your connection with a
 > certificate[3] if you don't want to identify to the NickServ after
 > connecting), permissions model (OFTC has coarse-grained RBAC
 > designed to reduce channel mismanagement), and NickServ and ChanServ
 > command syntax are among the challenges they're likely to face. The
 > same IRC nicks may also not be available in some cases, though due
 > to the generally smaller size of OFTC compared to Freenode this
 > hopefully won't come up for too many users. On a positive note, we
 > may be able to go back to not requiring nick registration just to
 > join channels, easing onboarding for newcomers.
 > 
 > If there is a consensus to move OpenDev's services to OFTC, or any
 > other IRC network for that matter, this will entail a bit of
 > development effort in order to accommodate the differences mentioned
 > above. Please do note, however, that moving to a network other than
 > OFTC would additionally mean we can't guarantee the availability of
 > the same IRC channel names either, so that needs to be weighed in
 > any decision. Some discussion[4][5][6] is already underway within
 > the OpenStack community as to what they'd prefer, but we haven't
 > heard much from other projects yet, so please do respond with your
 > thoughts on the matter.

Thanks, Jeremy and opendev team for starting the consolidated effort.

As you know, in OpenStack we are still discussing it on ML or in
openstack-tc channel (adding this in TC meeting agenda too).

One question though:

Does openinfra projects have their individual choice which one they want to move or continue OR 
there will be only a single network provided/supported by opendev based on the majority of openinfra
projects feedback.

-gmann

 > 
 > [0] https://fuchsnet.ch/freenode-resign-letter.txt
 > [1] https://www.oftc.net/
 > [2] http://lists.openstack.org/pipermail/openstack-dev/2014-March/028783.html
 > [3] https://www.oftc.net/NickServ/CertFP/
 > [4] http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022468.html
 > [5] http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022539.html
 > [6] http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23openstack-tc.2021-05-20.log.html#t2021-05-20T15:33:18
 > -- 
 > Jeremy Stanley
 > 


From fungi at yuggoth.org  Mon May 24 20:23:56 2021
From: fungi at yuggoth.org (Jeremy Stanley)
Date: Mon, 24 May 2021 20:23:56 +0000
Subject: Reevaluating choice of IRC network for our service bots
In-Reply-To: <YKs9pIgjCrXYG978@fedora19.localdomain>
References: <20210521192945.2p3qsmo2iihh2ckv@yuggoth.org>
 <YKs9pIgjCrXYG978@fedora19.localdomain>
Message-ID: <20210524202356.swzdgl7qbk5vfuis@yuggoth.org>

On 2021-05-24 15:46:12 +1000 (+1000), Ian Wienand wrote:
> On Fri, May 21, 2021 at 07:29:46PM +0000, Jeremy Stanley wrote:
> > If there is a consensus to move OpenDev's services to OFTC, or any
> > other IRC network for that matter, this will entail a bit of
> > development effort
> 
> Is there a procedure to obtain consensus?
[...]

If there's a clear consensus then I expect it to become readily
apparent. If there's no clear choice, I agree our provisional
governance document lacks a solution for reaching consensus.
-- 
Jeremy Stanley
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 963 bytes
Desc: not available
URL: <http://lists.opendev.org/pipermail/service-discuss/attachments/20210524/3f640ffd/attachment.sig>

From fungi at yuggoth.org  Mon May 24 20:34:19 2021
From: fungi at yuggoth.org (Jeremy Stanley)
Date: Mon, 24 May 2021 20:34:19 +0000
Subject: Reevaluating choice of IRC network for our service bots
In-Reply-To: <1799f2c5355.b70a93d1153288.1617320239230982797@ghanshyammann.com>
References: <20210521192945.2p3qsmo2iihh2ckv@yuggoth.org>
 <1799f2c5355.b70a93d1153288.1617320239230982797@ghanshyammann.com>
Message-ID: <20210524203419.keumv6e4z3wsgbte@yuggoth.org>

On 2021-05-24 11:19:12 -0500 (-0500), Ghanshyam Mann wrote:
[...]
> Does openinfra projects have their individual choice which one
> they want to move or continue OR there will be only a single
> network provided/supported by opendev based on the majority of
> openinfra projects feedback.
[...]

Technically those are two separate questions, so I'll do my best to
answer them both.

OpenDev doesn't mandate communication choices for projects, you're
always free to communicate where you like.

However, OpenDev doesn't provide nor plan to provide services
spanning multiple IRC networks, so we'll attempt to accommodate
whichever network the majority of our projects are using. If that's
Freenode then clearly we don't need to change anything. If it's OFTC
then that's a contingency we've planned for, and we can execute a
move of our services there fairly quickly. If it's some other IRC
network then expect longer delays and a fair bit more work on our
part.

The fact that we've kept channels registered for everyone on OFTC is
fairly significant, because that means we don't need to wrest
control from anyone who might already be hanging out in them.
Without that, some channels (particularly those not namespaced)
could take years to get under our control, if ever.
-- 
Jeremy Stanley
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 963 bytes
Desc: not available
URL: <http://lists.opendev.org/pipermail/service-discuss/attachments/20210524/d234ba80/attachment.sig>

From cboylan at sapwetik.org  Mon May 24 21:48:43 2021
From: cboylan at sapwetik.org (Clark Boylan)
Date: Mon, 24 May 2021 14:48:43 -0700
Subject: Team Meeting Agenda for May 25, 2021
Message-ID: <d9b0fe9f-2eaa-48ef-ad8f-fb71f0b3c6d2@www.fastmail.com>

We will meet May 25, 2021 at 19:00 UTC in #opendev-meeting with this agenda:

== Agenda for next meeting ==

* Announcements

* Actions from last meeting

* Specs approval

* Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.)
** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management]
*** topic:update-cfg-mgmt
*** Zuul as CD engine
** OpenDev
*** Gerrit account inconsistencies
**** All preferred emails lack external ids issues have been corrected. All group loops have been corrected.
**** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?)
**** Next steps
***** More "dangerous" list has been generated. Should still be safe-ish particularly if we disable the accounts first.
*** Configuration tuning
**** Reduce the number of ssh threads. Possibly create bot/batch user groups and thread counts as part of this.
**** https://groups.google.com/g/repo-discuss/c/BQKxAfXBXuo Upstream conversation with people struggling with similar problems.

* General topics
** Refreshing non LE certs that expire in just under a month (clarkb 20210525)
*** Shutdown/remove: ask.o.o
**** https://review.opendev.org/c/opendev/system-config/+/792789
*** Refresh as yearly cert or try to LE: ethercalc, wiki, translate, storyboard
**** https://review.opendev.org/c/opendev/system-config/+/792708
*** Meeting with foundation to discuss: openstackid and openstackid-dev
**** They are interested in taking on the hosting but due to timing we probably need to provision a cert for it. Will look into LE for this as well.
** Potentially Migrating away from Freenode to OFTC (clarkb 20210525)
*** Freenode is under new management
*** Freenode policies are changing
*** http://lists.opendev.org/pipermail/service-discuss/2021-May/000236.html
** Switch Vexxhost to provide only specialized labels in Nodepool (clarkb 20210525)
** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210525)
*** Enable Xenial -> Bionic/Focal system upgrades
*** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here
*** Zuul is done. Mailman next
**** Need to snapshot the server then perform in place upgrades on a test node based on the snapshot.
** Scheduling project renames (clarkb 20210525)
*** Our playbook(s) that do renames likely need updating since the last gerrit upgrade.
**** We can test this with our functional testing of gerrit too.

* Open discussion


From thierry at openstack.org  Tue May 25 09:21:25 2021
From: thierry at openstack.org (Thierry Carrez)
Date: Tue, 25 May 2021 11:21:25 +0200
Subject: Reevaluating choice of IRC network for our service bots
In-Reply-To: <20210521192945.2p3qsmo2iihh2ckv@yuggoth.org>
References: <20210521192945.2p3qsmo2iihh2ckv@yuggoth.org>
Message-ID: <99a62f39-1159-954a-b8e3-48d8e45bd517@openstack.org>

Jeremy Stanley wrote:
> [...]
> If there is a consensus to move OpenDev's services to OFTC, or any
> other IRC network for that matter, this will entail a bit of
> development effort in order to accommodate the differences mentioned
> above. Please do note, however, that moving to a network other than
> OFTC would additionally mean we can't guarantee the availability of
> the same IRC channel names either, so that needs to be weighed in
> any decision. Some discussion[4][5][6] is already underway within
> the OpenStack community as to what they'd prefer, but we haven't
> heard much from other projects yet, so please do respond with your
> thoughts on the matter.

My personal view on this would be (with bits stolen from Jim's proposal 
on Zuul's ML):

- the situation warrants relatively quick action (measured in days), so 
we should stay on IRC

- small preference for OFTC due to the level of Opendev team 
preparation, but also because Libera.Chat will likely be a hot target 
for a while

- once the dust settles we should definitely consider a move toward a 
more federated / decentralized / accessible system like Matrix

-- 
Thierry


From radoslaw.piliszek at gmail.com  Tue May 25 10:27:40 2021
From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=)
Date: Tue, 25 May 2021 12:27:40 +0200
Subject: Reevaluating choice of IRC network for our service bots
In-Reply-To: <99a62f39-1159-954a-b8e3-48d8e45bd517@openstack.org>
References: <20210521192945.2p3qsmo2iihh2ckv@yuggoth.org>
 <99a62f39-1159-954a-b8e3-48d8e45bd517@openstack.org>
Message-ID: <CAKZ_x79uZi-A6KEx9QhpuK6wcLwRJe_U53rP1WMx8UKbnyFFaA@mail.gmail.com>

On Tue, May 25, 2021 at 11:21 AM Thierry Carrez <thierry at openstack.org> wrote:
>
> Jeremy Stanley wrote:
> > [...]
> > If there is a consensus to move OpenDev's services to OFTC, or any
> > other IRC network for that matter, this will entail a bit of
> > development effort in order to accommodate the differences mentioned
> > above. Please do note, however, that moving to a network other than
> > OFTC would additionally mean we can't guarantee the availability of
> > the same IRC channel names either, so that needs to be weighed in
> > any decision. Some discussion[4][5][6] is already underway within
> > the OpenStack community as to what they'd prefer, but we haven't
> > heard much from other projects yet, so please do respond with your
> > thoughts on the matter.
>
> My personal view on this would be (with bits stolen from Jim's proposal
> on Zuul's ML):
>
> - the situation warrants relatively quick action (measured in days), so
> we should stay on IRC
>
> - small preference for OFTC due to the level of Opendev team
> preparation, but also because Libera.Chat will likely be a hot target
> for a while
>
> - once the dust settles we should definitely consider a move toward a
> more federated / decentralized / accessible system like Matrix

Great summary, I agree 100%.

-yoctozepto


From fungi at yuggoth.org  Tue May 25 13:47:06 2021
From: fungi at yuggoth.org (Jeremy Stanley)
Date: Tue, 25 May 2021 13:47:06 +0000
Subject: Reevaluating choice of IRC network for our service bots
In-Reply-To: <99a62f39-1159-954a-b8e3-48d8e45bd517@openstack.org>
References: <20210521192945.2p3qsmo2iihh2ckv@yuggoth.org>
 <99a62f39-1159-954a-b8e3-48d8e45bd517@openstack.org>
Message-ID: <20210525134706.zvmo4stuipdpjicl@yuggoth.org>

On 2021-05-25 11:21:25 +0200 (+0200), Thierry Carrez wrote:
[...]
> small preference for OFTC due to the level of Opendev team
> preparation, but also because Libera.Chat will likely be a hot
> target for a while
[...]

Also I should mention, with OFTC looking to be a likely destination,
I'm continuing to work through any obviously outstanding development
tasks to support such a move until we arrive at a conclusion, using
"oftc" as a consistent review topic in Gerrit for the changes I've
been pushing.
-- 
Jeremy Stanley
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 963 bytes
Desc: not available
URL: <http://lists.opendev.org/pipermail/service-discuss/attachments/20210525/6b6798a4/attachment.sig>

From jim at acmegating.com  Tue May 25 17:58:11 2021
From: jim at acmegating.com (James E. Blair)
Date: Tue, 25 May 2021 10:58:11 -0700
Subject: Reevaluating choice of IRC network for our service bots
In-Reply-To: <20210521192945.2p3qsmo2iihh2ckv@yuggoth.org> (Jeremy Stanley's
 message of "Fri, 21 May 2021 19:29:46 +0000")
References: <20210521192945.2p3qsmo2iihh2ckv@yuggoth.org>
Message-ID: <87sg2ag0gc.fsf@fuligin>

Jeremy Stanley <fungi at yuggoth.org> writes:

> Because the OpenDev Collaboratory exists to serve the projects it
> hosts, input from project representatives and the Advisory Council
> is critical in deciding major changes to services. If projects were
> to move their channels to OFTC, the transition would not be
> seamless. In particular, differences in authentication (no SASL
> support, but you can authenticate your connection with a
> certificate[3] if you don't want to identify to the NickServ after
> connecting), permissions model (OFTC has coarse-grained RBAC
> designed to reduce channel mismanagement), and NickServ and ChanServ
> command syntax are among the challenges they're likely to face. The
> same IRC nicks may also not be available in some cases, though due
> to the generally smaller size of OFTC compared to Freenode this
> hopefully won't come up for too many users. On a positive note, we
> may be able to go back to not requiring nick registration just to
> join channels, easing onboarding for newcomers.
>
> If there is a consensus to move OpenDev's services to OFTC, or any
> other IRC network for that matter, this will entail a bit of
> development effort in order to accommodate the differences mentioned
> above. Please do note, however, that moving to a network other than
> OFTC would additionally mean we can't guarantee the availability of
> the same IRC channel names either, so that needs to be weighed in
> any decision. Some discussion[4][5][6] is already underway within
> the OpenStack community as to what they'd prefer, but we haven't
> heard much from other projects yet, so please do respond with your
> thoughts on the matter.

I prepared the following strawman and asked for feedback from the Zuul
community:


1) The current situation warrants relatively quick action (measured in
days), and therefore we should choose an IRC network and not broaden the
discussion to alternate technologies.

2) We're not opposed to exploring alternate technologies in the future.

3) Staying on FreeNode seems risky due to the control issues raised, the
declining availability of operators, and the likelihood of our
collaborators and potential collaborators wanting to avoid FreeNode in
the future, so we should probably move to a different network.

4) Either OFTC or libera.chat are equally acceptable IRC networks.

5) We derive value from close collaboration with the OpenDev sysadmins
and other OpenDev users, and would like to move our official #zuul
channel to whichever network achieves consensus with the rest of the
OpenDev community.

Based on feedback, I would also add:

6) We value collaboration with the Ansible community and would like to
maintain ties there.

It would be great if Ansible and OpenDev ended up on the same IRC
network.  But if they don't, I think we can deal with being on two
servers.

-Jim


From fungi at yuggoth.org  Tue May 25 18:11:51 2021
From: fungi at yuggoth.org (Jeremy Stanley)
Date: Tue, 25 May 2021 18:11:51 +0000
Subject: Reevaluating choice of IRC network for our service bots
In-Reply-To: <20210521192945.2p3qsmo2iihh2ckv@yuggoth.org>
References: <20210521192945.2p3qsmo2iihh2ckv@yuggoth.org>
Message-ID: <20210525181151.q7rot6bkrwfhf46l@yuggoth.org>

On 2021-05-21 19:29:46 +0000 (+0000), Jeremy Stanley wrote:
[...]
> Because the OpenDev Collaboratory exists to serve the projects it
> hosts, input from project representatives and the Advisory Council
> is critical in deciding major changes to services.
[...]

Quoting here from a response posted on the kata-dev mailing list:

On 2021-05-25 19:16:00 +0200 (+0200), Fabiano Fidêncio wrote:
[...]
> We discussed this during the architecture committee meeting Today
> and we'll follow the path that OpenInfra is following. If the
> change will be done to OFTC, and OpenInfra will take care of
> setting things up for us (including the slack-irc and irc-slack
> bots), we're just super happy to go for that.
> 
> Some of us, a big portion of us, already are present at OFTC due
> to other communities we contribute to.

http://lists.katacontainers.io/pipermail/kata-dev/2021-May/001935.html

As an aside, I already replied to their list that folks at the
OpenInfra Foundation are taking care of the Slack bridge Fabiano
mentioned, but that they're also aware and I'll be keeping them in
the loop as well for any changes needed.
-- 
Jeremy Stanley
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 963 bytes
Desc: not available
URL: <http://lists.opendev.org/pipermail/service-discuss/attachments/20210525/e51acbc0/attachment.sig>

From gmann at ghanshyammann.com  Wed May 26 15:20:21 2021
From: gmann at ghanshyammann.com (Ghanshyam Mann)
Date: Wed, 26 May 2021 10:20:21 -0500
Subject: Reevaluating choice of IRC network for our service bots
In-Reply-To: <20210524203419.keumv6e4z3wsgbte@yuggoth.org>
References: <20210521192945.2p3qsmo2iihh2ckv@yuggoth.org>
 <1799f2c5355.b70a93d1153288.1617320239230982797@ghanshyammann.com>
 <20210524203419.keumv6e4z3wsgbte@yuggoth.org>
Message-ID: <179a94327b1.10f1fcdbe16716.2827949007538770104@ghanshyammann.com>

 ---- On Mon, 24 May 2021 15:34:19 -0500 Jeremy Stanley <fungi at yuggoth.org> wrote ----
 > On 2021-05-24 11:19:12 -0500 (-0500), Ghanshyam Mann wrote:
 > [...]
 > > Does openinfra projects have their individual choice which one
 > > they want to move or continue OR there will be only a single
 > > network provided/supported by opendev based on the majority of
 > > openinfra projects feedback.
 > [...]
 > 
 > Technically those are two separate questions, so I'll do my best to
 > answer them both.
 > 
 > OpenDev doesn't mandate communication choices for projects, you're
 > always free to communicate where you like.
 > 
 > However, OpenDev doesn't provide nor plan to provide services
 > spanning multiple IRC networks, so we'll attempt to accommodate
 > whichever network the majority of our projects are using. If that's
 > Freenode then clearly we don't need to change anything. If it's OFTC
 > then that's a contingency we've planned for, and we can execute a
 > move of our services there fairly quickly. If it's some other IRC
 > network then expect longer delays and a fair bit more work on our
 > part.
 > 
 > The fact that we've kept channels registered for everyone on OFTC is
 > fairly significant, because that means we don't need to wrest
 > control from anyone who might already be hanging out in them.
 > Without that, some channels (particularly those not namespaced)
 > could take years to get under our control, if ever.
 > -- 

Thanks for the details.

In today TC ad-hoc meeting, we concluded in OpenStack that we are moving to OFTC

Details and agreement with action plan are below etherpad:
- https://etherpad.opendev.org/p/openstack-irc

-gmann

 > Jeremy Stanley
 > 


From radoslaw.piliszek at gmail.com  Wed May 26 17:23:43 2021
From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=)
Date: Wed, 26 May 2021 19:23:43 +0200
Subject: Reevaluating choice of IRC network for our service bots
In-Reply-To: <179a94327b1.10f1fcdbe16716.2827949007538770104@ghanshyammann.com>
References: <20210521192945.2p3qsmo2iihh2ckv@yuggoth.org>
 <1799f2c5355.b70a93d1153288.1617320239230982797@ghanshyammann.com>
 <20210524203419.keumv6e4z3wsgbte@yuggoth.org>
 <179a94327b1.10f1fcdbe16716.2827949007538770104@ghanshyammann.com>
Message-ID: <CAKZ_x79xvT9kjT3n2nxxdAe9QFsORLT2XEM+-50M489ReLFmFQ@mail.gmail.com>

On Wed, May 26, 2021 at 5:20 PM Ghanshyam Mann <gmann at ghanshyammann.com> wrote:
>
>  ---- On Mon, 24 May 2021 15:34:19 -0500 Jeremy Stanley <fungi at yuggoth.org> wrote ----
>  > On 2021-05-24 11:19:12 -0500 (-0500), Ghanshyam Mann wrote:
>  > [...]
>  > > Does openinfra projects have their individual choice which one
>  > > they want to move or continue OR there will be only a single
>  > > network provided/supported by opendev based on the majority of
>  > > openinfra projects feedback.
>  > [...]
>  >
>  > Technically those are two separate questions, so I'll do my best to
>  > answer them both.
>  >
>  > OpenDev doesn't mandate communication choices for projects, you're
>  > always free to communicate where you like.
>  >
>  > However, OpenDev doesn't provide nor plan to provide services
>  > spanning multiple IRC networks, so we'll attempt to accommodate
>  > whichever network the majority of our projects are using. If that's
>  > Freenode then clearly we don't need to change anything. If it's OFTC
>  > then that's a contingency we've planned for, and we can execute a
>  > move of our services there fairly quickly. If it's some other IRC
>  > network then expect longer delays and a fair bit more work on our
>  > part.
>  >
>  > The fact that we've kept channels registered for everyone on OFTC is
>  > fairly significant, because that means we don't need to wrest
>  > control from anyone who might already be hanging out in them.
>  > Without that, some channels (particularly those not namespaced)
>  > could take years to get under our control, if ever.
>  > --
>
> Thanks for the details.
>
> In today TC ad-hoc meeting, we concluded in OpenStack that we are moving to OFTC
>
> Details and agreement with action plan are below etherpad:
> - https://etherpad.opendev.org/p/openstack-irc

Final published notice:
http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022718.html

-yoctozepto


From fungi at yuggoth.org  Wed May 26 18:15:00 2021
From: fungi at yuggoth.org (Jeremy Stanley)
Date: Wed, 26 May 2021 18:15:00 +0000
Subject: Moving IRC service bots to OFTC this weekend (was: Reevaluating
 choice of IRC network for our service bots)
In-Reply-To: <20210521192945.2p3qsmo2iihh2ckv@yuggoth.org>
References: <20210521192945.2p3qsmo2iihh2ckv@yuggoth.org>
Message-ID: <20210526181500.4cgkodj7jjoe57jl@yuggoth.org>

On 2021-05-21 19:29:46 +0000 (+0000), Jeremy Stanley wrote:
[...]
> Because the OpenDev Collaboratory exists to serve the projects it
> hosts, input from project representatives and the Advisory Council
> is critical in deciding major changes to services.
[...]

We've now heard back from three of the five confirmed Open
Infrastructure Projects (Kata, Zuul and OpenStack). This seems like
a sufficient majority to determine that we should relocate our IRC
service bots from Freenode to OFTC according to our long-standing
evacuation plan. The Zuul contributors indicated a preference for
moving as soon as possible, and OpenStack's leadership suggested
this weekend; it's a holiday weekend for many of their contributors
and their channels won't be as heavily used during that time anyway,
so gaps in logged discussions and notifications are more easily
tolerated. This also gives us a few more days to polish up the code
and config changes for our bots in order to ensure a reasonably
smooth transition.

For a majority of users, this move should be as simple as replacing
"chat.freenode.net" with "irc.oftc.net" in their client settings,
but read on for relevant caveats.

Nick registration won't be strictly required to join any channels
we're managing on OFTC, at least not initially (and hopefully never
unless spammers become a significant problem for us there), however
users are encouraged to go ahead and claim the nicks they want and
register them with OFTC's NickServ sooner rather than later:

    https://www.oftc.net/Services/#register-your-account

Do note that the SASL auth mechanism supported by Freenode is not
available on OFTC, but you can still send an identify message to
NickServ after connecting or set up certificate authentication for
your connection instead. See the OFTC Services FAQ for details on
these and other topics (like how to request control of an abandoned
nick if the one you want was already taken by someone else but is
apparently unused for some time):

    https://www.oftc.net/FAQ/Services/

There's also a marvellous write-up on the zuul-discuss mailing list
about using the Element WebUI to connect via Matrix's OFTC bridge,
for anyone who's interested in giving that a try instead of using a
traditional client and/or "bouncer" solution:

    http://lists.zuul-ci.org/pipermail/zuul-discuss/2021-May/001613.html

Please be aware that one of the tasks I'm personally planning for
over the next few days is to scrape a copy of all the channel topics
and apply them automatically to the channels we've pre-registered on
OFTC, so at least for now please refrain from changing your channel
topics to anything other than what you'll want them to appear a on
the new network. Further, there have been numerous reports of
Freenode staff taking over any channels which mention in their
topics that they're moving to a different network, therefore it
would help if some volunteers can stay connected to Freenode (most
modern IRC clients are capable of connecting to multiple networks
simultaneously) and joined to our more popular channels there to
help direct stragglers to our new home.

We'll have somewhat of a delay in bootstrapping channel-specific
operators since we want to push those through code review (in order
to avoid having to trust that people on a new network are really who
they say they are), so if you need a channel topic updated or a
problem user banned in the short term you'll need to reach out to
the sysadmins in the #opendev channel on OFTC for assistance until
we get the general admins and ops settled in.

I'll be following up shortly with a reply to the threads I started
on the project-specific discussion lists directing them to this
announcement. Further updates will be limited to the service-discuss
mailing list, so please respond here with any questions or comments.
-- 
Jeremy Stanley
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 963 bytes
Desc: not available
URL: <http://lists.opendev.org/pipermail/service-discuss/attachments/20210526/96e38c48/attachment.sig>

From fungi at yuggoth.org  Sat May 29 00:33:43 2021
From: fungi at yuggoth.org (Jeremy Stanley)
Date: Sat, 29 May 2021 00:33:43 +0000
Subject: Moving IRC service bots to OFTC this weekend (was: Reevaluating
 choice of IRC network for our service bots)
In-Reply-To: <20210526181500.4cgkodj7jjoe57jl@yuggoth.org>
References: <20210521192945.2p3qsmo2iihh2ckv@yuggoth.org>
 <20210526181500.4cgkodj7jjoe57jl@yuggoth.org>
Message-ID: <20210529003343.n6qe6fpv7cilw5o4@yuggoth.org>

On 2021-05-26 18:15:00 +0000 (+0000), Jeremy Stanley wrote:
[...]
> Please be aware that one of the tasks I'm personally planning for
> over the next few days is to scrape a copy of all the channel topics
> and apply them automatically to the channels we've pre-registered on
> OFTC, so at least for now please refrain from changing your channel
> topics to anything other than what you'll want them to appear a on
> the new network.
[...]

Just to update everyone, the channel topic replication has now been
completed.

> We'll have somewhat of a delay in bootstrapping channel-specific
> operators since we want to push those through code review (in order
> to avoid having to trust that people on a new network are really who
> they say they are), so if you need a channel topic updated or a
> problem user banned in the short term you'll need to reach out to
> the sysadmins in the #opendev channel on OFTC for assistance until
> we get the general admins and ops settled in.
[...]

If people want to start volunteering as global ops or
channel-specific ops, propose your additions to this file through
usual code review:

https://opendev.org/openstack/project-config/src/branch/master/accessbot/channels.yaml

There may be some delay before we have a chance to review them, but
if you have questions feel free to follow up to this thread or hit
us up in the #opendev channel on OFTC.

We're on track to start switching user-facing services over to OFTC
in roughly 13.5 hours, and will reply with further updates on our
progress over the course of the weekend.
-- 
Jeremy Stanley
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 963 bytes
Desc: not available
URL: <http://lists.opendev.org/pipermail/service-discuss/attachments/20210529/6fb51522/attachment.sig>

From fungi at yuggoth.org  Sat May 29 23:58:37 2021
From: fungi at yuggoth.org (Jeremy Stanley)
Date: Sat, 29 May 2021 23:58:37 +0000
Subject: Moving IRC service bots to OFTC this weekend
In-Reply-To: <20210529003343.n6qe6fpv7cilw5o4@yuggoth.org>
References: <20210521192945.2p3qsmo2iihh2ckv@yuggoth.org>
 <20210526181500.4cgkodj7jjoe57jl@yuggoth.org>
 <20210529003343.n6qe6fpv7cilw5o4@yuggoth.org>
Message-ID: <20210529235836.i4eq2weyyzvdvjab@yuggoth.org>

On 2021-05-29 00:33:43 +0000 (+0000), Jeremy Stanley wrote:
[...]
> We're on track to start switching user-facing services over to
> OFTC in roughly 13.5 hours, and will reply with further updates on
> our progress over the course of the weekend.

To update everyone, we spent a few hours today getting our primary
services cut over to OFTC. A few notes:

  * The meetbot (formerly named "openstack") is in channels as
    "opendevmeet" now.

  * The gerritbot (formerly named "openstackgerrit") is in channels
    as "opendevreview" now.

  * The statusbot (formerly named "openstackstatus") is in channels
    as "opendevstatus" now.

  * As of 15:00 UTC today (2021-05-29) channel logging on the
    http://eavesdrop.openstack.org/ site is now reflecting
    discussions for those channels on OFTC instead of Freenode.

  * The meetbot is currently incapable of changing channel topics
    without some additional development effort (or changing auto-op
    settings for channels, which would have some serious negative
    side effects), but hopefully this is at worst a minor
    inconvenience for now.

  * There are four channels we've not yet been able to completely
    update, because they were registered by other community members
    who will need to add our "opendevaccess" account to their access
    lists: #edeploy (sbadia), #openstack-sahara (SergeyLukjanov),
    #puppet-openstack (xarses), #rdo (apevec,jpena,spotz).

  * We've seen a bit of a spam flood today, seems to be entirely
    backscatter from the Freenode/Libera disagreements, and so we've
    set the +s flag on all channels as of 21:20 UTC in order to make
    it harder for the spammers to find our channels by not listing
    them in the public channel list, but as a side effect this  will
    cause them not to appear in the selection/autocompletion list
    for the OFTC Matrix bridge if anyone happens to be relying on
    that (you can still tell it to connect you to specific channels
    if you know the name and type it in).

We feel things are basically stable now, but will keep an eye on the
services over the course of the weekend and so may need to restart
some briefly to make minor adjustments. Also a reminder, if people
want to start volunteering as global ops or channel-specific ops,
propose your additions to this file through usual code review:

https://opendev.org/openstack/project-config/src/branch/master/accessbot/channels.yaml

There may be some delay before we have a chance to review them, but
if you have questions feel free to follow up to this thread or hit
us up in the #opendev channel on OFTC.
-- 
Jeremy Stanley
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 963 bytes
Desc: not available
URL: <http://lists.opendev.org/pipermail/service-discuss/attachments/20210529/07129185/attachment.sig>