Reevaluating choice of IRC network for our service bots
As most people reading this are likely aware, there has been a recent change in management for the Freenode IRC network. According to public statements[0], the former staff have all left and the network is in entirely new hands. Setting aside arguments about which parties are in the wrong, further stability of the service we provide is once again brought into question. We've seen evidence of (much) lower levels of abuse mitigation over the past few days, with a marked increase in registered accounts engaging in harassment over extended periods of time across our channels without getting banned from the network.
Due to disagreements between Freenode and other large IRC networks and frequent related attacks against Freenode's servers, our communities have experienced many extended periods of instability there. In order to insure against a potential collapse of the network, the OpenDev Collaboratory (and the OpenStack Project Infrastructure before it) has for many years maintained a sort of evacuation plan: For the multitude of channels overseen by OpenDev, we also keep equivalent channel registrations on the Open and Free Technology Community (OFTC) IRC network[1], and have communicated with its operators on multiple occasions about their willingness and ability to host our communities there should the need arise.
This week's events are, unfortunately, not an isolated incident. The systems administrators for OpenDev have encouraged a move to OFTC on multiple occasions, with the first formal proposal in March 2014[2]. In prior cases, the impact to the community for such a switch outweighed the perceived benefits, but as we've heard growing displeasure with Freenode from representatives of many of the projects we serve, it is time once more to revisit whether we should enact our longstanding evacuation plans.
Because the OpenDev Collaboratory exists to serve the projects it hosts, input from project representatives and the Advisory Council is critical in deciding major changes to services. If projects were to move their channels to OFTC, the transition would not be seamless. In particular, differences in authentication (no SASL support, but you can authenticate your connection with a certificate[3] if you don't want to identify to the NickServ after connecting), permissions model (OFTC has coarse-grained RBAC designed to reduce channel mismanagement), and NickServ and ChanServ command syntax are among the challenges they're likely to face. The same IRC nicks may also not be available in some cases, though due to the generally smaller size of OFTC compared to Freenode this hopefully won't come up for too many users. On a positive note, we may be able to go back to not requiring nick registration just to join channels, easing onboarding for newcomers.
If there is a consensus to move OpenDev's services to OFTC, or any other IRC network for that matter, this will entail a bit of development effort in order to accommodate the differences mentioned above. Please do note, however, that moving to a network other than OFTC would additionally mean we can't guarantee the availability of the same IRC channel names either, so that needs to be weighed in any decision. Some discussion[4][5][6] is already underway within the OpenStack community as to what they'd prefer, but we haven't heard much from other projects yet, so please do respond with your thoughts on the matter.
[0] https://fuchsnet.ch/freenode-resign-letter.txt [1] https://www.oftc.net/ [2] http://lists.openstack.org/pipermail/openstack-dev/2014-March/028783.html [3] https://www.oftc.net/NickServ/CertFP/ [4] http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022468.html [5] http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022539.html [6] http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23openstack-tc.2021-...
On Fri, May 21, 2021 at 07:29:46PM +0000, Jeremy Stanley wrote: 65;6401;1c> If there is a consensus to move OpenDev's services to OFTC, or any
other IRC network for that matter, this will entail a bit of development effort
Is there a procedure to obtain consensus?
so please do respond with your thoughts on the matter.
I feel like I've read more than a normal person would/should have, and I still have no idea what's going on. Whatever the behind-the-scenes reality is, the unfortunate result is that being involved with our projects has acquired the additional step of "overcome existential crisis of using freenode".
dmsimard's comment [1] is the best one I've seen tracking who is going where amongst peer projects. Despite communications like [2] it would seem to take a phoenix-esque reputation rehabilitation to revert these already made decisions [3].
To me, moving from freenode is looking less like something we need to decide, but more like action we need to take for self-preservation (i.e. not alienating our most important resource; developers).
Per the ~7 years of occasional messages, OFTC seems like a fine choice.
-i
[1] https://github.com/ansible-community/community-topics/issues/19#issuecomment... [2] https://freenode.net/news/freenode-is-foss [3] companies who have had contentious relationships with free software over a long period of time have undoubtedly managed to go on to later capture a large portion of open source development, even when their platforms don't provide for open tools or governance; so never say never...
-i
On 2021-05-24 15:46:12 +1000 (+1000), Ian Wienand wrote:
On Fri, May 21, 2021 at 07:29:46PM +0000, Jeremy Stanley wrote:
If there is a consensus to move OpenDev's services to OFTC, or any other IRC network for that matter, this will entail a bit of development effort
Is there a procedure to obtain consensus?
[...]
If there's a clear consensus then I expect it to become readily apparent. If there's no clear choice, I agree our provisional governance document lacks a solution for reaching consensus.
---- On Fri, 21 May 2021 14:29:46 -0500 Jeremy Stanley fungi@yuggoth.org wrote ----
As most people reading this are likely aware, there has been a recent change in management for the Freenode IRC network. According to public statements[0], the former staff have all left and the network is in entirely new hands. Setting aside arguments about which parties are in the wrong, further stability of the service we provide is once again brought into question. We've seen evidence of (much) lower levels of abuse mitigation over the past few days, with a marked increase in registered accounts engaging in harassment over extended periods of time across our channels without getting banned from the network.
Due to disagreements between Freenode and other large IRC networks and frequent related attacks against Freenode's servers, our communities have experienced many extended periods of instability there. In order to insure against a potential collapse of the network, the OpenDev Collaboratory (and the OpenStack Project Infrastructure before it) has for many years maintained a sort of evacuation plan: For the multitude of channels overseen by OpenDev, we also keep equivalent channel registrations on the Open and Free Technology Community (OFTC) IRC network[1], and have communicated with its operators on multiple occasions about their willingness and ability to host our communities there should the need arise.
This week's events are, unfortunately, not an isolated incident. The systems administrators for OpenDev have encouraged a move to OFTC on multiple occasions, with the first formal proposal in March 2014[2]. In prior cases, the impact to the community for such a switch outweighed the perceived benefits, but as we've heard growing displeasure with Freenode from representatives of many of the projects we serve, it is time once more to revisit whether we should enact our longstanding evacuation plans.
Because the OpenDev Collaboratory exists to serve the projects it hosts, input from project representatives and the Advisory Council is critical in deciding major changes to services. If projects were to move their channels to OFTC, the transition would not be seamless. In particular, differences in authentication (no SASL support, but you can authenticate your connection with a certificate[3] if you don't want to identify to the NickServ after connecting), permissions model (OFTC has coarse-grained RBAC designed to reduce channel mismanagement), and NickServ and ChanServ command syntax are among the challenges they're likely to face. The same IRC nicks may also not be available in some cases, though due to the generally smaller size of OFTC compared to Freenode this hopefully won't come up for too many users. On a positive note, we may be able to go back to not requiring nick registration just to join channels, easing onboarding for newcomers.
If there is a consensus to move OpenDev's services to OFTC, or any other IRC network for that matter, this will entail a bit of development effort in order to accommodate the differences mentioned above. Please do note, however, that moving to a network other than OFTC would additionally mean we can't guarantee the availability of the same IRC channel names either, so that needs to be weighed in any decision. Some discussion[4][5][6] is already underway within the OpenStack community as to what they'd prefer, but we haven't heard much from other projects yet, so please do respond with your thoughts on the matter.
Thanks, Jeremy and opendev team for starting the consolidated effort.
As you know, in OpenStack we are still discussing it on ML or in openstack-tc channel (adding this in TC meeting agenda too).
One question though:
Does openinfra projects have their individual choice which one they want to move or continue OR there will be only a single network provided/supported by opendev based on the majority of openinfra projects feedback.
-gmann
[0] https://fuchsnet.ch/freenode-resign-letter.txt [1] https://www.oftc.net/ [2] http://lists.openstack.org/pipermail/openstack-dev/2014-March/028783.html [3] https://www.oftc.net/NickServ/CertFP/ [4] http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022468.html [5] http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022539.html [6] http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23openstack-tc.2021-... -- Jeremy Stanley
On 2021-05-24 11:19:12 -0500 (-0500), Ghanshyam Mann wrote: [...]
Does openinfra projects have their individual choice which one they want to move or continue OR there will be only a single network provided/supported by opendev based on the majority of openinfra projects feedback.
[...]
Technically those are two separate questions, so I'll do my best to answer them both.
OpenDev doesn't mandate communication choices for projects, you're always free to communicate where you like.
However, OpenDev doesn't provide nor plan to provide services spanning multiple IRC networks, so we'll attempt to accommodate whichever network the majority of our projects are using. If that's Freenode then clearly we don't need to change anything. If it's OFTC then that's a contingency we've planned for, and we can execute a move of our services there fairly quickly. If it's some other IRC network then expect longer delays and a fair bit more work on our part.
The fact that we've kept channels registered for everyone on OFTC is fairly significant, because that means we don't need to wrest control from anyone who might already be hanging out in them. Without that, some channels (particularly those not namespaced) could take years to get under our control, if ever.
---- On Mon, 24 May 2021 15:34:19 -0500 Jeremy Stanley fungi@yuggoth.org wrote ----
On 2021-05-24 11:19:12 -0500 (-0500), Ghanshyam Mann wrote: [...]
Does openinfra projects have their individual choice which one they want to move or continue OR there will be only a single network provided/supported by opendev based on the majority of openinfra projects feedback.
[...]
Technically those are two separate questions, so I'll do my best to answer them both.
OpenDev doesn't mandate communication choices for projects, you're always free to communicate where you like.
However, OpenDev doesn't provide nor plan to provide services spanning multiple IRC networks, so we'll attempt to accommodate whichever network the majority of our projects are using. If that's Freenode then clearly we don't need to change anything. If it's OFTC then that's a contingency we've planned for, and we can execute a move of our services there fairly quickly. If it's some other IRC network then expect longer delays and a fair bit more work on our part.
The fact that we've kept channels registered for everyone on OFTC is fairly significant, because that means we don't need to wrest control from anyone who might already be hanging out in them. Without that, some channels (particularly those not namespaced) could take years to get under our control, if ever. --
Thanks for the details.
In today TC ad-hoc meeting, we concluded in OpenStack that we are moving to OFTC
Details and agreement with action plan are below etherpad: - https://etherpad.opendev.org/p/openstack-irc
-gmann
Jeremy Stanley
On Wed, May 26, 2021 at 5:20 PM Ghanshyam Mann gmann@ghanshyammann.com wrote:
---- On Mon, 24 May 2021 15:34:19 -0500 Jeremy Stanley fungi@yuggoth.org wrote ----
On 2021-05-24 11:19:12 -0500 (-0500), Ghanshyam Mann wrote: [...]
Does openinfra projects have their individual choice which one they want to move or continue OR there will be only a single network provided/supported by opendev based on the majority of openinfra projects feedback.
[...]
Technically those are two separate questions, so I'll do my best to answer them both.
OpenDev doesn't mandate communication choices for projects, you're always free to communicate where you like.
However, OpenDev doesn't provide nor plan to provide services spanning multiple IRC networks, so we'll attempt to accommodate whichever network the majority of our projects are using. If that's Freenode then clearly we don't need to change anything. If it's OFTC then that's a contingency we've planned for, and we can execute a move of our services there fairly quickly. If it's some other IRC network then expect longer delays and a fair bit more work on our part.
The fact that we've kept channels registered for everyone on OFTC is fairly significant, because that means we don't need to wrest control from anyone who might already be hanging out in them. Without that, some channels (particularly those not namespaced) could take years to get under our control, if ever. --
Thanks for the details.
In today TC ad-hoc meeting, we concluded in OpenStack that we are moving to OFTC
Details and agreement with action plan are below etherpad:
Final published notice: http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022718.html
-yoctozepto
Jeremy Stanley wrote:
[...] If there is a consensus to move OpenDev's services to OFTC, or any other IRC network for that matter, this will entail a bit of development effort in order to accommodate the differences mentioned above. Please do note, however, that moving to a network other than OFTC would additionally mean we can't guarantee the availability of the same IRC channel names either, so that needs to be weighed in any decision. Some discussion[4][5][6] is already underway within the OpenStack community as to what they'd prefer, but we haven't heard much from other projects yet, so please do respond with your thoughts on the matter.
My personal view on this would be (with bits stolen from Jim's proposal on Zuul's ML):
- the situation warrants relatively quick action (measured in days), so we should stay on IRC
- small preference for OFTC due to the level of Opendev team preparation, but also because Libera.Chat will likely be a hot target for a while
- once the dust settles we should definitely consider a move toward a more federated / decentralized / accessible system like Matrix
On Tue, May 25, 2021 at 11:21 AM Thierry Carrez thierry@openstack.org wrote:
Jeremy Stanley wrote:
[...] If there is a consensus to move OpenDev's services to OFTC, or any other IRC network for that matter, this will entail a bit of development effort in order to accommodate the differences mentioned above. Please do note, however, that moving to a network other than OFTC would additionally mean we can't guarantee the availability of the same IRC channel names either, so that needs to be weighed in any decision. Some discussion[4][5][6] is already underway within the OpenStack community as to what they'd prefer, but we haven't heard much from other projects yet, so please do respond with your thoughts on the matter.
My personal view on this would be (with bits stolen from Jim's proposal on Zuul's ML):
- the situation warrants relatively quick action (measured in days), so
we should stay on IRC
- small preference for OFTC due to the level of Opendev team
preparation, but also because Libera.Chat will likely be a hot target for a while
- once the dust settles we should definitely consider a move toward a
more federated / decentralized / accessible system like Matrix
Great summary, I agree 100%.
-yoctozepto
On 2021-05-25 11:21:25 +0200 (+0200), Thierry Carrez wrote: [...]
small preference for OFTC due to the level of Opendev team preparation, but also because Libera.Chat will likely be a hot target for a while
[...]
Also I should mention, with OFTC looking to be a likely destination, I'm continuing to work through any obviously outstanding development tasks to support such a move until we arrive at a conclusion, using "oftc" as a consistent review topic in Gerrit for the changes I've been pushing.
Jeremy Stanley fungi@yuggoth.org writes:
Because the OpenDev Collaboratory exists to serve the projects it hosts, input from project representatives and the Advisory Council is critical in deciding major changes to services. If projects were to move their channels to OFTC, the transition would not be seamless. In particular, differences in authentication (no SASL support, but you can authenticate your connection with a certificate[3] if you don't want to identify to the NickServ after connecting), permissions model (OFTC has coarse-grained RBAC designed to reduce channel mismanagement), and NickServ and ChanServ command syntax are among the challenges they're likely to face. The same IRC nicks may also not be available in some cases, though due to the generally smaller size of OFTC compared to Freenode this hopefully won't come up for too many users. On a positive note, we may be able to go back to not requiring nick registration just to join channels, easing onboarding for newcomers.
If there is a consensus to move OpenDev's services to OFTC, or any other IRC network for that matter, this will entail a bit of development effort in order to accommodate the differences mentioned above. Please do note, however, that moving to a network other than OFTC would additionally mean we can't guarantee the availability of the same IRC channel names either, so that needs to be weighed in any decision. Some discussion[4][5][6] is already underway within the OpenStack community as to what they'd prefer, but we haven't heard much from other projects yet, so please do respond with your thoughts on the matter.
I prepared the following strawman and asked for feedback from the Zuul community:
1) The current situation warrants relatively quick action (measured in days), and therefore we should choose an IRC network and not broaden the discussion to alternate technologies.
2) We're not opposed to exploring alternate technologies in the future.
3) Staying on FreeNode seems risky due to the control issues raised, the declining availability of operators, and the likelihood of our collaborators and potential collaborators wanting to avoid FreeNode in the future, so we should probably move to a different network.
4) Either OFTC or libera.chat are equally acceptable IRC networks.
5) We derive value from close collaboration with the OpenDev sysadmins and other OpenDev users, and would like to move our official #zuul channel to whichever network achieves consensus with the rest of the OpenDev community.
Based on feedback, I would also add:
6) We value collaboration with the Ansible community and would like to maintain ties there.
It would be great if Ansible and OpenDev ended up on the same IRC network. But if they don't, I think we can deal with being on two servers.
-Jim
On 2021-05-21 19:29:46 +0000 (+0000), Jeremy Stanley wrote: [...]
Because the OpenDev Collaboratory exists to serve the projects it hosts, input from project representatives and the Advisory Council is critical in deciding major changes to services.
[...]
Quoting here from a response posted on the kata-dev mailing list:
On 2021-05-25 19:16:00 +0200 (+0200), Fabiano Fidêncio wrote: [...]
We discussed this during the architecture committee meeting Today and we'll follow the path that OpenInfra is following. If the change will be done to OFTC, and OpenInfra will take care of setting things up for us (including the slack-irc and irc-slack bots), we're just super happy to go for that.
Some of us, a big portion of us, already are present at OFTC due to other communities we contribute to.
http://lists.katacontainers.io/pipermail/kata-dev/2021-May/001935.html
As an aside, I already replied to their list that folks at the OpenInfra Foundation are taking care of the Slack bridge Fabiano mentioned, but that they're also aware and I'll be keeping them in the loop as well for any changes needed.
On 2021-05-21 19:29:46 +0000 (+0000), Jeremy Stanley wrote: [...]
Because the OpenDev Collaboratory exists to serve the projects it hosts, input from project representatives and the Advisory Council is critical in deciding major changes to services.
[...]
We've now heard back from three of the five confirmed Open Infrastructure Projects (Kata, Zuul and OpenStack). This seems like a sufficient majority to determine that we should relocate our IRC service bots from Freenode to OFTC according to our long-standing evacuation plan. The Zuul contributors indicated a preference for moving as soon as possible, and OpenStack's leadership suggested this weekend; it's a holiday weekend for many of their contributors and their channels won't be as heavily used during that time anyway, so gaps in logged discussions and notifications are more easily tolerated. This also gives us a few more days to polish up the code and config changes for our bots in order to ensure a reasonably smooth transition.
For a majority of users, this move should be as simple as replacing "chat.freenode.net" with "irc.oftc.net" in their client settings, but read on for relevant caveats.
Nick registration won't be strictly required to join any channels we're managing on OFTC, at least not initially (and hopefully never unless spammers become a significant problem for us there), however users are encouraged to go ahead and claim the nicks they want and register them with OFTC's NickServ sooner rather than later:
https://www.oftc.net/Services/#register-your-account
Do note that the SASL auth mechanism supported by Freenode is not available on OFTC, but you can still send an identify message to NickServ after connecting or set up certificate authentication for your connection instead. See the OFTC Services FAQ for details on these and other topics (like how to request control of an abandoned nick if the one you want was already taken by someone else but is apparently unused for some time):
https://www.oftc.net/FAQ/Services/
There's also a marvellous write-up on the zuul-discuss mailing list about using the Element WebUI to connect via Matrix's OFTC bridge, for anyone who's interested in giving that a try instead of using a traditional client and/or "bouncer" solution:
http://lists.zuul-ci.org/pipermail/zuul-discuss/2021-May/001613.html
Please be aware that one of the tasks I'm personally planning for over the next few days is to scrape a copy of all the channel topics and apply them automatically to the channels we've pre-registered on OFTC, so at least for now please refrain from changing your channel topics to anything other than what you'll want them to appear a on the new network. Further, there have been numerous reports of Freenode staff taking over any channels which mention in their topics that they're moving to a different network, therefore it would help if some volunteers can stay connected to Freenode (most modern IRC clients are capable of connecting to multiple networks simultaneously) and joined to our more popular channels there to help direct stragglers to our new home.
We'll have somewhat of a delay in bootstrapping channel-specific operators since we want to push those through code review (in order to avoid having to trust that people on a new network are really who they say they are), so if you need a channel topic updated or a problem user banned in the short term you'll need to reach out to the sysadmins in the #opendev channel on OFTC for assistance until we get the general admins and ops settled in.
I'll be following up shortly with a reply to the threads I started on the project-specific discussion lists directing them to this announcement. Further updates will be limited to the service-discuss mailing list, so please respond here with any questions or comments.
On 2021-05-26 18:15:00 +0000 (+0000), Jeremy Stanley wrote: [...]
Please be aware that one of the tasks I'm personally planning for over the next few days is to scrape a copy of all the channel topics and apply them automatically to the channels we've pre-registered on OFTC, so at least for now please refrain from changing your channel topics to anything other than what you'll want them to appear a on the new network.
[...]
Just to update everyone, the channel topic replication has now been completed.
We'll have somewhat of a delay in bootstrapping channel-specific operators since we want to push those through code review (in order to avoid having to trust that people on a new network are really who they say they are), so if you need a channel topic updated or a problem user banned in the short term you'll need to reach out to the sysadmins in the #opendev channel on OFTC for assistance until we get the general admins and ops settled in.
[...]
If people want to start volunteering as global ops or channel-specific ops, propose your additions to this file through usual code review:
https://opendev.org/openstack/project-config/src/branch/master/accessbot/cha...
There may be some delay before we have a chance to review them, but if you have questions feel free to follow up to this thread or hit us up in the #opendev channel on OFTC.
We're on track to start switching user-facing services over to OFTC in roughly 13.5 hours, and will reply with further updates on our progress over the course of the weekend.
On 2021-05-29 00:33:43 +0000 (+0000), Jeremy Stanley wrote: [...]
We're on track to start switching user-facing services over to OFTC in roughly 13.5 hours, and will reply with further updates on our progress over the course of the weekend.
To update everyone, we spent a few hours today getting our primary services cut over to OFTC. A few notes:
* The meetbot (formerly named "openstack") is in channels as "opendevmeet" now.
* The gerritbot (formerly named "openstackgerrit") is in channels as "opendevreview" now.
* The statusbot (formerly named "openstackstatus") is in channels as "opendevstatus" now.
* As of 15:00 UTC today (2021-05-29) channel logging on the http://eavesdrop.openstack.org/ site is now reflecting discussions for those channels on OFTC instead of Freenode.
* The meetbot is currently incapable of changing channel topics without some additional development effort (or changing auto-op settings for channels, which would have some serious negative side effects), but hopefully this is at worst a minor inconvenience for now.
* There are four channels we've not yet been able to completely update, because they were registered by other community members who will need to add our "opendevaccess" account to their access lists: #edeploy (sbadia), #openstack-sahara (SergeyLukjanov), #puppet-openstack (xarses), #rdo (apevec,jpena,spotz).
* We've seen a bit of a spam flood today, seems to be entirely backscatter from the Freenode/Libera disagreements, and so we've set the +s flag on all channels as of 21:20 UTC in order to make it harder for the spammers to find our channels by not listing them in the public channel list, but as a side effect this will cause them not to appear in the selection/autocompletion list for the OFTC Matrix bridge if anyone happens to be relying on that (you can still tell it to connect you to specific channels if you know the name and type it in).
We feel things are basically stable now, but will keep an eye on the services over the course of the weekend and so may need to restart some briefly to make minor adjustments. Also a reminder, if people want to start volunteering as global ops or channel-specific ops, propose your additions to this file through usual code review:
https://opendev.org/openstack/project-config/src/branch/master/accessbot/cha...
There may be some delay before we have a chance to review them, but if you have questions feel free to follow up to this thread or hit us up in the #opendev channel on OFTC.
participants (6)
-
Ghanshyam Mann
-
Ian Wienand
-
James E. Blair
-
Jeremy Stanley
-
Radosław Piliszek
-
Thierry Carrez