Replace mailinglists, newsgroups and forum with Discourse

Abstract

The TYPO3 Server Team is responsible for managing the server infrastructure related to the TYPO3 project. We’re in the process of streamlining our infrastructure, especially to remove legacy software not widely used and/or maintained anymore. This applies to the current mailing lists, newsgroups and forum software, which were built a long time ago by former team members, and where we see no clear upgrade path. Therefore, we would like to replace mailing lists, newsgroups and forum with a single Discourse forum.

Current situation

Mailing lists

Update not yet encouraged by the Mailman Team:

While it is possible to upgrade existing Mailman 2.1 lists to run on Mailman 3, we are not yet officially sanctioning it. The plan is to officially support upgrades with Mailman 3.1, but for now, feel free to experiment with upgrading and let us know how it goes. It should be possible to run Mailman 2.1 on the same site as Mailman 3.

Newsgroups

  • nntp://news.typo3.org
  • Installed version: inn2 2.4.5 (Debian 5)
  • Available version: inn2 2.6.1
  • custom configuration for NNTP > Mailing list sync

Forum

  • https://forum.typo3.org/
  • Installed version: Fudforum 3.0.4
  • Available version: Fudforum 3.0.7
  • Posts are synchronised to our NNTP / mailing list server
  • Local modifications for NNTP sync: https://github.com/TYPO3-infrastructure/FUDforum
  • was never maintained by the TYPO3 Server Team, former maintainer not available anymore
  • Maintainer required, has to take care of updating Fudforum and fixing bugs without breaking our custom NNTP sync

Proposal

With the current size of our team, it is not possible to update and maintain all parts mentioned above in a feasible manner. The usage of NNTP and mailling lists is shrinking more and more, as there are other means of communication in place (Slack, Stackoverflow) more suited to the communication nowadays. Therefore, we propose to replace mailing lists, newsgroups and the current forum with a new forum based on Discourse.

Discourse

  • http://www.discourse.org/
  • modern forum
  • mailing list features available
  • already part of the TYPO3 infrastructure (https://decisions.typo3.org/)
  • the new TYPO3.org will use Discourse for comments on news. We could comibine this nicely with a new, central communication platform
  • widely known and used by others, and also well known to members of the TYPO3 Server Team

Numbers

To have some general idea, we compared numbers of messages on some of our mailing lists for April:

  • typo3-team-core: 2017: 4, 2016: 5, 2015: 35, 2014: 93
  • typo3-english: 2017: 103, 2016: 135, 2015: 215, 2014: 332
  • typo3-german: 2017: 216, 2016: 377, 2015: 696, 2014: 868

Those numbers reflect messages sent to the mailing list and newsgroup combined. We have no seperate logs for newsserver available, however looking at the number of open NNTP connections, we think this is used only by a very small number of people.

Impact

Pro

  • modern setup with a widely used technology
  • integrated into TYPO3.org user database through LDAP
  • the TYPO3 Server Team is able to maintain the whole stack
  • future proof solution
  • modern interface on desktop and mobile, mobile app available for notifications
  • we can keep current mailinglist features more or less
  • we could integrate Discourse as solution for comments on third-party websites (Wiki, News, …)

Con

  • people who still use NNTP exclusively by now will have to switch to another technology (use Discourse web frontend or watch categories once to get new posts by e-mail)
  • migration will be a lot of work, however we look forward to take that step to get rid of all those legacy software still in use

Organizational

Topic Initiator: TYPO3 Server Team
Topic Mentor: Andri Steiner

8 Likes

Totally agree with the plan and +1 on that! That was my wish already for some years now.

2 Likes

I guess I am one of the lonely guys that still use NNTP. It allows me to keep emails and newsgroups apart and is much faster than a webpage. But I must agree that one working and maintained solution is better than three half-maintained services.

1 Like

/me too
I’m also using the NNTP System and mostly liked to get the attachment issue fixed. :wink:

+1 for the plan - great summary and conclusion, thanks!

+1 this is a great idea.

I’m also using NNTP. It’s fast and simple. You can always download an existing list (not possible with forums and mailing lists).
Discourse is essentially a forum with mailing list functionality.

For users it means that email is getting used more for notifications or notifications have to go to a mobile app / chrome plugin.

I don’t think that Discourse will revive the forum/mailinglists/newsgroups; part of the support has moved to StackOverflow and Slack is still used a lot for conversations and support.
Anyway, I don’t expect any other outcome than that NNTP will be dropped and that Discourse will be used.

Thanks for your feedback.

While i agree that Discourse is mainly a forum with a bit of mailing list functionality, i don’t think that people will use if for notifications only. Actually, im running another forum (not related to TYPO3 but to VW campervans) where quite a fraction of people use e-mail only to read and reply. Most of those users are not very aware of computer stuff and just reply to notifications because they’re not able to distinguish between forum notifications and actual emails. It works fine for them, so i guess it will also work for us.

Its not so much about reviving the forum/mailinglists/newsgroups, but about removing the burden to keep those services running and up to date over the next years. As you said, a lot of former discussions moved over to StackOverflow or Slack, so i think it makes more sense to invest our energy in other things.

I absolutely agree about the fast and simple part. However, “simple” is already quite subjective, as i know a lot of people wo have not heard about NNTP, but who are perfectly at home in using web based services like Discourse. Its basically the same discussion as comparing Slack to IRC.

Regarding exporting lists: Maybe we should think about a global TYPO3 content archive, where we export content from all services, wherever they’re running. I think we had a IRC channel to HTML export once, this is something i miss in Slack, and certainly something we could expand to other services like a upcoming Discourse forum.

1 Like

Hey there.

Since a couple of other software projects either moved to Discourse from an existing forum implementation as communication platform or started with Discourse right away, I’m quite used to Discourse. As for personal opinion, using Discourse and dropping NNTP, Mailing-Lists and the Forum could be the way to go. So +1 for that.

But the thing is: It’s important to actually drop all other means of communication and try to not spread communication. Of course for lazy visitors it’s a nice thing just dropping a question wherever they are. So “as much channels as possible” might increase the number of questions asked a little bit. But to not simply drop a question anywhere but having a distinct place to search for answers before raising the same questions over and over, having as less channels as possible is imho the better approach.

My point is: Please use it as a replacement, not as an addition.

What about the guys trying to install StackExchange? Are they aware of this thread here? Is the current effort to install StackExchange an approved thing of “the TYPO3 guys” or is just one part of the community moving over there?

The same question goes to the Discourse effort: Is it somehow “approved” or just the infra structure guys trying to minimize their load?

Don’t get me wrong, I’m pretty in favor of Discourse and I’d really like to use it. But we shouldn’t take the current situation of having a limited number of complains compared to the number of actively involved people in completely different discussions of all platforms as a “the majority likes the idea”. It’s more like “the majority doesn’t care”, which is a completely different thing.

And that’s basically what happened to the StackExchange guys: They came out on 2016-09-13, basically by informing the community about having started a thing. That’s 8 months ago and they are currently at 41% of the “commitment” required to start a Public Beta.

I’d really like to see Discourse start and take over the places of all current channels of written communication. And I’d be really sad about seeing it living alongside all other channels and just pulling contribution from them.

Regards,
Stephan.

Hi Stephan!

[quote=“goli, post:9, topic:190, full:true”]
My point is: Please use it as a replacement, not as an addition.[/quote]
Years ago this was probably an issue. Today Google finds the solution, no matter if it’s posted on a blog, some mailing list or at Stack Overflow…

[quote=“goli, post:9, topic:190, full:true”]
What about the guys trying to install StackExchange? Are they aware of this thread here? Is the current effort to install StackExchange an approved thing of “the TYPO3 guys” or is just one part of the community moving over there?[/quote]
One could also say: If Stack Overflow does the job, then we can shut down the mailing lists without providing any replacement at all… But maybe it turns out that there are good reasons for both to coexist.

However, I must also say that I don’t know about the long term plan for using Stack Overflow and/or a dedicated Stack Exchange project. Communication wise, I’d say this could have been done in a much better way. So far, only a link on https://typo3.org/support/ and a mention in an Association news (https://typo3.org/news/article/diary-of-the-typo3-association-july-august-2016/) mention it.

[quote=“goli, post:9, topic:190, full:true”]
The same question goes to the Discourse effort: Is it somehow “approved” or just the infra structure guys trying to minimize their load?[/quote]
Well, the proposal you’re reading here is the formal way to approval. Waiting for feedback…

Greetings, Michael

Hey @steiner,

I like the plan, and decisions.typo3.org was actually planned to see how it could work at a small scale.

Still, there are some technical remarks that I’d like to see defined / in-place when removing the old legacy functionality - some you already adressed:

  • Migrate the old content please. If we do that, we could just shut down the old services without any grace period for having that data “found on google” still.
  • Fix LDAP / Single-Sign-On with t3o accounts.
  • Let’s also dig up the current communication channel lists and properly announce “Slack”, “StackOverflow” and “Discource” as platform everywhere (e.g. introduction package, t3o etc)
  • Find a way for the announce mailing list replacement - maybe we could use a newsletter system for that instead, where people could subscribe on t3o directly. So it does not have to be connected to discourse at all in the first place.

Please also: Do not integrate discourse in wiki.typo3.org - this is just not worth it, as wiki.typo3.org is also hardly maintained (organizational-wise).

Also, it would be great if we commonly decide who is responsible for maintaining the tool.

Overall, I give a 100% yes, let’s do it, let’s just sort out the details. Maybe people could jump in on a sprint to help you?

Thanks for the efforts again - Server team!

2 Likes

Btw there seems to be a NNTP plugin for discourse: https://github.com/sman591/discourse-nntp-bridge

1 Like

Are you able to provide me more informations here? There is nothing open, or do you know an error that I do not know?

I didn’t try Discourse but looking at the features it looks like it could replace also Slack for people that ask/give assistance.

IMHO Slack actually is not used in the right way (except for #typo3-cms-coredev). It’s good for team chat but, since the information that flows are not organized in threads (now there are threads but nobody uses them), it’s becoming a huge source of information but really hard to search and repetition will be more and more common.

Maybe #typo3-cms should be set read-only and people searching for help redirected to Discourse.

[EDIT] I add another problem of Slack: when someone ask a question, if the one person that could answer is not there reading, it’s highly probable that he will never see that question.

If we want to move to discourse, please provide an own instance under a dedicated sub-domain, otherwise the decision part will get lost in traffic.

Hi Benni,

Thanks for sharing your thoughts.

By now, my assumption was that we just keep the mailman archives (already available as HTML only) at some place. As it is HTML only, we could keep this forever without any hassle. Furthermore, we have every group available in mbox format, which we will keep as well.

However, your remark got me thinking and i just saw that there is a mbox importer for Discourse here. I will give it a try, but i cannot promise you that we’re able to import just everything 1:1.

As already pointed out by @avalarion_avarres, we’re currently not aware of any issues related to LDAP. If there are any, please let us know, im sure we’re able to fix it somehow.

I absolutely agree with those points :thumbsup:

Tacking a quick look, it seems that this plugin syncs messages through NNTP. This means that we would still ned to run and maintain a NNTP service. Personally, i don’t think this is worth it, anyone thinking otherwise please speak up :slight_smile:

This is not yet decided. However, there is already another point taken for keeping them apart: In here, you’re allowed to vote only if you’ve reached a certain trust level, which you can reach by participating in discussions. If we use the same installation for any communiction, you could reach this trust level also within other categories.

Hey there.

Please decide and commit carefully.

Don’t get me wrong, I don’t want to bash on the efforts taken here. Creating any kind of communication channel and providing services to make it work is a really important thing and I’m glad you guys do that.

My point is: The means of communication used by the community are drifting apart and to my opinion, we lack a strong decision and commitment about what the official channels are.

Just recently I stumbled across this post here:

This basically proves my point: It names seven channels to communicate TYPO3 related topics. Only one of that seven is “mailing lists, forum and slack” (which I consider as the official ones till now) and the whole “social media” is a single point of those seven, which is a whole bunch of individual channels itself.

Instead of deciding for a technology to replace existing technologies as part of the infrastructure cleanup, my strong opinion about all this is to decide for a communications strategy in the first place that covers all current channels and brings them in a distinct order.

Distinct order means: We know about all of those channels, we name all of those channels and we encourage everyone to prefer a couple of them over others.

That can be done either by a “government decision”, like asking all of but only the core team members.
Or it can be done by the whole community, asking on all currently available channels (emails, newsgroup, slack, twitter, stackoverflow) and giving it enough time to reach all of our community.

But making a technical decision, driven on infra structure changes and the manpower available for that, which was only discussed by a hand full of people is likely to be come just another option among several others when it comes to finding a place to drop an arbitrary question about TYPO3.

About integrating existing information

Whatever the platform will be, it should be a real successor of the current implementation, in all possible ways.
This include especially existing data.

Starting with a fresh installation and no content creates a feeling of something completely new that takes time to evolve.
We should avoid that feeling by filling it with all the existing threads and creating something that is mature right from the beginning.

Regards,
Stephan.

2 Likes

Hi Stephan,

We’re not creating a new channel, we’re replacing an existing one. With this in mind, I have little understanding for your problem. In fact, I understand that you’re unhappy with the spreading of information over various channels, like Stack Overflow, Slack or mailing lists. I agree that we can improve that by telling users where we expect information to be found.

However, besides this, our control over the Internet is quite limited. There are several TYPO3 forums which are maintained by 3rd parties, and we cannot forbid anyone to use Stack Overflow, Twitter etc. We can only decide if it should be recommended or not.

I invite you to draft such a strategy and mention the advantages you see by it. Please send it to the board of the TYPO3 association, because I assume that’s where the responsibility of such a concept would be.

Meanwhile, let’s decide if the lack of a concept is a blocker for continuing here, or not…

2 Likes

I am in favor of using modern tools for the communication channels.
I feel that the preferred channel to ask help for is now Slack - it is a lot since I answered or even looked at the mailing lists - because it is simple to get and offer help (too much simple?) so
my fear is that I don’t know if it will be possible to revert this “habit” and encourage people to use e.g. Discourse, instead.

I tend to agree with the remarks made by @goli