I'm currently working on a small web frontend for newsgroups (among other things) and while
newsgroups have a reputation of old dusty technology I was impressed by the underlying
protocol: NNTP.
Not that the protocol is very well done (I don't have the experience to judge that) but the fact that
it's directly usable for a human impressed me. And because I really like such technology get ready
for a small netcat session: we're going to read a newsgroup without a newsgroup reader. :)
Connect
First thing we need is a connection (it always starts like this nowadays, does it?). If you run Linux
old netcat will do the trick:
netcat news.example.com 119
For encrypted newsgroups socat will help:
socat stdio openssl:news.example.com:563,verify=0
Usually newsgroup servers listen on port 119 and if they use SSL encryption on port 563. But that's
all about these commands, there is no magic behind them. If you run them you'll get a plain TCP
channel to the NNTP server and the server will send you a welcome line. Something like that:
$ netcat news.example.com 119
200 news.example.com InterNetNews NNRP server INN 2.5.0 ready (no posting)
If you newsgroup requires authentication (a login) you can send it using authinfo. If you
don't know just skip authinfo. The server will complain if it needs to know who you are and the you
can tell it.
authinfo user newone
381 Enter password
authinfo pass secret12345
281 Authentication succeeded
Dive in
To get an overview of available newsgroups just type list and hit return. The server will then
send a list of all newsgroups you can read.
list
215 Newsgroups in form "group high low flags"
example.talk 0000005146 0000002417 y
example.announce 0000000123 0000000016 n
For each newsgroup we get the name, the numbers of the newest and oldest posts in the group and
finally a flag. If the flag is y we are allowed to post in there and if it's n we are not. The numbers
are also no real post IDs but more like indices. All posts in a group are numbered consecutively and the
two numbers just say us where we can start to look and where we can stop. Also a post in a newsgroup
can be deleted any time, therefore there can be holes in the list and it's more like a "watermark": we
know there has been a message with the number 5146 in the example.talk group but it might
have been deleted already.
Anyway, lets read the newest message in the example.talk group:
group example.talk
211 2617 2417 5146 example.talk
article 5146
…
From: Someone <someone@example.com>
Newsgroups: example.talk
Subject: Some demo message
Date: Thu, 27 Jan 2011 23:20:26 +0100
…
This is the content of the demo message. Usually just the same as with mails.
.
With the group command we can say which group we are interested in. The server confirms
that with a 211 status code (the first line of a server reply always starts with the status code) an
estimated post count for this group and the low and high watermarks again (this time in the reverse
order).
With article we can now read every message we want. It returns the contents of the message
with the given number followed by a line that just contains a dot. And this is the downside of newsgroups:
all the mess of mails are in there, too. I cut a lot of headers out of the sample above and some clients use
very funny ways to encode things like the subject and the body. Unfortunately this is a remnant of the
pre UTF-8 era and there isn't much that can be done about that.
On the bright side NNTP makes browsing easy. Just type last and return to select the previous
article. The server remembers which article you red last and last sets this pointer to the previous
message. If you now type article without a number the server will send you the contents this message.
next works the same way just into the opposite direction. However once you enter a newsgroup
with the group command you first have to select a message with a number for this browsing to work.
Have your say
Posting a message works similar but the other way around. Just type post and write your stuff,
followed by a line with just a dot. If you actually want to write a line with only a dot in your message
just make two dots out of it. The server will decode that later on.
post
340 Ok, recommended message-ID <u4yuyzyw@news.example.com>
From: New one <newone@example.com>
Subject: Hello there
Content-Type: text/plain; charset=utf-8
Newsgroups: example.talk
Hi,
I'm posting this using just a raw TCP channel.
.
240 Article received <u4yuyzyw@news.example.com>
And that's basically it. You don't really need the Content-Type header if you don't use UTF-8 characters
but if you don't type English text it's a good idea since most Linux terminals use UTF-8 today.
Some other nice stuff
To get a short overview of what's going on in a group you can ask for all subject headers of some
messages:
hdr subject 5100-5146
225 Header or metadata information for subject follows (from overview)
5100 An older message
…
5146 Some demo message
.
The hdr command will show the specified header of all messages in the range. On some older
newsgroup servers you might have to use xhdr instead of hdr since the command is kind of new.
In case you want to write your own newsgroup reader (just for fun of course) you might also find the
over or xover command useful. NNTP has quite some more stuff to offer so a look into
RFC 3977 can't hurt if you plan to spend some more time with NNTP.
Some thoughts
Even after some weeks and mixed experiences with NNTP I still like the fact that you can easily
talk some NNTP yourself. Of course I don't use a terminal and netcat to read real newsgroups
(if you want to see some obscurely formatted mails that might be interesting). I rather stick to
Thunderbird or my own newsreader. However it's a very nice thought to being able to just ditch
all news reading software if something goes wrong and see for yourself what's going on. This is a
somewhat refreshing change to the newer REST style APIs which are more machine friendly but
10 KiByte of JSON code in one line tend to be a bit difficult to read…