scrapbook

"In the early days of the Net, we built a nervous system, but nobody built an immune system."

E-mail lists choke on spam

By John Borland
Staff Writer, CNET News.com

For close to half a decade, entertainment executives and copyright-averse college students have debated the future of technology side by side on the "Pho" e-mail list. Now that forum is under siege.

Membership is falling, even though subscription requests are rising. In large part that's because so many e-mail addresses are choked with spam, or have fallen incommunicado behind bulk mail filters, and have had to be eliminated.

Recently, whole companies--including Time Warner and CNET Networks, publisher of News.com--have periodically started bouncing the list's messages. That's not only frustrated subscribers who miss out on their daily dose of digital music dish; network administrators say they sometimes have to clear their servers of thousands of returned messages a day.

Pho isn't alone. E-mail lists in general, long one of the most popular and useful online tools, are increasingly in danger of becoming collateral damage in the Net's war on unsolicited bulk mail.

"Our cures for some of these diseases are boomeranging and killing us," said Jim Griffin, chief executive officer of Cherry Lane Digital and co-founder of the Pho list. "What we're discussing is the passing of a medium. It is alarming to me that one of the most basic features of the Net has been threatened so badly."

It's far too early to write an obituary for e-mail lists. The 30-year-old medium has confronted crises before and has been reborn with the help of clever programmers and new technology. E-mail advocates say this process is already under way, as companies and list administrators figure out both how to keep spam under control without so much of an effect on mail lists and other desired e-mail messages.

"In the early days of the Net, we built a nervous system, but nobody built an immune system," said Marc Smith, a sociologist who studies communities such as Usenet and e-mail groups for Microsoft's research division. "What we're seeing now is the emergence of an immune system."

Nevertheless, it is undeniable that Pho and other groups are facing serious hurdles that could change the way the medium operates forever.

It's almost impossible to estimate how broad the e-mail list community runs. Experts say there are certainly millions, perhaps tens of million of lists. They cover every conceivable topic, from the most arcane scientific topics to the most basely sexual. Some have only a few subscribers, while others have as many as tens of thousands.

The growing problems are familiar to anyone with an e-mail box. The primary culprits are the avalanches of spam cluttering mailboxes with Viagra advertisements and XXX photos. The energy required to clear through that digital underbrush alone has taxed many people's patience for e-mail discussions, experts say.

But the response to the spam assault also has helped undermine mail lists. Many people move e-mail addresses routinely, creating dead boxes that bounce messages back to list administrators. Many people use Web-based mailboxes for e-mail list subscriptions, and these can quickly fill up with spam or even legitimate messages, again bouncing messages back to their original servers, filling administrator mailboxes and requiring substantial time to review and clear.

On the flip side are spam filters such as the popular SpamAssassin, used by many corporations. These routinely catch messages sent simultaneously to a large number of people, mistaking list messages for bulk advertisements. Subscribers have little or no way to tell that their mail is not getting through, or that, in some cases, they have been unsubscribed completely from a list.

Faced with these growing issues, many e-mail groups are changing their format to Web-based bulletin boards or augmenting their discussions with RSS (Really Simple Syndication) feeds, a popular content-distribution format used by bloggers and news sites. Internet pundit Clay Shirky, who teaches a graduate course in networking at New York University, said he's close to pulling the plug on his mailing list altogether in favor of RSS.

"The viability of mail lists is rapidly declining," Shirky said. "Fewer people are reading in e-mail directly. It's getting clear that the ordinary Web plus RSS feeds are better."

Periodic crises

This isn't the first time e-mail lists have flirted with collapse, however.

The first e-mail was sent by Ray Tomlinson in 1971, a simple test message to himself. His message evolved almost immediately into broader discussions, although only a few remained active for long. By the close of the 1970s, there were 17 public e-mail discussion lists on the ARPAnet, the precursor to today's Internet. By 1982, there were 44, according to at least one account. Others were springing up by the dozens on private academic networks such as PLATO and BITNET.

But as vibrant as these were, their own inefficiencies led to a crisis almost as dire as today's. At that time, lists were mostly run by hand, which meant that an actual human being had to respond to subscription requests and other problems. When these lists proliferated, it often took weeks for requests to be fulfilled.

Adding to problems were traffic jams caused by the era's still-scarce bandwidth. Single messages were sent out to hundreds of addresses at a time, clogging transatlantic lines so badly that e-mails between Europe and the United States sometimes took a week to be delivered. Some people on the lists started discussing whether e-mail discussion groups should be banned altogether.

The crisis soon passed, however. In 1986, a BITNET programmer in Paris named Eric Thomas wrote a tool called Listserv that automated the administrator's task of managing subscriptions. It also made message distribution more efficient, virtually eliminating the crippling traffic jams. The tool was quickly adopted elsewhere, and the number of e-mail lists on academic networks exploded.

A half-decade later, an American programmer named Brent Thomas started looking around for tools to help automate Internet-based mailing lists. He found Listserv, but decided he could write a new one as quickly as he could learn the old tool, and in a week created a program called Majordomo and a scant 3,000 lines of code. Both tools are still widely used today.

Over the ensuing decade, the shape of mailing lists has remained largely the same. Web-based services such as E-Groups, which Yahoo later bought and turned into Yahoo Groups, attracted hundreds of thousands of discussions, but the fundamental idea hasn't changed much.

An Internet immune system

That's why many in the e-mail list community think they'll survive, despite today's headaches. There is simply nothing that substitutes for the immediacy and simplicity of e-mail, advocates say.

"I don't really see too many people dropping off lists," said Chapman, whose Great Circle Associates consulting firm still manages the Majordomo software. "Mailing lists serve a very valuable purpose. They come to you. Certain Web sites I do check, but you have to go check."

Administrators are finding ways around the problems. Spam filters on list servers, and tools that ensure only list members can send to the list, help keep unwanted e-mail to a minimum. Automatic unsubscribe tools are helping reduce the amount of unwanted bounced messages.

E-mail software itself is getting better at filtering messages into folders, so that all list messages can be segregated away from spam. Web mail services such as Yahoo and Hotmail also support this feature.

Future-looking projects hold out hope of better improvements. Some programmers are working on pulling RSS and e-mail into the same interfaces, eliminating what appears to be competition between the mediums today.

Others are looking at ways to help people wade through the morass of online discussions more easily. Microsoft's Smith has written about new interfaces that would help highlight important or heated conversations among thousands of messages, for example.

These and other innovations, such as better tools to deal with the problems of spam and spam filters, will help keep e-mail lists and communities alive, he said.

"Machines have gotten us into this problem, and they're going to have to get us out," Smith said.

News.context

What's new:

E-mail lists, long one of the most popular and useful online tools, are increasingly in danger of becoming collateral damage in the Net's war on unsolicited bulk mail.

Bottom line:

Many e-mail groups are responding by changing their format to Web-based bulletin boards or augmenting their discussions with RSS feeds, a popular content-distribution format used by bloggers and news sites.

Posted by thinkum at April 17, 2004 07:52 PM