Why Is Email So Complicated? Part 362: Too Many Lazy Idiots

I'm probably going to spend the rest of my life completing this series of essays about why email is so complicated, but I'd really hate to go to my grave without mentioning one of the deepest, least soluble, and most frustrating reasons:  lazy idiots.  I rarely call anyone names, so let me explain.

As is probably clear to anyone reading these essays by now, email is inherently very complicated.  The best minds in the email world -- and some others -- have worked tirelessly and endlessly to figure out how to make it work well.  This has resulted in a host of standards, most of them well specified in excruciating detail.  They're not perfect, but if everyone complied with them fully, the world of email would be much more reliable and significantly less complicated.

Unfortunately, a remarkable number of people who write software that sends email not only don't follow the standards, and not only don't read the standards, but don't even know that the standards exist.  They look at a few messages, say "I can do that," and write software that produces things that look, from a distance, like email messages.  The myriad ways such software violates the standards is a significant, ongoing, and probably permanent factor in the complexity of email.

Consider, for example, the hotshot young programmer building a web site for his uncle.  He designs an HTML form that includes a box to input an email address, and when he gets one, he sends his uncle's information to that address in the form of something with a From, To, Subject, and Date field, but which only follows the standards if he gets extremely lucky. This happens all the time, every day.

A few such messages are an annoyance.  But now and then, the kid and his uncle get lucky.  Whatever it is they're doing, it catches on, and soon they're sending out tens of thousands of malformed messages every day.  What happens on the receiving end?

To a certain type of person, it might seem obvious that a mail server should simply reject or discard any malformed mail.  That, surely, would teach the offending party to pay more attention to the standard.  But traditionally, the Internet has been built on the philosophy of "be liberal in what you accept, and conservative in what you send."  This is the philosophy that has made the net function, and many implementors take it as gospel.

Even if this weren't the case, the marketplace forces mail servers to race each other to the bottom.   Consider the case of a third party that regularly sends out malformed mail that the recipients actually want.  If mail server A accepts it, and the customer changes to mail server B, the customer will be outraged if mail server B doesn't also accept the message.  They will consider mail server B's attempts to enforce the standard to be a bug, and inevitably B will be changed in order to keep the customer happy.  

This happens all the time, every day.

The result is that every mail server in the world has to take its best guess about what certain messages mean.  Is an incorrectly wrapped line supposed to be part of the previous header, a new header, or the beginning of the body?  Is an incorrect newline indicator supposed to be a newline or part of the previous line?  (I'll write more about these specific cases later, if I live long enough.)  Inevitably, different servers make different guesses; once you're outside the standard, there's no right answer.  And also inevitably, sometimes these guesses are wrong, and result in messages that appear malformed or unreadable to the end user.

This happens all the time, every day. It's unlikely that the world will ever run out of lazy idiots.  But it's relatively rare for one of them to decide to start naively spewing poor imitations of TCP/IP packets.  Unfortunately, sending email is such a user-visible function, and the format looks sufficiently simple, that email generation seems to be the project of choice for the protocol-challenged. We can do our best to educate them, but I think there's an endless supply.

 We can cope with it, of course.  But it does make things more complicated.