hpdz.net

Getting the Message to the People with HTML Email

Believe it or not, this is a subject sufficiently complex that I think it deserves a page in the Technical section. I've pounded as many fists on the table and pulled out as many newly-formed gray hairs over this issue as I have over any programming or mathematical problem, and I at least need to vent my frustrations -- and maybe this will be useful to someone who's trying to start their own email newsletter.

My Adventures with Email -- A Personal Saga

HPDZ.NET strives to keep its subscribers informed and happy, and to present a classy look to the world. The newsletter is a nice way to keep in touch and make sure everyone who might like to know about new projects is in the loop.

But I have learned that it’s not as easy as I thought to produce or distribute a good newsletter by email. I thought I’d share what I’ve learned so that others might not repeat the same mistakes, and also so that all the subscribers understand some of the changes I’ve made in how the newsletter is distributed and some of the changes I had to make.

Introduction

I have switched to using a commercial email distribution service rather than continuing to send the newsletter from Outlook by BCC. This solves several problems all at once.

1. Formatting; This is probably the biggest motivation for me to seek help from a professional service. HTML email is quite different than HTML for the web, and the new service helps get things formatted right.
2. Plain Text: At the opposite end of the beauty spectrum from HTML, plain-text email is still something people prefer. The new service makes it easy to track who wants HTML and who wants plain text, and it provides a convenient tool to convert the HTML into plain text.
3. Junk email: The newsletter was getting classified as junk email by several systems. The distribution service provides tools to help assess how "junky" the newsletter looks.
4. List management: I can now have a sign-up form and one-click unsubscribe without having to manage this myself manually.
5. Compliance with federal law: Although it’s possible to comply with federal laws regulating bulk email without using a commercial service, the service helps make sure each email has all the necessary elements to keep me out of trouble.

I’ll go over each of these issues in more detail below. First, a topic of paramount importance, not listed above because it's not a problem being solved, but rather an issue that is not relevant here.

Privacy

Commercial email services generally provide mechanisms to track email openings, clicks, etc. The service I’m using is no exception, but I have chosen to disable all that stuff. See the section below on Tracking for more details. You may also want to refer to the pithy (as these things go) HPDZ.NET privacy policy.

Formatting

HTML provides a set of sophisticated directives for formatting pages that allows a great deal of flexibility and control over page appearance, as well as mechanisms for making the code easy to maintain and modify. All modern web browsers support most of these features, but many email clients have only limited support, and this severely limits how HTML emails can be composed.

What I was doing – composing the newsletter in Outlook – was probably the worst thing I could do. Outlook (and Word) generates XML code that is full of complex, proprietary style information that just fails miserably in many other email clients.

Furthermore, many web-based email systems actually strip out all the formatting that is described at the beginning of an HTML document. This requires that each heading, paragraph, hyperlink, table cell, etc. have its formatting specified in the HTML code. This is a huge step backwards in terms of HTML evolution, and makes it very difficult to manage document formatting.

The result was that the newsletter would be arranged strangely and the text would be in the wrong color, with unpleasant, flaky-looking results like black text on a black background or random paragraphs in purple. Gmail is particularly bad at this.

The commercial service provides a way to convert standard embedded style information into inline style specifications, which saves a huge amount of time. This alone makes it worth using. They also provide views of how the newsletter looks in dozens of different email clients so I can see any formatting problems right away. I’ve created some test accounts of my own on many of the major free email services, but the service gives more views from many different email systems with different configurations.

Plain Text

For various reasons, many people prefer to receive email in plain text and have configured their email clients to show all emails in plain text. Depending on the system, this can result in mild changes, like links not being obvious and images being stripped out, to drastic problems like the raw HTML code showing up.

Generally, professional-quality newsletters (which HPDZ.NET strives for at all times) are sent with plain-text alternative text as a multipart MIME email. Outlook provides no way to do this. Furthermore, converting from HTML to plain text – while maintaining decent-looking formatting of headers and hyperlinks – is not easy to do. The new service provides a tool to do this that works pretty well.

Junk email

Many email systems were classifying the newsletter as spam, and either bouncing it altogether or sticking it into the recipient’s Junk Mail folder. The main reason for this was the way I was sending it – I was BCC’ing it to everyone from Outlook. Many spam filters examine the To: field of the email and verify that it contains the recipient’s name. Although I have everyone’s name of course, the names don’t show up in the To: field when you use BCC. I didn’t want to unblind the list and send by putting everyone in the To: field, so the only other alternative was to send it individually to one recipient at a time. That would be impractical, even for a small list like this.

The commercial email service solves this problem because it sends individual emails to everyone on my list, one at a time, with each person’s name in the To: field. They also run it through multiple spam filters and show the results so I can make any necessary changes before sending it.

And, just to add insult to injury, I have learned after much experimentation, that the mere presence of the word "spam" in an email is enough to trigger some spam filters. Not only that, but also references to federal spam regulations, unsubscribing from a list, etc. will score points on spam filters. That's why I had to completely remove the whole discussion of the spam issue from the November 2008 newsletter. The words present in the discussion itself were enough to get the newsletter categorized as junk. I need some \/|@gr.a after dealing with all this.

List Management

As the list grows, it becomes a hassle to add and remove people from it. Sa far, this has not been a problem, but it would still be nicer to have an automated system to handle all this. The service provides forms for signup, unsubscribe, etc. that I can modify and include right on my web site, which makes my life easier and also looks more professional.

Is it Spam?

There’s a federal law, the CAN-SPAM act, that regulates the distribution of commercial email, bulk email, and unsolicited email. It provides severe fines (up to $11,000 – yikes!) for violations.

Although this web site is currently not selling anything and is therefore not really commercial, there are plansto produce a DVD for sale at some point in the future, so I think I need to be careful.

There’s also the question of whether the newsletter is unsolicited. I think most people currently receiving the newsletter want to receive it, although many people were added to the list without really explicitly asking to be added. Nobody has complained, so I’ve kept everyone on the list after porting to the new service. With one-click opt-out, I think that anyone who doesn’t want to receive it anymore can easily unsubscribe.

For new subscribers, the service provides an automated system to comply with the double opt-in requirement. After someone enters an email address into the signup form, the service sends a confirmation email to that address with a link that must be clicked to activate the subscription. This helps protect me against allegations of spamming, either by individuals receiving the newsletter or by organizations who may be blocking it.

Tracking

Most commercial email distribution services provide tracking features so their clients can  see how many people are opening the email, how many are clicking on which links in it, etc. None of that is necessary for my purposes here, and it makes the newsletter look more like spam to the spam filters. It also makes people more suspicious, so I have disabled all the tracking features.

Let's be clear about this: All tracking features are disabled in the newsletter.

The main two types of tracking mechanisms are embedding of small tracking images into the HTML code (one reason why some people prefer plain-text emails over HTML) and link redirecting.

Embedded tracking images are just little GIF or JPG files, often 1x1 pixel in size, that are hosted on the email service’s site with file names that identify which email they belong to. When someone opens the email and downloads the images, the service records the hit on the image file and that gives the statistics. Any images in my emails will be hosted on my own site and are not used for tracking purposes.

Link redirecting means that all links in my newsletter are not sent directly to my website, but rather to the service’s web site, which tracks them and then redirects them to my site.

Both these methods really stand out when a spam filter scans an incoming email. I don’t need them, and I don’t want to annoy the spam filters, so I don’t use any of this tracking stuff.

Privacy

This web site has a simple privacy policy – any information you provide stays just within the site and is never given or sold to anyone else.

The commercial email service has a similar policy.