A-B testing in an email campaign

September’s Net Tuesday brought up a lot of questions about best practices for email campaigning.

Long or short subject lines? One ask or newsletter style? Weekly or monthly? Text or HTML layouts?

Now I may not be a grizzled old veteran yet, but I’ve already seen that “best practices” for one organization may not necessarily be the optimal strategy for everyone else. So, you have two choices when developing your email campaigns:

1. you can guess
2. you can test

I suggest test! It’s fast and easy with most email service providers.

Example using Mailchimp (but almost any major service will do)

NOTE: this was not a statistically valid experiment because of the small size of the test group. I did this as a test of Mailchimp’s functionality and to show how easy it is to setup an A/B test.

I setup an A/B test on the from field of a recent invitation to the Vancouver Net Tuesday group (instructions). The test sent the two variants of the email to a random 30% of the total list, and then after 24 hours automatically sent the winning variant of the email (based on open and clickthru rates) to the rest of the mailing list.

• Group A: NetSquared Camp
• Group B: Eli from Net Tuesday

And here’s the stats on the test. In this case “Group B” won with the more personal “Eli from Net Tuesday” from line.

As the French say, “et voilà!”