A-B testing in an email campaign

September’s Net Tuesday brought up a lot of questions about best practices for email campaigning.

Long or short subject lines? One ask or newsletter style? Weekly or monthly? Text or HTML layouts?

Now I may not be a grizzled old veteran yet, but I’ve already seen that “best practices” for one organization may not necessarily be the optimal strategy for everyone else. So, you have two choices when developing your email campaigns:

  1. you can guess
  2. you can test

I suggest test! It’s fast and easy with most email service providers.

Example using Mailchimp (but almost any major service will do)

NOTE: this was not a statistically valid experiment because of the small size of the test group. I did this as a test of Mailchimp’s functionality and to show how easy it is to setup an A/B test.

I setup an A/B test on the from field of a recent invitation to the Vancouver Net Tuesday group (instructions). The test sent the two variants of the email to a random 30% of the total list, and then after 24 hours automatically sent the winning variant of the email (based on open and clickthru rates) to the rest of the mailing list.

  • Group A: NetSquared Camp
  • Group B: Eli from Net Tuesday

And here’s the stats on the test. In this case “Group B” won with the more personal “Eli from Net Tuesday” from line.

As the French say, “et voilà!”

This entry was posted in Uncategorized and tagged , , , on by .

About admin

Eli strives to be a connector – the interstitial tissue that holds the muscle of a community together. He’s been a volunteer manager, an event organizer and a digital campaigner. Basically he’ll take any gig that allows him to enable a group of passionate people to create things they love. Currently he’s the NetSquared Community Manager supporting a global volunteer network of 50 monthly meetups for the nonprofit technology sector. Together they hold over 450 events per year.

2 thoughts on “A-B testing in an email campaign

  1. Calvin Taplay

    Thanks for this information. I have been researching Mailchimp for a few months now to see how effective it would be as an e-bulletin delivery platform. It is great to see the depth of the statistical analysis that the program provides. I personally prefer to receive information through RSS feeds or visiting websites as I am inundated with emails. It is good to receive some feedback on e-bulletin campaigns however. Thanks for the information.

    1. NetTuesdayVancouver Post author

      You can go in and test mailchimp for free. You can import 1000 records and send up to 3,000 emails/month for indefinitely at no charge.

      Us nerds love the RSS, but when I survey the David Suzuki Foundation’s online audience most have never heard of RSS. I suspect that’s true for much of the general public. So don’t give email newsletters a pass! Add them to your arsenal.

Leave a Reply