How to use A/B testing on your subject lines, using ConvertKit

A/B testing can be a helpful tool when you’re trying to figure out what best lands with your subscribers. It tends to be an ongoing thing rather than a once-and-done type of activity.

The basic premise

The feature of A/B testing is designed to give you two options (as the name suggests) to use as your email subject line and optimise the outcome by letting your email marketing platform (i.e. ConvertKit) determine which subject line will yield the best results for your campaign.

Let’s take a look at what this looks like when using ConvertKit…

💡 Note: The A/B testing feature is only available for ‘Broadcast‘ emails and cannot be used for emails contained in a ‘Sequence‘.

Create a broadcast

In ConvertKit, navigate to the menu option ‘Send – Broadcasts‘;

Click the ‘New broadcast‘ button to launch the email editor;

At the top of your email, you’ll see the ‘A/B’ testing feature, just to the right-hand side of the subject line;

Click on the ‘A/B‘, to the right of the subject line;

This will drop down a second text box, to be used as your second subject line. Now it’s time to come up with 2 subject lines, one for the ‘A’ option, and another for the ‘B’ option, e.g.;

From this point onwards, you just write your regular email – nothing else is different.

Once you’re ready to send it, either click the ‘Send now‘ option or schedule in advance and sit back and let ConvertKit do its thing.

The A/B test

Once you ‘send’ (i.e. ask ConvertKit to send) your email, ConvertKit takes 15% of the email’s intended recipients and sends them the email using the subject line ‘A‘.

Then it repeats the same process for another 15%, using the subject line option ‘B‘.

So, now that it’s sent your email to 30% of your recipient list, ConvertKit waits for 4 hours to see what the open rates look like for each subject line.

After 4 hours, ConvertKit determines which option worked best based on ‘open rates’ and then sends the remaining 70% of recipients the email, using the ‘winning‘* subject line.

*It’s worth noting that on occasion, the ‘winning’ subject line can change after all emails have been sent; for example, using the test above, option ‘A’ may eventually overtake ‘B’ when calculating open rates.

To explain in numbers, the final test results could be like this;

• 150 emails sent using option ‘A’ – with 75 opens, i.e. 50% open rate.
• 850 emails sent using option ‘B’ – with 400 opens, i.e. 47% open rate.

You should also be aware that the smaller your sample size (i.e. number of intended recipients), the less useful the test results may be.

​

​

In ConvertKit, the winning subject line will be marked by a ‘WINNER‘ badge, however, you should know that the badge is applied to the winning subject line at the 4-hour mark, even if the other subject line eventually proves more effective.

Full A/B test reports are available from inside your broadcast ‘Analytics‘ section.

Subject lines

Email subject lines have one job – to get the recipient to open the email. If they open it, it’s then over to your email content to take care of the next job.

They are designed to work in collaboration with your ‘preheader’ (or ‘preview‘) text. This is a snippet of text that appears in the email as listed in your subscribers’ email clients but doesn’t appear in the email itself.

Below is an example of an email showing both the subject line and the preview text;

The subject line is: “Email marketing – how I used it to drive £1,234 of sales last week…“

The preview text is: “Here’s the step-by-step process I used.” – however, this is not displayed inside the email itself, only in the ‘preview’ listing.;

You can see the email above, the ‘preview’ text is not shown.

Open rates and why you need to interpret them carefully

As useful as they can be, open rates are not necessarily accurate, so you should remember this when doing your test results analysis.

For example, and without getting too techy, the way email ‘opens‘ are detected (and counted) is by loading a tiny pixel in the form of an image (invisible to readers) – if that image gets loaded, it sends a signal back to your email marketing platform to say it’s been opened.

Any email subscribers who have the ‘load images’ option switched off, will never show up as having opened your emails, even if they do.

Another foible with the statistics is caused by Apple’s privacy policy that they deployed in recent years to protect their customers.

Essentially, what Apple does is open their customers’ emails (using software) to detect what’s included in the email – part of this check artificially loads the tracking pixel (i.e. the image) thereby triggering the signal back to your email marketing platform to say the email has been opened, even though it hasn’t.

⚡️Summary

If you’re sending out broadcast emails to your email list, especially if you’re sending hundreds of emails at a time, then you should be using the A/B testing feature that ConvertKit provides – it gives you a chance to make comparisons with the type of language you use and types of titles.

Using this feature on smaller email lists can be misleading, for example, if you send an email to 20 subscribers, your numbers would look like this;

• Option A – send 3 emails: 1 open = 33%
• Option B – send 3 emails: 2 opens = 66% (winner)
• Then send the remaining 14 emails using option B.

You can see here that the results are pretty meaningless no matter what happens.

If you haven’t yet joined ConvertKit, here’s the link to get your free account, including a 14-day free trial of all premium features: convertkit.com

​

The Freelancer’s Guide to Email Marketing gives you everything you need to know to design, launch, and run an effective email marketing system in your business. ​ Download your free copy now

​