Case Studies

We’ve talked a lot about how to theoretically improve email newsletter engagement, but can we really help? So we challenged email publishers to duels. We reached out to newsletter publishers with over 10k subscribers to participate. We then took their content and sent it to half of their list using our tactics and they controlled the other half. Here are the results.

Case Study 1

In our first duel, we spoke with a publisher who wanted to compete in 2 ways:

- Improve their clickthrough rate on opens (clicks/unique open) - Improve their clickthrough rate on a specific link in their emails.

The Background

This first competitor sends a weekly digest of articles from its site. The digest typically includes a quick note from the editor at the top, followed by 12-25 links to articles or events.

Our rocket boosters

In this challenge, we pit the publisher’s newsletter with a newsletter we created. Our newsletter was largely identical to the publisher’s, since we wanted to retain the same content. But, we made the following small changes:

- Shortened the letter from the editor

- Reduced the number of links in the editor’s note (from an average of 9.3 links to an average of 2.3 links)

- Added descriptive text (teaser text) following each of the links to articles so that you could read a bit about the articles before clicking

- Made the layout 1 column (instead of 2)

For privacy reasons, unfortunately, we cannot show you the two layouts.

Start your engines!

We divided the subscriber list into 2 groups. Each group received either the original layout or our layout for 3 consecutive weeks. We kept subject lines of both emails the same each time, and we sent all campaigns within 10 minutes of each other. We divided each group into

approximately 15k subscribers, and we sent out all campaigns through a third-party email service provider.

Game 1: Improving overall clickthrough rate

The publishers’ newsletter is represented by A, and ours is represented by B. After three weeks, the data shows a larger percentage of people who opened and clicked on something in our layout.

Generally speaking, content near the top of emails receives more clicks than content at the bottom. We also found that in our layout, the clickthrough rate of the lower sections was higher than the clickthrough rate of the comparable section in the original layout. In other words, the combination of adding “teaser text” to article links (which was in the lower section) and reducing the length of the editor’s letter drove more readers to engage with content throughout the

newsletter, including the very bottom section.

Lastly, subscribers to our variation, on average, clicked 8% more links than subscribers of the publishers’ version. For all three weeks, our layout encouraged readers to engage with more content.

LaunchBit 3, Publisher 0.

Game 2: Improving clickthrough rate on a specific link

In this second game, we wanted to see which newsletter would do better in directing clicks toward a specific link. Very often, email publishers may want readers to sign up for a event or buy a particular product, and it’s important to test how a layout does in driving clicks towards specific actions.

In game 2 of this challenge, the goal was to drive traffic to 2 high priority links. The first was a link to the article named in the subject line of the email. The second linked to a sign-up page for an upcoming event.

Subject line link – mini competition

In the original newsletter, the publisher typically mentions an article in the subject link and then places a link to that article as the very first link in their newsletter. As a result, we did the same in our layout. As you can see, the publisher wins 2 out of 3 matches.

LaunchBit 4, Publisher 2.

However, what is interesting is that for whatever reason, the publisher made the article link the 5th link in their newsletter during the 2nd week, while we placed that same link in the first position. And, the publisher still won that week. It would seem that people actively searched for that link after reading about it in the subject line.

Event link – mini competition

Despite the publisher’s apparent comeback, when it came down to driving traffic to a particular event link, our newsletter blasted the original into space. By cutting the number of links in the editor’s note from roughly 9 to 2, we made the event link standout. We increased the

clickthrough rate on their event-signup link by more than 2x.

FINAL: LaunchBit 7, Publisher 2.

Takeaways

The numbers show that your highest priority link should be the first link in your newsletter content. The editor’s note is an effective way to highlight important links to your readers, but reduce the number of links in this note to focus on a particular call-to-action.

Case Study 2

For this competition, the publisher wanted to compete in two ways:

- Increase open rate

- Increase click-through-rate The Background

The publisher sends out a digest-style newsletter that reports the most popular articles on their site from the last week. They also use the newsletter to tell their subscribers about new features and to highlight users and discussions.

Our Rocket Boosters

One of the first things that struck us was the sheer number of images in the publishers’

newsletter. A typical issue had 70 images!!! The average word count was roughly 125 words per issue. Though spam filters are largely a black box, best practices ask for a healthy text-to-image ratio of 7:3 (http://www.icontact.com/blog/the-hulk-no-bulk-text-to-image-to-text-ratio-tips/).

Frankly, this is not an exact science, but we thought that it was plausible to improve engagement by reducing the number of images in the layout. We decided to make these changes:

- Slim down the newsletter by removing extraneous images - Increase the text-to-image ratio by using CSS wherever possible - Add alt text to all images

- Move clickable items above the fold

- Make the email easier to read by making it more list-like Start your engines!

We split tested between the publisher’s layout and our version throughout January and randomly assigned half the list to their layout and the other half to ours.

Game 1: Improving overall open rate

The publishers’ newsletter is represented by A, and ours is represented by B.

After 4 weeks, the data shows that we were able to get higher opens overall.

LaunchBit 3, Publisher 1.

Game 2: Improving overall click-through-rate

These competitions are not rigged! We tied in this battle.

LaunchBit 5, Publisher 3.

Overall, this was a challenging match for us, but we found that by slimming the newsletter, we were able to drive them more opens, possibly by reducing the susceptibility of the email from going into spam (though this is speculation). Although an analysis of their open rates by email clients was inconclusive, for 3 of the 4 split test campaigns, our layout also yielded fewer spam complaints.

FINAL: LaunchBit 5, Publisher 3.

Takeaways

At the close of this challenge, we recommended that publisher adopt our layout as a starting point for continued tweaking to increase click-through-rates.

Case Study 3

In our third challenge, we competed with Snapette, a popular fashion discovery app, to see if we could beat their engagement in their newsletter. For this competition, the publisher wanted to increase the overall number of clicks back to their site.

The Background

Snapette sends a weekly digest-style newsletter that shows off fashionable apparel items. They had a very long and image heavy newsletter. As such, we suspected that a portion of their newsletters might be going into spam. So our strategy for this challenge was to see if we could increase their open rate first, and win this contest by getting extra clicks off the extra opens.

This publisher’s newsletter averaged 41 content images across 3 sections of the newsletter. In addition, there were several images for social sharing icons as well as the company logo and pictures to download the app. None of these images utilized the alt text attribute.

Start Your Engines

The first layout we tested reduced the number of content images to just 8 by removing the bottom section of the newsletter and the small, thumbnail images in the middle section. Instead of 2 large images in the first section that were one on top of the other, we made these images smaller and put them side by side, bringing them both above the fold. The overall width of the newsletter was also narrowed to bring it to a more traditional newsletter width. We also

encouraged the publisher to write a short note to the readers and made sure the images had text links in addition to alt text.

We blasted the open rate — 26.13% vs 22.75%. Unfortunately, we lost on clickthrough rate — 2.16% vs 2.68%.

LaunchBit 0, Publisher 1.

Having a significantly greater open rate across all four split tests strongly supported that new layout with reduced number of images was helping the newsletter avoid spam boxes. However, the overall decrease in clickthrough rate meant that while more people saw the email, the new layout was not as engaging.

In an effort to bring engagement back to original levels, we tested the following changes over the next few weeks:

- Removing the note to the readers

- Bringing back two vertically stacked large images

- Adding back photos by restoring a shortened version of the original bottom section

When the first two ideas didn’t move the needle on the clickthrough rate, a more in-depth click analysis revealed that a significant portion of readers clicked on the bottom content section exclusively. However, instead of restoring the entire section, which would have added 15 more images to the email, we opted to go with just 6 images as a compromise. Fortunately, this did just the trick and we were able to increase clickthrough rates (while still maintaining superior open rates)! However, it took us a while to get to this conclusion, so we lost this challenge overall.

FINAL: LaunchBit 1, Publisher 3.

Takeaways:

Going forward, we’ve recommended the publisher re-examine the priority of the different newsletter sections, perhaps moving the bottom section toward the top. They may also want to experiment with the number of images in each section to see if they can further optimize open rates.