Pattern #43: Long Ad Titles Save Pattern Bookmark

Pattern Author: Ronny Kohavi - Technical Fellow and VP, Analysis and Experimentation @ Microsoft

Based on 1 Tests, Members See How Likely This Pattern Will Win Or Lose And Its (?) Median Effect

Almost Certain Loser
-5
-4
-3
-2
-1
0
+1
+2
+3
+4
+5
Almost Certain Winner
Long Ad Titles

  1. Replace: Short Link Titles With Longer Ones Benefits

    The idea is to move ad text to the title line to make it longer. The additional copy could be something specific about the offer, or a key benefit.

Median Effects

?

Engagement

Ex: Any Action / Visit

?

Conversions

Ex: Signups, Leads

?

Sales

Ex: Transactions, Upsells

?

Revenue

Ex: AOV, LTV

(1 tests)

?

Retention

Ex: Return Visits

?

Referrals

Ex: Social Shares

Tests

Test #133 Tested on Bing.com by Ronny Kohavi Ronny Dec 13, 2017

Maybe It Worked Here

Listing

  •  

  •  

Zoom In
  •  
  •  
A
  •  
  •  
B

Source: exp-platform.com/2017abtestingtutorial/

In 2012 a Microsoft employee working on Bing had an idea about changing the way the search engine displayed ad headlines. Developing it wouldn’t require much effort—just a few days of an engineer’s time—but it was one of hundreds of ideas proposed, and the program managers deemed it a low priority. So it languished for more than six months, until an engineer, who saw that the cost of writing the code for it would be small, launched a simple online controlled experiment—an A/B test—to assess its impact. Within hours the new headline variation was producing abnormally high revenue, triggering a “too good to be true” alert.

HBR, September–October 2017 Issue, https://hbr.org/2017/09/the-surprising-power-of-online-experiments

Note: This experiment was a solid success and replicated multiple times over a period of months. It worked at Bing and had a profound influence. The only reason why we atributed a 0.25 point (a "Maybe") was because we don't have the exact sample size and conversion data.

 

For each pattern, we measure three key data points derived from related tests:

REPEATABILITY - this is a measure of how often a given pattern has generated a positive or negative effect. The higher this number, the more likely the pattern will continue to repeat.

SHALLOW MEDIAN - this is a median effect measured with low intent actions such as initiating the first step of a lengthier process

DEEP MEDIAN - this is derived from the highest intent metrics that we have for a given test such as fully completed signups or sales.