Guest Opinion: Don’t Test Whispers1 Apr, 2013 By: Doug Garnett Response
DRTV campaigns draw tremendous strength and power from the ability to measure phone and Web response. But this strength creates some amazing DRTV mythologies, with some agencies promising laser-like abilities to test every creative or offer change.
This is the first in a series of articles that will debunk testing mythologies in order to search out more powerful test realities. Let’s start with this mythology of testing creative changes with laser-like accuracy.
The Myth and the Math
This mythology doesn’t affect the initial campaign test — where we look at big issues like “did the phone ring.” It comes up when we “tweak” campaigns to improve results.
The mythology suggests that, somehow, media efficiency ratios (MERs) are so stable that testing can detect the impact of even the tiniest tweaks in advertising. Average MERs are stable over the long run. But they can be quite fickle in the short run.
We recently tweaked a long-running infomercial and found it difficult to detect direct impact from these changes. Looking carefully at past and present results, it all became clear: changes to the week-to-week MERs were far bigger than the impact we were trying to detect.
In fact, this MER changed so much that it had a .25 standard deviation. Given an average MER of 1.80, all we can assume is the high likelihood that our next MER will be between 1.55 and 2.05 — that’s a full 0.50 range — far larger than a hoped-for 10-percent MER increase.
What this means is that for creative tweaks to be detectible, they have to change the MER by 20 percent or more. Anything less is just a whisper.
DRTV’s Challenging Statistics
To understand what’s going on, consider one sample airing: Food Network at 5:30 a.m., on Monday, Feb. 18, 2013. The airing is entirely unique as there will never be another airing with the same audience profile — which is driven by news events, viewer preference shifts, station competition, etc.
DRTV is far different from direct mail (where they also “don’t test whispers”). In a way, direct mail teams have it easy. They can test an A/B creative split by sending each creative to 5,000 individuals.
By contrast, in an entire year of DRTV, there are only 52 Food Network airings at 5:30 a.m. on Monday. So can your agency get “laser-like” accuracy from testing creative tweaks with 20-30 airings? No.
Whispers That Strain Credibility
Don’t let anyone tell you that “Your price was $19.99, but the spot would test better at $19.95.” There are reasons $19.95 might be smart, but testing will never prove it.
And take care with anyone who says, “We changed five words and now the show works.” Five words? The only way this isn’t a whisper is by changing “You’ll pay $1,000” to “Yours free, just for calling.”
Some Suggestions for Testing
To tap into DRTV’s power, you need a clear testing vision. Allow me to suggest a few things:
- Keep your expectations in line: The only changes you can detect in testing are big ones.
- Test for longer periods to get better accuracy: Perhaps as long as three months.
- Resist the temptation to assume too much from too little: Testing is expensive, but so are mistakes.
It is ironic that we hear this mythology of test accuracy from an industry that still doesn’t have a solid understanding of the power of market research. But don’t let that scare you off.
DRTV delivers tremendous power for marketers, manufacturers, entrepreneurs and retailers. Learn to be humble — and succeed. DRTV has the capability to deliver great results to companies who treat it wisely. ■