The Importance of Testing Direct Mail

It still amazes me, after nearly 60 years in direct marketing, how few companies understand and actually do enough direct mail testing. The only logical explanation is that test is a four-letter word. And people don’t like to use four letter words, right?

Why Test

From a direct marketing standpoint, we can test the list(s), the offer, the package creative, and components of the package, the format, and so on.

The ability to test and measure results is what differentiates direct marketing from all other communication efforts. Testing, when done correctly, will help a company decrease the number of pieces mailed and intentionally increase its return-on-investment (ROI).

Testing allows us to determine, in a real world setting, what works, what doesn’t work and why. Like direct marketing itself, testing is only about numbers, ROI and data. A grid or matrix allows the direct mailer to test and track several lists, creative approaches and offers at once.

My advice is to set-up a testing matrix and test within budget parameters. The goal is to break even or make money on your testing while simultaneously learning as much as possible. A word of caution though – make sure that the test cells implemented are meaningful. If they are not, you will waste money. For instance, testing outer envelope teaser copy seldom produces significant lifts in response rates.

Testing does not have to be complicated

The practice of testing involves simple techniques to collect data. Data in turn becomes knowledge, which you need to succeed, move forward and grow.

Testing is a progressive art and can help make marginal programs more successful and successful programs more profitable.

What is a logical progression for various direct mail tests? What should you test first? Package? List? Price? Offer? What should you test next?

You could drive yourself crazy developing and assigning priorities for testing. In the process, you could develop one of two afflictions:

• Testiphobia (irrational fear of testing)
• Testmania (irrational love of testing)
• Testiphobia will prevent you from testing. That’s not good. Testmania will cause you to become test happy. That’s not good either. Either affliction diminishes your chance of success.

To simplify your testing decision, here’s a step-by-step checklist. I don’t claim that it’s the answer for all mailers, but it’s reasonable enough that you could follow it without undue concern.

• Test #1 – Test Mailing Lists – Create a direct mail package (or have one created) that you feel has the best shot at success. Use it as your initial control package to test various lists of businesses or consumers, recommended by a list broker. Each package sent to each list of prospects or customers (or list segment) must be identified.
• Test #2 – Test Package – Create an entirely different package from your control. Change the copy, theme, format and offer. Then take your most responsive lists and use them to test this new package against your control. The winner of this test becomes your control. (Test one and two can be combined using grid testing.)
• Test #3 – Test Price – If you have price flexibility, take your control package and change only the price. Nothing else. You might even test three different prices in a three-way test. The price package which wins (using whatever criteria you choose) becomes your control.
• Test #4 – Test Offer – Change the offer only. Nothing else (of course, you have to change some of the copy to reflect the change in offer. but only change the copy directly relating to the offer.) You might test a free bonus vs. no bonus, two bonuses vs. one bonus, different bonuses, soft offer vs. hard offer, half-price vs. two-for-one, etc.
• Test #5 – Test Copy – Not selected words, but the entire copy thrust. In copy tests, it’s usually best to change the primary appeals. You can test copy of individual components like the sales letter, brochure or order form. or everything. All other aspects of the package must remain the same. Do not change offers, colors, formats, paper, etc.
• Test #6 – Test Format – On format tests, focus on major changes, not minor ones. For instance, standard envelope mailing vs. a self-mailer, 9 x 12 package vs. a #10 package, personalized vs. non-personalized, (full color vs. one color), etc. Do not change copy thrust or offer. Keep in mind that 60-70 % of direct marketing success is finding the right audience.

Make an offer that cannot be refused.

Once you’ve defined your marketing you want to come up with an offer that will intrigue the prospect enough so that they will react in a positive fashion. Offers usually contain words like “Free” or “Money-Back If You’re Not Satisfied”, or “Guaranteed to.”, or “For a limited tine only”, etc.

Sell benefits, not attributes.

Don’t tell what your product is made of or how many pieces of equipment you have in your plant. Tell them what it will do for them. Will it make me happier, richer, more attractive, healthier, etc.? Use words that are easily understandable in a legible format. Fancy graphics are nice, but don’t let graphics overwhelm your basic message. On the other hand, strong graphics can help capture attention.

Current customers are your best customers

You will expend a lot more resources going after new customers with products they have never purchased versus retaining existing customers.

Mailing lists and offers are the two real biggies. They can make a huge difference. Don’t sweat the small stuff like color, paperweight, and teaser copy or envelope size.

It’s meaningless if you don’t track response and profit. If you don’t code your various tests so you can track response/profits there’s no sense in testing. You must be able to track and analyze tests so that you know what’s actually working and what isn’t.

How many responses are needed to make a test statistically valid? For a test to be statistically valid (meaning you can have confidence that the results you achieved on the test are likely to occur again when you mail again), you should have a minimum of 30 responses. The more responses you get, the more confident you can be.

Don’t rollout to a big list without a retest. Let’s say you tested package “B” against your control package “A” and “B” knocked the socks off it. Your inclination is to jump back in with both feet and mail the tail off of “B”. But before throwing caution to the wind, do a retest. Make sure the results and analysis were accurate. Rule of thumb. Don’t mail more than ten times your test quantity on a retest, just in case the initial results were flawed. So, if you tested 5,000 names, retest up to 50,000.

To play it safe, you can use a 75%/25% split. Because they know what to expect (profits) with the control package, many mailers opt not to test because they don’t want to give up those known profits. Of course, by not testing they may be missing out on even more profits. There’s a solution to this conflict. Instead of testing on a 50-50 basis, you can test at 75%-25% or 80%-20%. You’d mail your control to 75% of the list and the test package to 25% of the list. That way there is little “known” profit risk. but you still get to test. Remember though – for statistical validity, you need 30+ responses.

Some guidelines.

Use the same quantity of names for every list you test. Often this means 5,000 names, since many list owners will not rent smaller quantities for a test.

However, if your budget limits you to testing only one or two similar lists, select the larger list – because they have the most rollout potential.

Always test the “hotline” names – the most recent segments of any list – first. If they don’t work, no other segment will.

Test new lists early. New lists tend to deliver the highest response rates when they are first placed on the market.

Re-mailing the same offer to the same audience repeatedly over time will result in a decrease in response.

The smaller the target market, the faster the response will drop off. Varying the package or offer significantly with each new mailing is required to stimulate interest and response

Use your own customer file to profile against prospecting lists. Segments that have the greatest match are most likely to produce the best response

Test new products and offers against the most responsive segment of a list.

Every response device should bring back the label or inkjet address or email address from the list including a source code and date stamp. The source code tells you which list generated the reply. The date stamp tells you how long the name has been on the file and its lifetime value since being added to the file.

Customer lists, even when fatigued, tend to outperform prospect files. Therefore, even the oldest segments on your house file should be mailed as long as they produce more orders than the best prospect list.

Never throw away your inactive customer files. Hold them. Lists of inactive customers – even 5 to 9 years old often produce greater response than prospect lists.

It’s always better to mail a different segment of the same list, rather than make a repeat mailing to a segment already mailed.

If a list generates a good response, re-mailing the same promotion to it approximately 8 weeks after the first drop will generate approximately 50 percent of the original response. Example: If the first mailing pulled 5 percent, the second drop will produce around 2.5 percent.

No matter how often you mail to your house file, it is fairly certain you are not mailing enough. If you mail to your house file four times a year, try six or eight times.

Don’t test too many variables. If you do, you’ll wind up with results anyone can argue with. Remember, you can increase your return on investment by increasing response rates or by decreasing production costs.

Adding an insert will increase your product costs. However, if an insert is appropriate for your mailing and if it is executed properly, it could increase your response rate (and return on investment).

Conclusions

People must be randomly assigned between the various test and control groups. Random samples ensure valid samples across lists, demographics, geography, etc.

Test and control groups must be representative of the population plan to approach in their rollout.

Test and control groups must be treated identically outside of the factor(s) being tested. If not, it will be impossible to interpret and apply results.

Set decision criteria before starting the test and determine the direction you will take based on the decision.

A statistical test will tell you if results are different, but not whether those differences are important.

Testing is like proper exercise and diet; it will only provide a benefit if used properly over a long period of time…a little bit of testing will only help very little.

Earlier we said that “Test” is a four-letter word. But so is “Cash”.

Call Dunhill for a FREE list consultation

1-800-DUNHILL (386-4455) or email us at dunhill@dunhills.com. Visit our website www.dunhills.com for further details on any of our services.