Tests & Learning Loops

"Predictor variables with no relationship to the dependent variable can pass tests of significance." Freedman's paradox

Setting up and running tests correctly is as much of a science as the statistical analysis. It can go wrong in many ways, which can lead to erroneous and costly decisions. Just taking 3 dimensions - Timing, Selection, Creative - shows 1 way to be right with 6 ways to  be wrong: 

 

1) Timing correct but Creative Selection missed:
- Timing signals translated correctly but not to the right people the message package was not focused correctly.
+ Focus on better selection and relevancy.
 

2) Creative correct but Timing and Selection missed:
- Marketing has got it right but data has missed the brief.
+ Refine signals (timing) and traits (selection).
 

3) Selection correct but Timing and Creative missed:
- Creative brief not aligned and timing forced.
+ Focus on relevancy.
 

4) Creative missed:
- Data (timing & selection) are correct but creative has missed.
+ Is the appropriate message package available?
 

5) Selection missed:
- Transition from brief to selection to activation has missed.
+ Better test/control loops to remove misses.
 

6) Timing missed:
- Too early or too late by a metre or a mile.
+ Signal (timing) refinement.
 

7) Successful:
+ Excellent! Understand why it worked.

Setting up tests

Collate a test bible, Plan out the important dimensions and variables to test. Let data-led decisions do what its good at (the now) and humans will consider what could be important tomorrow. Write down the Hypothesis and KPIs of success. Set up the test cells properly, without accidental bias. Ringfence them to prevent contamination - a series of quick tests is better than a long one. Ensure the analysis is done by those who understand the context against the written down definitions of success. Check statistical conclusions and possible errors types. Report learnings and add to the test bible.

We need your consent to load the translations

We use a third-party service to translate the website content that may collect data about your activity. Please review the details in the privacy policy and accept the service to view the translations.