Building a Culture of Experimentation

Intuition is not the enemy. Experienced judgement is a legitimate and often superior input where data is thin. The problem is that intuition tends to win arguments it should have lost, for reasons that are structural rather than personal.

• Seniority outweighs evidence, because disagreeing with a senior view carries a career cost that being wrong quietly does not.
• Attachment to ideas. People defend proposals they authored more vigorously than proposals they merely support.
• Confirmation bias in analysis design, where the question is framed in a way that makes the desired answer likely.
• Post-launch narrative, where a result is explained after the fact in whatever terms make it look intentional.
• Market movement mistaken for impact. A rising category makes almost every initiative look effective.
• Failed concepts reframed rather than learned from, so the same idea returns in eighteen months with new packaging.

None of this is corrected by exhortation. It is corrected by making the evidence standard explicit and agreeing it before anyone knows the answer.

A good experiment is not primarily a statistical artefact. It is a decision-making device, and it should do six things.

1. 01State a hypothesis that could be false, in commercial rather than statistical language.
2. 02Predict the behaviour that should change, for whom, and through what mechanism.
3. 03Establish a credible comparison, so that the counterfactual is genuinely plausible.
4. 04Measure incremental effect rather than response, netting off what would have happened anyway.
5. 05Examine heterogeneous effects, because an average effect of zero often hides a strong effect in one segment and a negative one in another.
6. 06Produce a decision. Scale, refine or stop, agreed against thresholds set in advance.

The sixth point is where most experimentation programmes fail. A test that concludes with 'interesting, let us investigate further' has produced analysis, not evidence.

The academic evidence on experimentation's value is encouraging and should be read carefully. Research published in Management Science by Koning, Hasan and Chatterji, examining thousands of startups, found that firms adopting A/B testing tools saw meaningful performance improvements in the following year, with the strongest effects among firms running more tests. The authors note that the benefits accrued disproportionately to firms already positioned to act on results, and that the setting was technology startups. It would be a misreading to assume the same magnitude transfers automatically to a large bank or retailer with longer feedback cycles and heavier operational constraints.

The most valuable experiments I have been involved in were not digital. They were commercial and operational, and they were designed by people who understood the business rather than by specialists in test infrastructure.

Culture here is a function of what leadership visibly rewards, and there are eight practices that do most of the work.

• Reward learning explicitly, including the learning that came from a null result.
• Publish negative results with the same prominence as positive ones. This single practice changes behaviour faster than any training programme.
• Separate the quality of an idea from the status of the person who proposed it, structurally, by pre-registering measures.
• Build a searchable knowledge repository, so nobody pays twice to learn the same thing.
• Track test velocity as an operating metric, because learning rate is a genuine competitive variable.
• Quantify value captured from experimentation annually, in currency, and report it to the board.
• Prevent repeated testing of the same question by requiring a repository search before a test is approved.
• Make stopping safe. If cancelling an initiative damages a career, initiatives will not be cancelled.

The cheapest thing an organisation can buy is the knowledge that an expensive idea does not work.

Experimentation becomes strategically material when it is connected to planning. That requires four things that are organisational rather than technical.

Governance. A single forum that reviews test design before launch and results after, with authority to release or withhold funding based on the outcome.

Prioritisation. Tests are a scarce resource, constrained by traffic, customer contact tolerance and operational capacity. They should be prioritised by value at stake and by how much the answer would change the decision, not by which team asked first.

Capability. A small central group that designs and validates tests, embedded with commercial teams rather than isolated from them. Central expertise, distributed application.

Integration with annual planning. The most powerful change is to require that any initiative above a defined investment threshold arrives at the planning cycle with either experimental evidence or an explicit statement of why experimentation was not possible.

Underpowered tests are worse than no tests, because they produce confident conclusions from noise. A programme that runs many small tests and declares winners from them will systematically scale false positives, and it will do so while believing it is being rigorous.

Short-term optimisation is a genuine hazard. Experiments measure what can be observed within the test window, which biases organisations towards interventions with fast, visible effects and away from investments in brand, trust and product quality that compound over years.

There are ethical boundaries that should not be crossed for the sake of a clean measurement. Testing that exploits vulnerability, withholds a material benefit from customers who need it, or manipulates rather than informs, is not acceptable regardless of the statistical elegance. Financial services, healthcare and any context involving financial hardship require particular care.

Some decisions cannot be experimented on at all: acquisitions, market entries, platform replacements, brand repositioning. Here the discipline transfers rather than disappears, through explicit assumptions, sensitivity analysis, staged commitment and pre-agreed evidence thresholds for continuing.

Finally, an honest counterargument. Excessive experimentation can slow an organisation and create a culture in which nobody will commit without proof. Judgement remains essential, particularly where the window of opportunity is short. The aim is not to test everything, it is to know which of your beliefs are load-bearing and to test those.