CASE STUDY · COINTREAU · CRO
Testing programs across markets, brands, and campaigns.
Test program
Ongoing
Hypothesis-led
Hypothesis
Variant build
Honest analysis
Every test ends as
Ship it
Kill it
Run it longer
Structured
Tactical
Multi-market
Client
Anonymized — Premium spirits brand
Industry
Premium spirits
Service area
CRO
Engagement type
Project + retainer hybrid
Status
Ongoing
01 — The brief
Some of it was structured testing on the permanent site — the kind of test program where hypotheses are designed up front, variants are built with discipline, and decisions to ship or kill are made on data rather than gut. Some of it was tactical: a campaign launch in three weeks, a seasonal moment in five, a new market entry that needed the landing page to perform without time for a 12-week structured test. Both kinds of work matter; they need to be scoped and run differently.
02 — What we did
01
Structured testing program on permanent pages
Hypothesis design, variant development, post-test analysis. Tests built around real friction analysis — not green-button theatre. The discipline of killing tests that shouldn’t ship even when they technically “won” because the win didn’t translate to anything meaningful.
02
Tactical optimization for campaign moments
Seasonal landing pages, campaign hubs, market-entry pages. The optimization work that doesn’t fit a structured testing program because there isn’t time — but still needs to perform. Same discipline, shorter cycle. Sometimes the right call is “set up a test”; sometimes it’s “ship the best possible version and learn from what happens.”
03
Multi-market coordination
Tests run across multiple markets at once — different traffic profiles, different conversion behaviors, different platform setups. The work of figuring out which markets share enough behavior to test together, and which need to run separately to avoid muddying the data.
04
Honest analysis
Tests end one of three ways: ship it, kill it, or run it longer. We’re explicit about which result we got and why — including telling the team when a test that “won” probably shouldn’t ship because the result was statistically significant but practically meaningless.
03 — How the engagement runs
Project-based work for major campaign moments. Ongoing retainer for the structured testing program. Hybrid scoping when a project transitions into a longer-running program — which it often does.
The CRO work is integrated with the broader analytics work we run for the same client. Test design, hypothesis sourcing, and post-test analysis all sit on the analytics infrastructure rather than running in a separate silo.
04 — What changed
“
Decisions to ship or kill are made on the data the team trusts because the analytics underneath has been kept clean. Campaign moments perform without the team improvising the optimization at the last minute.
The more substantive change: a CRO discipline that treats killing a test as a real outcome. Most testing programs ship variants that won by tiny margins on irrelevant metrics because the team needs a win to justify the program’s existence. We’ve been comfortable saying “this won by 2% on click-through but didn’t move conversion — kill it.” The team is comfortable hearing it because the program over time has shipped enough real wins that the discipline pays back.