Growth.Talent
Deep Diveincrementality-testinggrowth-measurementattribution-modeling

The Incrementality Test 4 Growth VPs Run When Attribution Falls Apart

Incrementality, holdouts, and how senior growth operators actually measure what works

Apr 11, 2026|10 min read|By Growth.Talent|

Most growth teams are flying blind. They're optimizing metrics that don't matter, crediting channels that didn't drive the outcome, and scaling budgets into programs that would have happened without them. The problem isn't the data. It's that attribution itself is a fundamentally broken framework for understanding what causes growth.

That's the quiet consensus among a growing cohort of senior growth operators who have stopped asking "what did the user touch?" and started asking "what would have happened if we hadn't run this at all?"

The shift from attribution to incrementality isn't just a semantic upgrade. It's a rethinking of how modern teams structure experiments, allocate budgets, and resolve the thorniest disagreements in growth: which touchpoints actually matter, and how much credit they deserve.

Why last-click attribution is a storytelling problem, not a measurement one

The seduction of attribution models is that they offer narrative clarity. A user clicked an ad, signed up, converted—story complete. The problem is that the story is often fiction. Attribution tools assign credit based on proximity and observability, not causality. They tell you what happened, not what you caused.

Shruti Khatod, VP of Growth Marketing at Nutrafol, has spent years reconciling brand and performance marketing, and she's learned that separating measurement frameworks is where teams go wrong. She doesn't treat upper-funnel and lower-funnel channels as different species.

We also normally talk about brand activation and that measurement is very different than growth activation and that measurement, right? I think that's where it all begins because we don't know or don't feel like we can measure them in a way that the benchmarks can be connected, that's when the whole idea of let's keep them separate starts to begin.

— Shruti Khatod, VP of Growth Marketing at Nutrafol

Her team at Nutrafol has built a system that connects brand KPIs—aided awareness, favorability, unaided recall—to performance outcomes like cost per acquisition. The key insight is that upper-funnel work doesn't live in some unmeasurable ether. It shows up in the efficiency of lower-funnel spend. But only if you're measuring the relationship, not just the touchpoints.

This is a departure from the traditional split: brand teams run surveys, performance teams run ROAS reports, and the two groups nod politely at each other in planning meetings. Khatod's approach collapses that boundary. Her growth channels are brand amplifiers. The creative, the color scheme, the narrative arc—it's consistent from awareness to conversion, and the measurement stack reflects that continuity.

If you don't connect that back to how is that impacting your day-to-day optimization, that's where the miss is. When you build a connection, then you can actually take advantage of the brand KPIs that we are tracking.

— Shruti Khatod, VP of Growth Marketing at Nutrafol

The hypothesis framework: when teams disagree, test the counter-argument

Attribution debates are often proxy wars for deeper strategic disagreements. One team believes paid social is driving growth. Another thinks it's riding the coattails of organic. Both have dashboards to prove their point. The meetings multiply, alignment stalls, and the organization defaults to whoever argues loudest or ranks highest.

Laura Schaffer, VP of Growth at Amplitude, has a different protocol. When her team launched Amplitude's self-serve plan—a high-stakes pricing and packaging overhaul—she didn't try to force consensus on contested questions. She formalized the disagreement.

Where we disagree, instead of sitting in meetings over and over trying to get everyone to align, just recognize we're not gonna agree. What's the hypothesis we're gonna go with? And we're gonna do that. What's the strongest counter-hypothesis? Let's pay attention to that.

— Laura Schaffer, VP of Growth at Amplitude

Schaffer's team identified the top ten most divisive hypotheses and tracked them explicitly during soft launch. The structure is elegant: launch with the lead hypothesis, but instrument the counter-hypothesis so you know what signal would prove you wrong. This isn't A/B testing in the traditional sense—it's A/B reasoning. The team isn't just running experiments; they're pre-registering the conditions under which they'd change their minds.

This approach treats disagreement as information rather than friction. If the product team thinks feature X will drive adoption and the growth team thinks it's pricing, you don't need a tiebreaker. You need a measurement plan that captures both dynamics. Then you let the market settle the argument.

Holdouts and the incrementality question: what would have happened without you?

The core problem with attribution is that it conflates correlation with contribution. A user touched five channels before converting. Great. Which one caused the outcome? Attribution models assign weights—first-touch, last-touch, linear, time-decay—but none of them answer the causal question: what would have happened if you hadn't run that channel at all?

That's the incrementality question, and it requires a different architecture. You can't answer it by looking at touchpoints. You need a control group. You need holdouts.

Peter Caputa, CEO at Databox, has built a business on the premise that most teams don't know if they're winning or losing because they lack comparative context. Databox's benchmark groups allow companies to see how their performance stacks up—not against their own history, but against a cohort of peers. The insight is simple but powerful: you can't know if your 5% conversion rate is good unless you know what everyone else is converting at.

They can instantly see how their performance compares to a group of other companies. Oh, you're outperforming 52% of companies in this group for this metric, and you're outperforming only 20% of the companies for this specific metric.

— Peter Caputa, CEO at Databox

Caputa's framework is a version of the incrementality mindset: isolate the variable, measure the delta. Databox users connect their data sources and immediately see where they're above or below the benchmark. It's not causal inference in the strict experimental sense, but it shifts the question from "did this number go up?" to "did this number go up more than it should have, given the baseline?"

That shift is the heart of incrementality thinking. The metric that matters isn't absolute performance. It's the gap between what happened and what would have happened in the counterfactual scenario where you didn't intervene.

Where the experts diverge: long-term brand equity vs. short-term optimization

Not everyone agrees on how to balance the timescales. Khatod's framework at Nutrafol leans into the idea that brand metrics—favorability, awareness—should inform daily optimization. The feedback loop is tight. Brand work isn't just a long-term play; it's a lever you pull this quarter to improve efficiency next quarter.

Christine Segrist, VP of Consumer Marketing at Canva, takes a slightly different stance. She argues that strict binaries between B2B and B2C, or between brand and performance, are outdated constructs. People move fluidly between contexts. A user might see a Canva billboard, use the product at work, and recommend it to a friend—all in the same week. The brand doesn't live in discrete channels. It's ambient.

People are constantly moving between different spheres of their life. So this really strict binary of B2B and B2C, we just feel like that's a little bit of an outdated notion. People are experiencing our brand in the world in a way where they're not like, oh, what part of the org chart sent me this communication?

— Christine Segrist, VP of Consumer Marketing at Canva

Emma Robinson, Head of B2B Marketing at Canva, echoes this. Canva's marketing org is heavily matrixed—business units for B2C, B2B, and international, with shared centers of excellence for channel execution. The structure forces collaboration and prevents the kind of siloing that lets attribution debates fester. When everyone is working from the same creative brief and the same channel playbook, it's harder to argue that one audience deserves a fundamentally different measurement philosophy.

B2C gives us a lot of creative ideas for B2B. The channels are consistent across everybody. So it is important just to make sure we have that consistency across the channel regardless of the audience that you're speaking to.

— Emma Robinson, Head of B2B Marketing at Canva

The tension here is real. Khatod wants to pull brand insights into near-term optimization cycles. Segrist and Robinson argue that the brand is a continuous experience that doesn't segment neatly by funnel stage or business line. Both are right, and the disagreement points to the deeper challenge: there's no one-size-fits-all measurement system. The architecture you build depends on what kind of bets you're making and over what time horizon you're willing to wait for proof.

The real cost of broken attribution: misallocated budgets and false confidence

Attribution errors aren't just academic. They compound. A team sees that paid search has the highest last-click conversion rate, so they scale budget. But paid search was harvesting demand created by a brand campaign that started six months ago. The brand campaign gets defunded because it "doesn't perform." The pipeline dries up. The team panics and dumps more money into paid search, which is now bidding on a shrinking pool of intent. The unit economics collapse.

This death spiral is common. Schaffer has seen it play out across multiple companies, and her diagnostic is sharp: you need to know when you've got something real, not just something that shows up in a dashboard.

If you really have lightning in a bottle, you've got something effective. It tends to echo in a few different chambers, not just one thing that's really moving. It's a few.

— Laura Schaffer, VP of Growth at Amplitude

Schaffer's heuristic is that real growth rarely comes from a single lever. If you're seeing lift in multiple places—organic, paid, retention, word-of-mouth—that's signal. If only one metric is moving, you're probably looking at noise or short-term arbitrage. This is another version of the incrementality mindset: don't trust a result until you've seen it replicate across different measurement lenses.

Caputa's survey-based benchmarks at Databox offer a different angle on the same problem. He's not just measuring performance; he's measuring process. Databox runs roughly 50 surveys at any given time, often in partnership with agencies and consultancies, asking users how they've set up their tools, which features they're using, and how they're structuring their teams. The goal is to help companies see not just whether their numbers are good, but whether their operating model is sound.

It allows any HubSpot customer to come in and say, I want to compare what I'm using to other people and see if I'm missing something. Maybe I should be using this feature and I'm not. Or hey, maybe I haven't set up my HubSpot account correctly.

— Peter Caputa, CEO at Databox

This is incrementality thinking applied to infrastructure. The question isn't just "did revenue go up?" It's "are we doing the things that high-performing companies do?" If the answer is no, the attribution model is moot. You're not measuring the wrong thing; you're building the wrong thing.

What to do Monday morning: build the control group and name the counter-hypothesis

The common thread across these operators is a shift from passive measurement to active experimentation. Attribution is passive. It records what happened. Incrementality is active. It asks what you caused. The difference is the presence of a counterfactual—a world where you didn't run the campaign, didn't launch the feature, didn't change the pricing.

For teams ready to make the shift, the playbook is surprisingly concrete. First, stop treating every channel as equally measurable. Some channels—paid search, email retargeting—lend themselves to tight feedback loops and clear attribution. Others—brand campaigns, out-of-home, word-of-mouth—require longer windows and softer proxies. That doesn't make them less valuable. It makes them differently measurable. Build the instrumentation accordingly.

Second, run holdout tests wherever possible. If you're launching a new campaign, hold back a randomized segment and measure the difference. If you're scaling a channel, test a geographic or demographic control. If you're investing in brand, track leading indicators—search volume, direct traffic, survey metrics—and model their relationship to downstream conversion. Khatod's team does this at Nutrafol by connecting brand favorability scores to CPA trends over time. It's not a perfect causal chain, but it's a lot better than ignoring the relationship entirely.

Third, formalize disagreement the way Schaffer does. When your team can't agree on what's driving growth, don't force alignment. Write down the competing hypotheses. Decide which one you're going to act on. Then instrument the counter-hypothesis so you know what it would take to prove yourself wrong. This does two things: it keeps the organization moving, and it creates a learning loop that's more rigorous than consensus-building.

The end state isn't a single attribution model that everyone believes. It's a portfolio of measurement approaches, each calibrated to the type of bet you're making. Some bets pay off in weeks. Some take quarters. Some are legible in a dashboard. Some require qualitative triangulation. The mistake is pretending they're all the same, or worse, pretending that the easy-to-measure thing is the only thing worth doing.

The growth leaders who've moved past attribution haven't abandoned measurement. They've made it harder, more honest, and more willing to sit with uncertainty. They've stopped asking what the user touched and started asking what the company caused. That's a higher bar. It's also the only one that matters.

Related Insights