Comparing Meta and Google incrementality

Admaxxer is a DTC analytics platform with built-in Meta + Google ad ops. Platform-reported ROAS tells you correlation; incrementality tells you causation — the revenue that wouldn't have happened without the ad. The short answer: run a matched holdout or geo-lift test for 14–28 days, apply a two-proportion z-test (the method Admaxxer uses in v0.1), and expect Meta incremental ROAS to land 60–80% of reported; Google brand-search iROAS typically at 15–30% of reported. ## TL;DR - Incrementality = causal revenue lift from the ad; platform ROAS = correlated revenue - Two test designs: paid vs organic cohort holdout (fast) or geo-lift (slower but cleaner) - Meta prospecting incrementality usually 60–80% of reported ROAS - Google brand search incrementality usually 15–30% of reported (the rest would happen anyway) - Minimum 2,000 conversions per arm to detect a 10% lift at 95% confidence ## Test design: holdout vs geo-lift ### Paid vs organic cohort holdout The fastest method. You segment users who saw paid ads (treatment) from users who arrived organically in the same window (control), then compare purchase rates. The two-proportion z-test (used in the Admaxxer [incrementality module](/docs/incrementality)) asks: is the conversion rate in the paid cohort significantly higher than the organic cohort after controlling for time of day, day of week, and session length? Pros: fast (7–14 days), low-cost, doesn't require pausing spend. Cons: treatment assignment isn't random — paid-clickers are self-selecting. Works best for prospecting; falls apart on brand search because "users who searched your brand" are not comparable to "users who didn't." ### Geo-lift The cleaner method. Split your country into two matched cohorts (e.g. California + Oregon as treatment, Texas + Arizona as control), pause all paid spend in the control region for 14–28 days, and compare total revenue between regions. Meta's Lift and Google's Geo Experiments both do this natively. Pros: random(-ish) assignment, clean causal inference. Cons: expensive (you're zeroing spend in one region), takes weeks, requires matched geos (hard for small countries or niche products). ## Diagnostic steps ### Step 1: Pick the channel to test first Don't test everything at once. Pick the channel with the highest spend and the widest gap between reported ROAS and what feels true. Usually that's Google brand search or Meta retargeting — both are suspected of claiming credit they don't deserve. ### Step 2: Pre-register your hypothesis and sample size Before launching, write down: "I expect Meta prospecting iROAS to be between 0.6× and 0.9× of reported ROAS of 3.5, so between 2.1× and 3.15× incremental. I need 2,000 conversions per arm to detect a 10% lift at 95% power." This prevents p-hacking after the fact. ### Step 3: Run the test for the pre-committed window Don't stop early if results look good. Early stopping inflates false positives. Run the full 14 or 28 days. ### Step 4: Compute the z-test With conversion rates p1 (paid) and p2 (organic), sample sizes n1 and n2: - Pooled proportion p = (x1 + x2) / (n1 + n2) - Standard error SE = sqrt(p × (1 - p) × (1/n1 + 1/n2)) - z = (p1 - p2) / SE - |z| > 1.96 means p < 0.05 (significant at 95%) The Admaxxer incrementality dashboard computes this automatically; the [Claude agent](/docs/ai-agent) can pull the result via `query_metrics`. ### Step 5: Translate lift to incremental ROAS If the treatment cohort shows 2.8% conversion rate vs 2.2% in control, the lift is 0.6pp = 27% relative lift. Multiply the lift by the paid cohort's revenue to get incremental revenue. Divide by spend in the window to get iROAS. ## Sample size and statistical power Power calculation for a two-proportion z-test at 95% confidence, 80% power, baseline conversion rate 2%: - 10% relative lift detection: ~2,000 conversions per arm - 20% relative lift detection: ~500 conversions per arm - 5% relative lift detection: ~8,000 conversions per arm If your store does 100 conversions per day, you need 20 days of treatment traffic plus 20 days of control to detect a 10% lift. If you're doing 20 conversions per day, you need 100 days — at which point seasonality will swamp your signal. Small stores should test 20%+ lifts only. ## Common confounds Incrementality tests fail most often because of confounds: - **Email** — a new Klaviyo flow launched during the test window will inflate the paid cohort (who are on-list) vs. the organic cohort (who aren't). Freeze email during the window. - **Seasonality** — running a test across Black Friday means the baseline shifted mid-test. Avoid high-volatility weeks. - **Other channels moving** — if TikTok spend doubled during a Meta test, Meta's attribution will incorrectly claim some of TikTok's lift. Test one channel at a time. - **New customer vs returning mix** — if treatment cohort is 80% new and control is 60% new, purchase rates will differ for reasons that aren't ad-driven. Stratify. - **Day-of-week effects** — 7-day tests can catch you if treatment starts Monday and control starts Friday. Always match on day of week. For mature accounts, the Admaxxer [MMM channel contribution](/docs/mmm) is a useful cross-check — it decomposes revenue into channel contribution using OLS with geometric adstock, and the MMM-implied iROAS should roughly match your direct incrementality tests. ## FAQs **How is Meta Lift different from the holdout method?** Meta Lift is a geo-based holdout where Meta withholds ads from a subset of users; it requires $100k+ spend budgets. The holdout method works with any spend level but is slightly weaker statistically. **Can I trust vendor-run incrementality tests?** Be skeptical. Meta's reported Lift results use Meta's own attribution, which is the thing you're trying to measure. Run your own using Admaxxer or a third party. **How often should I re-test incrementality?** Quarterly is reasonable. Re-test sooner if you change creative strategy, add a channel, or see iROAS diverge from reported ROAS by more than 20% in subsequent months. **Does the Claude agent run incrementality tests?** It can query the result of a test via `query_metrics`, but setup and launch happen in the [incrementality module](/docs/incrementality). The agent will flag when a test completes. **What about brand lift tests?** Brand lift measures awareness, not revenue. Useful for upper-funnel creative decisions, not for iROAS. See [Cross-channel creative grid](/guides/cross-channel-creative-grid).