The sophisticated marketer’s guide to incrementality

August 30, 2020

As marketers, finding ways to successfully drive business metrics and organizational goals through marketing is our mission, and continued optimization of our marketing strategy and tactics is our daily work. In the last few years Incrementality and Lift as buzzwords in marketing have started to overshadow other marketing topics, especially given the increasing focus on “growth” marketing. Because these buzzwords put a premium on incremental growth, the notion of incrementality, and the tools to efficiently and effectively measure it, have become a critical focus for marketers.

The notion of incrementality has been around for a long time though. Most marketers have likely heard the 18th century quote from John Wanamaker:

Half the money I spend on advertising is wasted; the trouble is I don’t
know which half. – John Wanamaker

Wanamaker’s conundrum gets at the heart of incrementality in the digital age and leads us to ask the following questions:

  1. Does my marketing drive actual value or just claim credit for an action that would
    naturally occur?
  2. How far can I scale my efforts while positively impacting bottom line business

What is incrementality?

Incrementality in marketing refers to the incremental benefit generated per unit of action taken. Incrementality is the lift in desired outcome (web visits, conversion, revenue, etc.) provided by marketing activity (ads run, campaigns launched, etc.). Incrementality isn’t solely about assigning credit or value to a conversion (that’s attribution),  it’s about identifying the specific interaction that moves a user from passive to active and as such drives real business results.

It can be used to address the following types of questions:

  1. What happens if I increase or decrease budgets for channel X?
  2. Is campaign Y a key contributor to my wanted outcome?
  3. Is Z just cannibalizing my organic audience?
  4. How much more revenue am I generating with this marketing program?

Incrementality strives to identify the causal event of a conversion, allowing businesses to properly allocate budget, reduce wasted ad spend and optimize the marketing mix.

Doesn’t attribution indicate incrementality?

In today’s world it’s a must that marketers require some level of attribution for their paid marketing efforts (last touch, multi-touch attribution, etc.) and also likely provide some form of evaluation of effectiveness (ROI, ROAS, etc.). Measuring and determining the value of marketing has become a common and critical component of marketing organizations these days.

Unfortunately, Incrementality generally isn’t factored into existing marketing attribution or ROI calculations. We are so focused on the mechanics of HOW to measure we lose sight of the WHAT to measure. As a result marketeers often get derailed if asked about the incrementality of their efforts. You measured correctly and assigned credit logically, but how do you know your efforts fundamentally moved the needle? Are our retargeting efforts incremental? Are paid branded terms in search just cannibalizing SEO? Has the TV campaign shown a business lift? Knowing that a lead cost $10 is not as valuable as knowing that same lead generated an additional $100 in revenue. Incrementality testing helps address these questions.

Identifying the incremental value of remarketing (case study)

Remarketing has become increasingly popular led by cart abandonment campaigns, recommendations based on site visit behavior, etc. Evaluating based on classic attribution metrics looks great, largely driven by the high intent of the audience (we know they are interested), but incrementality is a bit more nuanced. In our own lead generation programs remarketing new home listings to potential buyers in a given geography we initially saw mixed results, I’ll provide a brief case study.

The ads were extremely compelling and relevant given our ability to leverage consumer intent info from past site visits and integrate into dynamic product ads across a number of ad platforms. The standard measure of success looked great with cost per lead and ROAS much better than our traditional new audience campaigns. So much so that we began shifting even more budget into the remarketing campaigns.

In measuring the incrementality of the campaigns however, it was a little bit shocking to see that we were in low double digits for truly incremental conversions. More importantly, the marginal costs of the leads generated were not consistently ROI positive despite some fairly significant revenue calculation on the true value of the leads. We kept experimenting with tweaking creative, segmenting and isolating audiences (prior channel, recency, similar audiences, geography, etc.), and of course measuring incrementality. We learned a few things along the way:

4 Key Learnings from Measuring the Incremental Value of Remarketing

  1. In remarketing recency is key. While conversion was much greater for more recent visitors, incrementality was lower. The opposite was true for longer horizon visitors or “lapsed users”.
  2. In our business, geography had a profound impact given the established value of a given lead in a specific market and marginal cost/marginal revenue calculations (In real estate the value of a lead can vary considerably largely driven by
    potential commission on home value). Some markets would never be profitable while others offered a great deal of headroom. National campaigns were blending results and under optimizing our marketing efforts.
  3. Even though the metrics for remarketing were great, shifting too heavily to remarketing exacerbated our incrementality issues as we spent more and more effort remarketing existing users and existing marketing channels. It’s important to maintain a pipeline of new users in each marketing channel/ad network.
  4. Remarketing impacted LTV. The remarketing program increased the volume of leads by a particular user (our business model valued each lead) which increased the Average Revenue per User (ARPU) which we identified by comparing the incrementally of users and leads.

Based on these insights and numerous others we were able to adjust our marketing spend to optimize for marginal cost of each lead by specific audience segments defined by geography, recency, and site behavior. The process provided valuable insights into not only our marketing efforts, but user behavior, our product experience, and our business model.

How to measure incrementality

There are three common ways to measure incrementality.

Marketing Mix Modeling (MMM) is an important practice in which historical regression can be used to derive incrementality, but this assumes the future will behave much like the past, so things like seasonality, a rapidly changing market, intense competition, etc. can impact reliability.

Multi-Touch Attribution (MTA) is a great framework for attribution, but is often challenged in today’s ad environment dominated by “walled garden” networks, current pixel tech (complicated by increasing privacy concerns), and multiple users’ presences across numerous devices. Attribution alone also often does not account for ‘immeasurable’ contributions like brand equity, offline marketing, and existing organic behavior.

The most common and generally recommended approach for measuring incrementality is a Design of Experiments (DoE) test.

How to measure incrementality with a Design of Experiments (DoE) test

In a typical Incrementality test, the target audience is segmented into a test group (exposed to the ads) and a control group (suppressed from seeing the ads or shown a “placebo”/Public Service Ad) and the conversion lift on the test group is compared to the  conversion lift on the control.

For example: Your test group saw 2% conversion whereas your control group saw a 1.5% conversion. So (2% – 1.5%) / 1.5% = 33% incrementality in conversions. Then of course you’ll then have to determine whether the 33% incremental conversions were worth the cost of the particular marketing effort.

Incrementality equation
Incrementality equation

This methodology accounts for the immeasurable or organic behavior and any other marketing activities, since both the control and treatment groups are being equally affected by those.

Incrementality is an even bigger problem in retargeting where there is potentially strong existing organic behavior from users. A common example of this approach are geographic audience holdouts often used for non-digital executions or the creation of separate and distinct audience cells in digital media.

Like many things, success is often in the planning. It’s important that you spend a bit of time thinking about the DoE incrementality test and what you specifically want to learn. The big challenge here is making sure you are asking the right questions before executing successfully so that your end results are not only accurate, but actionable.

Some things to consider when designing a DoE incrementality test:

  1. What is your primary desired outcome and what actions will you take with that knowledge? Where are you spending the most in marketing? Where are you most uncertain of your success metrics? Make sure you are willing and able to make changes and have the organizational alignment to do so.
  2. How confident are you in the outcome? Have you designed the experiment correctly (controlling variables, sample sizes, measurement, etc.)? Rigorous measurement is the key to understanding what investments are paying off and what exactly is influencing them. This is where having a good internal team is critical. It’s also an area in which success often requires collaboration across the organization as implementing incrementality tests often involve technology, analytics, product, as well as the marketing team. Many ad platforms also now regularly offer some very sophisticated tools or even data scientists and analysts to help conduct these experiments within their network to help marketers.
  3. How does the outcome align with existing beliefs? Do you have a good current understanding of your current marketing programs, the marketing channels, and the unique campaign management levers (marketing spend, campaign reach, impression frequency, audience quality, conversion rates, seasonality, etc.) within the channel? Fully understanding the dynamics behind your marketing and how it drives your business today (a baseline) is critical in understanding the nuances in evaluating and acting on the incrementality test.

Incrementality is neither good nor bad explicitly, it simply provides insights that have to be aligned with other marketing opportunities. Results are rarely perfect. Incrementality projects are very much It’s about the journey. Along this journey you learn to ask the right questions, analyse the results, and make yourself smarter about your marketing and what actions you take to scale your programs for success.

What to do after you implement an incrementality test

Once you’ve successfully implemented an incrementality test, what’s next? Any time when you run a lift test or incrementality test you are largely going to be evaluating two interrelated things: Marginal Cost and/or Cannibalization.

Marginal Cost: Most marketing reports look at a blended average for the campaign (or ad platform), which can conceal actual results. When you’re running an incrementality test, you want to see your “real” marginal cost. If too costly, it might not make sense for your business to scale up that channel.

Cannibalization: Cannibalization is related and can be a driver of increased marginal cost, but it can also be independent. Cannibalization occurs when an incrementality test shows that a user would have ultimately successfully converted and credit given to another channel/campaign (often un-measured organic), but the channel/campaign being measured “stole” the conversion as a result of existing business rules or attribution practices.

Combating Cannibalization with Incrementality (Cast Study)

An app retargeting campaign was added to our existing app acquisition efforts across a mobile display network where the primary KPI was the value of leads generated. Not surprisingly, the cost per lead was much lower with retargeting, and the blended average KPIs of the entire campaign improved. Success!

Looking at the incrementality of the campaign however revealed that less than 20% of the retargeting campaign’s impact was incremental. That fact alone meant the marginal (incremental) cost of those additional leads were actually 5X more expensive than originally assumed (pushing negative ROI), and we were simply taking credit for actions which would have eventually occurred through other channels (cannibalization). Failure?

Based on these insights however, we were able to further segment audience data and better understand timing of communication, specifically the interplay between app messaging, email, and the display campaigns. For a certain population (e.g. lapsed users, opt-outs, etc.) the display campaign showed significant incrementality, and augmented existing “organic” re-marketing efforts with acceptable marginal cost. Success!

In hindsight (and simplified for discussion) the outcome seems elementary, but in reality it becomes more complicated and requires a marketer embrace the complex dynamics of their campaigns.

Incrementality of paid search

Another classic question involves the incrementality of paid search, with plenty of well documented arguments on both sides. The short answer for search is usually neither a yes or a no, but a Maybe. For example, It’s highly likely paid search is not 100% incremental if you have even a minimal SEO presence. At the same time, attribution models often overlook the “brand” benefits of search beyond the simple click behavior.

Running incrementality tests on thousands (if not more…) keywords is likely unproductive, so focus on keyword segments like branded terms, category terms, product inventory, locations, etc. across your search portfolio and devise certain business rules for SEM efforts and use incrementality testing on those efforts. SEM is often evergreen and audience is not pre-determined (it’s driven by intent), making it difficult to suspend or segment for a test. For this reason geographic segmentations (hold outs)  are often used in addition to other segmentation.

Incrementality SEM test 2

Even though your focus is likely SEM (paid search efforts), start with SEO (organic search efforts) What’s your Share of voice? How often are you showing up in the top spot for a given keyword? Understanding this is often challenging and inaccurate, the general goal is to have better insights into which keywords and under which conditions you have a strong position.

Short answer is that if you are not ranking well in SEO the SEM ad is likely incremental. Likewise if you are ranking well in SEO, then you may choose to only show SEM ads to “new visitors” (i.e. not individuals who have previously visited and use search as a shortcut to URL) or make bid decision based on location, time, and of course keywords specifically to shift to greater incrementality. It’s not an all or nothing proposition and the effort will never be perfect. The goal is simply that continued learning and optimization to ensure your search efforts are working as hard as possible to drive business growth.

Incrementality: The North Star of marketing

Focusing on incrementality in your marketing efforts can have a profound impact on your overall marketing strategy and marketing mix. It can help you scale your marketing efforts and provide the justification for needed resources and budget.  It also provides a powerful foundation to engage and align the rest of the organization on the focus of incremental growth.

Incrementality can become a common benchmark that applies equally across the organization, providing a true value to the impact of initiatives. What CEO and CMO wouldn’t like a data driven marketing discussion on the marginal cost of acquiring a new user and the potential opportunity given additional marketing budget!

Think of incrementality as the North Star of marketing. By measuring for it, you can begin to understand the positive, negative, or neutral impact a marketing initiative has on your business. Implementing a focus on Incrementality testing can help:

  1. Prioritize data-driven decision-making and elevate its importance throughout the organization. Incrementality can serve as a focal point to a company-wide commitment to measurement and optimization, creating a test-and-learn culture that extends throughout organization and is led by engaged leadership.
  1. Shift hiring practices and put a premium on Hiring experienced, data-minded people. Marketing is an art and a science, but the word “science” is showing up in more marketing job titles. To succeed in today’s fragmented digital landscape you need people with expertise in sophisticated measurement techniques who can measure and learn from the results like a scientist and use that data to optimize each advertising channel more effectively. Hire talented people who can lead, with a focus on innovation and data in all aspects of marketing
  2. Invest in technologies for growth. Incrementality will likely require you to reconsider your marketing stack and potential changes in the technology and processes you use to measure your marketing. Current measurement platforms may be excellent attribution, but attribution alone paints an incorrect picture. This is an investment that comes with a cost, but the revenue gains from better understanding incrementality and the ability to facilitate a consistent consumer experience across digital inventory will be worth it.

It’s hard to argue against the benefits of incrementality testing. Not only is it the most accurate way to deduce the actual cost of acquired users, but it can also prevent you from cannibalizing your organic traffic. There’s a reason CMOs consider incrementality the “gold standard” of performance metrics. It’s the only way to accurately answer the question: “Did my advertising actually impact the business?” By that measure, it’s critical for marketing teams to get started with incrementality or risk leaving real revenue on the table.

Subscribe to our blog

Don’t miss a beat in the fast-changing local digital marketing landscape — sign up to stay ahead of the curve!