How to measure marketing effectiveness for acquisition and member campaigns

Marketing campaigns typically target one of two main audience pools: 1) Prospects, i.e., those a business aims to acquire; or 2) Members, i.e., those a business seeks to retain or engage. To measure marketing effectiveness, each group presents unique challenges.

In the case of acquisition marketing, information about prospects is often limited. Marketers must allocate spending across various channels to cast a broader net to acquire new customers. They can target addressable markets that meet their specific criteria (i.e., individuals aged 65+ for insurance companies that provide Medicare plans or those that live within specified locations for retail organizations). However, complete information about the prospect pool is often unknowable.

By contrast, member marketing benefits from having extensive data on current members. Much of the person-level data utilized is generally stored within an organization’s first-party data. This allows marketers to utilize personalized channels (such as digital outreach and content-specific mail) for more precise targeting and measurement.

In this blog, we will examine and compare these two types of marketing. For each type, we’ll touch on the broader objectives and measurement techniques, the data required for measurement, the typical challenges marketers face when attempting to gain insights from data, and the ways that analysts can offset these challenges to measure marketing effectiveness.

Why It’s Crucial to Measure Marketing Effectiveness

The measure of marketing effectiveness establishes the incrementality of our marketing, providing clarity into what we gained from a campaign, i.e., would not have happened otherwise. This is essential for discussing budget, ROI, and the net benefits marketing provides to the business. Effective measurement leads to educated, data-driven, strategic marketing decisions that account for unknowns and clarify the ideal focus areas.

Whether the marketing is directed at prospects (acquisition) or members, measurement allows marketers to answer big questions such as:

• “What is the value that marketing is providing?”
• “How can we increase our outcomes given budget constraints?”
• “What marketing tactics are the most effective at driving positive outcomes?”

These are complex questions that drive strategic decisions in organizations. The identification of insights that can provide better answers to these questions and make more informed decisions is invaluable.

Acquisition Marketing Campaign Measurement

Acquisition marketing is complicated. Marketers have a wide range of channels to use to engage new prospects. These include traditional channels, such as Direct Mail and TV (both branded and direct response); the wide array of digital channels, such as Display, Paid Search, Organic Search, Online Video, and Social Media, amongst many others; and more industry-specific channels, such as Events, Affiliates, or Out-of-Home (OOH, e.g. billboards or transit ads).

Measurement Techniques

With so many channel choices, finding the right mix of budget resources can be challenging. Marketing Mix Modeling (MMM) is an excellent method for answering mix-related questions. An MMM is an econometric model that helps measure the effectiveness of marketing activities and their impact on sales or other KPIs. Within MMM model development, audience granularity is not required. A set of aggregated data at various segment levels (e.g., time periods, geographies, product categories, and other types of well-defined metadata) can be used to build a valuable model.

While MMM helps provide a broad understanding of how to allocate spend across channels, it is limited in its ability to provide insights at the more granular customer level. Identifying how to attribute “credit” for each sale (or other output KPI) across marketing channels requires some attribution models.

The default, simpler methodology is last-touch attribution (LTA). This methodology attributes 100% of the sale to the last channel a customer engaged with before converting. This often leads to over-attribution of bottom-of-funnel channels in reporting, causing an overinvestment in these down-funnel channels that is more straightforward to measure; and atrophy within less measurable, upper-funnel channels that play an essential role in the acquisition process through their impact on brand health and audience attitudes.

The far more challenging approach is multi-touch attribution (MTA). MTA aims to solve the LTA problem by providing a more comprehensive view of the customer journey. The model apportions the credit of an application across the various channels that prospects engaged with throughout the buying process. A good MTA model can help marketers better understand the value of each touchpoint. With the complicated ecosystem of marketing and sales efforts, better information can significantly improve strategies.

Data Requirements

One key challenge in measuring acquisition marketing is finding signal within the sheer volume of data available. With a huge market and many potential customers, marketers have access to a wealth of information that can inform their strategies. However, the quality and granularity of the data across channels can vary significantly.

For example, in the case of Direct Mail as an acquisition channel, marketers have complete visibility into the “1s” and “0s” – that is, they know exactly who received a piece of mail and who did not. This level of granularity allows for more precise measurement and analysis.

Alternatively, poor ID resolution for upper- to mid-funnel channels, such as Digital channels and TV, is a crucial hurdle to overcome when building an acquisition MTA. Unlike Direct Mail, which allows for precise tracking, these channels often rely on cookies, device IDs, or other imperfect methods of identification. As a result, these channels usually only yield aggregated impression and response counts. As such, it can be difficult to connect individual users across different touchpoints on these platforms if they don’t use them to convert.

Finally, multiple application capture channels generate different levels of data, which can be challenging to reconcile with each other and the funnel activity. For example, a lead captured through a website form may provide different information than one captured through a call center. Integrating these disparate data sources into a cohesive MTA model can be complex and time-consuming.

Strategies for Overcoming Measurement Obstacles

Marketers can implement several strategies to overcome the challenges associated with poor data granularity within marketing channels. One strategy is an aggregated MTA approach, as opposed to the more traditional discrete MTA detailed above.

An aggregated MTA measures the effect of marketing investments on the last-touch attributed marketing channels or known segments over a given time period. This methodology estimates channel- or segment-specific dependent variables using multiple stimulus variables. In many ways, an aggregated MTA is more similar to an MMM in technical modeling methodology than a discrete MTA. In this instance, it measures the multi-channel impacts on siloed/coded response measures.

Model propensities offer another approach to overcoming the measurement challenges of channels without person-level impression data. This is applicable for channels, such as DRTV or Digital channels, where the individuals who convert (the “1s”) are known, but those who do not convert (the “0s”) are not. The aggregate number of impressions for these channels is known for a specific geography and time period. A model can be built to impute impressions within the broader audience pool. Individuals with higher propensities can then be treated as a “high likelihood” impression within a traditional MTA model.

Member Marketing Campaign Measurement

Member marketing can have many objectives, such as ensuring existing customers remain satisfied with their experience or engaging the individual to take certain actions to add value to the organization (e.g., extending their subscription, buying an add-on, etc.). While having a “known” population makes member marketing campaigns easier from a targeting perspective, these campaigns can face challenges in determining the prioritization of the testing agenda.

Measurement Techniques

A Randomized Control Trial (RCT) is the classic approach for testing incrementality. In this methodology, the population is randomly split into “test” and “control” groups, where the former receives the novel treatment to be evaluated, and the latter gets a comparative baseline. This baseline typically does not include marketing and is considered a universal holdout for establishing marketing’s incrementality in general. This approach allows the team to determine what conversions would have been lost if the campaign had not been run.

While RCT is the most straightforward approach conceptually, there are some opportunity costs to consider. In member marketing campaigns, the impact of traditional RCT universal holdouts tends to be “felt” more by the business because the members being withheld from marketing are known. Additionally, in its purest form, an RCT should ask a simple, one-objective question per test, which, depending on the outcome’s response timeline, can frustrate marketers who want to test and learn on big questions at a faster cadence.

Given the desire to preserve the whole addressable population in most member campaigns, MTA is a measurement route that can provide reads on incrementality while allowing the total population to be messaged. As discussed in our acquisition example, member-level data enables us to apply this modeling concept at a per member, per journey level, or a discrete MTA. The outcome is an understanding of incrementality per conversion while enabling aggregations to multiple different audience cuts given the discrete data origins.

In the member marketing world, a discrete MTA is an attractive foil to RCT, given that it mitigates some of the obstacles noted above. When using this approach, a universal holdout is not necessary to achieve incrementality. However, embedding RCT outcomes into the model itself and comparing the model output against RCT tests is a way to make your discrete MTA more mature.

Since discrete MTA models are built at an individual level and member populations can quickly get into the tens of millions, an advanced data science platform with scalable compute is necessary to enable this type of work (e.g., Databricks or Snowflake). Additionally, temporal constraints on the underlying data feeding these models can sometimes create a barrier to scoring and results. For example, if our campaign’s objective is first to have a member schedule an appointment with their doctor and, after that, schedule another appointment to get a specific test, such as a colonoscopy, the lifecycle of that final response is several months. Additional data complications and refresh cadences can delay this process even further. Given that MTAs look backward on conversions to see what drove the action, the campaigns receiving credit in this scenario would not get proper attribution for several months.

Data Requirements

In a member marketing scenario, we have distinct, discrete data on each person we are marketing to. There is also a clear line of sight into each individual’s outcome, i.e., whether they did or did not act. This means we can cleanly get to the “1s” and “0s,” reflecting the true outcome for a member. This foundation serves RCT and discrete MTA methods, allowing for tailored sampling and individual-level modeling.

Additionally, the marketing typically measured in member campaigns is not usually awareness-based outreach (e.g., TV ads). Instead, it consists mainly of direct response marketing. Channels such as email, outbound calling, SMS, and direct mail can all directly route members to the intended outcome. This facilitates member-level tracking of each marketing piece an individual received and intermediary metrics (opens, clicks) for the applicable channels.

Strategies for Overcoming Measurement Obstacles

Given the tradeoffs between the two outlined approaches, the marketing team should align with stakeholders on a clear learning agenda. If faster results are the priority, RCT would be the best solution despite its tradeoffs. To overcome hesitancy in the control population in the case of a universal holdout, the analytics team can provide various options on how sensitive (what level of confidence) and specific (how small of a difference) the test read will be—the more sensitive and specific, the larger the holdout. Often, a middle ground can be found to produce definitive reads at a reasonable population size.

If there is no desire to withhold individuals from marketing and the conversion metric being tracked has a reasonable lifecycle, a discrete MTA would be the favored method. Once it has been built and given consistent streaming inputs and a relatively swift conversion metric, discrete MTAs can migrate into real-time reporting of conversions reported to date.

Techniques & Strategies for Effective Campaign Measurement

It’s crucial to measure the marketing effectiveness of acquisition and member marketing campaigns to make informed, data-driven decisions that optimize budget allocation and drive positive outcomes. While each marketing method presents challenges, various measurement techniques and strategies can be employed to overcome these obstacles.

For acquisition marketing, a combination of MMM and MTA can provide a comprehensive understanding of how to allocate spend across channels and attribute credit for conversions. In the case of member marketing, RCTs and discrete MTAs can help establish the incrementality of campaigns and identify the most effective tactics for engaging existing customers. By leveraging the appropriate measurement techniques, utilizing the available data, and aligning on clear goals, marketers can make strategic decisions that drive business growth and demonstrate the value of their efforts.