How Marketing Mix Models Could Lose You $10,000,000
Or what I found by playing around with the team with our open source testing kit.
I was speaking to a few industry colleagues today, including someone quite sharp in media who had just deployed an open-source MMM model from a very big technology vendor. It was deployed by their very big and reputable media agency, and they were thrilled it had been so quick and so "out of the box."
To me, it all sounded too good to be true.
On Tuesday, we launched our Open Source MMM toolkit in APAC which was backed by many practitioners, so it was the perfect moment for us to pull our own kit out and run a comparison on five real-world, anonymized client datasets. The results are quite shocking.
Why We Must Test Marketing Mix Models
In marketing science, there is no "ground truth." Unlike training an AI to identify cats in photos, where you can easily check if the answer is right or wrong, you can't look up the "correct" ROI for your TV campaign last quarter. The entire purpose of an MMM is to estimate that truth.
This is a crucial distinction. If you don't have a known correct answer to check against, how do you know your model is any good? How do you know it's not just a complex random number generator, fitting to noise in your data and giving you a dangerously misleading picture of reality?
The answer is through rigorous testing and validation.
The Tests We Ran
To put the models through their paces, we quickly ran two fundamental tests that any marketer should be able to understand and ask their analytics provider about.
Out-of-Sample Accuracy: Think of this as "predicting the recent past." We take a chunk of historical data (say, everything up to two months ago), build the model, and then ask it to predict the sales for the last two months—a period it has never seen. A reliable model that has truly learned the underlying patterns of the business should be able to predict this unseen period with a low margin of error.
Cross-Validation: This is a more robust, almost paranoid, version of the out-of-sample test. Instead of just one test period, we systematically break our data into multiple sections or "folds." We then repeatedly build the model on some of the folds and test its prediction accuracy on the fold that was left out. By doing this over and over until every fold has been a test subject, we get a much more stable and trustworthy measure of the model's predictive power. It helps ensure the model's accuracy wasn't just a lucky fluke on one particular time period.
The Results
We ran our model against a popular open-source model on five anonymized datasets. The metric we used to judge them is the Mean Absolute Percentage Error (MAPE). In simple terms, MAPE tells you, on average, how far off the model's predictions were from the actual results. A 5% MAPE means the model was, on average, 5% off from the real number.
Here's what we found.
Test 1: Out-of-Sample Accuracy (2-Month Holdout)
This test reveals how well the model can forecast the immediate future. The errors for the open-source model were quite high out of the box. I’d note that this doesn’t include feature engineering or customisation that someone might do to improve the models - which many people will do as part of the MMM process, and most tech vendors also recommend doing.
An average error rate of 22.7% makes a model unusable for forecasting. For two of the runs, the open-source model was off by a staggering 40-50%.
Test 2: Cross-Validation Accuracy
This test speaks to the fundamental reliability and stability of the model's understanding of the business drivers.
While the average error is lower here, a nearly 10% error for Client E and over 12% for Client D are still significant red flags. Our validated model was consistently more accurate across the board.
This shows you how much variance can exist under the hood in MMMs.
The Implications: What This Means in Dollars and Cents
Percentage points on a slide can feel abstract. Let's translate this into the language of the CFO: money.
Impact on Financial Forecasting
Imagine you are the CMO for Client B, and you have an annual marketing budget of $20,000,000.
Using our model, with a ~12% forecasting error, your predicted sales outcomes for the year might have a variance of around $2.4M. That's a significant number, but it's a risk you can potentially manage.
Using the open-source model, with a 51% forecasting error, your predicted outcomes could be off by over $10,200,000.
You cannot build a credible budget, secure investment from finance, or set reliable targets for your team with that level of uncertainty. Your budget and forecast is essentially indefensible.
Impact on Channel ROI and Capital Allocation
This is where the damage truly multiplies. A high forecasting error isn't a separate issue from channel ROI; it's the alarm bell that tells you the ROIs are fundamentally wrong.
Here’s why: A model predicts total sales by adding up a baseline (what you'd sell with no marketing) and the incremental sales it attributes to each channel. If the model’s total prediction is off by 51% (as it was for Client B), it means it has profoundly misunderstood how much each of those pieces contributes. It has failed to learn the true drivers of your business.
That 51% error is a symptom of the model incorrectly attributing sales. It might have credited TV with a huge impact while completely missing the true effect of Search. This misattribution directly leads to wildly inaccurate ROI calculations.
Please note: the below is a completely hypothetical set of numbers simply to illustrate the point, as is the budget above.
Let's see how this plays out. The flawed open-source model, with its 51% error, tells you to allocate your $20M budget like this because it thinks TV is your best channel:
But in reality, Search was twice as effective. An accurate model, with a low forecasting error, would have correctly identified the true drivers and recommended this:
By trusting the model with the high forecasting error, you spent $20,000,000 to generate $47.5M in return. An accurate model would have guided you to a $57.5M return with the same budget.
The decision to use the unreliable open-source model cost your business $10,000,000 in lost revenue.
What This Means When You Deploy a Model
The appeal of a fast, "out-of-the-box" solution is strong, but the risks are immense.
Speed Matters Little: A fast answer is useless if it's wrong. In fact, it's more dangerous because it encourages you to make bad decisions, faster.
Inaccurate Models Create Financial Risk: As demonstrated, we are not talking about small statistical discrepancies. We are talking about multi-million dollar errors in capital allocation that directly destroy shareholder value.
Beware the Conflict of Interest: If your media agency—whose job is to buy media—is also running the model that measures the effectiveness of that media, you have a classic "marking your own homework" problem. They are inherently incentivized to produce a model that justifies their buying strategy, not one that is brutally objective and accurate. When you don’t have certainty and strong testing, this inevitably means models become a sales tool for media rather than a measurement tool.
The $247 Million Blind Spot: A National-Scale Risk
The risk of a single inaccurate model is clear. But what happens if this becomes the norm? Let's scale this up to the ASX 100 to understand the potential for national-level economic damage.
First, some conservative assumptions:
Let's assume the average ASX 100 company has an annual marketing budget of $15 million.
We'll use our out-of-sample test results, which measure a model's forecasting ability.
Our study found the average forecasting error (MAPE) for the open-source model was 22.7%, while our validated model's error was 6.2%. The difference in forecasting inaccuracy is 16.5%.
Now, let's apply that to the average ASX 100 company:
16.5% (Additional Error) x $15,000,000 (Budget) = $2,475,000
This means that for a single large company, relying on the less accurate model would introduce an additional $2.47 million variance into their financial forecasting compared to using a robustly validated model.
Now, multiply that across the entire ASX 100:
$2,475,000 (Error per Company) x 100 (Companies) = $247,500,000
If Australia's top 100 companies all adopted unvalidated, "out-of-the-box" models with this level of inaccuracy, it would create a $247.5 million blind spot in the nation's corporate forecasting. This represents a colossal amount of capital at risk of being misallocated, leading to systemic economic inefficiency, missed growth opportunities, and destroyed shareholder value on a massive scale.
The lesson is clear: demand more. Question the outputs. Ask about out-of-sample validation. Ask about cross-validation. Don't accept a black box, even if it comes from a big-name vendor. Your budget, your business, and maybe even a small piece of the economy, depend on it.