What are overfitting and data leakage?
Both problems produce the same symptom: great test results and poor real-world performance. They have different causes.
An overfit model is too flexible for its data. With many parameters and few data points, as in most marketing models, it can bend to fit every bump in history, including the random ones that won't repeat. A leaky model has a different flaw: something in its training data contains the answer, often information from the future, so it learns a shortcut that won't exist in use.
How do you catch overfitting and leakage?
| Check | What it catches | How |
|---|---|---|
| Hold-out test set | Overfitting | Keep data the model never sees until the final check |
| Time-based validation | Overfitting and look-ahead leakage | Train on earlier periods and test on later ones, rolling forward |
| Compare train and test error | Overfitting | A large difference means the model memorized |
| Feature timing review | Target leakage | Confirm every input is known before the moment of prediction |
| Simpler baseline | Both | If a simple model nearly matches the complex one, keep the simple one |
The bias-variance tradeoff sits underneath. A model that's too simple misses real patterns, which is bias. A model that's too flexible chases noise, which is variance. Regularization, priors, and fewer inputs pull a model back toward the middle.
What is an example of data leakage?
A conversion model that was too good
A team builds a model to predict which site visitors will buy. It scores 97% accuracy in testing, far better than anything before it.
One input turns out to be whether the visitor reached the order confirmation page, which only happens after a purchase. The model learned to spot buyers after they bought. With that input removed, accuracy drops to a believable level, and the model becomes useful for its real job.
Overfitting vs. data leakage
| Overfitting | Data leakage | |
|---|---|---|
| Cause | Model too flexible for the data | Training data contains information unavailable in use |
| Test results | Good on training data, weaker on new data | Good on both, if the test set leaks too |
| Caught by | Proper hold-out testing | Reviewing when each input becomes known |
| Fix | Simplify, regularize, add data | Remove or re-time the leaking inputs |
Why do overfitting and leakage matter?
Both produce models that win the bake-off and lose in production. In marketing, that means budget recommendations built on patterns that were never real.
- Thin data. Three years of weekly data is only 156 points, so marketing models overfit easily.
- Time order. Shuffled cross-validation lets a model learn from the future. Validate forward in time instead.
- Too-good results. Treat a sudden leap in accuracy as a bug until proven otherwise.
How does Alli do this?
Alli by PMG's Scenario Planner pairs a media response model built on historical sales with a brand equity model built on current signals. Two-stage estimation is designed to reduce the instability common in traditional models, so forecasts stay steadier as data changes.
Forecasts are modeled estimates with the assumptions in view, so teams can see what a recommendation rests on, and Alli Semantic Layer keeps the definitions behind every input consistent across the data.
Explore Alli Scenario PlannerOverfitting and data leakage FAQ
What is overfitting in simple terms?
A model that memorized its training data instead of learning general patterns. It looks accurate on the past and does poorly on anything new.
What is data leakage?
When a model is trained with information it won't have when it's used, often data from after the moment of prediction. It makes test results look far better than real performance.
How do you prevent overfitting?
Use more data, fewer or simpler inputs, regularization or informative priors, and test on data the model hasn't seen, ordered forward in time for time series.
Why can't you shuffle time-series data for cross-validation?
Because the model would train on later periods and be tested on earlier ones, effectively seeing the future. Time-series validation trains on the past and tests on what came after.
Last updated Back to the glossary