Drift is the bug nobody files
A broken feed throws an error. A drifting model keeps producing numbers that look fine. Nobody files a ticket, and the decisions built on it get a little worse every week.
By Nathan Barling Product Marketing Principal 5 min read
Starting with iOS 14.5, apps had to ask permission before tracking people across other companies' apps and websites. Models trained on the tracked data from before that change kept running. Nothing errored, and every number they produced still looked reasonable on its own. They were just describing a world that had moved.
Most drift doesn't come with a date anyone remembers. A forecast runs a little high one month and a little higher the next, and two quarters of targets get set on it before someone checks. Nothing crashed and no alert fired, so nobody reported it.
Models age even when nothing breaks
In 2022, Vela, Pianykh, and colleagues published what they presented as the first analysis of AI aging. They took 32 datasets from four industries, healthcare, weather, airport traffic, and finance, and trained four standard machine learning models on each. Then they watched what happened as each model got older.
Model and dataset pairs tested
Vela and colleagues, 2022
Pairs where quality degraded over time
Vela and colleagues, 2022
The striking part is the setup. The researchers chose datasets with no abrupt changes in them, specifically to avoid the obvious cause. Quality still degraded in 91% of cases.
Two other findings matter for anyone running models in production. Degradation can develop gradually, but it can also "escalate very abruptly, after a significantly long period of good model performance." And two models trained on the same data could age completely differently, one barely moving and the other degrading sharply. A model that's been fine for a year is not evidence it will be fine next month.
Marketing data moves on a schedule you don't control
The Vela datasets were chosen to be stable. Marketing data isn't. D. Sculley and colleagues warned about this in a 2015 paper on the hidden costs of machine learning, naming hidden feedback loops, data dependencies, and "changes in the external world" among the risks that build up quietly in production systems.
Marketing has all three, and the external changes come from platforms and regulators:
- Signals disappear. Apple's tracking permission is the best-known case, and browser limits and consent rules keep removing more.
- Definitions change. Standard Universal Analytics properties stopped processing data on July 1, 2023. Sessions, conversions, and engagement in Google Analytics 4 are measured differently, so a model fed the new numbers is reading a different language with the same words.
- Models change their own data. A bidding model that shifts spend changes which customers are reached, which changes the data it learns from next. That's a feedback loop, and it can drift without any outside change at all.
Why nobody files the ticket
Software bugs announce themselves. A pipeline fails, a page errors, someone gets paged. Drift does none of that. A drifting model returns a forecast, a score, or a recommendation in the same format and range it did before. It just gets less right.
Ownership is the other problem. The data team owns the pipeline, the analytics team owns the model, and the media team owns the decisions. When a forecast slowly goes wrong, each of them can reasonably assume someone else would have noticed.
Four places to look
1. The inputs
Compare the data coming in with the data the model was trained on: volumes, channel mix, audience mix, and the share of missing values. A sudden shift is usually a data problem. A slow one may be a real change in the business. Either way, someone should look before the next forecast goes out.
2. The definitions upstream
Keep a log of every change to a metric definition, tracking setting, or platform field a model depends on. A lot of marketing drift has a date on it, and that date usually lines up with a platform announcement nobody connected to the model.
3. The outcomes
When actual results arrive, compare them with what the model predicted. Track the error and its direction over time. A forecast that's drifting high by a little more each month is a trend, even if no single month looks alarming.
4. The loop
For any model whose decisions change its own future data, such as bidding, targeting, or recommendations, keep a small holdout it doesn't control. That gives you a clean read on whether it's still right, or just agreeing with itself.
Where Alli fits
Alli Alerts runs anomaly detection models built by PMG's data science team, which judge each metric against its own historical pattern and flag both unusual drops and unusual gains across every connected channel. Custom Alerts cover budget caps, pacing, and feed health, with notifications in Slack.
Alli Datasources shows connection status and feed health for every source in one place, so a team can tell a stale feed from a real change in performance. And when a platform changes a field or a team changes a definition, it's changed once in Alli Semantic Layer and applies everywhere in Alli, from dashboards to Ask Alli to planning tools. Categorizations catches naming variation as platforms change fields.
When it isn't drift
Not every miss is drift. A model can be wrong because it was never right, because a one-off event hit, or because the data feeding it broke yesterday. Check the feed and the definitions first, then the event calendar. Drift is what's left when those come back clean and the error keeps growing.
Pick the model your team trusts most. Find out when anyone last compared its predictions with what actually happened.
Frequently asked questions
What is model drift?
The gradual loss of a model's accuracy after launch, as the world moves away from the data it was trained on. It can come from changing inputs, changing relationships, or changes to how data is defined.
How common is model degradation?
Very common. In a 2022 study of four standard models on 32 datasets, quality degraded over time in 91% of the 128 model and dataset pairs, even though the datasets were chosen to have no abrupt changes.
Why is model drift hard to notice?
A drifting model doesn't throw errors. It keeps producing numbers in the usual format and range, so the problem usually surfaces as a bad decision rather than a failure anyone sees.
What causes drift in marketing models?
Platform and privacy changes that remove or redefine signals, shifts in audiences and prices, changes to metric definitions, and models whose own decisions change the data they learn from.
How do you monitor a model for drift?
Compare incoming data with training data, log upstream definition changes, track prediction error against actual results over time, and keep a holdout for models that change their own data.
Sources
- Scientific Reports, Temporal quality degradation in AI models, D. Vela, A. Sharp, R. Zhang, T. Nguyen, A. Hoang, and O. S. Pianykh, July 8, 2022.
- NeurIPS, Hidden Technical Debt in Machine Learning Systems, D. Sculley and colleagues, 2015.
- Apple Support, If an app asks to track your activity. Apple sells devices and services.
- Google Analytics Help, Universal Analytics is going away. Google sells advertising.
Last updated November 7, 2026. Figures are quoted from the sources above and were checked on that date.