Accuracy and fit
What Model Accuracy, MMM Fit, and Non-trained Accuracy each measure, and why the Backtest view ends earlier than the Current Fit view.
Model Center shows three scores for a model. MMM Fit and Model Accuracy tell you how closely a model follows your reported results in a period that includes data that the model learned from. Non-trained Accuracy is the one score for dates that a model did not learn from.
What does each score measure?
Section titled “What does each score measure?”Each score runs from 0 to 100, and a higher score is better. The difference is the model and the dates that each score uses.
| Score | Where you see it | What the number is |
|---|---|---|
| Model Accuracy | Each card of the overview | The fit of the current model for the selected period. It is the same figure as MMM Fit. |
| MMM Fit | The Current Fit view of the Model Health tab | How closely the predictions of the current model follow the reported totals in the selected period. |
| Model Accuracy | The Backtest view of the Model Health tab | The fit of the backtest model on the dates that it trained on. |
| Non-trained Accuracy | The Backtest view of the Model Health tab | The score of the backtest model for the most recent dates that it did not train on. |
The product describes the scores in these words.
| Score | Text in the product |
|---|---|
| MMM Fit | How closely your current model's predictions track reported totals over this window. Higher is better, on a 0 to 100 scale. |
| Model Accuracy | An estimate of how accurately this model forecasts future performance, learned from your historical data. Higher is better, on a 0 to 100 scale. |
| Non-trained Accuracy | Accuracy over the most recent ~30 days (about 4 weeks) the model did not train on. It stays fixed regardless of the date range above. |
A model can follow the dates that it learned from closely and still do less well on new dates. To see how a model does on dates that it did not learn from, read Non-trained Accuracy in the Backtest view.
What do in-sample and out-of-sample mean?
Section titled “What do in-sample and out-of-sample mean?”The product uses two terms for the data behind each view.
- In-sample data is data that the model has already seen. The model learned from this data, so a close match is expected.
- Out-of-sample data is data that the model did not train on. A close match on this data shows that the model can predict, not only describe.
The Current Fit view includes in-sample data. The Backtest view uses a separately trained model that did not train on the most recent period. In that view, Model Accuracy covers the dates that the backtest model trained on, and Non-trained Accuracy covers the dates that it did not train on.
Why does the Backtest view end earlier?
Section titled “Why does the Backtest view end earlier?”The two views cover different periods.
- Current FitEnds on the latest modeled dateThe current model, compared with reported totals. The period includes data that the model has already seen.
- BacktestEnds where the test period of the last backtest endsA separately trained model, compared with reported totals. Its most recent period is data that it did not train on.Most recent period that the model did not train on, about 30 days
Current Fit runs through the latest modeled date. Backtest uses a separate model run, and the view ends where the test period of that run ends. The most recent weeks are therefore only in Current Fit. A line under the Backtest chart gives the date when the backtest last ran.
What is the non-trained period?
Section titled “What is the non-trained period?”The non-trained period is the most recent period of about 30 days, or about 4 weeks, that the backtest model did not train on. The Backtest chart has a marker with the label Holdout for this period.
Two metrics of the Backtest view cover only this period.
- Non-trained Accuracy is the score for the non-trained period.
- Non-trained Forecast vs Observed shows what the model forecast for the period and what happened.
These two metrics do not change when you select a different period in the page header.
What do the colors of a score mean?
Section titled “What do the colors of a score mean?”The color of Model Accuracy and MMM Fit shows where the score falls. The colors are a quick guide to the number, and the number and the two views give the detail. For the limits of each color, see Colors of the score.
What do Daily and Weekly mean on a card?
Section titled “What do Daily and Weekly mean on a card?”Granularity: on a card is Daily or Weekly. It is the time unit of the data of the model. The product words this as how your data is grouped into time buckets.
Granularity changes the periods that you can select. A daily model offers periods in days. A weekly model offers periods in weeks. See Overview reference.