Part 2. MM4XL Tools > 1. Strategic Tools > Forecast Manager > 3. Technicalities > Opening the black box of Forecast Manager > Reliability and accuracy measures

Forecast Manager

Reliability and accuracy measures

Forecast Manager can print several indices useful for evaluating the accuracy of curve fit, the overall fit reliability, and the consistency of the quality of fitted values.

Accuracy is measured with the following indices:

Index

Formula

Meaning

MAD

Mean Absolute Deviation

Source: Ragsdale

Sales Forecast Software with Sales History Data

It assigns equal weight to all errors, so it is easy to compare, but it is difficult to interpret its scale of measurement.

It fails to take under- and over-representation into account. Meaning: the smaller the more accurate the fit.

MAPE

Mean Absolute % Deviation

Source: Ragsdale

Sales Forecast Software with Sales History Data

Like MAD but in percentage, so it overcomes the scale of measurement problem. Meaning: the smaller the more accurate the fit.

MSE

Mean Square Error

Source: Ragsdale

Sales Forecast Software with Sales History Data

It minimizes the occurrence of a major error. But penalizes techniques that produce only a small number of large errors, perhaps at start. Meaning: the smaller the more accurate the fit.

RMSE

Root Mean Square Error

Source: Ragsdale

Sales Forecast Software with Sales History Data

Like MSE. It is sometimes preferred to MSE because it is easier to interpret, for it has the same unit of measurement as the actual series. Meaning: the smaller the more accurate the fit.

R Squared

Coefficient of Determination

Source: Jarrett

Sales Forecast Software with Sales History Data

It represents the part of actual data explained with the fitted data. This is also the square of the correlation between actual and fitted data. Meaning: the larger the more accurate the fit.

Sales Forecast Software with Sales History Data

Overall reliability is measured with the following indices:

Index

Formula

Meaning

U-statistics

Source: Jarrett

Sales Forecast Software with Sales History Data

Sales Forecast Software with Sales History DataVariance actual series

Sales Forecast Software with Sales History DataVariance fitted series

If the method forecasts perfectly then U=0.

If generating erroneous forecasting U=1

Sales Forecast Software with Sales History Data

Durbin-Watson coefficient

Source: Jarrett

Sales Forecast Software with Sales History Data

DW tests the correlation of residuals (error terms).

If DW > U, conclude Ho.

If DW < L, conclude Ha.

Ho: error terms are independent.

Ha: error terms are correlated.

U = Upper level and L = Lower level can be found in DW tables. Sample size, significance level, and number of independent variables are needed to locate the appropriate values.

Consistency of fitting performance is measured with the following indices:

Index

Formula

Meaning

Turning-Point Performance

Source: Bails

False Signals: Sales Forecast Software with Sales History Data

Missing Signals: Sales Forecast Software with Sales History Data

A turning-point is a change in the direction of either the curve of actual data or of predicted values.

False signals occur when a change in the direction of the fitted data is not due to a change in the direction of the actual data.

Missed signals occur when a change in the direction of the actual data is not paired with a change in the direction of the fitted data.

Consistency of Performance

Source: Jarrett

Level set by the user and comprised in the range 0-100% of fitted data.

It counts the number of fitted values lying above and below the boundary level set by the user, typically 1-10% of fitted values.

Lifetime license:

Price: euro

Vote this tool
We proudly serve
Your vote
vote1 vote2 vote3 vote4 vote5
Email:
Gender:
M
F
Age:
Position:
Department:
Comment: