How does PendulumRock incorporate the model selection process in order to provide you forecasted estimates with the greatest accuracy, best fit, least error, and greatest parsimony?Well, let's walk through a simple example using a single instrument.Here, we will analyze the summary report section for the forecasted mean monthly sales spanning the time frame from September 2015 to July 2016.
The analyst sees from the start that a HoltWinter Multiplicative Method (HW Multip ES) has been chosen as the model for producing the forecasted estimates. This method exhibits multiplicative seasonality, hence, we are presented with 3 smoothing parameters: alpha, beta, and gamma for the level, trend, and seasonal factors, respectively. Supplementing the model parameters are the model fit statistics: standard error, AIC (Aikaike's Information Criterion), AICc (Corrected Aikaike's Information Criterion), & BIC (Bayesian Information Criterion).
Next, we see the accuracy measures for the model chosen here, HW Multip ES. These include ME, RMSE, MAE, MPE, MAPE, MASE, & ACF1. Our selection process places greatest importance on MAPE, with smaller values indicating more accurate models. Smaller values for the other measures also indicate greater accuracy. Along with these measure is a table of the final point estimates for mean monthly sales forecasted over a horizon of 10 months spanning September 2015 to June 2016. Notice that each estimate is supplemented with 80% & 90% confidence intervals, thus allowing for the values and bands in the first figure.
Now, let's compare these same accuracy measures across all possible models. It is quite apparent from the figure below that the HW Multip ES method has the lowest MAPE, thus it is ranked as the top model in the Validation Sample Accuracy Statistics table. Also notice that this method has the lowest values for ME, RMSE, MAE, and MPE, which further exemplifies this model has the greatest accuracy.
From the summary report, we are also able to assess model assumptions by checking the diagnostic plots. In the Holdout sample scatterplot of Actual vs. Predicted below and left, we see that the predicted values roughly follow the actual values, which bodes well for the normal assumption of our selected model. Notice, also, that the model residuals in the figure to the right exhibit homogeneity & uniform variation. Additionally, the figures of ACF & PACF vs. Lag do not appear to have any large spikes deviating outside the confidence bounds. This signals to us that all of the variation in monthly closing price has been accounted for by parameters in the model.
From the plot of Standardized Residuals vs. Normal Scores below and to the right, the residuals tend to follow the linear normality curve, with slight deviation at the extremes. However, the points comply enough to satisfy the normality assumption. The deviations are not extreme enough to violate this assumption.
