<< Click to Display Table of Contents >> Payoff Reporting 

The payoff report includes a variety of metrics for evaluating the outcome of an optimization, including the component contributions of each payoff element to the total payoff, and the sources of exogenous data (if any).
Metadata & Contributions
•Type  payoff element type, e.g., *P or *CR
•Component  the element specification as model_vardata_var/scale_or_weight
•Contribution  the element's contribution to the total payoff
•Percent  the element's percentage contribution to the total payoff; may be NA if the total is 0 or contributions have mixed sign
•Values  the element's contribution due to model values (for policy elements) or modeldata comparisons (calibration elements)
•Params  the element's contribution from scale parameter terms in the likelihood (some calibration distributions only)
•Used  number of points used
•Skipped  number of points skipped, either due to exceeding the model's time range, or invalid comparisons like LN(0)
•Source  data variable source identifier
Goodness of Fit
•RSquare  coefficient of determination, R2 = SSE/SST where SSE = sum of squared errors and SST = total sum of squares. Note that, while R2 is typically bounded between 0 and 1 in typical regression, bias and other factors make it possible for SSE > SST in a nonlinear model, so R2 may be negative.
•DW  Durbin Watson test statistic for detection of autocorrelation of residuals
•Autocor1  residual autocorrelations at 1 to 4 steps
•Autocor2
•Autocor3
•Autocor4
•(TimeGap)  time gap for residual autocorrelation steps
Theil Statistics
The Theil statistics decompose modeldata variance into three terms, from difference in means, difference in variance, and difference in covariance. See:
Sterman, J.D., (1984) Appropriate Summary Statistics for Evaluating the Historical Fit of System Dynamics Models. Dynamica, 10 (Winter), 5166. http://www.systemdynamics.org/dynamica/articles/102/4.pdf
Vensim reports:
•RMSE  the total root mean squared error between model and data
•Um  the fraction of MSE due to difference in means
•Uv  the fraction of MSE due to difference in variance
•Uc  the fraction of MSE due to pointtopoint covariance
Um + Uv + Uc = 1
All values will be reported as 0 if there are no data points.
If there are fewer than 2 data points, Uv and Uc will be 0, with Um = 1 and MSE reflecting the squared error of the single data point.
Absolute Errors
•MAE  mean absolute error, i.e. the sum over the valid model & data points of ABS(modeldata) divided by the count of comparisons.
•MAPE  mean absolute percentage error, the sum of ABS((datamodel)/data) multiplied by 100 to yield a percentage. If any data value is 0 this statistic is undefined and NA is reported.
•MAEoM  mean absolute error over mean, the MAE divided by the mean of the data, multiplied by 100 to yield a percentage. If the data mean is 0, NA is reported.
If there are no data points, 0s are reported.