Payoff Reporting

Top  Previous  Next


The payoff report includes a variety of metrics for evaluating the outcome of an optimization, including the component contributions of each payoff element to the total payoff, and the sources of exogenous data (if any).


Metadata & Contributions


Type - payoff element type, e.g., *P or *CR
Component - the element specification as model_var|data_var/scale_or_weight
Contribution - the element's contribution to the total payoff
Percent - the element's percentage contribution to the total payoff; may be NA if the total is 0 or contributions have mixed sign
Values - the element's contribution due to model values (for policy elements) or model-data comparisons (calibration elements)
Params - the element's contribution from scale parameter terms in the likelihood (some calibration distributions only)
Used - number of points used
Skipped - number of points skipped, either due to exceeding the model's time range, or invalid comparisons like LN(0)
Source - data variable source identifier


Goodness of Fit


RSquare - coefficient of determination, R2 = SSE/SST where SSE = sum of squared errors and SST = total sum of squares. Note that, while R2 is typically bounded between 0 and 1 in typical regression, bias and other factors make it possible for SSE > SST in a nonlinear model, so R2 may be negative.
DW - Durbin Watson test statistic for detection of autocorrelation of residuals
Autocor1 - residual autocorrelations at 1 to 4 steps
(TimeGap) - time gap for residual autocorrelation steps


Theil Statistics


The Theil statistics decompose model-data variance into three terms, from difference in means, difference in variance, and difference in covariance. See:


Sterman, J.D., (1984) Appropriate Summary Statistics for Evaluating the Historical Fit of System Dynamics Models. Dynamica, 10 (Winter), 51-66.


Vensim reports:


RMSE - the total root mean squared error between model and data
Um - the fraction of MSE due to difference in means
Uv - the fraction of MSE due to difference in variance
Uc - the fraction of MSE due to point-to-point covariance


Um + Uv + Uc = 1


All values will be reported as 0 if there are no data points.


If there are fewer than 2 data points, Uv and Uc will be 0, with Um = 1 and MSE reflecting the squared error of the single data point.


Absolute Errors


MAE - mean absolute error, i.e. the sum over the valid model & data points of ABS(model-data) divided by the count of comparisons.
MAPE - mean absolute percentage error, the sum of ABS((data-model)/data) multiplied by 100 to yield a percentage. If any data value is 0 this statistic is undefined and NA is reported.
MAEoM - mean absolute error over mean, the MAE divided by the mean of the data, multiplied by 100 to yield a percentage. If the data mean is 0, NA is reported.


If there are no data points, 0s are reported.