From [170, 171], validation is defined as:
”determining a simulation model to be an acceptable representation of the real system - given the purpose of the simulation model”
and by Schlesinger et al. [172] (as quoted in [18]) in the context of computer modelling as:
”substantiation that a computerized model within its domain of applicability possesses a satisfactory range of accuracy consistent with the intended application of the model”
.
For the model in this thesis, validation depends on capturing the real system’s values associated with model input parameters or the model results (or both). Two complications, however, must be addressed when determining the method of validation. These are:
(i) The probabilistic nature of the model and
(ii) The lack of empirical data that has encourage the modelling approach taken.
Addressing the second issue first, Kleijnen [170] discusses the use of statistical techniques for validating simulation models in relation to real system data availability. Splitting data availability into three possible situations of:
1. No data,
2. Only real data corresponding to the model output and
3. Both input and output data allowing for ’trace-driven’/correlated inspection.
The model developed in this thesis was motivated by the limited stock level data on ND building energy performance1 and limited data on the distributions in building form and component structure that influence thermal performance. As a result theoretical input values to the model were developed where adequate empirical data could not be found or realistically collected in the given project time scales.
In the light of no adequate real-life data for validation Kleijnen states [170]:
”there is still expert knowledge”
The statistical method for validating by expert (qualitative judgement) is considered later on.
A summary of validation techniques for simulation models is given in table 8.1 - used to identify appropriate techniques of validation for the stochastic modelling carried out in this thesis. From this, five validation methods were considered not only to be applicable to the modelling carried out, but also capable within the current state of knowledge and data availability of non-domestic building energy performance.
Degenerate tests and sensitivity analysis are considered of a similar nature and as such this form of model inspection/validation was considered in chapter 7. The results of sensitivity analysis showed varying sensitivity of the base case models according to the periods of construction. The varying sensitivities between building periods were explained by the varying impacts of thermal mass, u-values and initial base-case values.
Face validity and rationalism can only be used to give an indication of the model’s validity. Both require expert knowledge to assess the results and underlying model assumptions to provide a qualitative confidence in the results. Identifying experts is a key factor to the credibility of this validation, especially within a field where the real system’s data is not fully (or even adequately) appreciated.
Internal validity is associated with capturing the uncertainty in a system and seeing its influence on the model results. This, however, is the premise of the work undertaken in this thesis. This validation
Table 8.1: Summary of Validation techniques for simulation based modelling, from [18].
Validation Techniques Description Applicable to Model Currently Possible
Animation Display dynamic model behaviour graphically through time.
Comparison of Models Compare results against existing (valid) models
Degenerate Tests Model!s behaviour is tested by behaviour inspection when varying a given input parameter with known influence. This is in part covered by sensitivity analysis (see chapter 7).
Event Validity Compare the events of the model against the events in the real system. This could be used to compare the distributions in thermal performance of the model, against distributions of empirically based thermal models.
Extreme Condition Model output at extreme values is feasible. This is more appropriate to validate the dynamic simulation model used by the probability modelling method. These models undergo a strict set of validation tests and is not relevant here.
Face Validity Ask "experts! whether the behaviour is a reasonable reflection of their understanding of the real system. This is a qualitative measure only.
Historical Data Use data already collected on the real system as input to the model as well as relevant output. Kleijenen terms this as trace-driven where regression analysis can be used to determine whether theoretical input data can be statistically compared to real input data. This is only possible if adequate input and output data of the real system are known.
Historical Methods Rationalism - assumes that model assumptions are true and that logic can be used from the assumptions to derive the valid model. If the assumptions in the model are considered true then in this instance the model is a valid representation. This, however, requires wide spread agreement from expert opinion that the assumptions made are valid.
Empiricism - every assumption and outcome to be validated by empirical means. Again not enough data exists for this to be carried out.
Positive Economics - the model is only required to predict the future/
real system, regardless of the model assumptions and structure. This is not relevant to the probability modelling.
Internal Validity Assess the stochastic nature of the model to capture the variability in light of system uncertainty. This is the premise of the model - to capture the variability in the non-domestic stock. Inspection of results can be used to offer some validation of the modelling method.
Multistage Combining the three historical methods at different stages within the model
Operational Graphics Show model behaviour as it is running. this is of no consequence to validating the probabilistic results.
Sensitivity Analysis Assess the influence of input parameters on model behaviour. As sensitivity is dependent on the base case model to which it is applied the behaviour will vary for different building periods. this was covered in chapter 7.
Predictive Compare model predictions to real system data. This requires a large scale field study to be carried out that standardises the influence of varying occupant and HVAC system behaviour on the thermal performance in order to measure the base case thermal behaviour.
Traces Trace individual model components to determine if the logic is correct.
This is not appropriate here.
Turing Tests Ask knowledgeable "experts! to discriminate between the real system and model results. This requires adequate real system data.
approach can be used to determine the appropriateness of building period as a category for investigating building thermal performance. This was used by results inspection in relation to a building case study.
The methods of validation summarised in table 8.1 are clearly presented for deterministic model valida-tion in [18]. Validavalida-tion for probability models is more complex, [170, 173], as the model is accounting for the uncertainty of the real system. According to Lind [174], probability statements cannot themselves be verified as the real system uncertainties are what force the probability statement. It is the model or method of producing the probability statement that can be validated by comparing real or simulated data.
Real system data is considered inadequate for validation and therefore any approach must use simulated data (either from other existing models or from expert judgement).
8.1.1 Validation for Probability Models by Expert Judgement
Expert judgement is required in the light of inadequate data on the real system being simulated. Kleijnen states in [170],
”If no data on the real system are available, then strong validation claims are impossible.”
Kleijnen’s statement can be expanded for uncertainty modelling to say: without a statistically significant amount of data strong validation claims are impossible. This is acknowledged as a limitation in validating the modelling method within this thesis.
Kleijnen describes a sensitivity analysis approach to be taken when no real system data is available -using expert judgement to determine if the sign of the sensitivity in relation to parameters agrees with qualitative knowledge. This, however, is not fully applicable to uncertainty modelling.
Lind, [174], discusses the use of likelihood (L) and Shannon information (I) for use in relative validation.
The likelihood of the model is the probability on the observations when considering the probability of the model as true. Comparing the likelihood of two models gives a relative measure known as the likelihood ratio. This requires real system observation to which a true/false statement can be applied to each outcome of the model.
The Shannon Information is the negative natural log of the likelihood (see equations 8.1 and 8.2),
L= p1p2p3⋯pn (8.1)
I= − log L = − log p1− log p2− log p3...− log pn (8.2) where piis the probability of an independent prediction of the observed value being true, for n predictions.
The third validation measure presented by Lind is that of relative entropy (or Kullback-Leibler diver-gence) [175]. This measures the difference of two distributions, Pr(the real system distribution) and Pm
(the modelled distribution). The measure is given as:
S(Pr∣∣Pm) = ∑Pr(i) log Pr(i)
(i) (8.3)
for discrete random variables, and as:
S(Pr∣∣Pm) = ∫i∞pr(x) log pr(x)
pm(x)dx (8.4)
for continuous distributions.
8.1.2 Validation Design
The Kullback-Leibler measure can be used to show the variation in the model’s parameter distributions (derived from historical building regulations and other theoretical measures taken) against the current best understanding of the non-domestic (office) stock’s distribution of building characteristics (result-ing from large scale empirical and modell(result-ing studies). The real system be(result-ing represented by expert knowledge.
The expert knowledge can also be used to identify typical component constructions according to building period. Those constructions presented by the experts can be compared to the construction databases developed for this modelling study. Depending on the similarity of construction a level of credibility in the construction databases can also be given.
Another method identified for validating was to downscale the probability model by physical characteris-tics. The downscaled model can be compared to detailed models of real buildings that are within a given construction period and of similar form to the downscaled probability model (with the same occupancy and internal load patterns as applied in the probability model). If the thermal loads calculated in the real models are comparable with the downscaled probability model then (if only partially) some confidence can be applied to the probability models ability at representing true thermal loads.
Figure 8.1 represents the two methods used to provide credibility to the probability model used in assessing a stock level influence of building construction period on thermal loading. The figure highlights the limitation in that no direct path for comparing the real system with the modelled system is available.