Model Validation

In order to validate and verify the developed model, we use a number of common techniques, including tracing, graphical representation, model-to- model comparison and statistical significance. More details about these tech- niques will follow.

Predictive Validity

For the predictive validation of the daily behaviour data (occupancy and ac- tivities), we refer to TAPAS (Take A Previous Model and Add Something) principle [123], which is one of the startegies to validate simulation models. This incremental strategy is one of the most successful strategies for model creation, where a new model is built upon a previously validated model. In this case, the predictive validity of the previous model (the PM in our case) is passed to the new one (the developed ABM). In order to verify that the implemented ABM actually generates the same data as the previous PM, we use the Model-to-Model comparison technique [124]. In this technique the outputs of each of the models are compared and the difference is calculated and/or graphically represented. Figures3.4and3.5show the plots for occu- pancy data and activities data (watching television and using the computer) generated by the PM and the implemented ABM. The shown data represent the average of 100 simulations of the scenario "one adult aged 25-39 with a

3.4. Experiments and Results 59 full-time job" given that the two models use uniform random number gen- erators with different seeds. The figures clearly show that the implemented ABM is able to generate identical data to the one generated by the existing PM [37]. To statistically prove that the data sets generated by the two mod- els come from the same distribution, we perform Kolmogorov-Smirnov test [125]. The results of the test are shown in table 3.5, which shows that the p-value is close to 1. This indicates that the models produce the same dis- tribution of data, thus the predictive validity of the occupancy and activities data is passed from the existing PM to our developed ABM.

Away

PM SleepPM ActivePM AwayABM SleepABM ActiveABM

4AM 12PM 20PM 4AM 0 0.2 0.4 0.6 0.8 1 (A) Weekdays Occu- pancy 4AM 12PM 20PM 4AM 0 0.2 0.4 0.6 0.8 1 (B) Saturdays Occu- pancy 4AM 12PM 20PM 4AM 0 0.2 0.4 0.6 0.8 1 (C) Sundays Occu- pancy

FIGURE 3.4: Average Occupancy Data Comparison between the developed ABM and the existing PM

As the PER attribute is added in the developed ABM to simulate energy waste, the energy consumption data generated by the existing PM was dif- ferent from the one generated in the ABM. Therefore, testing the predicitive validity of energy consumption data requires knowing the distribution of the

4AM 12PM 20PM 4AM 0 0.05 0.1 0.15 0.2 0.25 PM ABM (A) Weekdays Watching Television Activity 4AM 12PM 20PM 4AM 0 0.05 0.1 0.15 0.2 (B) Saturdays Watch- ing Television Activ-

ity 4AM 12PM 20PM 4AM 0 0.05 0.1 0.15 0.2 0.25 (C) Sundays Watch-

ing Television Activ- ity 4AM 12PM 20PM 4AM 0 0.02 0.04 0.06 0.08 (D) Weekdays Using

the Computer Activ- ity 4AM 12PM 20PM 4AM 0 0.01 0.02 0.03 0.04 0.05 (E) Saturdays Using the Computer Activ-

ity 4AM 12PM 20PM 4AM 0 0.02 0.04 0.06 (F) Sundays Using

the Computer Activ- ity

FIGURE3.5: Average Activities Data Comparison between the developed ABM and the existing PM

PER attribute in the Belgian community where the time-use survey was col- lected, and comparing the simulated energy consumption data with the real

3.4. Experiments and Results 61

TABLE 3.5: Daily Behaviour Predictive Validity: Kolmogorov- Smirnov Test Results

Tested Dataset p-value

Occupancy 0.999

Watching television activity 0.988 Using the computer activity 0.975

ones. Obtainig this distribution was not possible since the survey took place in 2005. Therefore, predictive validation of the energy consumption data was not possible. Taking this limitation into account, we follow the approach of comparative analysis experiments by observing the changes in results when the inputs of the model change. This approach is common among ABMs that simulate energy cosumption, as the collection of data is not always possible [59]. Besides, this is also compatible with the mediator role of the model that we described in Chapter 1. Mediator models are developed when a high level of understanding of the system is not possible, where the model is used to gain insights that are then tested in reality.

Internal Validity

To prove the internal validity of the model, we compare the data generated by the model when using different seeds. If the replicated scenarios with different seeds generate the same distribution of data, then the model is in- ternally valid [124]. For this purpose, we run 5 pairs of 5 different house- hold types each with different random seed and calculate the p-value using Kolmogorov-Smirnov test [125]. The results of the test are reported in Ta- ble 3.6. The table shows that the p-value for all the simulated scenarios is close/equal to 1, which indicates that each tested pair of datasets come from the same distribution. This proves the internal validity of the model.

Structural Validity

The structural validity of the occupancy and activities modelling is done us- ing the tracing technique [59]. The agent behaviour is followed per time step to ensure that it is behaving as designed. We start by observing one occu- pant agent in a household, and proceed to observe the collective behaviour of several family members who may be sharing activities and appliances. A rule-based method has been followed to observe whether the agents are log- ically behaving based on values of parameters in the model. An abstraction

TABLE 3.6: Internal Validity: Kolmogorov-Smirnov Test Re- sults

Household Type p-value

One adult 25-39 years old with full-time job

0.968 One adult 40-54 years old with full-time

job, and one child 12-17 years old

1 Two adults 40-54 years old one with full- time job and the other is unemployed, and one child 12-17 years old

1

Two adults 55–64 years old both with full- time job

1 One elderly 65-75 years old who is retired 0.999 of these rules is presented in Relation (3.6)

c1∧c2∧ · · · ∧cn → B , (3.6)

where c1, c2, . . . cn are the rule conditions, and B is the expected behaviour if the conditions are true. For example, when the occupant finishes the watch- ing television activity, it is not supposed to turn OFF the television if another occupant is still watching television. All the rules that affect the agent be- haviour and reaction have been tested and approved, thus validity by tracing is confirmed.

To prove the structural validity of the electricity model, we run experi- ments of a single occupant household and observe the results graphically. This is done by studying how the occupant consumer type affects the energy consumption of the house. Households with one occupant were simulated varying ages and employment of occupants. Figure 3.6 represents the re- sulting weekday average energy consumption of lights, TV and PC for an occupant who is 25-39 years old in a full-time job. Each of the sub-figures in Figure3.6 includes the results of 5 scenarios: the basic model that is the ideal scenario with 100% PER (referring to the existing PM [37]), and four scenarios, each having a different consumer type.

In the basic model (i.e. ideal scenario), it is observed that every time the occupant is sleeping or away, the energy consumption is very low or almost zero. For the other four scenarios, it is observed that the implemented model produces similar trend of daily energy consumption. The only difference is due to the addition of the PER attribute, which has caused the line graph to level up in a proportional way based on the percentages in Table 3.4.

3.4. Experiments and Results 63

Basic Model Follower Greens Concerned Green Regular Waster Disengaged Waster

4AM 8AM 12PM 4PM 8PM 12AM 4AM Time of day 0 50 100 150 200 250 300 350 400 450 500 Light Consumption (W) (A) Lights Consumption

4AM 8AM 12PM 4PM 8PM 12AM 4AM Time of day 0 10 20 30 40 50 60 70 80 TV Consumption (W) (B) TV Consumption

4AM 8AM 12PM 4PM 8PM 12AM 4AM Time of day 0 2 4 6 8 10 12 14 16 18 PC Consumption (W) (C) PC Consumption

FIGURE3.6: Appliances energy consumption of one occupant (25-39 years old / full-time job)

The energy consumption of Follower Green and Concerned Green occupants are almost similar because their mean PER is very close (74% and 72% re- spectively). While the other two waster occupants are much higher with the Regular Waster being more efficient than the Disengaged Waster who turns off appliances and lights only 25% of the time. Same observations were noticed for Saturdays and Sundays and for all age groups and employment types. These results prove the structural validity of the implemented model, which produces energy consumption trends similar to the basic model, and reflects the various consumer types levels of occupants.