NatHERS Benchmark Study

(1)

NatHERS Benchmark Study

February 2014

Prepared for the Department of Industry by: Floyd Energy, Wayne Floyd

With assistance from Tony Isaacs and Rodger Hills

(2)

ii | P a g e

And last but not least, the study team would like to thank all participating assessors who generously took time away from their business or leisure to participate in this study. This study represents an important step forward for the NatHERS assessor industry and will help all assessors to produce consistent and reliable ratings for their clients. As the first study of its kind this study was on its own learning curve, and as a result there were inevitable glitches in its implementation. The study team appreciates your dedication and patience for sticking with us to the end.

(6)

6 | P a g e

1 Executive Summary

1.1 Study design

This study creates a national benchmark by measuring the accuracy of NatHERS assessments.

Assessors were randomly allocated one of four houses to rate and their assessments were compared with a solution set prepared by a committee of expert assessors. Approximately 100 questions about NatHERS data entry for each house were developed to specifically test how assessors applied

Technical Notes 1 and 2, whilst also testing accuracy of general data entry techniques.

A study website was developed to provide a portal for assessors to access the study, enter responses to questions, ask questions and gather feedback. A webinar – available through YouTube – and the study website provided instructions on participating in the study and using the data entry portal. Assessors were initially contacted through Assessor Accrediting Organisations (AAOs) and software providers who endorsed the study and encouraged assessors to participate. AAOs provided

Continuing Professional Development (CPD) points for participation as an incentive.

1.2 Key findings

1.2.1 Statistical significance of participants

Statistical significance depends on the sample size relative to the total population. The measure of statistical significance used for academic publications is 95% certainty and a 5% margin of error. This study obtained a sample of 344 assessors from an estimated sample of 1816 assessors which has an error margin of 4.8% at 95% certainty, so the sample was statistically representative of the assessor population as a whole. As the sample was self-selecting and not random, the nature of the bias, that is, whether one would expect a self-selected sample to be more or less accurate than the general population is not clear.

Sub samples by jurisdiction, AAO and software tool were also examined to determine statistical significance. Only the sub sample of ABSA assessors fell within the 95% confidence and 5% error margin limit. Sub samples of BDAV assessors, assessors from WA and Victoria and each of the software tools fell within an error margin of 10% at 95% confidence. Assessors who were

unaccredited represented only 10% of participants. This does not mean that comparisons between sub samples had no significance, but that the differences between these subsamples would need to be greater to be able to conclude that the difference was statistically significant.

1.2.2 Star rating accuracy

Star rating accuracy is the most important element of a NatHERS assessment as this is directly related to compliance with building regulations and the National Construction Code (NCC).

The study found that around 21% of assessments obtained the correct rating around 37% of ratings were within 0.25 stars, 58% within 0.5 stars and 77% were within 1 star (Table 1). Alternatively, 64% of assessors had an error greater than 0.25 of a star.

(7)

7 | P a g e The high error rate in star ratings was consistent with assessors correctly answering 65% of the questions about data inputs.

Table 1 Proportion of ratings within various star rating error bands

Error in star rating Cumulative

Within 0.25 stars 37% Within 0.50 stars 58% Within 0.75 stars 70% Within 1.00 stars 77% Within 1.50 stars 86% Within 2.00 stars 91%

More than 2.00 stars 100%

1.2.3 NatHERS Technical Notes

NatHERS Technical Notes 1 and 2, cover rules for data entry including zone types and conditioning, site exposure and calculation of the area of uninsulated ceiling created by ceiling penetrations. The errors found in this study suggest that these documents are not well understood, or widely used by the assessor industry. For example, depending on the complexity of the house plan only between 3% and 29% of assessors correctly estimated the area of ceiling penetrations.

1.2.4 Application of Technical Note 1 Zoning rules

Arguably, the most important element of accurate modelling using NatHERS software is to correctly zone the house plan. A large proportion of assessments were not zoned correctly according to the zoning rules in Technical Note 1. Rooms were incorrectly combined or split, the allocation of other day or night time-zoned rooms were not well understood and heating and cooling was incorrectly turned off in these zones. Inspection of individual files showed that around 60% of assessors had made errors in zoning house 1 and 4 and 85% incorrectly zoned house 2.

1.2.5 House plan complexity

Assessor errors increased significantly with the increasing complexity of design and documentation. 57% of assessors were within 0.25 stars of the correct rating for house 1, but only 41% for house 2 and 26% for house 3. Houses 2 and 3 were two storey houses (which are inherently more complex to rate than single storey houses) and house 3 had more complex documentation than the simpler house 1.

House 4 was an apartment on the 15th floor and only 19% of the assessors got within 0.25 stars of the correct rating. In addition to zoning errors a significant contributing factor to the errors was found to be selecting incorrect wind exposure and height above ground. 41% of assessors selected the wrong exposure for house 4.

1.2.6 Net Conditioned Floor Area (NCFA)

The rating is calculated on the basis of energy loads per square metre of NCFA, so getting this right is vital to an accurate rating. One in five ratings reported incorrect NCFA (> 5% difference), which resulted from factors such as failing to turn on heating and cooling in other day and other night classified zones.

(8)

8 | P a g e 1.2.7 Overshadowing

Overshadowing refers to the shading of the rated building by surrounding objects, typically adjacent houses and fences. Incorrectly modelling overshadowing on a heavily overshadowed site can result in rating errors of over 1 star.

Houses 1 to 3 each had significant overshadowing from adjacent buildings and fences. Most

assessors did not accurately model these obstructions. Errors ranged from miscalculating the height and depth of the overshadowing obstruction to not counting the overshadowing impact of roof ridges and fences. Over 75% of assessors that rated house 1 did not accurately model obstructions. 1.2.8 Systematic process

A wide variety of other errors were also observed that indicated the lack of a systematic approach to the rating process by assessors. For example, 25% of assessments reported incorrect window areas, 85% reported incorrect area of ceiling penetrations, 35% had incorrect roof solar absorptance, 25% had the incorrect R-value for waffle pod slabs and the area of different wall constructions and floor coverings within each house showed error rates ranging from 10 to 70%.

1.2.9 Differentiation by AAO or tool

The responses of assessors were also checked to see whether the use of a particular tool or

membership of AAO was correlated with different accuracy levels. While differences were found, the small sample sizes meant that the differences found could be explained by the error margin of the samples.

1.2.10 Information about the assessor industry

Assessors who registered for the study were asked a number of questions which provided valuable information on the assessor industry. Around a quarter of assessors work solely as an assessor. Two thirds of assessors are self-employed and around half of these assessors were building designers or architects. The average assessors had been in the job for 5.5 years and rated 90 houses per year. Almost all assessors had some building site experience with around 63% having direct on site experience.

1.3 Recommendations

It is important that NatHERS assessments are consistently accurate to ensure compliance with building regulations, effectiveness in saving energy, for the reputation of the industry and to ensure consumers get the service they pay for. The requirement that all assessors complete the Certificate IV in NatHERS Assessment by 1 July 2015 is an important step towards improving the skills of assessors.

This study highlighted areas where enhancements could be made to NatHERS to increase the

accuracy and consistency of NatHERS assessments. These can be grouped into four broad categories: 1.3.1 Mandatory accreditation of assessors

This study found that there was a high level of error in NatHERS ratings irrespective whether assessors were accredited or not. The quality assurance (QA) processes of accreditation is a mechanism that could contribute to more accurate assessments, however, there is no process

(9)

9 | P a g e available to improve the accuracy of unaccredited assessors. Making accreditation mandatory for all assessors is essential to ensure to improve the accuracy of assessments.

The error rate showed that there are benefits for further CPD training for assessors in Technical Notes 1 and 2. This additional training should incorporate evaluation to ensure that the content is understood by assessors. The Technical Notes could be improved by providing detailed examples to better explain the principles.

1.3.2 NatHERS tool improvements

NatHERS Software Tools could be further improved to minimise error rates. This includes improvements such as automatic calculation of ceiling penetration area, better guidance on the allocation of zone occupancy types and the application of heating and cooling, automatic calculation of waffle pod slab R values and simplifying data entry for overshadowing. The calculation of NCFA is not consistent across tools and should be harmonised as the rating is determined on the basis of this area.

1.3.3 New resources for assessors

A number of new resources for assessors would help to increase accuracy:

 The NatHERS assessor industry has no standard data entry and error checking procedures. Development of such procedures would facilitate a more systematic approach to rating. This would help to eliminate many of the errors found in this study;

 A comprehensive technical manual for assessors which explains how to model all the design and site features which assessors may encounter in the field; and

 Work with product suppliers to develop trade literature which better meets the needs of assessors (69% reported that they had difficulty finding the information they needed in trade literature).

1.3.4 Improving future NatHERS assessor benchmark studies

The proposed Universal Certificate extracts data from the rating file and displays this on the Certificate. A similar tool which extracts a larger set of data than the proposed Universal Certificate from NatHERS rating files automatically would have significantly reduced the extent of analysis required, facilitated analysis in greater detail and made participation in this study faster and easier for assessors. It would also avoid issues of assessor misinterpretation of questions. The proposed universal certificate generator could be modified to do this. It would also significantly reduce the resources needed for QA checking by AAOs and assist in the marking of exams and tests for trainers.

(10)

10 | P a g e

2 Introduction

In May 2013, The Department of Industry commissioned Floyd Energy to undertake the first national benchmark study to measure the accuracy and consistency of assessments undertaken by

Nationwide House Energy Rating Scheme (NatHERS) assessors. This study followed a study in 2012 where the Association of Building Sustainability Assessors (ABSA) was contracted by the Western Australian Office of Energy to conduct a workshop to trial state-based interim software modelling principles and defaults. The results of this workshop identified a number of areas where the accuracy and consistency of NatHERS assessments could be significantly improved, such as how to consistently treat zoning and floor covering, and how to make consistent judgements regarding insulation and external screening.

As the results were unlikely to be unique to Western Australia, the Department of Industry has sought to establish a national baseline of current assessor practices. The intent of the study was to more fully understand problem areas, monitor and evaluate improvements to the Scheme, and to work with all governments to continually improve the consistency and accuracy of NatHERS assessments nationally.

3 Background

NatHERS provides a national framework that allows various computer software tools to rate the potential energy efficiency of Australian houses at the design stage. NatHERS software tools fulfil a regulatory function for assessing compliance of a building design with the thermal performance requirements in the National Construction Code (NCC). The correct application of NatHERS software tools is critical to the success of the Scheme. Although each state and territory government uses the NatHERS accredited software to provide assessments, minimum standards in all other aspects of the Scheme including application of the use of tools can vary.

Main components of the Scheme include:

 The use of accredited software tools that provide consistent and accurate modelling of building shell performance (assessments) across a diverse range of climates to determine heating and cooling demands;

 Minimum standards for assessors:

o Accreditation of Assessor Accrediting Organisations (AAOs) incorporating CPD and quality assurance of assessors, Code of Conduct and mandatory Professional Indemnity Insurance;

o Certificate IV in NatHERS Assessment qualification;

o Consistent use of NatHERS tools through nationally-applied technical notes; and o Working toward a minimum consistent standard across all states and territories, and  A process of continuous improvement for all aspects of the Scheme.

3.1 The Industry

The NatHERS Assessor industry is still in the early years of its development. National Regulations for minimum NatHERS performance were introduced in 2003, and a number of states delayed the introduction of these regulations. Prior to 2003 a number of jurisdictions had introduced minimum requirements: NSW’s Energy Smart Homes program required a minimum 3.5 stars, the ACT required

(11)

11 | P a g e a minimum 4 stars and extended this as a disclosure rating for houses for sale and rent, while in Victoria houses requiring a planning permit had to be at least 4 stars.

Training requirements to become an assessor have been minimal: a 4 day course covering thermal performance theory, software use and professional practice. A new Certificate IV course has been developed for NatHERS assessors and all accredited assessors will be required to meet this new standard by 1 July 2015.

Implementation of Quality Assurance (QA) programs has been inconsistent across states and territories:

 In NSW, Assessors were required to be accredited with ABSA for thermal performance assessments conducted under BASIX. In 2006 ABSA became the first recognised AAO for NatHERS. Assessors were required to pass entry exams, participate in CPD and some assessors work was subject to QA checks each year.

 In Victoria there was a minimal accreditation requirement which involved passing an exam run by training authorities with occasional QA random checks. When FirstRate5 was released, SV ceased accrediting assessors and this task was given to the Building Designers Association of Victoria (BDAV). BDAV initially accredited assessors under the requirements of SV but became recognised as a NatHERS AAO in 2011.

 The ACT has required licensing for assessors from March 2011.

 Other states have required no accreditation, but some have recommended that assessors accredited with an AAO be used in preference to unaccredited assessors.

The software and regulations around NatHERS ratings have changed several times:

 2nd generation software was introduced between 2006 and 2008. All assessors were required to retrain to be able to use the new software.

 AccuRate and BERS Pro released new versions with non-regulatory assessment of appliance efficiencies and water use.

 Accreditation for BERS Pro and FirstRate5 was provisional and when the fully accredited versions were released the ratings for houses changed slightly.

 National regulation stringency has increased from 4 to 5 stars and then to 6 stars in 2010.  ABSA’s rating procedures have been updated three times culminating in the release of the

NatHERS Technical Notes 1 and 2 in 2013.

 The basis of the Window Energy Rating Scheme (WERS) changed without corresponding amendments to the window libraries in NatHERS tools making the job of identifying the required minimum properties of glazing more difficult.

The NatHERS Assessor Industry is a young and rapidly changing industry. This environment and the inconsistent application of QA across Australia does not lend itself to establishing high standards across the country without a consolidated and consistent approach.

(12)

12 | P a g e

4 Objectives

4.1 Purpose

The purpose of this study was to:

 Collect relevant data about NatHERS assessments to be able to determine levels of consistency and accuracy of assessments nationwide;

 Design a methodology to assess the accuracy of NatHERS assessments;

 Identify ways to improve NatHERS tools and the skills, accuracy and consistency of assessors; and

 Provide a benchmark to measure current and future improvements to the NatHERS.

4.2 Deliverables

Deliverables for this study included:

 Designing a sound methodology to gather the required NatHERS assessor/assessment information;

 Ensuring participation of assessors was statistically significant;  Guaranteeing the confidentiality of participants;

 Ensuring the study was conducted with the highest integrity;  Identifying and manage risks;

 Analysing data to highlight trends and summarise outcomes;

 Advising of appropriate actions to improve the accuracy of assessments; and  Providing a final report.

4.3 Assumptions

4.3.1 NatHERS Technical Notes 1 and 2

The NatHERS administrator released two Technical Notes in early 2013 which defined the way in which all assessors using NatHERS tools are required to rate dwellings. Technical Note 1 gives a general description of data entry techniques, and Technical Note 2 explains how to allow for loss of ceiling insulation due to the clearances required around some ceiling penetrations. The Technical Notes specifically address the poor rating practices of assessors which had been identified by previous studies (such as the Floyd Energy project in WA) and AAO QA findings.

4.3.2 Bias

Participation in the study was voluntary. Samples which are self-selected are not random and this will introduce some bias into the study. It is difficult to determine whether this bias would lead to participants being more or less accurate than the general population of assessors. If only those assessors who were confident they did accurate ratings participated then the bias might be expected to cause greater accuracy. On the other hand it may be that, because accuracy in this study had no impact on accreditation, those who struggle with the rating process participated because they want to learn how to improve their ratings so the accuracy of the sample could be worse than the

population. Due to the high number of CPD points allocated by AAOs for participation in the study, assessors’ main motivation to participate may simply have been to earn CPD points, in which case

(13)

13 | P a g e the impact on the accuracy of ratings is not clear. The self-selection of assessors for participation in this study is an inherent limitation to the methodology of this study which is acknowledged, however, the nature of the bias this introduces is unclear.

4.3.3 Accredited assessors vs. non-accredited assessor participation

Although both non-accredited and accredited assessors were invited to participate in this study, it was assumed that most participants would likely be accredited assessors as they received a significant allocation of CPD points for their efforts at no cost.

5 Methodology

The methodology for this study involved three key stages (1) System design, (2) Participation and (3) Data collation and analysis.

5.1 Stage One: System design

Assessors were randomly assigned one of four house plans that were uniquely designed and peer-reviewed. Assessors then answered around 100 questions (see section 8.6) about their specific assessment through a website designed specifically for this study. Assessors were also asked to upload their rating files to the study website to facilitate deeper analysis if the questions asked did not provide sufficient information. This also helped to resolve any issues around misinterpretation of questions.

5.1.1 House plans

Houses were selected to test the application of specific clauses in the Technical Notes as well as general rating techniques. In developing the plans the common rating errors reported by AAOs were examined to make sure they were tested. The plans were typical of the different types of dwellings which assessors face in the field:

 House 1 was a single storey 196 m2 4 bedroom volume builder house provided by Dennis Family Homes;

 House 2 was a two storey 311 m2 4 bedroom volume builder house provided by Dennis Family Homes it has some more complicated features such as lower floor roof spaces adjacent to upper floor rooms and walls which change construction from Brick Veneer to cement sheet clad over their height. Two storey homes are also, by nature, more

complicated to assess with NatHERS tools because overshadowing must be adjusted to take into account the floor height of the zone and zone adjacencies between floors are more complicated;

 House 3 was a two storey 107 m2 3 bedroom town house which was part of a multi-unit town house development (NCC class 1) and was provided by volume builder AV Jennings. Its documentation is for the whole medium density development and assessors needed to locate the rated unit within these documents; and

 House 4 was a 166 m2 3/4 bedroom apartment on the top floor of a 15 storey building (NCC class 2). This design was developed for this study specifically to test the application of Technical Note zoning rules to do with the creation of zones for common corridors in multi-unit buildings.

(14)

14 | P a g e Section 8.5 shows the plans of the houses and the specific areas of assessor data input these houses were designed to test.

5.1.2 Peer review committee

A peer review committee examined the documentation for each house to ensure the house documentation included everything to undertake the rating task. Expressions of interest for participation in the committee were requested through AAOs. A number of nominations were received. In addition to examining the nominee’s experience advice was sought from AAOs and tool developers on the suitability of each nominee to undertake this task. Experts were selected for the committee from each of the software tools; 3 for AccuRate, 4 for BERS Pro and 2 for FirstRate5. 5.1.3 Development of data entry questionnaires for each house

A full QA check of each rating assessment was too time-consuming for the purposes of this study. For this reason, a series of questions were developed for each house to cover those data inputs which were considered essential to the accuracy of the rating. The questions were designed to focus on key new areas of assessor practices introduced in the Technical Notes released in 2013 and based on AAO observations of typical assessor rating errors. The full set of questions asked for each house in each software tool is shown in section 8.6. Around 100 questions were asked for each plan covering:

 Key rating parameters: star rating, energy loads and NCFA, climate zone and site exposure;  House zoning including: total number of zones and number of zones of each different type;  Areas and types of key construction elements: walls, floors, ceilings and windows;

 Insulation R values: particularly where these involved calculation by the assessor e.g.

combined foil and batt insulation in a wall with BERS Pro or calculation of ceiling penetration areas;

 Number of air leakage sites of different types; and  Details of overshadowing obstructions.

5.1.4 Study website

A website was developed for the study (www.nathersbenchmarkstudy.com.au), and was the assessors’ portal for the study. The site:

 Provided key messages about the importance of participation;

 Explained how the study would run and what would be required of assessors;  Allowed assessors to register their interest in the study;

 Gave a contact email to allow assessors to ask questions about the study;

 Provided downloads of key background materials: NatHERS Technical Notes 1 and 2 and instructional notes from the webinar, together with a link to the video of the webinar; and  Gave study participants access to the Frequently Asked Questions compiled as a result of

participant feedback during the study.

5.1.5 Totara system for recording answers to questions on each house

The Totara system is a popular learning management platform used in the corporate sector. It was used for the NatHERS Benchmark Study to provide participating assessors with a website to enter

(15)

15 | P a g e the answers to questions for each house and to upload their rating files. The questions were divided into subsections and assessors were given feedback at the completion of each subsection on

whether their answers were correct. At the conclusion of the questions a summary report showing all correct and incorrect answers was provided for each assessor. Totara was selected because it is robust, scalable and fully supported in Australia. The system allowed the study team to load content specific to the study quickly and easily. It provided sophisticated assessment and reporting to ensure tracking of participant progress, scores and final results of the study. Issues and trends that were determined from Totara included:

 Assessor demographics;  Typical assessment errors;  Typical errors by rating tool;

 Typical errors by demographic (state / accredited ABSA / accredited BDAV / unaccredited / age / gender etc.);

 Details of assessment practices;

 Overall deviation of assessors from benchmarks; and

 Proportion of questions correctly answered by each assessor. 5.1.6 Additional data gathered for this study

The study also provides a unique opportunity to gather more data about the assessor industry. In addition to asking questions about the rating of each house assessors were also asked to provide information on their background through the registration process e.g. number of ratings per year, whether self-employed, number of years since last trained etc. Section 8.2 lists the questions asked and analyses assessor responses on assessor background. Assessors who completed their ratings were asked a number of questions about assessor practices such as how long assessors spent checking their ratings, the nature of procedures they use when rating a house and the ease of finding appropriate thermal performance data in trade literature. Section 8.4 lists the questions asked and analyses the responses to these questions on assessor practices.

5.2 Stage Two: Participation

5.2.1 Access to assessors

Access to accredited assessors was facilitated through the cooperation of AAOs and software suppliers. To comply with privacy requirements, all communication with assessors regarding recruitment into the study was done through these sources. AAOs and software providers added their support to the study in these communications to encourage assessor participation. Reminder emails were sent once per week through these channels. A total of over 17,000 emails were sent to recruit assessors into the study.

5.2.2 Voluntary participation

To maximise participation a number of strategies were employed such as AAOs offering CPD points for participation, which saved assessors around $400. The benefits to assessors and to the industry of the study were highlighted during recruitment and explained on the website (see Section 8.1 for a description of the benefits to assessors in participating in this study). Assessors were also

guaranteed confidentiality. A webinar was conducted in the early stages for assessors who

(16)

16 | P a g e assessors to get feedback on the accuracy of their work for free without the potentially negative consequences of going through an AAO QA process.

AAOs and software developers played a major role by sending out reminders to assessors to (1) participate and/or (2) complete their assessment.

5.2.3 Assessor engagement

A key benefit for assessor participation was to improve the standard of their ratings. On completion of the study, assessors were given access to the report and/or report summary as well as the solution sets for all houses. This feedback encouraged participation and was included to start the process of assisting assessors to improve their standards.

5.2.4 Responding to assessor rating queries during the study

The Zendesk system was used by the study team to respond to assessor queries. Zendesk is a cloud-based customer service system. It is typically used to provide Help Desk services in the IT industry and industries which need to provide technical support for their products. It was selected because it is an efficient method to receive and respond to participant enquiries. It allowed the study to identify participant’s issues quickly, allocate the query to the study team member with the appropriate expertise to ensure rapid response. The Zendesk online portal allowed both the submission of queries and responses as well as allowing members of the study team to communicate directly via email regarding the study.

5.2.5 Sample size

Establishing the number of study participants that would constitute a statistically valid sample of assessors required selecting a confidence level and confidence interval. The following description of these terms was taken from the Survey System1 web site:

The confidence interval (also called margin of error) is the plus-or-minus figure usually reported in newspaper or television opinion poll results. For example, if you use a confidence interval of 4 and 47% percent of your sample picks an answer you can be "sure" that if you had asked the question of the entire relevant population between 43% (47-4) and 51% (47+4) would have picked that answer.

The confidence level tells you how sure you can be. It is expressed as a percentage and represents how often the true percentage of the population who would pick an answer lies within the confidence interval. The 95% confidence level means you can be 95% certain; the 99% confidence level means you can be 99% certain. Most

researchers use the 95% confidence level.

When you put the confidence level and the confidence interval together, you can say that you are 95% sure that the true percentage of the population is between 43% and 51%.

People sometimes think that the 95% level is sacred when looking at significance levels. If a test shows a .06 probability, it means that it has a 94% chance of being true. You can't be quite as sure about it as if it had a 95% chance of being be true, but the odds still are that it is true. The 95% level comes from academic publications, where a theory usually has to have at least a 95% chance of being true to be

(17)

17 | P a g e

considered worth telling people about. In the business world if something has a 90% chance of being true (probability =0.1), it can't be considered proven, but it is probably better to act as if it were true rather than false. 1

As explained above, in most academic research a 95% confidence level with a 5% confidence interval is used to define a statistically significant sample i.e. you can be 95% sure that the results represent the population within +/- 5%. To meet this level of confidence for the total assessor population would require a sample size of 317 assessors of the total, 1816 assessors.

It would also be ideal to ensure that the results of the study were statistically valid for a variety of assessor sub samples e.g. assessors in in different jurisdictions (Table 2) or individual AAOs (Table3).

Table 2 Number of participants required to achieve statistically valid samples from each state

State within 5% within 10%

ACT 49 36 NSW 136 66 NT 15 14 QLD 213 80 SA 102 57 TAS 53 38 VIC 246 84 WA 120 62 TOTAL 934 437

Table 3 Number of participants required to achieve statistically valid samples for each type of accreditation

Accreditation: ABSA BDAV None Total

within 5% 200 234 251 685

within 10% 78 83 85 246

5.3 Stage 3: Data collection and analysis

5.3.1 The Totara System

The Totara system collected assessor responses to questions in a form that could be imported into Excel for analysis. This study assessed the accuracy of assessments in terms of star rating outcomes and the extent of errors in data inputs that have led to divergence from the correct star rating. Where sub samples were statistically significant, differences between sub-samples were also examined e.g. the differences in accuracy and data input errors between:

 Each AAO and non-accredited assessors;  NatHERS tools;

 Assessors in different states; and

 Each of the four house plans used in this study.

The impact of data entry errors on star rating outcomes was also examined e.g. the average star rating for assessors with correct answers to a particular question was compared with those with

1

(18)

18 | P a g e incorrect answers. In some cases the impact of the extent of error in data inputs was correlated with average star rating outcomes as well e.g. the average ratings of those assessments with too many thermal zones was compared to the average rating for those with the correct number and those with too few.

5.3.2 Analysis

The analysis for this study focussed particularly on new areas of the Technical Notes:  Zoning: the process of dividing the rooms of a house into unique thermal zones and

assigning occupancy and conditioning, particularly the application of other day and night time occupancies and combining of rooms into single zones;

 Modelling of surrounding buildings and fences which cast shade on the dwelling;  Setting of site exposure to wind;

 Modelling of building elements shared with common areas; and

 Calculation of the area of ceiling left uninsulated due to ceiling penetrations.

Other critical areas of rating accuracy were also examined such as ensuring that: the areas of different construction elements were correct, building element construction details were correctly identified and researching product thermal performance properties.

5.3.3 Maximising assessor understanding of rating questions

For the study results to be valid, it was important that the questions asked were unambiguous. To try and ensure the questions were as easy to understand the study provided:

 Instructions for the study to ensure study participants understood how to find the data they needed to enter; and

 A webinar was developed to explain to how to prepare the rating and extract data inputs needed for the study.

Despite all efforts to ensure questions and documentation were easily understood, it became clear that some questions were interpreted incorrectly, which was not anticipated. This was the first time such a study has been attempted and, with the best will in the world, it was likely that the questions would not be clear to all participants.

To further assist with interpretation of assessor responses, each study participant was required to upload their ratings data file. This allowed files to be checked to gain a better understanding of where assessors were making mistakes, to check outlier results and to check whether assessors understood questions when unusually high rates of incorrect answers were observed.

5.4 Limitations of the methodology

Ideally a complete check of each rating file would have been undertaken to determine the extent of data entry errors. The technique used in this study of asking assessors to only extract key data saves a substantial amount of time in analysis but does present a limited view of the rating. If errors are made in areas of data input which were not included in the data extracted for this study then the rating error will not be able to be explained. In addition, if assessors didn’t understand the question or how to extract this data from the software then they may report incorrect data entry, where they

(19)

19 | P a g e may have actually entered the correct information. This study attempted to overcome these limitations by examining data in the rating files where assessors were found to struggle with

particular questions in feedback given through the support website, or where there seemed to be an unusually high error rate.

The documentation provided for this study was designed to be sufficiently comprehensive to include all data required for ratings to be completed. Documentation standards vary considerably in the residential building industry. The documentation provided for this study is likely to be more

comprehensive than assessors would usually receive, and it does not therefore test assessors’ ability to deal with inadequate documentation. Furthermore, information may not be presented in the way assessors are used to. Assessors develop relationships with their clients, and become accustomed to the way in which their clients present information e.g. how details of weatherstrips are shown, taking window dimensions from elevations rather than from a schedule. Therefore, some errors may be made in data entry for this study which assessors may not normally make when they are dealing with their regular clients.

This study was an artificial construct. There was not an opportunity for client feedback in the usual way that assessors would be accustomed to when dealing with their clients. Furthermore, this study was undertaken by assessors on top of their usual workloads and contained an element of

evaluation of their performance which may have caused some anxiety. Again, these conditions may see assessors make errors where they may not in the field. Assessors may have also been assessing houses in climate zones they were not accustomed to working in. Consequently the feel they may have developed for what appears to be a reasonable rating outcome for dwellings like this in the climate zones they are used to working in may not be useful in this case. The study may therefore suffer from a Hawthorne effect where the subjects being studied do not behave as they would normally simply because they know they are being studied.

Finally there is one key aspect of NatHERS assessments which was not tested in this study, that is, how assessors optimise the performance of a house to achieve regulatory compliance. This can only be effectively tested if the data entry is correct in the first place.

5.5 Risk Management

5.5.1 Participation rates

There were a number of strategies employed to maximise the participation rate in the study including:

 Engaging AAOs and software tool providers to support the study. Endorsement from these bodies demonstrated the importance of the study to the industry. Assessors were

approached to participate through AAOs and software providers. Assessors were sent weekly reminders throughout the recruitment phase encouraging them to participate.  Allocating CPD points by each AAO to participants allowed assessors to earn these CPD

points in one study at no cost, providing an annual saving of about $400.

 A 4-week timeframe to complete the study was used. This minimised the intrusion into assessors work time so that it would not require more than 2 hours per week to participate in the study. An extension of 5 days was granted at the end of the study to encourage those assessors who had commenced the study to complete it.

(20)

20 | P a g e  Outlining the benefits of assessor feedback on their practices and in helping to set priorities

for training and CPD based on assessors actual needs. (see section 8.1) 5.5.2 Collusion

There was some risk that assessors would discuss the study and enter responses to the study which they had not worked out themselves. To some extent, if this happened, it would reflect what assessors actually do in the field as around half of assessors said they would consult other more experienced assessors when confronted with unfamiliar situations. However, this would mask some of the information needs of assessors so a number of strategies were put in place to address this:

 Four different plans were designed and randomly assigned;

 Assessor were assured that the results were confidential so that there were no negative consequences for incorrect responses; and

 Assessor responses were scrutinized to isolate any cases with substantially identical responses.

5.5.3 Misunderstanding study questions

A number of strategies were put in place to ensure that assessors understood the questions to ensure that their answers properly reflected their rating:

 Plans were peer reviewed to ensure they did contain all the information required;  An information webinar was held to explain the study and the approach to rating houses

required for the study. A link to this webinar was included in the study website so that all participating assessors could gain access as often as required;

 A web-based help system was developed to deal with assessor queries. Over 500 queries were addressed during the study and most queries were handled within 24 hours; and  When error rates were high, individual rating files were reviewed to ensure that assessors

had entered the correct data. Where assessors did not understand the question, data was taken directly from the rating files.

5.5.4 Confidentiality

Once registered, each assessor was allocated a 12 digit personalised security code. This unique identifier was used to associate assessor responses with rating files. All personal data was removed. Confidentiality was vital to ensuring a high participation rate so that assessors felt assured that there would be no negative consequences from participating in the study.

(21)

21 | P a g e

6 Findings summary

Note: this section represents a summary only. More comprehensive discussion and analysis can be found in section 8.3.

Assessors enter thousands of individual data items to establish the rating of a house. Each data item has a different effect on the rating, so the extent of accuracy required by assessors for each data item in order to maintain an accurate assessment also varies. To reflect the varying impact of different types of data on the accuracy of the assessment different levels of acceptable variation to the correct answers were used. For example, a variation in the NCFA of the building of 2.5% would lead to an error of 0.1 stars while a variation in the area of ceiling penetrations of 20% would lead to a similar error. In the following sections, the measure of acceptable variation to data entry used was that which would lead to an error of less than or about 0.1 star, so different percent error limits are considered acceptable for different types of data entry.

6.1 Statistical significance of sample

To obtain a statistically-significant sample for the total assessor population the minimum number of assessors needed for the study was 317 i.e. the number of assessors needed for 95% confidence at a 5% margin of error. 547 assessors registered for the study and 344 completed the rating and entered the details into the study web site. The study sample is therefore a statistically-significant sample of the total assessor population.

Table 4 shows the final participation rates of assessors by accreditation status and Table 5 by jurisdiction.

Table 4 Participation rates in the study

AAO Membership Registrants Completions % of

registrations completed ABSA 414 309 208 67.3% BDAV 682 181 112 61.9% Other 720 57 24 42.1% Total 1816 547 344 62.9%

(22)

22 | P a g e

Table 5 Statistical analysis of sample and sub-sample sizes

State/Territory Accreditation status Total Study Participa nts* 95% confident answers are within +/-

ABSA BDAV None

ACT 5 1 50 56 29 12.8% NSW 174 36 0 210 102 7.6% NT 6 1 9 16 3 50.0% QLD 22 11 446 479 19 22.1% SA 7 7 125 139 34 14.7% TAS 16 15 30 61 10 28.3% VIC 55 596 30 681 119 8.1% WA 129 15 30 174 64 9.7% TOTAL 414 682 374 1816 344 4.8% Study Participants 207 115 22 344 95% confident answers are within +/- 4.8% 8.4% 20.6% 4.8%

* this includes assessors registered/licensed ACT and SA whether they live in that state or not. As a result the total in each state adds up to more than the total participants in the study.

The statistical significance of the sample can be summarised as follows:

 The overall sample will give answers which are expected to be within 4.8% of total population of assessor’s answers at a 95% confidence limit;

 The sample for ABSA members will give answers which are expected to be within 4.8% of total population answers at a 95% confidence limit;

 The sample for BDAV members will give answers which are expected to be within 8.4% of total population answers at a 95% confidence limit;

 The sample size for non-accredited assessors was not statistically significant with a confidence interval of over 20%; and

 Only samples from NSW, VIC and WA provide answers which are expected to be within 10% of total population answers at a 95% confidence limit.

The number of non-accredited assessors was difficult to assess, and the numbers of accredited assessors is changing as new assessors are trained and old assessors leave the industry. Therefore, it was important to establish how errors in the estimation of the population size would affect the number of assessors required to achieve a statistically-significant sample. If there were 2,000 assessors the number of study participants needed would increase from 301 to 322. This was still below the total number of participants so the results from this study are still robust even if assessor numbers have been underestimated.

To determine whether the difference in answers between members of different AAOs were

statistically significant, the proportion of assessors with different answers would need to exceed the sum of the confidence intervals for the two AAOs. To compare the responses of ABSA and BDAV members it may be that the sample was in error by +8.4% for BDAV members and -4.8% for ABSA members so only differences outside the range of 13.2% could strictly be considered a statistically-significant difference.

(23)

23 | P a g e The numbers of current software users are derived from information provided by the NatHERS software tool providers. This does not add up to the same number as the total number of accredited assessors because:

 Some assessors are accredited in multiple software packages;

 Some assessors work for other assessors and use their copy of the software; and  Almost a quarter of AccuRate assessors report they have not upgraded to AccuRate

Sustainability V2.0.2.13SP1. Hearne Scientific (distributors of AccuRate) were only able to provide details regarding the number of users of the most current package which did not include those who have not upgraded so the number of assessors using AccuRate may be underestimated.

The statistical significance of the samples obtained for each tool was also evaluated. Table 6 shows the confidence interval at a 95% confidence level for the samples of assessors using each tool in this study.

Table 6 Statistical significance of samples of assessors using different NatHERS tools

Software Package Total number of users Number of participants in study

Confidence Interval

AccuRate 350 76 10.0%

FirstRate5 670 147 7.1%

BERS Pro 650 121 8.1%

None of the samples meet the standard of +/- 5% at 95% confidence; however, all are significant at a 10% confidence interval. To determine whether the differences in answers between users of

different packages are statistically significant, the proportion of assessors with different answers would need to exceed the sum of the confidence intervals for the two packages being compared. To compare the responses of AccuRate and BERS Pro users, for example, it may be that the sample was in error by +10.0% for AccuRate users and -8.1% for BERS Pro users so only differences outside the range of 18.1% could be considered a statistically-significant difference.

(24)

24 | P a g e

6.2 Queries

Over the duration of the data entry phase of the study, there were 311 enquiries logged requiring a total of 592 responses (including follow-up clarifications). These enquiries were broken down into the categories represented in Table 7.

Table 7 Enquiries

e-Ticket Type Number of tickets

Technical enquiries needing expert assessor response

House 1 Accurate 10

House 1 BERS Pro 4

House 1 FirstRate5 10

Total Technical 86

Other enquiries

Withdraw from study 19

General help enquiry 206

TOTAL 311

6.3 Assessor demographics

This analysis includes all 547 assessors who registered for the study. This the first time such

information has been available on the assessor industry and it provides a very useful snapshot. The following is a summary to the responses to the assessor practices question shown in section 8.24.

 Assessors registering interest in this study reported that together they rate around 52,000 houses each year. The Australian Building Codes Board’s (ABCB) 6 star Regulatory Impact Statement (RIS)2 estimated that the number of building permits assessed using ratings would be around 71% of the 130,000 houses constructed in 2011. The assessments conducted by study participants therefore represent about a half of all new houses assessed using NatHERS tools in Australia each year.

2

Centre for International Economics, Consultation RIS 2009-3, Proposal to Revise the Energy Efficiency Requirements of the Building Code of Australia for Residential Building – Classes 1, 2, 4, 10. Australian Building Codes Board, Canberra, September 2009 see page 17 for proportion of houses assessed using NatHERS tools, and p. 190 for numbers of new houses

(25)

25 | P a g e  While the most frequent response to number of houses assessed per year was under 10, the

average reported number of assessments per registrant was 90 per year, or 2 per working week.

 Two-thirds of assessors are self-employed and only one-quarter do ratings as their main business. Of self-employed assessors, almost half are building designers or architects.  The average assessor has been doing NatHERS assessments for 5.5 years and was last

trained 2.5 years ago.

 FirstRate5 was the most commonly used package and the largest number of assessors in the study came from Victoria.

 80% of the study sample came from Victoria, NSW or WA.

 ABSA members represent 57% of the registrants; BDAV was 33% and 10% were not accredited.

6.4 Assessor Practices

The following is a summary of responses to the assessor practices question shown in section 8.4.  Assessors reported spending on average 22.5 minutes checking their rating for accuracy

once the assessment was completed. Assessors reported an average time to rate the houses in this study of 3 to 5 hours depending on the tool used.

 The most common approaches to checking ratings were through using a formal procedure such as (33%) using a rating checklist (28%) and following an informal procedure (22%).  Almost half of assessors do not or rarely seek feedback from their clients on their

performance.

 When assessors are presented with a complex house that is beyond their current experience just over half of the assessors would proceed with the rating without obtaining any advice from a more experienced assessor.

 Only 1% of assessors never had any building site experience while 63% have either supervised construction, inspected or approved construction or have worked as a tradesperson or labourer on site.

 Around one in five assessors report some difficulty in visualising a building in 3D from 2D plans.

 69% of assessors report some difficulty in finding appropriate product thermal performance properties in trade literature.

 On average assessors keep trade literature for 22 different products in their professional library.

(26)

26 | P a g e

6.5 Star Ratings

There was a wide spread of results in star rating accuracy. Figure 1 shows the proportion of the overall sample at various ranges of star rating error. Note that BERS Pro only reports half stars, not decimal stars as the other two tools report. This will slightly skew results to show a greater level of error.

Figure 1 Frequency distribution of star rating errors showing the difference between assessor rating and the correct answer

One third of the sample was able to obtain a rating that was within a quarter of a star of the correct result. The average of all rating errors was -0.02 stars. As can be seen in Figure 2 the number of assessors rating too high was greater than the number of assessors who rated too low. The size of the errors for those who rated too low was greater than for those who rated the house too high. The result is that on average these errors cancel out. Table 8 presents this information in another way. It shows the percentage of ratings within specific star rating error levels e.g. 20.6% of assessors in the sample obtained the correct rating. At star differences under 0.5 stars the level of inaccuracy for BERS Pro is slightly overstated because BERS Pro only reports ratings in half stars, not decimal stars as is reported by AccuRate and FirstRate 5. To better reflect the results on a similar basis for all software Table 9 shows the error ranges down to half a star.

(27)

27 | P a g e

Table 8 Proportion of ratings at various star rating error levels

Error Range* % of sample

From To 3.8 3.0 2.3% 2.0 2.9 6.1% 1.5 1.9 2.9% 1.0 1.4 4.1% 0.5 0.9 7.3% 0.0 0.4 7.6% 0.0 0.0 20.6% -0.5 -0.1 28.2% -1.0 -0.6 11.9% -1.5 -1.1 4.4% -2.0 -1.6 2.6% -3.0 -2.1 1.2% -5.4 -3.1 0.9%

*-ve indicates that rating was higher than the correct answer

Obtaining the correct rating does not necessarily mean that the rating was correct in all aspects. If the heating energy is in error by around the same amount as the cooling energy load then the overall star rating will be correct.

Table 9 Proportion of ratings within various star rating error bands

Error in star rating Cumulative Within 0.25 stars* 37% Within 0.50 stars 58% Within 0.75 stars* 70% Within 1.00 stars 77% Within 1.50 stars 86% Within 2.00 stars 91% More than 2.00 stars 100%

*For 0.25 and 0.75 stars, the level of inaccuracy is slightly overstated as BERS Pro only reports ratings in half stars, not decimal stars as is reported by AccuRate and FirstRate 5.

6.5.1 House plan complexity

The rating errors found in this study increased with the complexity of the house and documentation. House 1 was a volume builder single storey house, House 2 a volume builder two storey house with complex roof configuration and mixed wall types, House 3 was a small 3 bedroom townhouse where documentation was provided for the entire development and not just the dwelling to be rated, and house 4 was an apartment on the top (15th) floor.

Assessors were far better at rating house 1 than the other houses in the study, as shown in Table 10 57% obtained a rating within 0.25 stars of the correct answer and 95% within 1 star. As the

complexity of the houses and documentation increased, the level of accuracy reduced. Assessments were within a quarter of a star in only 41% of cases for house 2, 26% for house 3 and 19% for house 4.

(28)

28 | P a g e

Table 10 Proportion of ratings within various star rating error levels

Error in star rating House 1 House 2 House 3

Townhouse House 4 Apartment Within 0.25 stars 57% 41% 26% 19% Within 0.50 stars 77% 70% 45% 32% Within 0.75 stars 94% 80% 62% 41% Within 1.00 stars 95% 90% 74% 46% Within 1.50 stars 97% 95% 91% 57% Within 2.00 stars 100% 97% 95% 72% From +3.8 to -5.4 stars 100% 100% 100% 100%

Assessors clearly found rating an apartment very difficult with less than half the sample obtaining a rating within one star of the correct result for house 4. Error levels for houses 2 and 3 show that assessor error levels increased as the complexity of the house and documentation increased.

6.6 Software tools

Errors in star rating for the three NatHERS tools were different. Table 11 shows the proportion of assessors within 0.25 and 0.75 stars for each software package.

Table 11 Accuracy of assessors using different NatHERS tools

AccuRate BERS Pro FirstRate5

Within 0.25 stars 43% 36% 32%

Within 0.75 stars 78% 72% 62%

Some care must be taken interpreting these results as the samples are not representative at a 95% confidence level with a 5% confidence limit. The differences in the level of accuracy between AccuRate and FirstRate5 are similar to the error margin (confidence limit) of the samples, however, the error level differences could, in the total population, be negligible.

(29)

29 | P a g e

6.7 AAOs

The differences in star rating errors between assessors with different types of accreditation were also examined. Figure 2 shows the star rating errors for ABSA members, BDAV members and unaccredited assessors.

Figure 2 Rating errors by type of accreditation

Note that the small numbers of assessors who are not accredited means that this subsample is not statistically significant so comparisons may not reflect actual performance in the field. ABSA assessors achieve a rating within 0.25 stars at a significantly higher rate (41%) than BDAV members (28%) or unaccredited assessors (32%). However, at a limit of 0.75 of a star, the results are much closer though ABSA assessors are still more accurate (70% within 0.75 stars) than BDAV (63%) and unaccredited assessors (59%). The differences in error levels observed at 0.25 stars between ABSA and BDAV assessors are around the same as the sum of the error margins for the BDAV and ABSA samples. The actual difference could be negligible if total population statistics are all negative for one and positive for the other AAO.

This difference in accuracy between ABSA and BDAV members may be more due to the software used than the performance of the assessors from each AAO. 81% of BDAV members use FirstRate5 while only 22% of ABSA members use FirstRate5. In section 6.6 above, FirstRate5 was found to have a higher rate of error in ratings compared to AccuRate. Note that the average proportion of

(30)

30 | P a g e

6.8 Question scores

On average, assessors gave correct answers for two-thirds of the questions. Figure 3 shows the proportion of assessors achieving various score levels.

Figure 3 Overall proportion of questions answered correctly across the sample

The data items covered by each question do not have similar impacts on the star rating. For example, the impact of correctly calculating the area of uninsulated ceiling due to penetrations is different to the impact of the orientation of the dwelling. As a result, there was little correlation observed between the score and the star rating. Assessors made mistakes across all data entry areas. Only 4% of the assessor sample provided correct answers in 80% or more of questions. During the rating phase, questions received from assessors showed that some questions were open to misinterpretation. In these cases data was extracted directly from files.

6.9 Net Conditioned Floor Area (NCFA)

The basis of the energy rating is the total energy load of the house divided by the NCFA. If the assessor makes mistakes in calculating the NCFA, the star rating will also be incorrect. The three main errors associated with in estimating NCFA included:

 Incorrect data entry for zone sizes; and/or

 Heating and cooling zones which should not be heated and/or cooled; and/or  Failing to apply heating and cooling to zones which should be heated and cooled.

(31)

31 | P a g e Figure 4 shows the distribution of errors in estimating the NCFA of the dwellings, expressed as a percentage of the correct NCFA across the sample.

Figure 4 Errors in estimating Net Conditioned Floor Area (NCFA)

An error in the estimation of the NCFA of up to 2.5% would not have a major impact on the star rating of the house (around 0.1 stars for the houses in this study). 64 percent of the ratings were within of 2.5% of the correct area NCFA and 81% fell within 5% of the correct NCFA. This means that almost one in five assessors was making significant errors in calculating the NCFA.

Checking rating files for houses 1 and 2 showed that a major source of the errors in estimation of the NCFA were from assessors not applying conditioning to zones consistent with Technical Note 1. BERS Pro and FirstRate5 users import the plan, scale it and trace over to define the zones. There was some concern that incorrect scaling would lead to significant errors, however, no evidence of poor application of scaling in BERS Pro or FirstRate5 could be found. No significant differences were found in the estimation of NCFA between the three software packages.

A small systematic error in the estimation of NCFA consistency for BERS Pro and FirstRate5 was found in developing the solution sets; however this is not shown in Figure 4 because the graph shows areas relative to the correct answer for the tool. Because FirstRate5 and BERS Pro do not subtract the area of floors in upper storeys above walls in rooms below some significant differences were identified. Table 12 shows the NCFA calculated by each tool in the solution set files.

(32)

32 | P a g e

Table 12 NCFA calculated by each software for each house in the master rating files

House Software

AccuRate BERS Pro FirstRate5

1 163.2 170.7 163.4

2 257.6 270.9 266.4

3 84.6 88.3 86.4

4 149.9 149.4 144.9

In calculating NCFA, errors were greatest in house 2 (a two storey house) because fewer walls on the lower floor lined up with walls on the upper floor and so this house has a greater floor area above ground floor walls.

6.10 Zoning

Zoning refers to the allocation of spaces within the building to particular hours of occupancy and use of heating and cooling. Different types of zones can have significantly different energy loads. The energy loads of bedrooms/night time occupied spaces are usually much lower per square metre than in a living/day time space due to differences in the time of occupancy and thermostat settings. If heating or cooling is turned off in a zone, its performance has little impact on the rating.

NatHERS Technical Note 1 was specifically designed to improve assessor practices with regard to zoning. AAOs have previously reported significant errors with zoning in QA of assessments: 32.5% of BDAV QA checks and 6.7% of ABSA QA checks found errors in zoning. Some of the common zoning errors made included combining too many rooms into the one zone (which effectively breaks the cross ventilation model), turning off heating and cooling to zones which should have been conditioned and allocating living and circulation spaces incorrectly to night-time occupancy.

This study found that assessors were making significant errors in the number of zones and allocation of occupancy to zones. These errors were similar to those that have previously been reported. Zones were inappropriately combined and the allocation of zones to ‘other day time conditioned’ showed that error rates were between 86% and 100% depending on the house. The average ratings of assessors who allocated the number of zones in the house within 1 of the correct number of zones were closer to the correct rating than those who had too few or too many zones. The energy ratings of assessors who had too few zones were significantly higher than those who had too many zones. The high error rate observed raised issues regarding assessor interpretation of the questions. To ensure that assessor interpretation issues were not clouding the results the rating files were further examined to investigate zoning practices. Table 13 shows the extent of incorrect zoning found in Houses 1, 2 and 4 by software type. House 3 was omitted from this analysis to save time – manually checking rating files is time consuming - and because house 3 did not present any unique zoning issues that were not covered by the other houses.

NatHERS Benchmark Study