Coherence – emphasising the user perspective

10.2.1 Definitions in theory

As stated previously, statistics are estimates of finite population parameters (FPPs). Ingredients in such a parameter are

• statistical measure (total, mean, median, etc);

• variable (production, number of hours worked, etc);

• unit (enterprise, kind-of-activity-unit, etc);

• domain (sub-population, for example defined by a standard classification like NACE Rev. 1);

• reference times; both units and variable values relate to specific times.

The reference times are mostly time intervals, like a calendar year, a quarter, or a month. (However, some variables may refer to a point in time, for example the starting point of the period.) Usually reference times agree for all variables and units in a FPP. This means for example for monthly statistics that the delineation of units should refer to the current month. It follows from the above that units, classifications, other auxiliary variables, and reference times are essential to consider whenever using statistics.

In a joint use of several sets of statistics, the user wishes to keep some of the ingredients of the FPPs constant and vary one or more of the others. Some typical examples, with emphasis on what is varied:

− comparison over time: reference times, for example every month from a given one

onwards;

− comparison of countries: domains are Member States or other countries;

− comparison between non-geographical groups: domains like industries are varied;

− new statistics using several surveys: combining statistics from different business surveys

(production & employment, annual & short-term) for further analysis of industries for

example.

A simple example of a complex setting is: first taking ratios between production and number of hours worked using two surveys and then comparing those relative quantities over different aspects of space: geographical areas, industries, size groups etc. To this end, the surveys should be equal in their units, domains, and reference times. The domains are defined by for example an industrial classification that needs to be the same for all surveys.

When a user is judging coherence, definitions of the target finite population parameters (regarding units, population and domain delineation, variables, and reference times) play a

primary role. Accuracy is important, but it plays a secondary and different role. The more accurate the statistics, the smaller the disturbances; the study is more easily performed, and the conclusions drawn are usually stronger.

10.2.2 Definitions in practice

As described in the previous section, joint use of sets of statistics builds on some ingredients of the target statistics being the same. The difficulties meeting the user often depend strongly on the “distance” between the statistics used jointly. It may not be trivial even within a single survey, since definitions can vary (for example for production and employment, reference times could be a period for one and a point in time for the other). Normally, however, the problems increase considerably when using several surveys.

Even if definitions are the same in principle – as far as the user can see – they may differ in practice. One survey may have the reference time of the domains equal to that of the variable and the other use that of the frame (which the quality reports should show). A further example is the enterprise unit; it has to be defined and applied in the same way in both surveys. In a comparison between MSs, the enterprise definition may vary a lot, in spite of there being a Regulation on statistical units.

In practice there is an influence from the methodology used for example in data collection and estimation. Hence, the user needs information also on such influential factors.

10.2.3 Accuracy and consistent estimates

Accuracy has, of course, to be considered when studying for example how the ratio between production and hours worked varies over industries, so that differences that can be due only to “noise” are not stated to be significant. The user needs a measure of the overall accuracy in the joint use. This means an assessment of inaccuracy from all sources, not only due to taking a sample. It is important that the measure is realistic.

If there is a relationship between the FPPs involved, many users find it convenient if the estimates also fulfil this relationship. Two simple examples:

(i) The number of employees in two different surveys (on employment and production)

with definitions such that the FPPs are equal.

(ii) Monthly and annual production statistics with definitions such that the sum over the

twelve calendar months equals the annual value.

The expression consistent estimates will be used here to emphasise that the estimation procedures have forced the estimates to have the same relationship as the FPPs, see Section 10.3.3 for some detail. Obviously, statistics can be coherent without giving consistent estimates. This is normally the case with preliminary and definitive statistics. Note that the concept of consistent estimates is different from consistency in asymptotic theory.

If a user has two statistics that he/she believes estimate the same FPP and these estimates differ more than expected, from the inaccuracy measures given, the user should suspect

deficiencies in coherence. A simple example is as follows. Without going into technical details, assume that uncertainty intervals are given.

1) The figures are 750 ± 25 and 705 ± 10

These are not coherent from what can be seen.

This signals that there are differences in definitions that have not been stated or the user has not observed. Another possibility is that one or two of the intervals is too short.

2) The figures are 700 ± 25 and 705 ± 10

These are coherent from what can be seen.

It would be more convenient for the user to have a single figure (consistent estimates),

say 704 ± 9

The discussion in this section has emphasised the random part of the estimation error. There may also be systematic errors to take into account when using statistics. Such errors could be caused for example by the data collection. The distinction between definitions and systematic deviations is not always clear-cut, though, since definitions in practice are influenced by many factors in, for example, data collection and estimation.

10.2.4 Comparability over time

Comparisons over time are frequent. There are often two conflicting user interests as to the statistics to be produced:

− stability of definitions to compare the present with the previous for a special issue;

− the current state should be well described.

The first one works in the direction of comparability, whereas the second one goes in the opposite direction. This may be a cause of tension in statistical systems. When a change is made, special actions are often taken to improve the comparability, for example by producing statistics in both ways on one occasion or even re-estimating a part of the old series in terms of the new definitions.

There may be different opinions as to whether it is more important to estimate the level or the change accurately – different statistics may have different priorities. Short-term statistics often emphasise changes. To make that possible, comparability is needed over the time period that the changes refer to. Users of annual statistics may find the level to be more important. The National Accounts need to describe both level and change.

A further aspect of comparability over time is that certain users (for example using economic statistics indicating short term changes) are anxious to be able to separate for example

♦ trend and

♦ regular seasonal variations.

Technical means for this purpose are seasonal adjustments and calendar adjustments. To include such parameters is an enrichment of the statistics.

10.2.5 International comparability

A particular, important aspect is comparability between Member States, other countries and geographical areas in general. This involves not only different producers of the statistics but also further differences due to inherent dissimilarities between countries: labour market rules, economic practices, tax rules, etc.

Attempts to reduce differences – to increase comparability for the benefit of the user – by using similar concepts and definitions have been going on internationally for a long time; they are time-consuming tasks. There are many activities for harmonisation in business statistics in Eurostat and other international authorities, see Section 10.3.

10.2.6 Some user-based conclusions

In summary, comparability and coherence within and between sets of statistics require some definitions to be the same, for example units, variables, or reference times, depending on the particular joint use. The user needs information on differences and their consequences from the producer. The quality report for a certain set of statistics should provide such information with regard to comparability over time and coherence with other sets of statistics. It is not possible to include all other sets but experience should be used to list uses that are frequent and where users are likely to need help.

Comparability and coherence in general depend on definitions. Accuracy plays a different role. There is, however, not always a clear-cut distinction. Definitions may seem clear and unambiguous in theory but still vary in practical work. There may be a tendency not to include such deviations when measuring accuracy, although that should be done. If, for example, there is an undeclared systematic deviation in one survey but not another, there will be deficiencies in coherence between the two sets of statistics.

As a consequence of the above, comparability and coherence depend on the “distance” between producers; the deficiencies mostly increase in the following order: parts of a single survey, different surveys at the same agency, different organisations in the same country, statistical offices in different countries.

It is important for the user to have accuracy measures when using statistics together. It is convenient if the joint use has been foreseen and prepared, for example so that estimates are consistent. Explanatory comments in cases of differences are helpful, for instance when there are substantial revisions.