Judgemental sampling

As noted by Eurostat (1996:04), “several [EU] countries use judicious samples based on a high coverage of relevant characteristics (for example, production, employment, and turnover). This mainly concerns production and output price indices for which there is no register of products.” In effect, such samples are based on expert judgement as to representativeness rather than full probability sampling.

An example of how this may arise occurs in one or more stages of the sampling process supporting the creation of producer price indices. For instance, Eurostat (1998:07, abbreviated E98) contains an extensive discussion of methodological aspects of estimating producer prices on the export market; most of the material in this section is based on this document.

4.4.1 Producer price index construction in the EU

Background on the problem addressed by export-market producer price indices is as follows. “Producer price indices in general should cover the prices of all commodities produced in a given country in order to be consistent with [the country's overall index of production]. ... While total producer price indices (PPI) show the evolution of prices for goods

produced on the domestic market, irrespective of whether they are sold on the domestic

market or abroad, producer prices on the export market (PPIx) only take into account the prices for those commodities which are sold abroad. ... The main purpose of the PPIx is to provide rapid information on business cycle movements, that is, to serve as an economic

indicator. Furthermore PPIx also serves as a deflator for foreign trade data and for national accounts. ...

[The] PPIx for a given industry group should be calculated as a weighted average of commodity price indices, based on a sample of enterprises and samples of representative commodities. Thus the first step in the compilation of a PPIx is the selection of a basket of representative “goods,” that is, headings at a given level of a product nomenclature (such as PRODCOM or HS). In accordance with the selected goods, enterprises have to be chosen which produce these goods on a regular basis destined to be sold abroad. The last step consists in defining in each enterprise the products representing these goods, for which prices will then be reported each month.” [E98, pp. 4−5]

In other words, the creation of a PPIx typically involves three stages of sampling: (i) choosing

a kind of market-basket of goods, (ii) selecting enterprises (companies) making those goods, and (iii) taking a sample of actual products representing the goods made by the enterprises. In practice each stage of selection in this hierarchy may use one or more sampling methods in a more or less formal way, for example, stratification, probability proportional to size, cut-off sampling, and/or expert judgement. Here are two examples from specific EU Member States: 1. In the Netherlands, “The selection of products and reporting units is based on detailed

base year production and consumption data from different statistical sources, such as production statistics and foreign trade statistics. ... In order to guarantee a minimum quality of price indices, the following rule applies: per commodity group the selected reporting unit should on average cover 80% of sales (cut-off method). If for a particular commodity more than 25 reporting units are required in order to attain 80% coverage, a random sampling method can be applied. ... Once the reporting units have been selected, the next step is to select for each reporting unit certain products within a specified commodity group. The price statistician knows for what kinds of products he wants to gather prices from the reporting unit. So, with the help of a field surveyor, a visit is made to the reporting unit. The reporting unit is asked to specify the price of a product, within the commodity group, which is representative for the export. At least one, but normally two or more, prices are asked for. ... At present about 7,000 export price quotations are

collected at frequent intervals from about 5,500 reporting units.” [E98, pp. 23−24]

2. In Sweden, “The sample of representative items is revised annually and is made in four steps: (i) Industrial activities (as specified by [the Swedish version of] SIC92) are sampled by cut-off according to export value. Within each activity (ii) commodities (as specified by HS) are then also sampled by cut-off according to Foreign Trade Statistics which have been processed for the national accounts. (iii) Producers of selected commodities are then sampled by cut-off from the Foreign Trade Statistics register of exporters. (iv) Finally, representative items are selected [judgementally] in consultation with the respondent (producer). They are selected with preference to products with high sales values, which could be expected to be sold during all months, and if possible are representative of price movements within the commodity group (HS number).” [E98, p. 44]

As these excerpts demonstrate, the choice of detailed commodity specifications is likely to involve discussion with each enterprise as a basis for expert judgement. The Swedish example shows that these commodities are typically chosen to be representative of price changes, and to be sold both frequently (so that monthly data are available) and for a long period of time. It is important to assess the accuracy of the types of samples just mentioned. For example, if products are chosen because they have enjoyed frequent sales, this may be due to low prices, and those prices may, during periods of rising inflation, increase more than others.

It does not appear that many EU Member States are attempting at present to assess the bias or

sampling variability with which their PPIx are estimated. The effects of judgemental sampling

are normally difficult to quantify, but there are several approaches which can be adopted, some of which rely on the existence of other information, and some of which are only available through additional studies. We conclude this section with a discussion of some methods currently in use in the UK.

4.4.2 The UK experience

The first point to note, in the context of price indices, is that there is rarely a frame with product information from which commodities can be selected. As mentioned above, this means that sampling is usually restricted to choosing an enterprise, and then identifying a “representative” product on a judgemental basis. There has been a tendency in the UK PPI to obtain more than one quote from businesses for similar products, which in practice gives little additional information, since businesses usually have consistent pricing policies; it would be better to obtain quotes for different products, or to sample a new business. This is especially important if the sample size in terms of number of price quotes is fixed or constrained.

Small-scale studies of the effect of this sampling can be made by enumerating the products manufactured by a business, selecting a probability-based sample, and then looking at the price movements over a short period in comparison to the existing judgemental sample. This approach is expensive in collecting additional information and forming the product list to sample from.

The UK is in the process of transition from a judgemental sample to a sample based on this concept. Lists of product sales at the detailed (8-digit) level of the PRODCOM classification are obtained as part of the PRODCOM survey for a (probability) sample of businesses from the IDBR. These will then be used to form a frame from which sampling of 8-digit products can take place according to a probability mechanism in the PPI, giving a two-phase design. There is still an issue of which product to choose within an 8-digit heading, but at least the business-product pair will be selected by a probability mechanism from the PRODCOM sampling, and appropriate weighting can be used to give a design-unbiased estimator of the “population PPI”. The first stage in the introduction of this design is underway in the UK, and results comparing the current judgemental system (which also inherits many characteristics of a previous voluntary survey) and the new probability-based system are expected around April 1999.

There are particular problems with the products of some industries which may make judgemental selection of a “representative” product extremely difficult. In the clothing industry, for instance, items and fashions change on a seasonal basis, and getting a continuous price quote for a transient line is impossible. Thus there will be a tendency to select continuously-produced products, even when these do not accurately represent the overall price movement under the appropriate heading.

In a similar way it might be expected that “typical” rather than representative products are identified, and that for this reason minority production (which might have a more volatile price) may be missed. This is very difficult to assess: the information required is about the proportion of extreme price movements, which requires a large sample for estimation. However, in cases where product identification instructions draw attention to this problem, it should be noted as part of the quality assessment that this may be an issue.

Some assessment of the quality of a judgemental sample can also be made using the model- based approach by invoking the ignorable sampling assumption (see Chapters 2 and 9). If we assume (probably falsely) that the judgemental sample is approximately representative, then we can calculate the variability of prices in product categories (choosing a higher or lower level depending on the sample size available so as to obtain a reasonable estimate). This helps to assess the “sampling variability” of the judgemental sample, and by reallocating the sample using a Neyman-type allocation and calculating the expected variance (noting that the expected variance is smaller than what will be achieved in practice because it uses the same data for allocation and sampling variance estimation), the two can be compared. This

approach has been adopted in the optimisation of the UK CPI, where − for example − the

number of quotes for potatoes was increased because of the variability induced by the high price of imported new potatoes at certain times of the year.