Appendix 1
This technical summary describes the methodology employed in the second stage quantitative survey.
Method
The quantitative survey was conducted using Computer Assisted Telephone Interviewing (CATI). The use of CATI for this type of survey had a number of advantages. The telephone research was easily controlled and supervised and allowed for a relatively short fieldwork period. The interview was conducted electronically with all questions and routing programmed automatically, meaning interviewers were free to concentrate on the respondent’s answers and data was recorded accurately.
Rather than use quota sampling for the survey, the survey aimed to maximise the response obtained from the sample in field, using a variety of techniques. The telephone fieldwork encouraged participation while also allowing the respondent to participate at a time that suited them. Respondents were able to schedule
appointment times for the interviewer to call, ensuring the sample and the
interviewer’s time was used most efficiently and respondents were more committed to taking part.
The greater rigour of a random probability approach enables statistical significance tests to be carried out on the quantitative data, so one can be confident that any differences identified between the baseline and post implementation samples reflect real differences in consumer understanding during and after the implementation of the oral disclosure rule.
The achieved interviews were also monitored to ensure that the target number of interviews was carried out with standalone and rider benefit policy purchasers and those who had bought their policy direct and via an intermediary.
Sample design
The sample design took into consideration several issues that applied to both stages of the survey. First, the samples at each stage of the survey had to be representative of recent CIC consumers. Second, the samples needed to be sufficiently well
matched to enable the results from the two stages to be compared. Third, the sample at each stage needed to be large enough to ensure confidence when examining differences between the two samples, in that they reflected real differences, rather than being attributable to sampling error. Last, the sample needed to be structured in such a way that one could compare the responses of those buying CIC as a ‘rider benefit’ with those buying standalone policies.
The sample for the second stage survey consisted of consumers who had purchased CIC between March and May 2009, and where part of the sales process involved oral communication (in effect, all advised sales). Details of all CIC purchasers in this time period were not provided, since the FSA wished to minimise the burden on providers. So the FSA limited the sample to eight major CIC providers – these were the same providers as in the baseline survey.
All of the records supplied were divided into strata based on sample provider, product type and sales channel. Figure 34 shows how the records were distributed across these three strata.
Figure 34
ORIGINAL SAMPLE COUNTS, BY SAMPLE PROVIDER, PRODUCT TYPE AND SALES CHANNEL
Direct/tied Via intermediary Total
Sample provider Rider Standalone Rider Standalone
A - - 86 73 159 B 198 23 8702 586 9509 C 246 114 - - 360 D 1784 - 1784 E 12492 3196 - - 15688 F 3012 - - - 3012 G - - 8980 717 9697 H 14594 104 - - 14698 Total 30542 3437 19552 1376 54907
A large proportion of records supplied did not have telephone numbers. However, given the sheer number of records to begin with, rather than conducting a tracing exercise, these records were excluded from the survey.
Additionally, a comprehensive check for duplicate records was done. This was based on full telephone number. Where duplicate telephone numbers were identified, one of the records was randomly selected for removal. These were typically where partners were listed on the sample database under a joint policy. When it came to the
interview stage, even if one partner had been randomly selected to be kept in, if it was the other partner who had gone through the sales process then the interview was attempted with the other person instead. Figure 35 shows how the eligible records for selection were distributed across the three strata of sample provider, product type and sales channel.
Figure 35
ELIGIBLE SAMPLE COUNTS, BY SAMPLE PROVIDER, PRODUCT TYPE AND SALES CHANNEL
Direct/tied Via intermediary Total
Sample provider Rider Standalone Rider Standalone
A - - 55 49 104 B 71 14 4162 331 4578 C 217 97 - - 314 D - - 1267 - 1267 E 5966 1876 - - 7842 F 2245 - - - 2245 G - - 2713 224 2937 H 8755 69 - - 8824 Total 17254 2056 8197 604 28111
Once the process of eliminating ineligible and duplicate records was completed, a final sample was drawn. This was done by applying a selection probability specific to each stratum. Although the weighted results at each stage needed to be
representative of the CIC market, with different types of policy in their correct proportions, specific groups of purchasers in the sample needed to be analysed separately. Therefore disproportionate sampling was employed in order to over- represent these specific groups, to allow sufficient numbers for analysis. The proportion of standalone CIC purchasers in the sample was boosted to take into account that they only represent a small proportion of the market. Figure 36 shows the sample size selected within each cell.
Figure 36
SELECTED SAMPLE SIZES, BY SAMPLE PROVIDER, PRODUCT TYPE AND SALES CHANNEL
Direct/tied Via intermediary Total
Sample provider Rider Standalone Rider Standalone
A 3 15 18 B 3 4 196 100 303 C 10 29 39 D 60 60 E 281 564 845 F 106 106 G 128 67 195 H 413 21 434 Total 813 618 387 182 2000 Fieldwork
The survey fieldwork was conducted in October 2009 and involved two main stages.
Stage One: Despatching advance letters to consumers of CIC who had purchased their policy in March to May 2009. The letters contained a brief outline of the objectives of the research.
Stage Two: Interviewing consumers of CIC who had purchased their policy in March to May 2009.
Advance letter
An advance letter was sent to the respondent before they took part in the main interview. The letter was despatched on FSA headed paper. This helped to reassure respondents of the genuine nature of the research and therefore encourage
response. The letter explained the purpose of the research and the importance of the research in terms of collecting information to help understand the sales process.
The letter gave assurances that the results would be treated confidentially in accordance with the Data Protection Act. Contact details were also provided for a member of the research team at TNS-BMRB and the FSA so that any respondent
Questionnaire
The questionnaire consisted of six main sections and was largely unchanged from the baseline survey. There were a few questions added for the second stage survey but these were accompanied by a few deletions of existing questions. The net result brought down the average interview length by two minutes from the baseline survey. The average interview length in the second stage survey was 21 minutes.
Response
Figure 37 shows the final outcomes for the total sample issued at the start of the second stage survey.
Figure 37
RESPONSE RATE
N % %
Total sample issued 2000 100
Out of scope
Business number 37 1.9
Number unobtainable 272 13.6
Incorrect sample details 34 1.7
Modem/fax/beeper 5 0.3
Ineligible 162 8.1
Total out of scope 510 25.5
Total eligible sample 1490 74.5 100
Unproductive outcomes
Refusals 367 24.6
Unavailable during fieldwork 84 5.6
Unresolved sample
No answer/ answering machine 401 26.9
Telephone number changed 59 4.0
General call back (GCB) 56 3.8
Busy 6 0.4
Stopped interview 2 0.1
Total unproductive/ unresolved sample 975 65.4
Of the 2,000 records issued, one quarter (26%) were out of scope of the survey for various reasons. From the remaining sample a total of 515 interviews were achieved, representing a response rate of 35%, a slight increase from 33% in the baseline survey. The second stage survey allowed for a slightly longer fieldwork period so perhaps this was a factor in encouraging participation among consumers.
The main reason for non-response was due to sample being loaded into the survey but not being resolved before the close of fieldwork (35%). The refusal rate was comparable with the baseline survey – one quarter (25%) of in-scope consumers refused to participate, compared with 27% in the baseline survey. Just 6% of consumers were unavailable to do the survey during the fieldwork period.
Data preparation and data output
The use of CATI removed much of the requirement for post-fieldwork data cleaning, since range, logic and consistency checks were built into the CATI program at the start. This approach had the advantage that interviewers had to resolve any inconsistencies with respondents during the interview.
All verbatim answers at ‘other (specify)’ and open-ended questions were inspected by coders. This resulted in some additional codes being added to the code frames of some questions. In all questions, the aim was to reduce the proportion of answers left in ‘other’ to below 10%.
A data file was supplied to the FSA as an SPSS for Windows system file, which consisted of 515 records.
Weighting
The weighting process involved two stages.
1. Design weights
These were attributed to each respondent based on their probability of selection. This was dependent on the sample provider, product type and sales channel stated in the
2. Overall weights
After applying the weights to take account of the over-sampling of standalone policy purchasers, further weights were applied to the entire sample to calibrate it to be representative by sample provider, product type and sales channel as originally provided by the insurance firms. This used the statistical technique of rim weighting. Finally, the weights were scaled so that the weighted total was the same size as the number of respondents that responded.
Appendix 2
Figure 38
FINANCIAL SOPHISTICATION LEVELS
Label Description
Higher
I have a very good knowledge and understanding of financial products and services; I like reading the financial pages of the newspapers and I like to make my own choices about financial products and services
Medium
I have a reasonable knowledge of personal finance products and services and I am able to weigh up the advice of finance
professionals when choosing a product to suit my personal circumstances
Lower Financial issues are best left to the experts – I generally rely on the advice of financial advisers, friends or relatives about which products are best for me
Appendix 3
Analysis process using ‘Matrix Mapping’
Material collected through qualitative methods is invariably unstructured and
unwieldy. Much of it is text based, consisting of verbatim transcriptions of interviews and discussions. Moreover, the internal content of the material is usually in detailed and micro-form (for example, accounts of experiences, inarticulate explanations, etc). The primary aim of any analytical method is to provide a means of exploring
coherence and structure, within a cumbersome data set, while retaining a hold on the original accounts and observations from which it is derived.
Qualitative analysis is essentially about detection and exploration of the data and making sense of the material by looking for coherence and structure within the data. Matrix Mapping works from verbatim transcripts and involves a systematic process of sifting, summarising and sorting the material according to key issues and themes. The process begins with a familiarisation stage and includes a researcher’s review
of the audio files and/or transcripts. Based on the coverage of the topic guide, the researchers’ experiences of conducting the fieldwork and their preliminary review of the data, a thematic framework is constructed. The analysis then proceeds by summarising and synthesising the data according to this thematic framework, using a range of techniques such as cognitive mapping and data matrices. When all the data has been sifted according to the core themes, the analyst begins to map the data and identify features within the data, defining concepts, mapping the range and nature of phenomena, creating typologies, finding associations, and providing explanations. The analyst reviews the summarised data, compares and contrasts the perceptions, accounts or experiences, searches for patterns or connections within the data and seeks explanations internally within the data set. Piecing together the overall picture is not simply aggregating patterns, but it involves a process of
weighing up the salience and dynamics of issues, and searching for structures within the data that have explanatory power, rather than simply seeking a multiplicity of evidence.
Appendix 4 The Quiz
Consumers were asked the following questions
What type and severity of cancer does their policy cover them for? 9 Malignant
8 Pre malignant
8 Low malignant potential
8 All instances covered
8 Non invasive
8 Cancer in situ
8 Borderline malignancy
What type and severity of heart attack does their policy cover them for? 9 Major heart attack
8 All instances covered
What type and severity of HIV/ AIDS does their policy cover them for?
9 Occurring due to blood transfusion as part of medical treatment, physical assault (with appropriate testing)
9 Transmitted during specified employment (e.g. emergency services)
9 In the case of assault, appropriate testing must support claim
9 Incident causing infection must have occurred inside the UK 8 Covered for transmitted infection due to drug use or sexual
activity
9 if infection takes place during employment, the claim can be void if incorrect investigation procedures are followed
What type and severity of stroke does their policy cover them for? 9 Resulting in permanent symptoms
8 Resulting in non permanent symptoms
What exclusions are relevant?
9 Pre-existing medical conditions
9 If you do not inform the provider about any of the following changes that happen between completing the application from and when your plan starts
9 Change to health
9 Family history
9 Occupation
9 Start smoking or other lifestyle changes
9 Inappropriate use of alcohol or drugs, including but not limited to the following
9 Consuming too much alcohol
9 Taking an overdose of drugs, whether lawfully prescribed or otherwise
9 Taking Controlled Drugs (as defined by the Misuse of Drugs Act 1971) otherwise than in accordance with a lawful prescription
9 Taking part in (or practising) boxing, caving, climbing, horse – racing, jet skiing, martial arts, mountaineering, off-piste skiing, pot-holing, power–boat racing, under-water diving, yacht racing or any race, trial or timed motor sport
9 Living abroad
9 Living outside of the European Union for more than 13 consecutive weeks in any 12 months
9 Self inflicted injury
9 Intentional self inflicted injury
9 Unreasonable failure to follow medical advice Are there any other factors that can affect a claim?
9 Notification of illness or condition must be received by the policy issuer within six months
9 Claim form is to be completed and received by the policy issuer within 28days
9 Policy issuer must receive all relevant certificates and information
9 Claimant must agree to undergo medical examinations
9 All info and evidence is agreed by policy issuer How is payment to be received in the event of a claim?
9 A lump sum
Appendix 5
Client • FSA
Conducted by • BMRB Ltd
Objectives • Principal objective: to explore consumer understanding of CIC policy
Universe • General public Sample size • 16 consumers
Fieldwork
period • August to November 2009 Method • Face to face depth interviews Recruitment • From sample
Incentives • Yes Interviewers • Three
Interviewer
validation • Yes
Appendix 6
Screening Questionnaire
This form is confidential property of: Job Number:45107449 BMRB Social Research Job Name: FSA ICOBS
Ealing Gateway Date: 29/06/09
26-30 Uxbridge Road, Ealing, London,
W5 2BP
PLEASE WRITE IN BLOCK CAPITALS
Mr/Mrs/Miss/Ms: Initials: Surname:
First name: Address: Postcode Tel home: Tel work: Depth Details:
Face to face recruitment 1
QUOTAS
[Please list in the order that they appear on the screener]
Area 1. F2F London Birmingham Gender Male Female
CIC Policy Sale Type
Stand alone
Rider
Purchase Point
Direct from insurer
Intermediary Channel of Sale Telephone Face to face Financial Sophistication of Consumer High Low
Date: ………..
Time: ………..
Location: ……..………..
Researcher: ……….……….. Recruiter tel no ……….
RECRUITER’S DECLARATION
The person named above has been recruited by me in accordance with the instructions and within the Market Research Society Code of Conduct.
Signed:
Print name: Date:
BACKCHECKED
Signed:
Print name: Date:
“Good morning/afternoon, I am from BMRB, we sent you a letter recently about research we are carrying out for the Financial Services Authority (FSA) about people’s experiences of taking out or purchasing protection policies. The FSA is the independent financial services watch dog.
We understand that you have recently taken out a Critical Illness Cover policy. As part of this research, we have been asked to explore people’s experience of taking out Critical Illness Cover. We are not trying to sell you any financial products. We are simply interested in your experience of how financial products were sold to you and how you made decisions based on information given at the time.
The research is completely anonymous. Any of the information you share with BMRB will be kept completely confidential, and your personal details will not be passed on to FSA or any other organisation. BMRB are totally independent of FSA. May I ask you a few questions please?”
Explain to everybody
The nature of the methodology – depth interviews Confidentiality and anonymity of respondents That depths will be recorded
About BMRB
Note: If respondent does not want to participate could you ask them to please provide a reason as to why and record their answer below:
_____________________________________________________________
Recruiter please fill in
1. Area London 1 Birmingham 2 2. Note gender: Female 1 Male 2 3. Policy Purchase
Can I just check that you have taken out this Critical illness Cover within the last 4 months from [see SAMPLE for company]?
No 1
Yes 2
4. CIC Policy Sale Type
Was this taken out as a standalone policy by itself (standalone), or were you taking out or buying another financial product at the time (like a mortgage, taking out a loan or another insurance product such as life insurance)?
Standalone 1 Rider benefit 2 Not a quota variable. Got to Q2 Not a quota variable. Go to Q3
If no, prompt with Q4. If still no: Thank and close
Go to Q4
Recruit to quota then go to Q5
5 Purchase Point
Did you buy the CIC policy directly from an insurance company or someone employed by the insurance company or from someone selling it on behalf of the insurer such as a financial adviser or a bank (intermediary)?
Insurer Direct 1
Intermediary 2
6. Channel of Sale
Were you sold the policy over the telephone or face to face?
By telephone 1
Face to face 2
7. Financial Sophistication
Which of the following three statements would you say comes closest to describing your own attitude towards financial products?
A) I have a good knowledge of financial products and services; I like reading the financial pages of the newspapers and I like to make my own choices about financial products and services.
B) I have a reasonable knowledge of personal finance products and services and I aim able to weigh up the advice of finance professional when choosing a product to suit my personal circumstances.
C) Financial matters are best left to the experts - I generally rely on the advice of financial advisors, friends, or relatives about which products are best for me.
8. We would like you to take part in an interview to talk about your