Results

For the purposes of pre-testing the instrument was dived into three parts that are items on cultural values, idea fairness, and fairness of the actual law. The items were numbered sequentially. Section 1 contained items 1 through 29 measuring cultural values. Section 2 contained items 30 to 34 measuring ideal fairness. Section 3 contained items 35 to 39 measuring fairness of the actual law. The items for control variables and the social desirability scale were not tested by the cognitive interviews due to time constraints.

Several lexical problems with questions related to cultural values in the cognitive interviews were discovered. The first and most frequent problem was related to the multiple meanings that respondents assigned to capitalized words that were describing the meaning of the values. Among the most problematic words were the following: MODERATE; CAPABLE, AND INFLUENTIAL. Even though additional words were used to clarify the meaning of capitalized words, the respondents routinely reported completely different semantics for these words. For instance, the verbal label INFLUENTAL was interpreted as someone who has influential personal or political connections and thus is “above the law.” This negative connotation was

far from the positive meaning of qualified and skillful employee that was assumed in the study for this label. In these situations, because the main labels played a tremendous validity role in the instrument, a list of synonyms for the problematic labels had to be generated from the existing dictionary of synonymous. Words from the list had to be tested for equality of meanings through a pilot study and the most consistent meanings of the words replaced the ambiguous verbal labels.

Questions on cultural values were constructed from the capitalized word indicating the main meaning of the value and additional verbal clues (words that were placed in parenthesis to clarify the meaning of the main word). The interviews revealed a different pattern of comprehension logic among the respondents related to these two parts. Some respondents were evaluating both main and additional words when answering the question. The other respondents refer to the additional words only in cases of confusion. The logic of the instrument developers was to use additional words to clarify the meaning of the main verbal clues, but not supersede it. In this case it was reasonable to include additional instructions for a respondent when answering such questions. The instruction referred the respondent to the words in the parenthesis only if the meaning of the main word was not clear.

Pre-testing techniques for questions on cultural values revealed that respondents in general had no problems with the 9-point mixed-word scale. Respondents accurately indicated the differences between the verbally described and non-described scaled items (e.g. scale items 7, 6 and 5, 4) and had no selection problems. A few respondents, who also revealed a low confidence in their answers, preferred to have a

6-point scale rather then a 9-point scale. As a result, recommendations were made to make no changes to the scale for questions on cultural values.

Questions related to the criminal procedure showed the most serious problems with statement comprehension. The time of latency increased two times for every question and the concurrent verbal protocols indicated serious confusion about question comprehension. The respondents exhibited considerable confusion, when encountered with the special criminal procedure terminology. However, when probed, respondents revealed the ability to understand the meaning of the statements and distinguish between the fair and unfair ones. From the probes it was evident that some simple explanation of the scenario as well as the exclusion of any special terms was needed. Based on these findings, the questions about ideal fairness and of the actual law were restructured in the following way. First, a short simple introductory statement was made explaining the scenario then the question was asked whether the legal statement presented in the introduction was fair.

The example of the question revision is presented below: Original question:

Thinking about a FAIR criminal justice system in your opinion what should be the purpose of a criminal trial?

1) The purpose of a criminal trial should be to impose punishments on criminals based on the established truth about the crime

2) The purpose of a criminal trial should be to negotiate possible punishment based on the available evidence from the prosecution and defense sides

Revised question:

To impose punishment on the criminals, a court needs to establish the facts of the crime. Modern law gives us two main ways to do it. Thinking about IDEAL criminal justice system, in your opinion, assess both ways of establishing facts of the crime. First way: The judge is the person who establishes the facts in the trial. He or she does it on the basis of the evidence that is provided by the prosecutor and defense counsel.

Second way: The prosecutor and defense counsels bring their evidence to the court and decide together what the facts of the case are. The judge supervises them in this process.

Cognitive interviews also revealed the need to remove all legal terms and jargons from the questions to make the comprehension of the instrument easier. Such terms as “parties”, “remand”, “suppressing evidence”, “presiding judge” and others were removed or replaced by more general and simple terms such as “defense counsel”, “prosecutor” “judge”, and “send case back for the new investigation”.

Pre-survey interviews provided a few indications of the scaled item problems used for the questions about ideal fairness and of the actual law. No respondents indicated preferences for the 5-point scale and few respondents acknowledged the need for a longer scale for these question. No change in the scale was performed based on results of the cognitive interview.

Results of the temporal indicators statistics are presented in Table 15 (page 115), Tables 16 and 17 (page 116). Table 15 shows the average response latency for six interviews in the second group of respondents. Table 15 also provides the information

about the increases of the response latency for particular questions compared to the average for each section. From this data, it may be concluded that some problems of comprehension occurred for questions 4, 11, 13, 16, 17, and 21 (cultural values items). The most serious comprehension issues were expected for the questions about ideal fairness and fairness of the actual law (questions 30 to 39).

Table 15. Response latency and increase in response time

# of

interview Average minutes per question for each section Increase (particular questions)

Sec 1 Sec 2 Sec 3 30% 50% 100%

1 1.12 0.56 2.06 Q11 - Q 30 2 3.21 3.07 5.59 Q4,Q13, Q17 Q36, Q34,Q35 Q31,Q32, Q 39 3 1.56 1.02 2.13 Q13 Q31,Q34 - 4 1.34 1.20 1.57 Q21 - - 5 2.05 2.16 2.45 - Q32,Q30 Q33,Q35 6 1.11 1.19 1.55 Q33 Q21,Q25 -

Results on the coding of concurrent protocols are presented in Tables 15 and 16. Results on the concurrent protocols are consistent with the data received from probing and temporal indicators. From section 1, only questions 8, 11, and 21 revealed some issues in comprehension and confidence. Much larger problems were disclosed in the questions from sections 2 and 3. All questions of section 2 on the variables about ideal fairness (Qs 29-34) caused numerous problems for respondents on both the meaning and task performance. One question from section 3 on the fairness of the actual law (Q 35) faced uncertainty in answers from respondents.

Table 16. Concurrent verbal protocols

# of intervi

ew

Repetition forget confidence can’t say don’t know 1 Q21, Q32, - Q31, Q33,Q34 Q11 Q31,Q32 2 Q34,Q35 Q 32 Q14, Q16 Q34,Q35 3 - - Q34 Q31 Q33 4 - - Q33 - Q21, 5 - - Q33, Q21,Q34 - - 6 Q31 Q35 Q21 - Q 8

Table 17. Concurrent non-verbal protocoles

# of

interview Questions Broken utterances Unintelligible utterances

1 Q21,Q22 Q11 - 2 Q24 Q22 Q4 3 Q21,Q13 Q23 - 4 - - - 5 Q21,22,24 - Q13 6 - Q21,Q22 -