Reliability and Validity

Earlier in this part a few vital questions concerning the methodology were left unanswered. One question remains: How can the generalization be justified when the discussion was started from one case, and ended in an analysis of the firm in general? The rationale justifying the generaliza-tion builds on three arguments. First, the design of the quesgeneraliza-tionnaire was clearly addressing the general management aspect of the firm. Second, even though the discussion was initiated by talking about one case, the outcome in the interview was that the interviewee had spoken about several

projects. In each case study, there are recorded references to experience from several projects. Third, even though product innovations were the key issue in the interviews, the empirical material consistently shows evidence of attention to the broader view of the firm.

The quality of the study depends both on the reliability of the underlying material that is assessed and the conclusions that are sensible and acceptable to draw from the scale—the validity of the scale. The analytical approach builds on the assumption that the data reveal several things about leadership in innovative firms. For one thing, the leaders spend most of their interview time on the topics that concern them most. Furthermore, it is assumed that the time the leaders spend on the interview topics reflects how they have devoted their attention when they master the innovation activity in the context of the entire firm. The idea of focusing on stories about specific events was to enhance the connection between actual action and their talk. The reliability of the

132

empirical material is partly addressed in this way. Discussions about how things should be done, which naturally also occurred in the interviews, came to have more of a predictive value, and ran a higher risk of being more loosely connected to actual action. The interview was designed to give the informants the freedom of emphasizing whatever was on their minds. The semi structured interview approach served as a mechanism that yielded, at minimum, comparability across interviews. When several informants were interviewed per case, it was assumed that the time spent discussing a given issue with informants evens out—that one informant would devote a great deal of time to that issue and another informant would have little to say about it The coding exercise demonstrated that in the end the entire field of topics was reasonably well covered.

One might also wonder if the logical basis for the categories and properties are fair. On one hand, it is a matter of how consistently the coding work has been done and the logical order of the structure of categories and properties. On another hand, the study of the statistical variation of attention across categories is not independent of the decision about the width of a category. A narrowly defined category runs the risk of being less easily recognized than a broadly defined category. The logical bundle of key words has been the leading thought when the categories and the properties have been composed. This notion was later tested in the study. When it comes to the validity of the scale, the data reduction revealing the key scope of the leaders does cover potential pitfalls. Can a secret—something about which nothing is said—not also be a vital piece of information? The same question applies to implicitly communicated pieces of information. Surely the research method applied in this study does have blind spots. The concern of the validity of the empirical structure is addressed through a transparent and comprehensive description of the empirical material in Part 4: Empirical Observations.

Furthermore, the design of the research strategy to test the outcome of the analysis is designed specifically to test issues of validity. The six stories of each case and the exercise to connect these stories to relevant theory

133

are found at the end of this book. The result of that test provides an open view of the merits, to allow the reader to judge if the result of this study is acceptable and worth being treated seriously. Eventually, it cannot be avoided; the conclusions and the outcome of this study are, in any case, dependent upon some portion of fair judgement. That is why a great deal of energy has been devoted to ensuring good traceability of evidence in this research report.

Summary

In summary, the purpose of the description of the empirical domain is to become familiar with the ideology of the leaders interviewed. The empirical material consists of 509 pages of transcribed text, equal to 24,773 lines of text, or 229,034 words from the total of 24 individual interviews conducted in the six case firms. Similar keywords were assembled and were later given the labeling property of a category. The categories emerged when similar and related groups of properties were assembled. When the categories were assembled, the structure of the empirical material in this study was born. That is, the keywords generated the properties, and the properties ended up as categories, and the

categories stand for the structure of the empirical findings. The reporting of each category gives a description of the uniqueness of the category, a description of the observations, and a comparison between the cases. In the process of categorizing the empirical material, the individual interviews were combined into one text body for each firm. Hence, the respondents of each company become ’one voice’ talking in the empirical material.

That is, the leaders collectively introduced shared stories of their firm.

The connection of the empirical observations to the existing theory in each field of knowledge is made in the summaries of the Part 4: Empirical Observations; in Part 5: Analysis, in connection to the found critical areas of attention and in connection to the test in which each case story is examined; and finally, in Part 6: Conclusions.

134

Empirical observations from the all six case firms are highlighted in this part of the book. The purpose of describing the empirical domain is to familiarize the reader with the thinking of the leaders, i.e. of those who lead the firm and who are engaged with the innovations of the firm.

A further purpose is to introduce evidence of consistency among the structures across cases. Part 4 should provide the reader with a deeper understanding of the leaders’ thinking when they talk about leading innovation in their firms. On the most general level, four domains of discussion emerge. The leaders describe the leadership of innovative firms in terms of 1) external rulers, 2) two internal dependencies of motivation–

driven and rational systemic—the general dependencies of time aspects and barriers of innovation. Figure 30 illustrates an overview of the thinking of senior managers.

Figure 30. Three Domains in the Thoughtworld of Leaders in Mature Innovative Firms.

general

motivation–driven

external rulers internal rulers

rational systemic

135

Each domain of attention comprises various categories of discussion. The external dependencies include the use, the user, the location of the use, and the appearance of the innovation. Each category is like a room in a house.

The properties, subordinated to the categories in this description, are like the furniture in the room. The category of use, for example, is described more specifically by the activity–related properties connected to the use:

the experience of using and the outcome of the use of the innovation.

This further elaboration introduces a more specific landscape of attention.

Figure 31 illustrates this landscape and forms the main point of reference throughout Part 4 of this book.

Figure 31. Landscape of Attention in the Thoughtworld of Leaders in Mature Innovative Firms.

136

A separate section is dedicated to each category of discussion:

• an introduction to the discussion,

• a description of empirical observations of each category,

• a comparison of findings across the six cases, and

• examples of barriers encountered.

The findings are illustrated in a number of ways. A comparison of key findings among the cases is illustrated in a table of observations on a general and a detailed level and the general matrix illuminates the attention that leaders have devoted to each subject in the matrix. Significant level of attention has the symbol (S) in the table, the low attention symbol (L), and the mid level of attention symbol (M). A positive description of the observation and a description of the negative barriers are presented in order to enhance an understanding of the discussions.

The presentation of the empirical findings was a journey. The more deeply I delved into a category of discussion, the more visible the connections to other categories became. Thus one discussion led to another, resulting in a tremendous network of dependencies among the categories observed. The question arose: ’Does this matter actually belong to this or that discussion?’ As everything appeared to be connected—directly or indirectly—to everything else, a sequentially written story of the whole assumes that choices were made when the structure of the report was decided. Looking at the landscape figure, one could ask: ’Where should the story begin? Along which line of thinking should it proceed? To what should attention be paid?’

The structure of this book represents my personal choices, based on prior knowledge and by the support of the theoretical frames presented in chapters. The grouping of the categories on a detailed level was another such choice. A reading of Part 4: Empirical Observations should clarify the meanings and merits of the choices made, yet it does not resolve the debate over what is the heart of the matter and what is the best line of thinking.

In the end, it is evident that the construct is based to some extent on my

137

own judgment. The matrix analysis is used to reduce any tendencies for subjective judgment.

To summarize the remainder of this book, Part 4 focuses on empirical observations; Part 5 presents the analysis of the main discussions and their dependencies, as well as a test of the validity of the results. The conclusions and connection to theory is presented in Part 6: Conclusions and

Discussion.