Chapter II: Theoretical Framework
3.4 Methodological Limitations
As Barbara Geddes argues, the pitfalls of violation of the norms of ―general convention‖ are often caused by inherent deficits of the ―conventions themselves.‖4 We agree that deficits are unavoidable, nevertheless must reduce them by increasing the degree of the measurement validity. The conceptual setting established in previous chapter provides us with the logic that appears to be more beneficial for our study and encompasses both questions rather than just one:
1 Arend Lijphart, ―Comparative Politics and the Comparative Method*,‖ The American Political Science Review LXV, no. 3 (September 1971): 685.
2 Collier, ―Translating Quantitative Methods for Qualitative Researchers: The Case of Selection Bias,‖ 464.
3 Ibid., 465.
4 Geddes, ―How the Cases You Choose Affect the Answers You Get: Selection Bias and Related Issues,‖ 89–90.
61
What caused the outcome X?
What was the effect of the cause Y?
The vast amount of information we collect must be appropriately marshaled and interpreted.
One way to do it is to gather more observations on the dependent variable or by observing the same variable in another context.1 This is very much critical to our study, due to concerns about the validation of scores generated in one contextual setting that have been put within another context. Additionally, national units, subunits or subgroups may also significantly vary in their political, social or cultural context.2 Consequently, we do not apply a large number of explanatory variables but reduce them to those that carry similar underlying characteristics, as suggested by Arend Lijphart.3 By doing so, we, in fact, compare the political phenomenon of institutional influence between the units of analysis (ministries of defence) on the basis of common trait - configuration (political functional and ideational/cultural change). We largely follow the objective to keep our analytical concept simple, which is only possible if we restrict the number of variables to the really major ones.
It will prevent us from endangering the entire quality of the study by overwhelming readers with large number of variables and currents of data that human intelligence has ―difficulty to cope with.4‖ To meet the requirements of the general convention we selected the variables that are kept constant for all cases, as shown in our research design (analytical concept). They are purposefully selected to measure, what they are supposed to measure – the outcomes of NATO - influence in three key dimensions of institutional performance: political, functional and soliological/ideational. Similar to King, Keohane and Verba‘s claim we place the research design at the center of our study as the key provider of appropriate units/objects of analysis and the variance of explanatory variables.5 This enables us to conduct measurement predominantly across qualitative categories (Elites, Bureaucracy, state agencies etc.), with some combination of quantities of the same variable. Numbers are important but not the central issue. An intensive research within each case, performed in the way of multidimensional observation, helps significantly to improve the measurement - stability, and makes sure that scores gathered will meaningfully capture the ideas contained in our concept.6
1 King, Keohane, and Verba, ―The Importance of Research Design in Political Science,‖ 479.
2 Sartori, ―Concept Misformation in Comparative Politics,‖ 1048.
3 Lijphart, ―Comparative Politics and the Comparative Method*,‖ 687.
4 Ibid., 690.
5 Caporaso, ―Research Design, Falsification, and the Qualitative-Quantitative Divide,‖ 457.
6 Mahoney, ―Qualitative Methodology and Comparative Politics‖; Adcock, ―Measurement Validity,‖ 530.
62 Obviously, the term of result - generalizability refers to the need of theoretical parsimony that is to explain more with less, and which is scientifically difficult to neglect. Knowing well that any political phenomenon is hardly possible to recur in different contextual settings, its degree of generalisibility remains always questionable. By narrowing down the scope of our case-analysis we naturally conditioned the applicability of expected results in other areas.
However, the strong validity of the applied theory and concept can be very well tested against the cases and evidence that are representative in their subject (contextual area). By developing our explicit questions and inferences we also increase the utility of our approach to the falsification principle. Thus, the results generated from our study will be judged against prior relative expectations by avoiding the risk of Sartori‘s conceptual stretching. The example of the realist school theory is the best manifestation of a strong parsimony of qualitative studies as confronted with those of quantitative scholars.1 Our cases reveal strong relation to certain causal linkage, show liitle potential to become idiosyncratic and less informative with questionable external validity.
As frequently mentioned afore, the political phenomenon and institutional performance, in particular, are very difficult to measure. Competing concepts speak for the preferable and distinct pool of data - sources. The questions we pursue to answer, to greater extent, encompass an intensive interplay between bureaucracies and various political groups, as well as individuals and public players. Monitoring these activities helps to shed light on particular preferences and interest - areas. Yet, exactly this aspect is an ongoing issue among political scientists.2 For, within the domain of tnternational relations, as well as for domestic affairs, monitoring function is largely associated with ―gathering and analyzing empirical data‖ from the members or domestic organizations.3 In fact, these ―monitoring reports‖ represent the primary and most important source of information that are performed periodically and reflect best the qualities of national institutional performance. NATO, however restricts the access to this kind of documents due to sensitive (security and defence related) nature of information contained. National bodies (defence ministries and other agencies) also refuse to disclose related information both for the reason of national interest and bilateral (with NATO) obligations. We do not expect security policies to change soon in this regard and make certain
1 Levy, ―Qualitative Methods and Cross-Method Dialogue in Political Science,‖ 203, 205.
2 Janet Johnson Buttolph and H.T. Reynolds, Political Science Research Methods, 7th ed. (Washington, D.C.:
CQ Press, 2012), 14–15.
3 Carrington, DeBuse, and Lee, ―The Theory of Governance and Accountability,‖ 4.
63 documents public on routine base as it was done by the World Bank by 1993.1 Therefore, we will heavily rely on a broad spectrum of information sources consisting of three distinct major data – pools, making the function of triangualation, i.e. data – comparison for proving the strength of evidence, quite possible.
The available NATO – documents (reports, communiques, assessments etc.) represent a first strong body of available data, we naturally refer as primary sources. National official sources such as official documents, government decisions, statements and internal agency reports are the second pool of available data. Finally, contributions of respected think – tanks (national and international), academic and policy - centered journals (articles, surveys and analytical papers), as well as interviews with key – individuals complete the third pool of data.
Interviews are of critical importance to our study due to large amount of information already collected and analyzed, and the ability of respondents to indicate causal, relational and motivational linkages between facts, individuals and behavior. We pay high importance to relate collected data to certain period of observation, and to distill concrete outputs from each interview conducted. This approach increases the visibility and structure of observations as well as the degree of responsiveness of a reader and his ability to understand the behavioral and motivational aspects of institutional decisions. Our interviews will be centered on facts, i.e. ―observable implication‖ presented within the personal opinion of a respondent, and reflect motivations and preferences of people working at NATO, national defence ministries and other state agencies.2 Whenever the data availability is limited or, as in our case, the phenomenon is especially difficult to observe, the relevance of interviewing techniques becomes much more understandable. For with each more interview conducted, a stronger consensual evaluation on certain issues and processes will be constructed. This, in fact, increases the degree of objectivity of findings, since the more individuals among those ―in best position to know‖ agree on certain issue, the more objective conclusion can be generated that must be taken very seriously.3
1 Ibid., 12.
2 Brian C. Rathbun, ―Interviewing and Qualitative Field Methods: Pragmatism and Practicalities,‖ in The Oxford Handbook of Political Methodology, Box-Steffensmeier Janet M. et al. (New York: Oxford University Press, 2008), 691; Lijphart, ―Comparative Politics and the Comparative Method*.‖
3 Rathbun, ―Interviewing and Qualitative Field Methods: Pragmatism and Practicalities,‖ 692.
64