Program
Qualitative Comparative Analysis -
Social Science Applications and Methodological
Challenges
Date:
January 14 – 16, 2015
Venue:
Tilburg University | The Netherlands
Webpage:
www.tilburguniversity.edu/qca2015
Organizers:
Betina Hollstein
(University of Bremen),
Claudius Wagemann
(Goethe
University Frankfurt) &
Jörg Raab
(Tilburg University)
Wednesday
14 January
09.00 – 17.00 Jonas Buche, Markus Siewert QCA Introduction
Thursday
15 January
09.00 – 12.30 Jonas Buche, Markus Siewert QCA Introduction
14.00 – 14.15
Begin Conference, Welcome
14.15 – 15.45
Analyzing Social Inequality
Camilla Borgna
Different systems, same inequalities? The social
stratification of cognitive skills in 18 OECD
countries
Ramsey Wise
Does Market Orientation in Education Improve
Performance or Increase Inequality?:
Developing a Conceptual Framework for Testing
(Un)Intended Outcomes with Fuzzy Set
Qualitative Comparative Analysis (fsQCA)
Charles Ragin, Peer Fiss
Set-Theoretic Methods for the Study of Social
Inequality: Test Scores, Parental Income, and
Poverty
15.45 – 16.00
Coffee
16.00 – 18.00
Applications in
Organization Studies and
Political Science
Grégoire Croidieu,
Charles-Clemens Rüling
Why are exemplary and successful practices
sometimes not imitated? A configurational study
of Penfolds Grange and the Australian wine
industry
Stefan Cloudt
Organizational configurations and performance
of Dutch social housing associations
Markus Siewert
When do Presidents Prevail and Fail within the
Legislative Arena - Evaluating President George
W. Bush's Success on the Substance of
Legislation
Stefan Verweij, Lasse Gerrits,
Christian Kroll, Jack Meek,
Daniel Schraad-Tischler
The Relationships between Governance Capacity
and Environmental Policy Performance: A
Comparative Evaluation of 41 OECD and EU
Countries with Fuzzy Set QCA
18.00 – 19.00
Reception
20.00
Dinner
Friday
16 January
09.15 – 10.45
QCA in Mixed Methods
Research
Jonas Buche, Antje Buche,
Markus Siewert
Comparing OR combining? Assessing the Small
Value of Valuing fsQCA Versus Regression
Analysis
Alrik Thiem, Michael
Baumgartner, Damien Bol
Still lost in translation: A correction of three
misunderstandings between configurational
comparativists and regressional analysis
Sho Niikawa, Andreas
Corcaci
Conceptual coordination in applied QCA-based
mixed methods research: Reconsidering slippage
from the analysis of consensual government
formations
10.45 – 11.00
Coffee
11.00 – 12.30
Modelling and Methods
Issues in QCA
Michael Baumgartner, Alrik
Thiem
When there is more than meets the eye: Model
ambiguities in configurational comparative
research
Lakshmi Nair, Michael
Gibbert
Combining Comparative Outlier Analysis and
QCA: A review of 26 years (missed)
opportunities for theory building in
Carsten Schneider, Ingo
Rohlfing
Over- and Underfitting in Qualitative
Comparative Analysis: Theoretical foundations
and proper simulations
12.30 – 13.15
Lunch
13.15 – 14.15
Validity and Criticism of
QCA Analyses and Results
Alessia Damonte
Interesting results - but are they valid?
Benoît Rihoux
On trench warfare and hand grenades. Essay
from the battlefield of critiques against QCA
and set-theoretic methods
Analyzing Social Inequality, Thursday 14.15-15.45
Camilla Borgna
Different systems, same inequalities? The social stratification of cognitive skills in 18 OECD countries
In stratification studies, the ambivalent role of education – able to enhance social mobility, but also to legitimize intergenerational inequalities – is well established. Empirical evidence indicates that the extent to which social origins influence educational attainment varies across countries. In recent years, the availability of internationally comparable measures of cognitive skills has made systematic comparisons of education systems and their effects possible. Many studies have investigated the institutional determinants of skill stratification during compulsory schooling, finding that, for instance, family background effects are particularly strong in systems that track students into differentiated curricula at early age. Instead, due to a lack of comparable data, the social stratification of skills after compulsory schooling has been long overlooked. However, given the nearly universal nature of primary and secondary schooling and the expansion of tertiary education, this is a topic of high relevance for stratification research.
In this paper, I analyze social inequalities in literacy skills of young adults in 18 OECD countries, using internationally comparable data collected by the Programme for International Assessment of Adult Competences (PIAAC) in 2012. I adopt a mixed-methods, two-step analytical strategy. In a first step, by means of individual-level, within-country regressions, I assess the degree to which family background affects the current level of literacy skills of 24-29 year-old individuals in each country. The association of family background and skills becomes the explanandum in the second step of analysis, where I address the question of which kinds of educational systems produce the most unequal outcomes. Since educational systems can be conceived as complex entities of interconnected elements, in this second step I rely on fuzzy-set Qualitative Comparative Analysis as a method particularly apt at detecting complex causality patterns. Preliminary results suggest that different institutional configurations can bring about similarly unequal outcomes: on the one hand, if participation rates to tertiary education are low and institutional barriers to access detriment individuals of disadvantaged family background, literacy skills of young adults are strongly affected by family background. On the other hand, if tertiary education is widespread and open to individuals of different background, but highly differentiated, skills are also strongly stratified by social origin.
Ramsey Wise
Does Market Orientation in Education Improve Performance or Increase Inequality?: Developing a Conceptual Framework for Testing (Un)Intended Outcomes with Fuzzy Set Qualitative Comparative Analysis (fsQCA)
Since the introduction of PISA in 2001, profound changes in the governance of European education systems have been introduced to improve educational performance. The implementation of competence-based standards as well as a greater emphasis on privatisation, decentralisation and accountability, have all been well documented as clear indications of a transition towards market orientation in education, where choice is the key mechanism by which schools are pressured to improve. Although some researchers argue that related trends may contribute to better performance, others claim that it may unintentionally contribute to further educational inequality. Subsequently, this study develops a new conceptual framework that first identifies core indicators of market orientation in education, including school diversification, parental choice, school autonomy and school accountability. Using this framework, data from PISA are aggregated to first assess the magnitude of market orientation in 14 European countries. In a second step, fuzzy-set qualitative comparative analysis (fsQCA) is employed to test whether market orientation is both a necessary and sufficient condition to explain educational outcomes. As this method allows for the analysis of complex causality, market orientation is modelled in conjunction with additional causal conditions, including expenditures, stratification, social origin and social inequality. The results identify theoretically and empirically relevant causal pathways that explain cross-national variation of educational outcomes based on the presence or absence of causal conditions. Results demonstrate how market-oriented conditions alone are not necessary conditions for higher performance, although they are sufficient when combined with stratification and high social origin. However, this combination is extremely
likely to contribute to high inequalities in education, especially when further combined with stratification and high social inequality.
Charles Ragin, Peer Fiss
Set-Theoretic Methods for the Study of Social Inequality: Test Scores, Parental Income, and Poverty
The data analysis presented by Herrnstein and Murray in The Bell Curve is surprisingly simple. Many observers describe it as ‘underspecified’ because it is so lean, and this charge is in fact one of the main complaints lodged by Fischer et al., the research team that mounted the most extensive data-based challenge to Herrnstein and Murray. Most of the Bell Curve statistical analyses are focused on two independent variables, test scores and parental SES, and most of the analyses are little more than contests between these two variables, along with one or two control variables (e.g., age of respondent). This chapter also presents a relatively lean analysis, focusing on the main characters in the Bell Curve drama: test scores, parental background, and poverty. The key difference is that we set aside conventional quantitative methods and instead take advantage of the analytic nuance that can be gained using fuzzy sets and set theoretic methods. We first address several questions concerning the impact of single causal conditions: (1) Is there a connection between test scores and poverty? (2) Does this connection differ by race and gender? (3) Is there a connection between parental income and poverty? (4) Does this connection differ by race and gender? While these questions seem simple and straightforward, we conduct the analysis of the empirical evidence in a way that is much more nuanced than is typical of conventional forms of quantitative analyses.
Applications in Organization Studies, Thursday 16.00-18.00
Grégoire Croidieu, Charles-Clemens Rüling
Why are exemplary and successful practices sometimes not imitated? A configurational study of Penfolds Grange and the Australian wine industry
The paper studies why exemplary and successful practices are sometimes not imitated. We consider this question in the context of the Australian wine field in the second half of the 20th century. Based on a theoretical framework inspired by fundamental positions as well as more recent developments in institutional theory, we seek to understand the configurations of conditions that explain the non-imitation of practices. We employ fuzzy-set qualitative comparative analysis (fsQCA) to identify distinct field positions that can be associated with non-imitation. We selected a sample of 62 Australian wines, for which we collected information on ten winemaking practices that were key in the success of Penfolds Grange, the most exemplary and successful wine in the history of Australian winemaking. We operationalized five structural and cultural conditions geographical distance, structural equivalence, date of product creation, collective logic, and professionalization --, and we collected organization-level data to analyze the configurations of conditions that were associated with non-imitation. Our analysis so far yielded four types of configurations, which we interpret as field positions associated with non-imitation, and which we characterize as marginal insiders, outreaching insiders, insular outsiders, and boundary-pioneering insiders. These four configurations relate to distinct, complex fields positions which go beyond the division in the literature between central and marginal actors. The field positions identified in the study allow us to contribute to the literature on the (non-)imitation of exemplary practices by unpacking the institutional complexity of field positions, and by integrating the structural and cultural conditions associated with exemplarity and non-imitation. In terms of prior studies in the domain of organization studies, our research 3 distinguishes itself by interpreting the different solution paths not as organizational configurations, but as alternative field positions. A further distinctive feature of the study is that it is firmly anchored in a tradition of qualitative research and seeks to "make sense" of the solution paths by theorizing and interpreting them in the light of rich firm-level textual data.
Stefan Cloudt
Organizational configurations and performance of Dutch social housing associations
In the paper, fsQCA is applied in the analysis on organizational form and performance of social housing associations in the Netherlands. Organizational form is operationalized - in a qualitative stance - as key features of the organization. Combinations of certain key features where considered organizational configurations. We used standardized data on client satisfaction and operating expenses as performance indicators. The main goal of the research project was to get empirical insight on the theoretically developed organizational features of housing associations and to investigate in what way these features differ in respect to organizational performance. More specifically, the paper addresses two questions: 1) which - out of seven - organizational features (or configurations of features) are found in housing associations with high client satisfaction? And 2) which - out of six - organizational features (or configurations of features) are found in housing associations with low operating expenses? By means of a newly developed questionnaire, data was gathered on 61 housing associations and their organizational forms while existing data on performance outcomes could be used. The analysis revealed several necessary conditions for high performance, which are additionally evaluated on the amount of deviant cases by means of a plot. By using intermediate solutions we discovered two solution paths for low operating expenses and one for high expenses. We found three solution paths for high client satisfaction - containing the remaining two necessary conditions - and one path for low client satisfaction. The paper contributes to the use of QCA in Organization Studies by adding more conservative evaluation criteria (than e.g. Fiss' (2011) evaluation of number of cases in truth-table rows) in case of fuzzy-set scores with small variance. The theoretical contribution is on organizational forms in Dutch social housing associations and their interconnected features with respect to performance differences and the paper therefore responds to the call by Fiss (2011) for a configurational approach to organizational analysis.
Markus Siewert
When do Presidents Prevail and Fail within the Legislative Arena - Evaluating President George W. Bush's Success on the Substance of Legislation
The paper analyzes presidential success on the substance of major domestic legislation during the administration of George W. Bush. Focusing on the content of important pieces of legislation, presidential success is conceptualized as the degree to which the final bill contains the president’s preferences. The study addresses the question to what extent and under what conditions the president can act successfully in the legislative arena through a mixed-methods design, combining a macro-qualitative case comparison using fuzzy-set Qualitative Comparison Analysis (fsQCA) and in-depth case study analyses. As explanatory framework the paper applies a neo-institutional approach combining congressional and presidential factors. The following conditions are included in the analysis to explain President Bush’s levels of success: 1) the partisan composition of Congress, 2) the distribution of ideology within Congress and across the branches, 3) the president’s popularity, 4) the bargaining strategies between the White House and Congress, and 5) the going public tactics of the White House. Empirically, the study rests on a mix of quantitative and qualitative data, such as ideology scores, survey data, socio-economic information, and content analyses of the legislative histories of the bills under study provided by CQ Weekly and CQ Almanac, newspaper articles as well as Statements of Administrative Policy and other resources from the Public Papers of the Presidents.
Stefan Verweij, Lasse Gerrits, Christian Kroll, Jack Meek, Daniel Schraad-Tischler
The Relationships between Governance Capacity and Environmental Policy Performance: A Comparative Evaluation of 41 OECD and EU Countries with Fuzzy Set QCA
One of the central assumptions in the disciplines of political science and public administration is that national governments have the capacity to influence policy performance (Jann & Seyfried, 2009). Governments are, however, not closed systems; they operate in governance networks and this means that their capacity may be dependent upon the capacity of other actors in the network (Kickert, Klijn & Koppenjan, 1997; Koppenjan & Klijn, 2004; Pierre & Peters, 2000; Rhodes, 1997; Stoker, 1998; Teisman, Van Buuren & Gerrits, 2009). In this paper we study how the capacities of national governments (i.e. ‘the executive’) and other actors (i.e. ‘citizens’,
‘intermediary organizations’, and the ‘legislature’) combine in configurations, and which configurations are related to lower and higher policy performance. Specifically, we evaluate the environmental policy performance of 41 OECD and EU countries, and by comparing the configurations of governance capacities in these countries we are able to identify (configurations of) capacities that are necessary and/or sufficient for policy performance. This evaluation of the performances of the government’s environmental policies can serve the identification of best practices and can promote ‘learning’ between countries (Happaerts, 2009). The data we use is the 2014 index of the Sustainable Governance Indicators (SGI) project (Schraad-Tischler & Azahaf, 2014). The research approach we use is fuzzy set Qualitative Comparative Analysis (fsQCA) (Ragin, 2008), because it is suitable as an evaluation method (e.g. Verweij & Gerrits, 2013; see for an overview of QCA in the evaluation literature: Gerrits & Verweij, UnderReview), and for studying how configurations relate to an outcome, and comparing configurations resulting in the identifications of necessary and/or sufficient capacities for performance. In the paper we use the recently introduced QCA package in R (Duşa & Thiem, 2014; Thiem & Duşa, 2013a; 2013b). The analysis will be structured along the four basic subroutines in QCA (cf. Grofman & Schneider, 2009; Verweij, OnlineFirst): constructing the data matrix, calibration towards the truth table, pairwise comparison, and interpretation. Each subroutine is addressed and discussed using the QCA package in R.
QCA in Mixed Methods Research, Friday 09.15-10.45
Jonas Buche, Antje Buche, Markus Siewert
Comparing OR combining? Assessing the Small Value of Valuing fsQCA Versus Regression Analysis
Even in times of preaching the gospel of mixed method research, Qualitative Comparative Analysis (QCA) is still compared to standard quantitative techniques. In a recent article on women’s legislative representation, tockemer (2013) compares “the value of qualitative comparative analysis (fsQCA) versus regression analysis”. He concludes that “OLS regression analysis performs somewhat better than fsQCA” (ibid p. 86) as the latter a) suggests complex configurations with low coverage instead of two statistically significant variables and b) shows a high sensitivity to coding. In our paper we, firstly, show that these results arise a) from a non-nderstanding of the set theoretic foundation of QCA, namely the principles of conjunctural causation, equifinality and asymmetry, and b) from a non-informed use of QCA. Thus we, secondly, apply both fsQCA and regression analysis to the original data used by Stockemer and address the importance of an informed calibration procedure avoiding the allocation of the 0.5-anchor as the point of maximum ambiguity. Finally, we discuss shortcomings of stand-alone fsQCA and regression analysis in the study of women’s legislative representation. We conclude that comparing the value of these fundamentally different approaches is rather arbitrary and identify ways how to overcome the shortcomings in a mixed method design.
Alrik Thiem, Michael Baumgartner, Damien Bol
Still lost in translation: A correction of three misunderstandings between configurational comparativists and regressional analysis
Even after more than a quarter-century of debate in political science and sociology, representatives of Configurational-Comparative Methods (CCMs) and those of Regressional-Analytic Methods (RAMs) continue talking at cross purposes. In this article, we clear up three fundamental misunderstandings that have been common within and between the two communities, namely that 1) CCMs and RAMs use the same logic of inference, that 2) the same hypotheses can be built and tested with one or the other set of methods, and that 3) multiplicative RAM interactions and CCM conjunctions constitute the same concept of causal complexity. In providing the first systematic correction of these widespread and persistent misapprehensions, we clarify their formal connections. Our objective is to contribute to a more informed debate than has been the case so far, which should also lead eventually to progress in dialogue and more accurate appraisals of the possibilities and limits of each set of methods.
Sho Niikawa, Andreas Corcaci
Conceptual coordination in applied QCA-based mixed methods research: Reconsidering slippage from the analysis of consensual government formations
By analyzing coalition formations, this article reconsiders a conceptual issue of QCA-based mixed methods research against the notion of conceptual slippage. While QCA can be understood as an alternative to correlation-based research strategies, set theoretical approaches cannot easily answer the question whether expected causal mechanisms are actually present. Therefore, follow-up case studies that elucidate causal processes play an important complementary role to QCA. In order to combine these methods, it is however worth recalling that they differ at the conceptual level, which raises the question of how to realize the transfer from theory to observation in this context. In this article, the issue of conceptual coordination in QCA-based mixed methods is addressed by analyzing the formation of consensual governments. Our approach focuses on linking three different conceptual elements: theoretical framework, empirical framework, and observation of empirical processes. Methodologically, it was already suggested that the connection between the theoretical and empirical framework, which is based on theoretical expectations and QCA solution terms, is indispensable for model evaluation. The same holds true for connecting the empirical framework and the observation of empirical processes, which is realized via QCA-based case selection and plays an important role for the development of more sophisticated models. However, comprehensive empirical studies need to consider linking all three elements, which will be carried out as follows. Firstly, a theoretical framework will be derived from coalition theory, which allows posing the research question: Why do political parties not seek to form simple majority governments? Based on conceptual work, an empirical framework is then developed via QCA. Subsequently, in-depth case studies will be evaluated regarding their contribution to more sophisticated models of consensual government formations. Finally, a conclusion is drawn from the case studies that applies to the new empirical framework and, more generally, also to the established theoretical framework.
Modelling and Methods Issues in QCA, Friday 11.00-12.30
Michael Baumgartner, Alrik Thiem
When there is more than meets the eye: Model ambiguities in configurational comparative research
Sociologists have been at the forefront of configurational causal modeling by regularly making use of Qualitative Comparative Analysis (QCA). This article contends that, although QCA methodologists have developed comprehensive guidelines to help researchers avoid many of the pitfalls that lurk along the method's procedural protocol, one of the severest pitfalls has gone unnoticed so far: model ambiguities. Configurational data can often be accounted for by multiple causal models that fare equally well with respect to all parameters of fit. The degree of ambiguity sometimes reaches such extreme proportions that no causal conclusions are possible. Mainly due to deficiencies in standard QCA software, however, researchers are typically unaware of the whole model space and report only one distinguished model. As a result, there is an indeterminable risk for QCA studies published over the last two decades to have drawn conclusions beyond what the data would have warranted. Using purposefully assembled data we first analyze the source and extension of ambiguities and expose the software deficiencies. Then we show that model ambiguities are not a mere theoretical possibility but a pervasive phenomenon in applied sociological research. We argue in conclusion that QCA guidelines should require full transparency with respect to model ambiguities.
Lakshmi Nair, Michael Gibbert
Combining Comparative Outlier Analysis and QCA: A review of 26 years (missed) opportunities for theory building in Management and Organizational Research
Outliers or inconsistent cases are clear signals for lacking theory-data fit, and thereby prime candidates for theory building. Despite this, they are often ignored in Management and Organizational Research. As such, QCA appears to be a particularly promising method in this regard. It ‘tends to give explanations without dismissing exceptions or outliers’ by taking into consideration even a combination of conditions which explain only a single
case. Despite this, even in QCA, sometimes researchers reject inconsistent cases believing that these exceptional circumstances are unlikely to be repeated elsewhere. Clearly, from a QCA perspective it would be injudicious to take this stance: a cross-tabulation of cause and effect can be considered absolute only if all cases (be they deviant or not) are accounted for. A Mixed Method Research design taking outliers into account would therefore help enhance the theorizing potential of QCA. We review all published QCA studies in the top 65 Management and Organizational Research journals from 1987 (inception of QCA) till 2013 (26 years), investigating to what extent QCA researchers address inconsistent cases. We also conceptually detail the post-QCA research strategies which we term ‘Comparative Outlier Analysis’ (‘COA’); and discuss their relative merits in terms of building more rigorous theory. Specifically, we detail three distinct approaches based on ‘Replication Logic’, process tracing of single cases versus the comparative analysis of several cases. As the outlier analysis techniques would differ in csQCA and fsQCA; and also as regards the necessity and sufficiency of conditions, we present subsequent sections subdivided along these lines.
Carsten Schneider, Ingo Rohlfing
Over- and Underfitting in Qualitative Comparative Analysis: Theoretical foundations and proper simulations
Currently, the Quine-McCluskey algorithm (QMC) is at the heart of Qualitative Comparative Analysis (QCA) as a widely used technique for set relational analysis. Based on simulations, QMC has been recently and repeatedly criticized for performing poorly in reproducing the correct data generating process (DGP). Non-causal conditions often are part of the QCA solution and conditions that belong to the DGP do not display in the solution. We show that all existing simulations do not produce relevant insights on QCA’s ability to derive the correct DGP because the simulations misrepresent the DGP in QCA on sufficiency. Consequently, the results of these simulations are of questionable use for judging the performance of QMC. Based on this, we first address the problem of overfitting, i.e., the inclusion of non-causal conditions, from a theoretical point of view. We derive the proper circumstances under which QMC derives the correct DGP in the presence of overfitting. In the next step, we deal with underfitted QCA models that omit a relevant condition. For over- and underfitting, we further derive the requirements for deriving the correct DGP in the presence of limited diversity and when our interest lies the conservative, intermediate, and parsimonious solution. After we have laid the theoretical ground, we perform Monte Carlo simulations consistent with QMC’s theoretical foundation in order to arrive at a valid assessment of its performance. The goal of this paper is not to defend QMC against all criticisms and argue that it is the best algorithm. However, the evaluation of QMC and, in fact, any algorithm first needs to get the DGP straight. This paper aims to make a step in this direction for a more balanced discussion of algorithms in QCA and the performance of QCA more generally.
Validity and Criticism of QCA Analyses and Results, Friday 13.15-14.15
Alessia Damonte
Interesting results - but are they valid?
QCA does not claim any external validity for its results, yet sets five requisites of internal validity: (1) the analysis should be run on a population satisfying some clear scope condition; (2) calibration should be justifiable; (3) conditions should be meaningful; (4) the truth table should be non-contradictory; (5) minimizations should make a consistent use of logical remainders. A rich literature has been developing to deal with problems which make these requisites hard to meet in actual research; however, not each of them has been addressed equally, and not every solution seems convincing when applied. Indeed, sometimes recommendations seem to excessively borrow from case-oriented and from variable-oriented practices. So, on the one side, the justifiability of thresholds for calibration has been stretched to include statistical definitions; non-contradictions may accommodate some remarkable consistency outliers; and conditions that approximate to a constant can be considered as meaningless. On the other side, the population criteria is seldom recalled, and minimizations can be deemed consistent as they lead to results which confirm the starting theory – despite this could imply the use of false positive or negative counterfactuals. The paper will use administrative data about bureaucratic control devices in 17 European countries to address the problems raised by the 5 requisites of validity and, whenever
possible, to advance solutions consistent with the limits imposed by the basic nature of QCA – that of a «difference-oriented» strategy not for testing the universal causal power of some variable of interest, but for uncovering those actual causal configurations explaining the (non-)occurrence of an outcome.
Benoît Rihoux
On trench warfare and hand grenades. Essay from the battlefield of critiques against QCA and set-theoretic methods
A reflection on the context of current (harsh) critiques vis-à-vis QCA. It is not a technical discussion, rather a reflection on 'territorial battles' around QCA, in particular examining the contrasts in the mood in North America v/s Europe, & trying to explain why some current critiques are so 'virile'.