Impact Assessment Using Counterfactuals

An even better method for assessing the impact of international research collaboration on research team performance is by comparing teams of similar

characteristics in all relevant aspects and, in particular, on the probability of collaborating

14 _{The mathematical structure of binary models is not explored here, as a discussion on the statistical model} will be out of scope of this dissertation. See Long, S. (1997) Regression Models for Categorical and Limited Dependent Variables. Thousand Oaks, CA: Sage Publications.

internationally. In fact, collaborating teams may be more productive for the same reasons they collaborate internationally. That is, foreign partners may prefer to collaborate with those teams and in those R&D projects that are expected to generate new knowledge and outputs. This is technically called ‘endogeneity’ or reverse causality, and it may result from selection bias.

Hence, to estimate the “real” effect of international research collaboration, it is necessary to address the basic question: How would the teams collaborating

internationally have performed had they not participated in a collaborative experience with international partners? To the author’s knowledge, no study on the impact of research collaboration (whether local or international) has attempted to model this counterfactual situation.

To solve this endogeneity problem, comparable groups both in terms of their internal characteristics and particularly in their propensity to collaborate internationally is used.

The use of tools to control for selection bias and endogeneity using comparison groups and propensity scores is not new in the S&T policy evaluation literature. Klette, Moen, and Grilches (2000) provide a comprehensive survey on the ways public subsidies affect firm productivity, private R&D investment, patent applications, fixed-asset

investment, returns on capital, returns on sales, and growth of sales or employment (Klette, Moen et al. 2000). Focusing on crowding-out effects, Almus and Czarnitzki (2003) investigated the average causal effect of all public R&D schemes in Eastern Germany using a nonparametric matching approach to pay attention to the possible interdependence between public R&D funding and R&D performance of firms. To do

that, the authors compared the potential outcome of a group of subsidized firms to a matched control group of nonsubsidized firms (Almus and Czarnitzki 2003).

Busom (2000), considered the problem of selection bias by applying a two-stage econometric treatment model in which the first stage consists of estimating a probit model on the participation probability in public funding programs and in the second stage the R&D activity is regressed on several covariates, including a selection term that accounts for the different propensities of firms to be publicly funded. This second equation is estimated separately for participants and nonparticipants. The difference in expected R&D expenditure of both groups is according to this approach the result of public funding (Busom 2000).

A similar approach is followed in this dissertation using team output and teams’ ability to contribute to local knowledge as the dependent variables. Team characteristics, sector of affiliation, scientific field, size of home institution, and size of city or region of location are the control variables. The key independent variable is international research collaboration.

3.4.2.1 The Impact Assessment Framework: Propensity Score Matching

The Propensity Score Matching (PSM) is an approach to estimate causal

“treatment” effects (Caliendo and Kopeining 2008). The PSM is useful to overcome the fundamental question in every evaluation attempt and address the possible occurrence of selection bias. We would like to know the difference between the participant’s outcome (say, team’s productivity) with and without treatment (International Research

We clearly cannot observe both outcomes for the same teams at the same time. And taking the mean outcome for non-collaborating teams as an approximation is not advisable, since collaborating and non-collaborating teams usually differ even in the absence of ‘treatment.’ This is what the selection bias is all about. In fact, motivated teams, with strong support from their home institution, and led by someone with long experience and reputation have a higher probability of collaborating internationally and being productive than comparable teams. The matching approach, which simulates an experimental context, is one possible solution to the self-selection problem.

Since we are interested in how teams would have performed had they not

collaborated internationally [ץi = Yi(1) – Yi(0)] as a way of assessing the impact of IRC, and since only one of the potential outcomes is observed for each team, we need to construct the unobserved outcome or counterfactual to know the population average treatment effect. To do this, we need to find in a large group of non-collaborating teams those teams who are similar to the participants in all relevant pre-treatment characteristics (X). Once that is done, differences in outcomes of this well selected, and thus adequate control group and of ‘participants’ or counterfactuals, can be attributed to the

collaborative activity.

Since conditioning on all relevant covariates is limited in the case of a high dimensional vector X, Rosenbaum and Rubin (1983) suggest the use of the so-called balancing scores b(X), that is, a function of the relevant observed covariates X such that the conditional distribution given b(X) is independent of ‘assignment into a treatment’

(Rosenbaum and Rubin 1983). That balancing score is the Propensity Score: the probability of collaborating internationally given observed characteristics X15.

To estimate the population’s average treatment effects by matching teams on their propensity scores we need to focus on the treated, and more precisely on the Average Treatment Effect on the Treated (ATT) parameter, which is defined as:

ץATT = E(ץ | D=1) = E[Y(1) | D=1] – E[Y(0) | D=1]

Where D=1 for participating teams; and Y(1) and Y(0) are the outcomes of participating and non-participating teams respectively.

As the counterfactual mean for those being treated - E[Y(0) | D=1] – is not observed we have to invoke some identifying assumptions. According to Caliendo and Kopeining 2008, one possible identification strategy is to assume unconfoundedness, that is, given a set of observable covariates X which are not affected by treatment, potential outcomes are independent of treatment assignment. This is called the Conditional Independence Assumption (CIA) and it implies that selection is solely based on

observable characteristics and that all variables that influence treatment assignment and potential outcome simultaneously are observed. We’ll come back to this rather strong assumption later.

Besides the CIA, an additional requirement is the common support or overlap condition, which rules out the phenomenon of perfect predictability of D | X.

15 _{The decision whether to apply PSM as opposed to Covariate Matching is not discussed here. See Zhao,} Z. (2004). "Using Matching to Estimate Treatment Effects: Data Requirements, Matching Metrics, and Monte Carlo Evidence." The Review of Economics and Statistics 80(1): 91-107. for Mahalanobis distance

used to calculate similarity of two individuals in terms of covariate values, where the matching is done on these distances.

As the probability of collaborating given a set of team characteristics falls between 0 and 1, the common support condition (CSC) ensures that teams with the same X values have a positive probability of being both collaborating and non-collaborating teams. In other words, the CSC ensures that any combination of characteristics observed in the treatment group can also be observed among the control group.

Since the CIA holds and there is overlap between both groups, the PSM estimator for the Average treatment Effect on the Treated (ATT) can be written as:

ץATT,PSM = EP(X) | D=1{E[y(1) | D=1, P(X)] – E[y(0) | D=0, P(X)]}

Where P(X) is the Propensity Score P(D =1 | X); and D=0 for non-participating teams.

In other words, the PSM estimator is the mean difference in outcome over the common support, appropriately weighted by the propensity score distribution of participants.

3.4.2.2 Estimation Using International Research Collaboration (IRC) as the Treatment of Interest

Since the propensity score represents the discrete ‘choice’ of collaborating (or not) internationally, a logit or probit model can be used to estimate it. Either a logit or a probit model is preferred over a linear model due to the shortcomings of the latter model in terms of the unlikeliness of the functional form when the response variable is highly skewed and predictions that are outside the 0,1 bounds of probabilities.

The estimation of the probability of collaborating internationally is discussed in chapter 4. It uses the same set of variables plus two new variables that help to explain the determinants of international research collaboration: a) a dummy variable coded 1 if the

team leader writes well in a second language, 0 otherwise; and b) a dummy variable coded 1 if the team leader studied overseas, 0 otherwise.

All of the variables considered are both unaffected by participation and are not influenced by the anticipation of participation. For theoretical or empirical reasons discussed, they all influence simultaneously the participation decision and the outcome variable. They credibly satisfy the CIA and justify the matching procedure. Their impact on the probability of collaborating internationally is discussed in chapter 4. For the purpose of the analyses done in chapters 5 and 6, these variables are used to compute the propensity score for matching.

To contrast the productivity and probability of involving Colombia of a

collaborating team with productivities and probabilities of comparison group teams, the kernel matching algorithm is used16. The Kernel matching is a non parametric matching

estimator that uses weighted averages of all teams in the control group to construct the counterfactual outcome. As Caliendo and Kopeining 2008 note, citing Smith and Todd (2005), “kernel matching can be seen as a weighted regression of the counterfactual outcome on an intercept with weights given by the kernel weights.” Weights depend on the distance between each individual from the control group and the participant

observation for which the counterfactual is estimated. (…) The estimated intercept provides an estimate of the counterfactual mean” (Smith and Todd 2005; Caliendo and Kopeining 2008).

16 _{The technical details of this matching algorithm are not discussed here. See Imbens, G. Ibid."Nonparametric Estimation of Average}

Treatment Effects under Exogeneity: A Review." 86: 4-29. According to Caliendo & Kopeining, since the sample is large enough, it doesn’t matter what matching algorithm is used. Kernel seems to be the most popular in the literature.

Both a kernel function and a bandwidth parameter are used. As Caliendo and Kopeining 2008 warn, the selection of bandwidth values have implicit the trade-off that a high value yields a smoother estimated density function, therefore leading to a better fit and a decreasing variance between the estimated and the true underlying density function. On the other hand, underlying features may be smoothed away by a large bandwidth leading to a biased estimate. The opposite is true: a small bandwidth may decrease bias but increase variance which means decrease efficiency. The bandwidth choice is therefore a compromise between a small variance and an unbiased estimate of the true density function (Caliendo and Kopeining 2008). To avoid the risk of using observations that are bad matches, the common support condition is imposed.

To perform the analyses, the STATA module developed by Leuven and Sianesi (2003) is used to estimate the full model and test the balancing hypothesis using an iterative process to ensure that the estimated model is consistent with this requirement (Leuven and Sianesi 2003).

Since we do not condition on all covariates but on the propensity score, the

matching procedure is checked to see if it is able to balance the distribution of the relevant variables in both the control and treatment group. This is done by comparing the situation before and after matching to see if there are any differences remaining after conditioning on the propensity score.

Finally, the bootstrapping procedure is used to test the statistical significance of treatment effects and to compute their standard errors in case analytical estimates are biased or unavailable. Each bootstrap draw consisted in the re-estimation of the results, including the estimation of propensity scores, common support, etc. The bootstrapping

was repeated 999 times, which led to 999 bootstrap samples and 999 estimated average treatment effects.

The next chapters discuss about the determinants of team performance, and particularly about how international research collaboration affects team productivity and team orientation. However, before exploring such effects, it is worth investigating about the factors that explain who collaborates internationally in Colombia and who does not.

CHAPTER 4

DETERMINANTS OF INTERNATIONAL RESEARCH