Procedia Computer Science 55 ( 2015 ) 186 – 195
1877-0509 © 2015 Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
Peer-review under responsibility of the Organizing Committee of ITQM 2015 doi: 10.1016/j.procs.2015.07.031
ScienceDirect
Information Technology and Quantitative Management (ITQM 2015)
Multiple Criteria Data Envelopment Analysis – An Application
to UEFA EURO 2012
Rubem A. P. S.
a,*, Brandão L. C.
baDecision-Aid Department, Center for Naval Systems Analysis, Praça Barão de Ladário, s/n, Ilha das Cobras, Centro, 20091-000, Rio de
Janeiro, RJ, Brazil.
bPost-Graduation in Production Engineering, Fluminense Federal University,Rua Passo da Pátria 156, São Domingos, 24210-240,
Niterói, RJ, Brazil. Abstract
In this work, we evaluate the final performance of UEFA EURO 2012 national teams, by means of a methodology that combines Data Envelopment Analysis and Multiple Objective Linear Programming, known as MCDEA (Multiple Criteria Data Envelopment Analysis). Nonetheless, the formulation herein proposed is an extension of the original MCDEA model, as it considers the radial output orientation. The use of the MCDEA model seeks to improve the discrimination power among the evaluated units (national teams), a well-known limitation of traditional DEA models. The results reveal that the MCDEA framework actually enhanced the discrimination, considerably reducing the number of efficient units.
© 2015 The Authors. Published by Elsevier B.V.
Selection and/or peer-review under responsibility of the organizers of ITQM 2015
Keywords: Data envelopment analysis; multiple criteria data envelopment analysis; sports; football; UEFA EURO.
1. Introduction
Data Envelopment Analysis (DEA) [1] is a non-parametric method based on mathematical programming for measuring the relative efficiency of production units, referred as decision-making units (DMUs). DEA calculates the relative efficiency of a DMU as a weighted sum of its outputs divided by the weighted sum of its inputs, on a bounded ratio scale.
Two main disadvantages of DEA are the eventual lack of discrimination among DMUs and the poor multipliers schemes [2]. The first problem generally occurs when the number of evaluated DMUs is small, as compared to the total number of inputs and outputs used in the evaluation [3], giving rise to draws and/or a large number of efficient DMUs. The second arises when efficient DMUs attach a few large multipliers (weights) to some inputs/outputs, and very small or even null multipliers to the remaining inputs/outputs [4].
* Corresponding author. Tel.:+55-21-2638-2425. E-mail address:[email protected].
© 2015 Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
In fact, one issue relates to the other, as DEA´s optimization problem (described in Section 3) allows each DMU to select a few inputs/outputs to attach positive multipliers, discarding all the others, to maximize its own relative efficiency. In this sense, the weights derived from the DEA analysis may be counter-intuitive, since all inputs and outputs should be regarded for achieving the final evaluation [5].
The literature comprises several proposals to overcome the said issues. Focusing on this topic, [4] reviewed some of these approaches. Among them, we point out the following: weight restrictions [6-7], super-efficiency [8], and cross-evaluation [3, 9]. Nevertheless, these approaches present some limitations, such as sensitiveness to the variation range of inputs and outputs (weight restrictions), possibility of unfeasible solutions for efficient DMUs, especially under variable returns-to-scale (super-efficiency), and computational complexity (cross-evaluation).
Alternatively, [2] developed the Multiple Criteria DEA (MCDEA) model, which uses multiple objective linear programming to surpass the problems of low discrimination and poor multipliers schemes. In this work, we propose the use of the MCDEA model to evaluate the performance of the national teams, which participated of UEFA European Football Championship 2012 (UEFA EURO 2012). For that, we take into account market´s expectations and favoritism, as inputs, while, as output, we use the cardinal ranking values obtained by means of the multiple criteria method named MACBETH [10], based on the final stage achieved by each team. This input-output configuration is the same used in [11], but with a different mathematical formulation.
Finally, it is important to remark that the version of the MCDEA model used herein represents an extension of the original MCDEA model [2], once that, in our formulation, we consider the radial output orientation, deemed as more compatible with the problem under analysis, than the original input orientation.
2. DEA in Football
DEA applications for sports evaluation are considerably widespread, especially when it comes to football. For instance, in [12], the author used traditional DEA models to measure the productive efficiency of 20 football teams from the English Premier League in 2000/2001 season, while [13] analyzed the 2004/2005 season of the Spanish Professional League on- and off-field with a tree-stage DEA model.
More recently, [14] studied the football players of the German Premier League in 2002/2003 and 2008/2009 seasons, using a non-concave DEA-based meta-frontier approach. The model showed a positive relation between the average efficiency of players and the rank of the team at the end of the season. In [15], the author evaluated the efficiency of top 42 scorers who played in the UEFA Champions League (UCL) over a period of six years, using input and output-oriented models. In [16], the author examined Greek football clubs, before and after the Euro 2004 victory, using a bootstrapped DEA approach, which revealed that those clubs generally exhibited lower efficiency scores after the Euro victory, and that financial health appears to be crucial for the performance. In [17], the authors used a two-stage double bootstrapped DEA to analyze how the current value and debt levels of the top 25 European football clubs influenced their performance. In [18], the authors applied a two-phase DEA approach to select and rank the best football players of the English Premier League 2010/2011 season, while [19] used a super-efficiency DEA model to find out who were Real Madrid’s all-time most efficient players. In [11], the authors proposed advances to the Smooth DEA frontier theory, and applied them to the evaluation of national football teams in UEFA EURO 2012.
3. DEA and MCDEA
3.1. DEA: general aspects
The basic premise of DEA [1] is the homogeneity of the DMUs, meaning that they must perform similar activities and produce comparable products [20]. The method individually optimizes the efficiency score of each DMU by single objective linear programming, comparing the resources used (inputs) and the quantities produced (outputs) to the levels of other units. The result is the construction of an efficient frontier. The DMUs lying on it are efficient (unitary score), the other are inefficient (score of less than unity).
The most common DEA models are the CCR [1] and the BCC [21]. The CCR model works with constant returns-to-scale and obeys the unbounded ray property [22], so that any variation in the inputs produces a proportional variation in the outputs. The BCC model assumes variable returns-to-scale, replacing proportionality by convexity, and thus it does not meet the unbounded ray property.
Traditionally, for the calculation of radial measures of efficiency, we may use the input orientation, which seeks to minimize the resources while the production levels remain fixed; or the output orientation, which implies the increase in quantities produced while the resource levels remain unchanged. Herein, we use a radial output-oriented BCC formulation.
Thereby, considering a production process where each DMUk(k= 1,..., n) consumes rinputsݔ(i= 1,…, r)
to produce soutputsݕ(j = 1,…, s), the linearized output-oriented BCC model [21], is given by the linear program problem in (1). Min ݄=σ ݒݔ ୀଵ + ݒכ s. t. σ௦ ݑݕ ୀଵ = 1 σ ݑݕെ σ ݒݔ ୀଵ ௦ ୀଵ െ ݒכ0,݇ (1) ݑ,ݒ0,݆,݅ ݒכא ࣬
In (1),݄is the reciprocal of the relative efficiency (i.e., ݄= 1Τܧ݂݂) of the DMU under evaluation (DMUo);
ݒand ݑare, respectively, the multipliers of the inputs and outputs; ݔand ݕare, respectively, the inputs and outputs of DMUo; and ݒכrepresents the scale factor. In (1), the DMUois efficient if, and only if, ݄= 1.
Thus, defining a deviation variable ݀= (݄െ1), and using it in place of ݄, so that DMUobe efficient if,
and only if, ݀= 0, the problem in (1) can be reformulated as in (2). From the second constraint in (2), we may observe that the deviations for each DMUk(k= 1,..., n) are given by ݀= σୀଵݒݔെ σ௦ୀଵݑݕ. Then, for
DMUo, and considering the first constraint in (2), as well as the definition of ݄presented in (1), we have ݀=
݄െ1, as previously mentioned. Min ݀ s. t. σ௦ ݑݕ ୀଵ = 1 σୀଵ௦ ݑݕെ σୀଵݒݔ + ݀െ ݒכ= 0,݇ (2) ݑ,ݒ,݀ 0,݆,݅ ݒכא ࣬
As discussed in Section 1, in traditional DEA models (as those exhibited above), since each DMU optimizes its own set of multipliers, so that its efficiency score be as great as possible, many DMUs may lie on the frontier. The fact that several of the optimal multipliers, derived from the DEA analysis and used to compute the efficiency
measure, may possibly be null, relates to the same problem. These two features combined denote the benevolence of the method, which, in certain cases, may compromise its discrimination power.
3.2. MCDEA: an alternative to enhance discrimination
In this work, we opted for the MCDEA model [2], for the reasons addressed in Section 1. The MCDEA model adds two objective functions to the optimization problem, beyond the traditional DEA one. Thence, the new conditions used to measure the efficiency tend to restrict the results obtained by the DMUs, by reducing the flexibility in the optimization process [23-24]. In most cases, no single optimal solution meets all the conditions simultaneously, thus requiring a set of non-dominated solutions. For further details on multi-objective linear programming see, e.g., [25].
Herein, due to the nature of our application, it was necessary to formulate the output-oriented MCDEA, which differs from the original input-oriented formulation developed in [2]. In addition, distinctively from those authors, who based the MCDEA on the CCR model, we use the BCC, as done in [26], though still combined to an input orientation. Hence, we incorporate the two additional objective functions of the original MCDEA model [2] to the problem exhibited in (2). These objective functions comprise the minimization of the maximum deviation (minimax) and the minimization of the sum of deviations (minisum). In (3), we present the output-oriented MCDEA-BCC in the multipliers formulation, first proposed here and applied hereafter in this work.
Min ݀ Min ܯ Min σୀଵ݀ s. t. σ௦ ݑݕ ୀଵ = 1 (3) σ ݑݕെ σ ݒݔ ୀଵ ௦ ୀଵ + ݀െ ݒכ= 0,݇ ܯ െ ݀ 0,݇ ݑ,ݒ,݀ 0,݆,݅ ݒכא ࣬
In (3), the variable Min the second objective function (minimax) denotes the maximum value of deviations dk(k= 1,..., n), and the insertion of the third constraint does not alter the region of viable solutions, only ensures
that max ݀0. It is noteworthy that, although using the BCC assumption in (3), we avoid negative efficiencies, due to the output orientation. As reported in previous works (see, e.g., [26-30]), negative efficiency values, though implicit, usually appear when combining the BCC assumption to advanced input-oriented DEA models.
In the evaluation of the results, a DMU is minimax efficient if, and only if, the value docorresponding to the
solution that minimizes the second objective is null. Analogously, a DMU is minisum efficient if, and only if, the value docorresponding to the solution that minimizes the third objective is null [23, 24]. Thus, when a DMU
is minimax or minisum efficient, it must be necessarily efficient in the traditional DEA sense, once, by definition [2], both minimax and minisum efficiencies require do= 0.
4. Case Study
In this section, we first apply the traditional DEA model in (1) and then the MCDEA model in (3) for the performance evaluation of UEFA EURO 2012 national teams. Finally, we compare the results obtained.
The DMUs are the sixteen national teams that participated of the said football championship. Here, following [11], we evaluate the DMU´s performance based on market´s expectation and favoritism, once that in
tournaments among national teams, expectation and favoritism usually rely on historical aspects, such as the tradition of the team. Hence, in synthesis, our aim is to measure the extent in which expectation and favoritism towards the national teams turns into actual performance in the championship.
4.1. Modeling
In this work, as done in [11], we applied a three dimensional model, composed by two inputs and one single output, to represent the problem under analysis. To evaluate market´s expectation and favoritism in the said football championship, we used two proxies as inputs: the sum of the players´ market value and the total points in FIFA´s ranking. We calculated the first and second inputs using data available at www.financefootball.wordpress.com/ and www.fifa.com/, respectively. For representing the outcome of the national teams at the end of the championship, we used the final tournament ranking, available at www.uefa.com. Table 1 displays the input and output data.
However, the final ranking represents an ordinal scale, and therefore we shall convert it into a cardinal scale, and then use the result of this conversion as output. We followed the same procedure used by [11], relying on the M-MACBETH software [31], version 2.3, which helps the decision-maker to grade different alternatives, by comparing them in terms of attractiveness [10]. Notwithstanding, before using M-MACBETH, we categorized the tournament ranking as follows: first place; second place; eliminated in semifinals; eliminated in quarterfinals; eliminated at group stage; and nonparticipant. Then, we considered that the difference in terms of attractiveness between the first place and second place is greater than the difference between the second place and the eliminated in semi-finals, and so forth. This is the same logic applied and in [32-34] for the Olympic Games evaluation. Finally, based on these judgments, M-MACBETH suggested the cardinal scale exhibited in the last column of Table 1, which we used as output.
Table 1. Data for the UEFA EURO 2012 efficiency evaluation
DMU Input 1: market value (million €) Input 2: points in FIFA ranking Final tournament ranking Output: M-MACBETH ranking
Spain 625 1456 1 100 Germany 475 1288 3 30 England 415 1145 5 17 Portugal 350 996 4 30 France 345 964 8 17 Netherlands 320 1234 15 10 Italy 310 977 2 54 Russia 165 975 9 10 Croatia 155 1053 10 10 Sweden 130 910 13 10 Ukraine 110 572 12 10 Czech Republic 105 771 6 17 Poland 95 518 14 10 Denmark 90 1019 11 10 Greece 85 953 7 17 Ireland 70 907 16 10
Since it seems unreasonable to presume proportionality between the two inputs and the output, and due to the bounded ray of the output variable as well, in this application, we used the BCC formulation, as mentioned in Section 3. Furthermore, once the goal of the teams is to improve their performance, rather than reducing their market value and FIFA scores, we resorted to an output orientation, as also addressed in Section 3.
4.2. Results
To avoid any distortions arising from different input-output ranges, and allow a more consistent analysis of the multipliers, we normalized the data of Table 1, dividing each input and output by its corresponding maximum value. Thus, the results hereinafter reported derive from the normalized data. We started applying the output-oriented BCC model in (1), to the input-output data of Table 1. For that, we used the SIAD software [35]. Table 2 displays the results for the traditional DEA analysis.
In Table 2, we note that out of the sixteen DMUs in the set of analysis, six obtained unitary scores, and thus are tied in first position. The BCC efficient DMUs are Spain, Italy, Czech Republic, Poland, Greece and Ireland. A known distortion of the BCC model is that if a DMU is the unique consuming the smallest amount of an input or producing the greatest amount of an output, it will be classified as efficient, despite of other input-output values [36]. Some authors refer to these DMUs as efficient by default (see, e.g., [37]). In our case, Ireland has the smallest market value and Ukraine has the least amount of FIFA points. Spain has the greatest output, however it may not be considered efficient by default because it is CCR efficient, as shown in [11].
From Table 2, we may also note that eight DMUs assigned null multipliers to at least one input. Among them, two DMUs, namely Spain and Ireland, managed to reach a unitary score. The case of Spain stands out, as the traditional DEA analysis allowed the assignment of null multipliers to both inputs, what seems completely unreasonable.
Table 2. Multipliers and efficiency scores derived from the traditional output-oriented BCC model in (1)
DMU v1 v2 u1 Eff DMU v1 v2 u1 Eff
Spain 0 0 1 1 Croatia 10.277778 0 10 0.350740 Germany 3.0423280 0 3.3333333 0.384146 Sweden 10.158584 2.6005974 10 0.424697 England 5.3688142 0 5.8823529 0.245192 Ukraine 9.9368301 3.114627 10 0.738569 Portugal 0.0298722 8.1788047 5.8823529 0.304388 Czech Republic 5.8451942 1.8321336 5.8823529 1 France 0.0298722 8.1788047 5.8823529 0.322022 Poland 24.362817 3.6484937 10 1 Netherlands 9.1269841 0 10 0.180309 Denmark 10.277778 0 10 0.561097 Italy 0.0094042 2.5748089 1.8518519 1 Greece 5.9756375 1.5297632 5.8823529 1 Russia 10.277778 0 10 0.331614 Ireland 29.166667 0 10 1
For the resolution of the MCDEA´s triple objective linear programming in (3), we applied the iMOLPe software, available at www.uc.pt/en/org/inescc/products, and use the method based on the weighted sum of the objective functions to obtain the non-dominated solutions for each DMU. The graphical interface of the said software shows the weight space decomposed into indifference regions (as will be seen in Figure 1), that is, the regions in which the weights assigned to the objective functions may vary without changing the values found for those functions.
Table 3 presents the results derived from the MCDEA analysis, from which we may note that all efficiencies reported for solution 1 correspond to the traditional DEA efficiency values of Table 2. This occurs because solution 1 refers to the non-dominated solution that optimizes at least the first objective function (traditional DEA objective).
Table 3. MCDEA efficiency scores for each non-dominated solution of (3)
DMU Solution 1 Solution 2 Solution 3 DMU Solution 1 Solution 2 Solution 3
Spain 1 0.945180 Croatia 0.350741 0.318826
Germany 0.384146 0.369772 Sweden 0.424683 0.360825 0.409836
England 0.245192 0.238541 Ukraine 0.738533 0.403329 0.473684
Portugal 0.304365 0.284085 0.280631 Czech Republic 1 0.706464 0.837897
France 0.322025 0.287594 0.284492 Poland 1 0.442416 0.536353
Netherlands 0.180309 0.179712 Denmark 0.561097 0.457184
Italy 1 1 Greece 1 0.804054
Russia 0.331614 0.304642 Ireland 1 0.527638 0.688073
Fig. 1 shows the decomposition of the weight space into indifference regions for each DMU. From Fig. 1, we may also see the corresponding area in the indifference region for each non-dominated solution. As remarked in [24], this area gives a clear idea of the possible weight combinations for each objective function, thus allowing to evaluate the stability of each non-dominate solution. In this sense, large indifference regions indicate that the evaluation does not alter with moderate changes in the weights of the objective functions.
(a) (b) (c) (d)
(i) (j) (k) (l)
(m) (n) (o) (p)
Fig. 1. (a) Spain; (b) Germany; (c) England; (d); Portugal; I France; (f) Netherlands; (g) Italy; (h) Russia; (i) Croatia; (j) Sweden; (k) Ukraine; (l) Czech Republic; (m) Poland; (n) Denmark; (o) Greece; (p) Ireland
From the MCDEA analysis, we selected for each DMU the non-dominated solution that exhibits the largest indifference region. Notably, using this criterion, for every DMU of this case study, the solution chosen optimizes the third MCDEA objective, which minimizes of the sum of deviations. In particular, for Italy, Russia, Croatia, Denmark, and Greece, the solutions also maximize the traditional DEA efficiencies. These solutions may be directly identified in Fig. 1, and their respective efficiency scores are in bold at Table 3.
With these MCDEA solutions, we managed to improve the discrimination among the DMUs, as the number of efficient units falls from six in the traditional DEA analysis to two in the MCDEA approach herein proposed. The efficient units were found to be Italy, second place in the tournament, and Greece, eliminated in the quarterfinals. Remarkably, MCDEA additional objectives also eliminated the efficiency by default distortion. Table 4. Multipliers for the MCDEA non-dominated solution corresponding to the largest indifference regions
DMU v1 v2 u1 Eff DMU v1 v2 u1 Eff
Spain 1.027778 0 1 0.94518 Croatia 10.27778 0 10 0.35074
Germany 3.425926 0 3.333333 0.369762 Sweden 10.27778 0 10 0.409836
England 6.045752 0 5.882353 0.238541 Ukraine 10.27778 0 10 0.473684
Portugal 6.045752 0 5.882353 0.280631 Czech Republic 6.045752 0 5.882353 0.837897
France 6.045752 0 5.882353 0.284492 Poland 10.27778 0 10 0.536353
Netherlands 10.27778 0 10 0.179712 Denmark 10.27778 0 10 0.561097
Italy 1.903292 0 1.851852 1 Greece 6.045752 0 5.882353 1
On the other hand, the MCDEA-based analysis performed has not succeeded in providing better multipliers schemes, as shown in Table 4, where we may see that, except for Spain, the number of null multipliers for all DMUs remained unaltered or even increased. Thus, by the criterion adopted in this work of choosing a preferable non-dominated solution, the second input (points in FIFA ranking) was constantly disregarded by the DMUs. Among other things, this may indicate that the two inputs used in the modeling of the problem are redundant.
5. Conclusions
In this work, we evaluated the performance of national teams participating of UEFA EURO 2012. For that, we used the MCDEA model [2], under the BCC assumption, which we adapted for the radial output orientation. We resorted to MCDEA due to the low discrimination power provided by traditional DEA models. As there were no solutions optimizing all the MCDEA objectives simultaneously, we had to appeal to non-dominated solutions. For selecting a preferable non-dominated solution, we used its stability as a criterion (the higher the better).
The results revealed that, with the use of MCDEA, the number of efficient units decreased considerably. In particular, only Italy and Greece remained efficient, whereas the tournament champion just achieved an efficiency score of 94%. However, our MCDEA-based approach has not managed to offer better multipliers schemes. Thus, future work may either include a different input configuration for the modeling procedure or even further investigate alternative processes for calculating positive multipliers, based on their non-uniqueness.
Acknowledgements
We thank CNPq for the financial support.
References
[1] Charnes A, Cooper WW, Rhodes E. Measuring the efficiency of decision making units, Eur J Oper Res 1978; 2:429–444. [2] Li X-B, Reeves GR. A multiple criteria approach to data envelopment analysis, Eur J Oper Res1999; 115:507–517.
[3] Sexton T, Silkman RH, Hogan AJ. Data Envelopment Analysis: critique and extensions. In: Silkman RH, editors. Measuring efficiency: an assessment of data envelopment analysis, San Francisco: Jossey-Bass; 1986, p. 73–105.
[4] Angulo-Meza L, Lins MPE. Review of methods for increasing discrimination in data envelopment analysis, Ann Oper Res2002;
116:225–242.
[5] Ghasemi MR, Ignatius J, Emrouznejad A. A bi-objective weighted model for improving the discrimination power in MCDEA, Eur J Oper Res2014; 233:640–650.
[6] Thompson R., Langemeier L, Lee CT, Thrall RM. The role of multiplier bounds in efficiency analysis with application to Kansas farming,
J Econometrics1990; 46:93–108.
[7] Charnes A, Cooper WW, Huang ZM, Sun DB. Polyhedral cone-ratio models with an illustrative application to large commercial banks,
J Econometrics1990; 46:73–91.
[8] Andersen P, Petersen NC. A procedure for ranking efficient units in data envelopment analysis, Manag Sci1993; 39:1261–1264. [9] Doyle JR, Green RH. Efficiency and cross-efficiency in DEA: derivations, meanings and uses, J Oper Res Soc1994; 45:567–578. [10]Bana e Costa CA, Vansnick JC. MACBETH: an interactive path towards the construction of cardinal value functions, Int Trans Oper
Res1994; 1:489–500.
[11]Brandão LC, Andrade FVS, Soares de Mello JCCB, 2013. “2012 UEFA Euro efficiency evaluation based on market expectations”, 4th
International Conference on Mathematics in Sport. Leuven, Belgium, p. 32-37.
[12]Haas DJ. Productive Efficiency of English Football Teams - A Data Envelopment Analysis Approach, Manage Decis Econ2003;
24:403–410.
[13]García-Sánchez IM. Efficiency and effectiveness of Spanish football teams: a threestage-DEA approach, Cent Eur J Oper Res2007;
15:21–45.
[14]Tiedemann T, Francksen T, Latacz-Lohmann U. Assessing the performance of German Bundesliga football players: a non-parametric metafrontier approach, Cent Eur J Oper Res2011; 19:571–587.
[15]Papahristodoulou C, 2013. “Evaluating the performance of UEFA Champions League scorers”, 26thEuropean Conference on Operational
Research (Euro 2013). Rome, Italy.
[16]Kounetas K. Greek football clubs’ efficiency before and after Euro 2004 victory: a bootstrap approach, Cent Eur J Oper Res2014;
22:623–645.
[17]Halkos GE, Tzeremes NG. A two-stage double bootstrap DEA: the case of the top 25 European football clubs' efficiency levels, Manage Decis Econ2013; 34:108–115.
[18]Arabzad SM, Ghorbani M, Shahin A. Ranking players by DEA: the case of English Premier League, Int J Ind Syst Eng2013; 15:443– 461.
[19]Santín D. Measuring the technical efficiency of football legends: who were Real Madrid’s all-time most efficient players?, Int Trans Oper Res2014; 21:439–452.
[20]Dyson R, Allen R, Camanho AS, Podinovski VV, Sarrico CS, Shale EA. Pitfalls and protocols in DEA,Eur J Oper Res2001; 132:245– 259.
[21]Banker RD, Charnes A, Cooper WW. Some models for estimating technical and scale inefficiencies in data envelopment analysis, Manag Sci1984, 30:1078–1092.
[22]Cooper WW, Seiford LM, Tone K. A comprehensive text with models, applications, references and DEA-solver software. 2nd ed. New York: Springer; 2007.
[23]Clímaco JCN, Soares de Mello JCCB, Angulo-Meza L. Performance Measurement - From DEA to MOLP. In: Adam F and Humphreys P, editors. Encyclopedia of Decision Making and Decision Support Technologies, Hershey: Information Science Reference; 2008, p. 709–715.
[24]Soares de Mello JCCB, Clímaco JCN, Angulo-Meza L. Efficiency evaluation of a small number of DMUs: an approach based on Li and Reeves's model, Pesqui Oper2009; 29:97–110.
[25]Clímaco JCN, Antunes CH, Alves MJ.Programação linear multiobjectivo: do modelo de programação linear clássico à consideração explícita de várias funções objectivo. Coimbra: Imprensa da Universidade de Coimbra; 2003.
[26]Silveira JQ, Soares de Mello JCCB, Angulo-Meza L. Evaluación de la eficiencia de las compañias aéreas brasileñas a través de un modelo híbrido de análisis envolvente de datos (DEA) y programación lineal multiobjetivo, Ingeniare2012; 20:331–342.
[27]Appa G, Argyris N, Williams HP, 2006. A methodology for cross-evaluation in DEA. Working Paper LSEOR 06-81, London School of Economics and Political Science, p. 31.
[28]Wu J, Liang L, Chen Y. DEA game cross-efficiency approach to Olympic rankings, Omega2009; 37:909–918.
[29]Soares de Mello JCCB, Lins MPE, Gomes EG, Construction of a smoothed DEA frontier, Pesqui Oper2002; 22:183–201.
[30]Soares de Mello JCCB, Angulo-Meza L, Silveira JQ, Gomes EG (2013). About negative efficiencies in cross evaluation BCC input oriented models, Eur J Oper Res2013; 229:732–737.
[31]Bana e Costa CA, De Corte J, Vansnick J. M-Macbeth Versão 1.1, Guia do Utilizador (julho/2005). Available at: < www.M-MACBETH. com> (Acessed 18 Feb.2015).
[32]Soares de Mello JCCB, Gomes EG, Angulo-Meza L, Biondi Neto L. Cross evaluation using weight restrictions in unitary input DEA models: Theoretical aspects and application to Olympic Games ranking, WSEAS Trans Syst2008; 7:31–39.
[33]Soares de Mello JCCB, Angulo-Meza L, Silva BPB. A ranking for the Olympic Games with unitary input DEA models. IMA J Manag Math2009;20:201–211.
[34]Soares de Mello JCCB, Angulo-Meza L, Lacerda FG. A DEA model with a non discritionary variable for Olympic evaluation, Pesqui Oper2012; 32:21–29.
[35]Angulo-Meza L, Biondi Neto L, Soares de Mello JCCB, Gomes EG. ISYDS - Integrated System for Decision Support (SIAD - Sistema Integrado de Apoio a Decisão): A software package for data envelopment analysis model, Pesqui Oper2005, 25:493–503.
[36]Ali AI. Streamlined computation for data envelopment analysis, Eur J Oper Res1993, 64:61–67.
[37]Gomes EG, Abreu UGP, Soares de Mello JCCB, Carvalho TB, Zen S. DEA performance evaluation of livestock systems in Brazil. In: Charles V, Kumar M, editors. Data Envelopment Analysis and Its Applications to Management, Newcastle upon Tyne: Cambridge Scholars Publishing; 2012, p. 224–238.
