MADM Accuracy Evaluation Framework

4.5

MADM Accuracy Evaluation Framework

This section specifies an evaluation methodology for MADM techniques that allows to assess the accuracy of a Multiple Attribute Decision Mechanism. With the MADM accuracy evaluation framework it is possible to compare MADM techniques more ef- ficiently and without relying on sub-representative evaluation metrics, such as han- dover ratios, in the path selection problem.

The MADM accuracy evaluation framework employs Design of Experiments (DoE), also known as experimental design, [Montgomery, 2008], which allows to plan exper- iments, in such a way that facilitates analyses and conclusions. DoE has different tech- niques to promote analyses, specifically the 2k _{factorial design to assess the effect of}

several variables over a response. In the path selection problem, the several variables include benefits and costs criteria, and the response corresponds to the path score of a MADM technique. In detail, the 2k_{factorial design specifies full factory experiments}

for the k main effects, (k₂) two-factor interactions, (k₃) three-factor interactions, and so on, in a total of 2k− 1 effects. By applying full factorial, a decision matrix is obtained for the k effects, considering two levels: (-) representing the minimum values and (+) the maximum values.

Table 4.8: Decision matrix for 3 criteria with 2k _facto-

rial design. Id x1 x2 x3 Effect 1 - - - (1) 2 + - - x1 3 - + - x2 4 + + - x1x2 5 - - + x3 6 + - + x1x3 7 - + + x2x3 8 + + + x1x2x3

Table 4.8 exemplifies the decision matrix for 3 factors (x1, x2, x3), considering a 2k

factorial design. The nk factorial design considers n levels of the criteria. In the path selection problem with 3 paths, the n levels can correspond to the maximum values of the diverse criteria, maxp1(x1), maxp2(x1), maxp3(x1) and so on.

Algorithm 4.3- MADM accuracy evaluation framework

Require: Dn[m, k]#n Decision matrices for n paths with m measurements and k criteria

1: m(Dj) = min{Di[, j] |i = 1, · · · , n; j = 1, · · · , k}#Minimum level (-) for criterion j 2: M (Dj) = max{Di[, j] |i = 1, · · · , n; j = 1, · · · , k}#Maximum level (+) for criterion j 3: if m(Dj) 6= 0and M (Dj) 6= 0then

4: a = 2k#Follow 2kfactorial design

5: Fa,k= a

2 !

with m(Dj)and M (Dj)levels #Factorial design matrix

6: else

7: a = nk#Follow nk_{factorial design}

8: M (Dj) =c max(D1[, j]), · · · , max(Dn[, j]) with j = 1, · · · , k

9: Fa,k= a

n !

with cM (Dj)levels #Factorial design matrix 10: end if

11: Wz,k= {Wi,j|i, · · · , z; j = 1, · · · , k; andP Wj= 1}#Set weights matrix for z experiments

12: Ia,k+z =        k1 ··· kk z1 z2 ··· zz 1 e1,1 · · · e1,k s1,k+1 s1,k+2 · · · s1,k+z 2 e2,1 · · · e2,k s2,k+1 s2,k+2 · · · s2,k+z .. . ... ... ... ... ... ... . ..

a ea,1 · · · ea,k sa,k+1 sa,k+2 · · · sa,k+z

      

#Input matrix with sa,zscore

13: s = β0+ β1x1+ β2x2+  #Run ANOVA for s

14: o = β0+ β1x1+ β2x2+ β1,2x1: x2 #Run ANOVA with p-value < 0.05

15: R2andF-statistic and p-value #Perform statistical analysis

With the results of several experiments, s (score), the response variable, can be estimated through a regression model, as depicted in Equation 4.20, where x1, x2and,

x3represent effects/criteria, β0 is the intercept coefficient, β1, β2are effect coefficients

and σ is the error estimate. Experiments are based on the same criteria values (+) and (-) levels, but with different weight sets.

s =β0+ β1x1+ β2x2+ β3x3+ β4x1x2+ β5x1x3+ β6x2x3+ β7x1x2x3+  (4.20)

Analysis of Variance (ANOVA) applies regression to formulate a linear model in the form of the Equation 4.20 and has associated statistical values that determine the efficiency of the model. Such statistical values include the goodness of fit and F- statistic. The goodness of fit can be assessed by the coefficient of determination R2, which corresponds to the total variance in response variable (s) due to effects/criteria. Higher values of R2_{, close to one, indicate that the model explains almost 100% of the}

4.5 MADM Accuracy Evaluation Framework

variation in s due to the effects/criteria and their possible interactions.

The F-statistic is also important to assess the variation between groups and within groups. Such groups represent the different experiments. For instance, higher val- ues of the F-statistic indicate that mean variation between experiments is greater than variation within experiments. If variation is between experiments, it highlights that the score varies due to the different configured weights.

The MADM accuracy evaluation framework is summarized in Algorithm 4.3. The different phases of the MADM accuracy evaluation framework are detailed in the next paragraphs.

Phase 1 - Gather data of the different paths for each criterion. Such step can be

performed in a controlled way or relying on data collected by others, outside control of a researcher employing the proposed accuracy evaluation framework. In this step n decision matrices Dn[m, k] are obtained, with m measurements for the n paths with

k criteria.

Phase 2- Determine the levels of each criterion for the diverse paths. Levels corre-

spond to the minimum, minj, and maximum, maxj, for path i in the n overall paths.

m(Dj) corresponds to the minimum level (-) while M (Dj) corresponds to the maxi-

mum level (+), and are determined according to Equation 4.21 and Equation 4.22 for the n paths with k criteria, respectively.

m(Dj) = min{Di[, j] |i = 1, · · · , n; j = 1, · · · , k} (4.21)

M (Dj) = max{Di[, j] |i = 1, · · · , n; j = 1, · · · , k} (4.22)

This step determines the logic of employing 2k_{or n}k_{factorial design. If there are no}

zeros in both levels, 2k_{factorial design can be followed, otherwise n}k_{factorial design}

must be employed. Data with zeros can represent issues in ANOVA, such as outliers. With a 2k_{factorial design, the levels correspond to the vectors m(D}

j) and m(Dj). On

a nk _{factorial design, cM (D}

j), the levels for criteria j are based on the maximum (+)

values for the n paths, assuming maximum values are different from zero, and are determined according to Equation 4.23.

c

M (Dj) =max(D1[, j]), · · · , max(Dn[, j]) with j = 1, · · · , k (4.23)

Phase 3- Determine factorial design matrix Fa,k, with a relying on the factorial

design, a = 2k or a = nk. For instance, Table 4.8 depicts the combinations of three criteria under 2k _{factorial design, resulting in a = 8, F [8, 3]. If a n}k _{factorial design is}

matrix.

Phase 4- Specify weights matrix for the different z experiments. Each j criterion in

the k criteria has associated a weight. Wz,k, the matrix with weight sets is determined

for the z experiments. Weights define how important a criterion is over another, tai- loring the final ranking determined by MADM techniques.

Phase 5- Run MADM technique for the full set of factors specified in the Fa,kma-

trix with the respective weight sets in the Wz,k matrix, for the z experiments. The

output of MADM in each experiment is the respective experiment score sa,z. Such

experiments scores are combined with the full set of factors to form the input ma- trix Ia,k+zas illustrated in Matrix 4.24, where ea,kholds the minimum and maximum

levels, determined as follows ea,k =< m(Da,k); M (Da,k) >.

Ia,k+z =       k1 ··· kk z1 z2 ··· zz 1 e1,1 · · · e1,k s1,k+1 s1,k+2 · · · s1,k+z 2 e_2,1 · · · e_2,k s_2,k+1 s_2,k+2 · · · s_2,k+z .. . ... ... ... ... ... ... . ..

a e_a,1 · · · e_a,k s_a,k+1 s_a,k+2 · · · s_a,k+z

      (4.24)

Phase 6- Perform ANOVA where the response variable corresponds to the sa,z

score determined by the MADM technique according to the diverse covariates (k cri- teria). The initial linear model must include all the covariates and their possible in- teractions, as exemplified in Table 4.8 and Equation 4.20 for 3 covariates (x1, x2, x3).

Interactions are important as the values of one criterion might be related with the val- ues of other criteria. For instance, the score, besides being based on bandwidth, round trip time and IPDV can be based on a relation between these parameters. Interactions between criteria are important, as they can be typical in path selection problems. For instance, higher bandwidths have associated lower RTT, as well as lower IPDV values.

Phase 7- Reformulate the linear model by including only the effects that are sig-

nificant, those with p-value < 0.05. Run ANOVA with the reformulated model and validate if assumptions for ANOVA models are fulfilled. Namely the model must comply with normality, homogeneity and independence assumptions [Montgomery, 2008]. Normality assumes that under the same conditions, the observations are nor- mally distributed for each value of X. Homogeneity assumes that the variance for all X values is the same. Independence means that Y values of one observation (Xi) should

not influence the Y values for other observations. In DoE, with the factorial design, the independence assumption is assured. The normality assumption can be checked

4.5 MADM Accuracy Evaluation Framework

via histograms, where bars must follow the trend of the normal curve. Homogeneity can be checked by plotting the residuals versus the fitted models, determined with ANOVA. If the model complies with normality and homogeneity assumptions, then statistical analysis of the regression model must be performed as detailed in the next step.

Phase 8- Analyse the model regarding its completeness, if all the criteria are in-

cluded, as well as interactions. The analysis must also rely on coefficient of determina- tion, R2, that assesses how the model explains the variance of score and F-statistic that complements R2 _{in the sense that it measures if variance is inside experiments or be-}

tween experiments. F-statistic assesses how a MADM technique deals with weights. Higher values of R2(close to one) and higher values of F-statistic are preferred. In ad- dition, the significance of the effects and interactions must be considered. Significant effects indicate strong contribution to the score.

Table 4.9: Summary of performed evaluations

Evaluation MADM Typea_Criteriab_Goals Evaluation

metrics Scenarios

MADM accuracy

MeTHODICAL,

TOPSIS, DiA A subset

Accuracy, Weights Impact R2_{, F-statistic} Dropbox, Heteroge- nous Analytical MeTHODICAL, TOPSIS, DiA, NMMD A all all CRRc, RHRd Dropbox-A, Dropbox-B, Operator VoIP quality MeTHODICAL, TOPSIS, DiA A subset, Rscore all MOSe, steadiness of quality Normal, Wireles Cloud Testbed MeTHODICAL, TOPSIS T all, faulty server Accuracy Transfer time, server usage ratio Testbed a_{(A)nalytical; (T)estbed;}

b_{all - Benefits and costs criteria proposed by MeTHODICAL; subset - only a set of the proposed} MeTHODICAL criteria

c_{Correct Rankings Ratio (CRR)}

d_{Required Handover Ratio (RHR)}