To further examine performance of the proposed methodology, it is tested over a broad range of decision problems.
3.4.1 Settings
We characterize each decision problem with a set of parameters: number of criteria (M); num- ber of alternatives (N) that some of them will be evaluated by a simulated DM; number of
pairwise comparisons (p) made by the DM; and finally, the complexity level in judgment policy of the simulated DM (D). A larger value ofDsignifies a more complex value system used by the simulated DM. Later in this section we will explain how this complexity is modeled. The values considered for the parameters defining each decision problem are presented in Table 3.4.1.
Table 3.4.1: Different decision problem settings considered in the experiment
N M p D
{6,8,10,
12,14} {2,3,4,5}
{4,6,8,10,12,
complete ranking} {1,2,3,4}
The experiment has been conducted as follows:
Step 1: The performances of alternatives for criteria are generated randomly from a uniform distribution in the range[−10,10].
Step 2: Thenppairs of alternatives are randomly selected to be compared by the simulated DM. To guarantee that there exists at least one compatible value function for each decision setting, we generated polynomial marginal value functions with D degrees in the following form: v(x) = p0 D ∏ d=1 ( x 10 +pd), (3.23)
in whichp0,p1,· · · ,pD ∈ [−1,1]are random coefficients generated from a uniform distribu-
tion. Moreover, in order to better define complexity of the marginal value function byD, the coefficients are selected such that p0 ̸= 0 and p1 ≠ p2 ̸= · · · ̸= pD. The value function
is obtained by adding the marginal value functions, and the selected pairs of alternatives are evaluated according to the generated value function.
Step 3: For the generated decision problem, the optimization problem (3.13) is solved to infer a set of marginal value functions compatible with the provided pairwise comparisons. The
transformation (3.9) is then used to normalize the value function. In all decision problems in the experiment, discretization of criteria is performed by defining a breakpoint for each distinct performance value.
To increase reliability and provide statistically invariant results, each decision setting is re- peated100 times, i.e. for each configuration 100 different decision problems are generated. Overall, 48000decision problems are solved. To measure quality of the results obtained by solving each decision problem, three different measures are used:
• γ, maximum value of (γm
j )T across all the interior breakpoints and all the criteria, i.e.
maxm,j(γmj )T,averaged over the100repetitions of the same configuration,
• MC, maximum number of changes in monotonicity direction across all the criteria (i.e. the extreme value across the marginal value functions), averaged over the100repetitions of the same configuration,
•ε∗Tminimal difference in standardized comprehensive values of alternatives, as an indicator of discriminatory power of the inferred preference model, averaged over the100repetitions of the same configuration.
3.4.2 Results
Summaries of the results are presented in Table 3.4.2. In this table, mean values ofγandMC show that the complexity of value functions are well-controlled. Maximum value of0.17for the variable γimplies that the upper bound of change in slope angle is less than10◦, according to Figure 3.3.10. Moreover, maximum value of2.46for the variableMC implies a nearly three times change in monotonicity, which can happen in the scenarios where4degree polynomials are used to simulate DM preferences. Moreover, the results show thatε∗Thas a large mean value with a narrow confidence interval at99%level of confidence. A large mean value ofε∗T implies
that the inferred preference models have high discriminatory power, and the narrow confidence interval indicates thatε∗Tis not greatly affected by the simulation design parameters.
Table 3.4.2: Summaries of experimental results
Mean Minimum Maximum 99% Confidence Interval
γ 0.0250 0.0 0.17 0.021 0.029
MC 0.270 0.0 2.46 0.225 0.316
ε∗T 0.104 0.01 0.25 0.098 0.111
The impact of simulation design parameters, i.e. number of alternatives, criteria, pairwise comparisons, and degree of the polynomials simulating DM preferences, on the three variables
γ,MC, andε∗Tis presented in Figure 3.4.1.
The figure shows that ε∗T decreases when the number of pairwise comparisons increases; however, it is not influenced by the remaining design parameters. In fact, when more pairwise comparisons are provided by the DM, it becomes more difficult to separate the alternatives. Complexity of inferred value functions, on the other hand, increases when a greater number of alternatives or pairwise comparisons are presented in the decision problem. In this case, in order to better separate the alternatives and maintain discriminatory power, the value function needs to hold a more complex form. However, a larger number of criteria provides greater degrees of freedom for the value function in order to separate the alternatives while maintaining the level of ε∗T. Therefore the complexity level of the value function decreases while its dis- criminatory power is maintained. We need to emphasize that in our experiment the criteria are assumed to be independent. The results might be different in the presence of correlated criteria and noisy data. This requires a separate investigation using a comprehensive set of real data.
Further analysis demonstrated an interaction effect between pand M. The analysis shows that for higher values ofM, the association between the complexity in the inferred value func-
Figure 3.4.1: The impact of numbers of alternatives (N), criteria (M), pairwise comparisons (p), and degree of polynomials employed to simulate DM preferences (D) on γ,MC, andε∗T
tions,MC, and the amount of supplied preferences,p, diminishes. In other words, the effect of
ponMCis weaker for higher values ofM. This is indicated in Figure 3.4.2.
Most importantly, Figure 3.4.1 shows that complexity of value functions increases by the de- gree of polynomials. The figure demonstrates that when linear preference models are employed to simulate DM preferences (D = 1), in all the corresponding 12000decision problems the inferred value functions are monotonic (MC = 0) and linear (γ = 0). By increasingD, the complexity in judgment policy of the DM will be reflected in the inferred value functions.
In Figure 3.4.3, the two variables measuring complexity,γ andMC, are standardized with respect to their mean and standard deviation, and are plotted against degrees of polynomials.
Figure 3.4.2: The effect of ponMC diminishes by increasingM
The plot shows that both measures similarly reflect the complexity inherent in a preferential system of the DM.
Figure 3.4.3: Average value of standardized γ andMC versus different degrees of polyno- mials employed to simulate DM preferences