Results

(a) Comparing the PII and SNR with different covariance matrix estimators

In this section, we investigate how applying different covariance estimators influence the screening performance of the PII and SNR. To this aim, in cal- culating SNR and PII, each time we utilised one of the following covariance estimators: optimal estimator ˆCopt introduced through Equation (3.4.10), the

While calculating the thresholded and shrunk covariance estimator, ˆChs, we

considered three different values of tuning constant h = 0.01, 0.005, 0.001 de- noted by hs1, hs2 and hs3 respectively. As pointed out earlier, we illustrate the comparisons between covariance shrinkage methods by comparing specificity percentages of predictive power and SNR values while we fix the sensitivity at levels (100j/10)%; 1 _{≤ j ≤ 10. The higher the specificity, the more accurate} the screening. In other words, a high specificity percentage signifies that a high proportion of unimportant predictors have been detected and discarded correctly by the screening procedure. The following result show the specificity values when sensitivities are fixed.

Sensitivity-Specificity plots of Predictive Information Index (PII)

(a) (b)

Figure 3.5.1: Comparing PII performance with different covariance matrix estimators. Re-

sults obtained from 50 simulations where (p, n, J,|T |) = (2000, 88, 20, 10) for settings with

(a) Highly correlated B. (b) Weakly correlated B. Here, sho corresponds to PII built upon

ˆ

Copt estimator and hs1, hs2 and hs3 refers to PII built upon the thresholded and shrunk

estimator ˆChs with tuning constants h = 0.01, 0.005, 0.001 respectively. Also eig1 and eig2

correspond to the PII built upon ˆCeig1and ˆCeig2 estimators.

Simulation results in Figure 3.5.1 show that in both settings with highly correlated and weakly correlated B, the PII has a higher specificity when it is built upon the estimators ˆCeig1 and ˆCeig2. However, in setting 1 with highly

correlated B, Figure 3.5.1 (a), the specificity is significantly higher than the specificity obtained for setting with weakly correlated B, Figure 3.5.1 (b). The

reason that PII performs exactly the same for both ˆCeig1 and ˆCeig2 estimators

and same for other four estimators, is the particular simulation setting which is used here. Note that we cannot give a specific reason about why PII performs better with ˆCeig1 and ˆCeig2 estimators and we just rely on numerical results.

Justification of this phenomenon is a complicated task and beyond the scope of this thesis.

Sensitivity-Specificity plots of Signal-to-Noise Ratio (SNR)

(a) (b)

Figure 3.5.2: Comparing SNR performance with different shrinkage methods applied on co-

variance matrix. Results obtained from 50 replicates where (p, n, J,_{|T |) = (2000, 88, 20, 10)}

for settings with (a) Highly correlated B. (b) Weakly correlated B. Here, sho corresponds

to the SNR built upon ˆCopt estimator and hs1, hs2 and hs3 refers to SNR built upon the

thresholded and shrunk estimator ˆChs with tuning constants h = 0.01, 0.005, 0.001 respec-

tively. Also eig1 and eig2 correspond to the SNR built upon ˆCeig1and ˆCeig2 estimators.

Simulation results in Figure 3.5.2 reveal that when columns of B are highly correlated there is no significant difference in SNR performance using different covariance estimators whereas, in the setting with weakly correlated B the in- fluence of using different estimators is more noticeable. SNR screening based on the shrunk estimator ˆCopt and also thresholded and shrunk estimator ˆChs,

result in a more precise detection. These estimators shrink the overdispersed sample covariance eigenvalues more efficiently due to the particular shrinkage intensity placed on the target matrix which is discussed in Section 3.4.3. How- ever, the screening accuracy does not change much by using ˆChs instead of

ˆ

since we set the constant h to different values of _{{0.01, 0.001, 0.005}. Similar} to part (a) the reason that SNR has the same performance based on both ˆCopt

and ˆChs for all values of h, is a result of our particular simulation setting. In

settings that the noise part in data is more substantial, the difference becomes more noticeable (Zhang and Liu, 2015).

As it was mentioned earlier, our proposed method has explored the estima- tion of response covariance in each screening procedure. The results depicted above validate this declaration perfectly. Scenarios wherein there exist a high correlation in the coefficient matrix have much higher specificity level than the weakly correlated cases. The reason is that the simulated response inher- its some high correlations from the coefficient matrix and this high correlation magnifies the screening accuracy of both PII and SNR. The reason is that these high correlations provide more information leading to a higher accuracy. (b) Comparing PII with SNR performance

The optimum screening results for PII and SNR presented earlier are not based on the same covariance matrix estimator. In other words, PII performs better based on ˆCeig1 and ˆCeig2 estimators, whereas SNR does not. This makes it

difficult to make a faire comparison on the performance of these statistics. Therefore, to come to a final conclusion about which of these statistics gives more reliable screening results, we compared the specificity of each of these statistics based on all covariance estimators where we fixed the sensitivity at levels (100j/10)%; 1 ≤ j ≤ 10. Results presented in Figure 3.5.3, uncover that in both scenarios with high and weak correlation structures in coefficient matrix, the SNR-based screening procedure outperforms the PII-based screen- ing. Although the PII based on ˆCeig1 and ˆCeig2 estimators performs well, it

cannot gain the accuracy of the SNR all the time and specificity percentages justify the correctness of SNR screening in both scenarios with high and weak correlations. Accordingly, we opt for SNR as our screening statistic.

Box plots of Signal-to-Noise Ratio and Predictive Information Index

(a)

(b)

Figure 3.5.3: Comparing SNR with PII statistics based on different shrinkage methods

applied on covariance matrix. Results obtained from 50 replicates where (p, n, J,|T |) =

(2000, 88, 20, 10) for settings with (a) weakly correlated B and (b) Highly correlated B.

Here, sho corresponds to the statistics built upon ˆCopt estimator and hs1, hs2 and hs3

refers to the statistics built upon the thresholded and shrunk estimator ˆChs with tuning

constants h = 0.01, 0.005, 0.001 respectively. Also eig1 and eig2 correspond to the statistics

(c) Comparing SNR-based screening with likelihood-based marginal screening (LMS)

In this section, we compare the SNR-based screening based on ˆChs with

the likelihood-based marginal screening procedure introduced in Section 3.3. The comparison was accomplished based on the corresponding sensitivity and specificity values for each method. We considered the multivariate regression model (3.5.1) with 10 nonzero or active coefficients. In order to carry out the LMS, we fitted a single multivariate regression to each covariate xk; 1≤ k ≤ p

and the multivariate response variable denoted in the matrix form as Yn×J.

We estimated the corresponding J-dimensional coefficient vector ˆbk; 1≤ k ≤ p

which is expressed in the Equation (3.3.3).

Having found the estimates, we calculated the squared Euclidean norm of estimated coefficient vectors _kˆb1k22,· · · , kˆbpk22. We took the same approach as

previous Sections to threshold the norm values at different levels. We indicate the norm of 10 active predictors in an increasing order as _kˆb(1)k22 ≤ kˆb(2)k22 ≤

· · · , ≤ kˆb(10)k22. We thresholded the values of kˆbkk; 1 ≤ k ≤ p at levels of

kˆb(j)k22; 1 ≤ j ≤ 10 respectively. Therefore the reduced model corresponding

to each level 1_{≤ j ≤ 10 was obtained as} M_{(j) ={1 ≤ k ≤ p s.t kˆbkk}2

2 ≥ kˆb(j)k22; 1≤ j ≤ 10}, (3.5.2)

Since norm of active predictors are ordered increasingly, by thresholding at the level ofkˆb(1)k22, the selected subset of covariates contained all active covariates

which gave a sensitivity value of 100%. However, setting the threshold level at the largest value kˆb(10)k22 gave a sensitivity of 10%. This way, we obtained a

set of 10 different sensitivity values of 10%, 20%, 30%,_{· · · , 100%. We then cal-} culated the specificity values corresponding to each of these sensitivity values. We repeated this procedure for 50 simulations wherein data were simulated from a multivariate regression model according to the settings explained in Section 3.5.1. We then standardised the data according to what explained in Section 3.3 and set the number of active predictors to |T | = (10, 100).

(a) 0 25 50 75 100 50 100 Sensitivity Specificity SNR LMS (b) 0 25 50 75 100 50 100 Sensitivity Specificity SNR LMS

Figure 3.5.4: Box plots of specificity corresponding to SNR-based screening and likelihood-

based marginal screening (LMS). Results obtained from 50 replicates where (p, n, J,_{|T |) =}

(2000, 88, 20, 10) for settings with (a) Weakly correlated B (b) Highly correlated B.

Results illustrated in both Figure 3.5.4 and Figure 3.5.5 are another ev- idence of SNR-based screening efficiency. These results also reflect the ben- eficial effect of employing the covariance matrix of the response variable in enhancing the screening accuracy. Existing high correlations in the coefficient matrix, and as a result in the response variable, does not improve the LMS performance since the correlation is not taken into account in this type of screening. However, this high correlation can substantially increase the SNR- based screening accuracy.

In the following results which are depicted in Figure 3.5.5, we can see that in the setting with a larger number of active variables |T | = 100, identifying active predictors becomes more difficult in both approaches. This is a result of a higher correlation structure caused by a larger number of active predictors. This effect is reflected in having lower specificity values for setting with 100

active coefficients. This phenomenon called mask effect is studied in more detail in Section 5.7.2.1

(a)

(b)

Figure 3.5.5: Box plots of specificity corresponding to SNR-based screening and likelihood-

based marginal screening (LMS). Results obtained from 50 replicates where (p, n, J,|T |) =

(2000, 88, 20, 100) for settings with (a) Weakly correlated B (b) Highly correlated B.