Singular Value Decomposition

In contrast to the χ2 and IG that rely on evaluating term importance, Singular Value Decomposition (SVD) (Wall et al. 2003) ensures that the data variance of the higher dimensional feature space is preserved in the lower feature dimensional space. Further- more, SVD utilizes the unsupervised approach that does not require any knowledge about categories of the instances.

For instance, with the BoW approach for feature representation, we might have a sparse representation consisting ofddimensions. Using the SVD approach we can generate ˜X

the approximation of the original feature space X such that ˜X has ˜d feature variables, where ˜d < d. One of the most important phases before employing the SVD approach is to go through the pre-processing phase which involves data translation and data scaling.

Data translation is a procedure known as mean centering because it translates the data center to the origin. For a given set of feature vectorsX ={x(1),x(2), . . . ,x(n)}, where x(i) = [x(₁i), x(₂i), . . . , x(_di)]∈_Rd_{, data translation of each feature vectorx}(i) _{is performed} as follows x(i) ←x(i)−µ, (2.8) where µ= 1 n n X i=1 x(i). (2.9)

Data scaling, also known as variance scaling, normalizes each feature variablex(_ji) of a feature vector x(i) to ensure that all feature variables have the same variance. This is done by dividing each feature variable by its variance σ_j2 calculated over the data set,

i.e., x(_ji) ← x (i) j σ2 j , (2.10) where σ2_j = 1 n−1 n X i=1 (x(_ji))2. (2.11)

The normalization is more important when feature variables have varying scales. For instance, in the text processing problem where the BoW is employed for feature rep- resentation, the value of each feature variable x(_ji) corresponds to the total counts of a termtj in a documentEi. This can result in a case where all the feature variables have a different scaling. Hence, the data pre-processing phase plays an important role.

One of the advantages of SVD is that the feature vector x(i) ∈_Rd _{can be transformed} into a lower dimensional feature vector ˜x(i)∈_Rd˜_{, where ˜d≤d, ensuring that the most}

important information that explain x(i) is preserved in ˜x(i). This is done by initially decomposing X as:

X=UΛVT, (2.12)

where the symbol U represents a left singular vector ofX, which is given by

U = h

u(1),u(2), . . . ,u(n) i

∈Rn×n.

The symbol V is given by

V =hv(1)_,v(2)_{, . . . ,v}(d)i_∈Rd×d

and it is known as a right singular vector belonging to X. u(i) and v(i) are the unit orthonormal vectors. Λ is an n×ddiagonal matrix consisting of the singular values. The ˜X can be generated by assigning all low ranked singular values in the the singular matrix Λ to zero values. This means we have

˜

X ≈X, (2.13)

where all the low ranked columns in ˜X consist of zero values. If we discard all zero columns, then ˜X will be decomposed as:

˜

X =U_d˜Λ_d˜V_d˜T, (2.14)

where ˜d < d.

One of the mostly used multivariate statistical technique in the text processing problems which is associated with the SVD is known as the Principal Component Analysis (PCA) (Abdi & Williams 2010). The main aim of this technique, PCA, is to transform the fea- ture representationX into the new space of uncorrelated variables. Those uncorrelated variables are known as principal components.

The SVD approach can be employed to calculate those principal components (Xie et al. 2017). PCA is one of the feature reduction technique that has been employed previously to ensure that hidden patterns about feature representations generated are revealed. That has let to acceptable performance of classifiers for identifying spam emails and other malicious activities throughout the Internet.

Masud et al. (2007) employed PCA for reducing feature size of the vector space model which was constructed for email worms detection. The email worm is a code used for malicious purposes by infecting a computer device and also distributing its copy throughout the Internet. In this study (Masud et al. 2007) authors make use of PCA to extract meaningful features which will be helpful for identifying email worms.

Features being used includes availability of the

• HTML tags;

• images to avoid attacks through buggy image processor;

• hyper-links leading to the infected sites;

• binary attachments.

Masud et al.(2007) further explained that PCA can be used for finding hidden patterns of the data. In their study, Masud et al. (2007), a decision tree C4.5 was employed for feature reduction and it is found to lead to better performance in identifying email worms compared to when PCA alone is used for feature reduction. Classifiers which were taken into consideration includes SVM and Naive Bayes (NB).

Goodman et al. (2015) in contrast to Masud et al. (2007) employed PCA for feature reduction. The objective of this study (Goodman et al. 2015) was to tackle two cyber- crime problems, which are spam detection in Short Message Service (SMS) and malicious movements detection in the network. This study (Goodman et al. 2015) is related to email spam detection because in both email spam detection problem and SMS spam detection problem, message content can be used for feature extraction. The generated features can be used to train learning models that will be able to identify patterns of spam related messages and legitimate messages.

In contrast toMasud et al.(2007), for feature representationGoodman et al.(2015) used anomaly scores. Use of anomaly scores has resulted in an improved Receiver Operating Characteristic curve (ROC) (Wright 2005) when employed on spam detection problem. One of the major disadvantages in this study (Goodman et al. 2015) is that N-gram approach was used for feature extraction where N was set to 2 and 3. The major problem is that SMS users these days make use of slangs and many noisy words. This can result in a case where we have a very sparse representation. Furthermore, words which are more related to each other with meaning although are different with character co-occurrence may not be captured. This becomes a challenge for feature reduction techniques to determine more robust features with the evolving vocabulary in the SMS platform.