Speech signals carry an enormous amount of information apart from the intended message. Researchers agree that speech signals also carry vital information regard-ing the emotional state of the speaker [27]. However, researchers are still unde-cided over the right set of features of the speech signals, which can represent the underlying emotional state. This section contains the details of feature sets which are heavily used so far in SER research and performed well in the classification stage. There are three prominent categories in speech features used in SER : (1) the prosodic features, (2) the spectral or vocal tract features, and (3) the excitation source features. The following sub-sections will discuss these features in detail.
2.3.1 Prosody Speech Features
The human speech production system is a very sophisticated apparatus. Humans, while speaking can utilize different tools available in this system for varying the duration, pitch, and intensity of the spoken utterances, called prosody alteration, to express their various feelings in words. Prosody features are the characteristics of the speech sound generated by the human speech production system, for exam-ple, pitch or fundamental frequency (F0) and energy. Researchers used different derivatives of pitch and energy as various prosody features [130–132]. These are also called continuous features and can be grouped into the following categories
[35,76, 93,94]: (1) pitch-related features; (2) formants features; (3) energy-related features; (4) timing features; and (5) articulation features. Several studies tried to establish the relationship between prosody speech features and the under-lying patterns of different emotions [33–37,94,133,134].
Most of the early studies of SER considered the fundamental frequency (F0) as the most prominent attribute which represents different emotions [86,87,135–
137]. After that, other important features for SER like energy, speech duration, for-mants are also introduced by researchers along with F0and their derivatives [61, 122,135,138,138–140]. Several studies tried to establish the relationship be-tween prosody speech features and the underlying patterns of different emotions [33–37,133].
2.3.2 Excitation Source Features
The features used to represent glottal activity, mainly the vibration of glottal folds, are known as the source or excitation source features. These are also called voice quality features because glottal folds determine the characteristics of the voice.
Some researchers believe that the emotional content of an utterance is strongly related to voice quality [38,94,135]. Speakers have their unique voice quality signature, and different voice qualities can convey relevant information like inten-tions, attitudes, and emotions.
Human vocal folds vibrate to generate quasi-periodic impulse-like excitation in the vocal tract system during speech production. Glottal vibrations or excita-tion source signal can be extracted by using inverse filtering (IF) technique on the speech signal to remove the vocal tract contribution [141]. The signal received after inverse filtering speech signal is also called linear prediction (LP) residuals, which contain only higher order relations. The relations present among the distant speech samples are treated as higher-order relations, whereas the adjacent relations are treated as lower-order relations.
Cowie et al. [94] grouped acoustic correlates, related to voice quality, are grouped into the following categories. 1. voice level: signal amplitude, energy and duration
have been shown to be reliable measures of voice level; 2. voice pitch; 3. phrase, phoneme, word and feature boundaries; 4. temporal structures.
Voice quality measures for a speech signal includes harshness, breathiness, and tenseness. The relation of voice quality features with different emotions is not a well-explored area, and researchers have produced contradictory conclusions.
For example, Scherer [38] associated anger with tense voice whereas Murray and Arnott [35] associated anger with a breathy voice. Many SER researchers [56, 142,143] extracted features from the glottal waveform for emotion classifications.
However, deriving the accurate transfer function by canceling out the effect of the vocal tract system, and obtaining the closed phase duration of the glottal cycle [144,145] is a challenge.
2.3.3 Spectral Features
Spectral features are the characteristics of various sound components generated from different cavities of the vocal tract system. They are also called segmental or system features. Spectral features extracted in the form of 1. ordinary linear pre-dictor coefficients (LPC) [39], 2. one-sided autocorrelation linear prepre-dictor coef-ficients (OSALPC) [146], 3. shorttime coherence method (SMC) [147], and 4.
least-squares modified Yule–Walker equations (LSMYWE) [42].
However, the extracted spectrum is often needed to pass through a bank band-pass filters [93]. The filters’ bandwidths are usually evenly distributed with re-spect to a suitable nonlinear frequency scale such as the Bark scale [40], the Mel-frequency scale [40,148], the modified Mel-frequency scale, and the ExpoLog scale [42] because a human being does not perceive pitch in a linear scale.
Researchers claim that the sequence of shapes of the vocal tract system also carries emotion-specific information, along with the information related to the sound unit [29]. The spectrum characterized by formant frequencies and their respective bandwidths is extensively analyzed for emotional speech [36,87,149].
It is inferred that the first formant(F1) for angry speech has a higher mean than the neutral speech [87]. Researchers [87,150,151] also observed association
among changes in the spectral component and glottal source excitation; for ex-ample, higher F0in angry speech tend to have smaller F1amplitudes. Some studies [58,152,153] have shown that properties of formants like magnitude and shift vary across vowels for different emotional states.
There is a particular type of spectral features called the cepstral features which are extensively used by SER researchers. Cepstral features can be derived from the corresponding linear features like linear predictor cepstral coefficients (LPCC) is derived from LP. Mel-frequency cepstral coefficients are one such cepstral feature which along with its various derivatives is widely used in SER research [43,72, 154–156].
2.3.4 Deep Feature Learning Methods
The advent of deep learning has proven to be a paradigm shift towards looking at feature extraction stage in the machine learning process. The ability of DL meth-ods to learn underlying representations from data has already proven to be very robust to variability in data such as speech signals [157,158]. One such feature extraction technique Generalized Discriminant Analysis (GerDA) is proposed by Stuhlsatz et al. [159] to learn discriminative features of low dimension. Han et al.
[160] used DNN to extract high-level features from raw data. Researchers [71, 161] also employ a 1-layer CNN trained with a Sparse Auto-encoder (SAE) to extract affective features for speech emotion recognition.
End-to-end deep learning systems are becoming very popular among SER re-searchers, where raw speech is fed into a deep neural network model. These end-to-end SER models usually combine CNN with RNN where the CNN layer is re-sponsible for feature learning. For example, some researchers [69,162] proposed end-to-end models where they stacked CNN layers before Long Short-Term Mem-ory (LSTM) layers. However, researchers suggested that shallow 1-layer or 2-layer CNN structures may not be able to learn effectively the affective features which are discriminative enough to distinguish the subjective emotions [70]. So, a deep structure is recommended.
Drawbacks of Deep Feature Learning
Feature set learned using DL methods usually needs a very high number of at-tributes to be provided as input. That means global speech features such as energy, F0, needs to be further broken down into different derivatives. Moreover, feature sets learned through DL methods usually becomes very high in dimension, and it runs into thousands sometimes [160,163].