Investigating In-domain Data Requirements for PLDA Training
Md Hafizur Rahman
1, David Dean
2, Ahilan Kanagasundaram
3, Sridha Sridharan
4Speech and Audio Research Laboratory
Queensland University of Technology, Brisbane, Australia
{m20.rahman1, a.kanagasundaram3, s.sridharan4}@qut.edu.au, [email protected]
Abstract
This paper analyzes the limitations upon the amount of in-domain (NIST SREs) data required for training a probabilistic linear discriminant analysis (PLDA) speaker verification system based on out-domain (Switchboard) total variability subspaces. By limiting the number of speakers, the number of sessions per speaker and the length of active speech per session available in the target domain for PLDA training, we investigated the rel-ative effect of these three parameters on PLDA speaker verifi-cation performance in the NIST 2008 and NIST 2010 speaker recognition evaluation datasets. Experimental results indicate that while these parameters depend highly on each other, to beat out-domain PLDA training, more than 10 seconds of ac-tive speech should be available for at least 4 sessions/speaker for a minimum of 800 speakers. If further data is available, considerable improvement can be made over solely out-domain PLDA training.
Index Terms: speaker verification, PLDA, in-domain, out-domain, in-domain data requirement
1. Introduction
Over the last few years the state-of-the-art text independent speaker verification has been greatly influenced by cosine sim-ilarity scoring (CSS) i-vector and probabilistic linear discrimi-nant analysis (PLDA) [1, 2], which resulted in excellent perfor-mance on recent NIST speaker recognition evaluations (SREs). But for any domain other than the standard NIST SREs, current speaker verification systems performance are not satisfactory if sufficient development/training speech data is not available in the target domain.
One of the key requirements to achieve state-of-the-art speaker verification performance is the use of huge amount of speech data during development [3]. But in real world appli-cation it is very difficult and often unrealistic to gather huge amount of speech data to develop a state-of-the-art speaker ver-ification system. Most of the previous research focused only on finding effect of limiting the number of speakers for PLDA speaker verification [4, 5]. But in reality not only the number of speakers but also the number of sessions per speaker and the active length of the speech have a direct influence on the devel-opment of speaker verification systems.
Recent studies focused on short utterances with differ-ent speaker verification techniques like joint factor analysis (JFA) [6], support vector machines (SVM) [7] and PLDA [8, 9] showed that speaker verification performance degrades when short utterances are used for evaluation. Recently, Kanagasun-daramet al. [10] studied the speaker verification performance with limited number of sessions, limited number of speakers
of microphone data [11] and proposed techniques for reliable estimation of PLDA parameters.
In 2013, the Speaker and Language Recognition Workshop at the Johns Hopkins University (JHU) [12], introduced the Domain Adaptation Challenge, designed to evaluate the sys-tem performance built on out-domain data (data not from NIST SREs). Preliminary results presented at that workshop showed that PLDA system trained on the SWB dataset produces higher error rate than a PLDA system trained on earlier NIST datasets. To cope with this challenge of adapting two different datasets in the i-vector domain, methods like inter dataset variability compensation (IDVC) [13, 14], with-in class covariance cor-rection (WCC) [15] have been proposed recently. Also,
Garcia-Romeroet al.[4] proposed four supervised PLDA domain
adap-tation techniques. They also proposed agglomerative hierarchi-cal clustering (AHC) [5] for unsupervised PLDA domain
adap-tation. Villalbaet al. [16] proposed Bayesian adaptation of
PLDA with limited data. In most of these previous research in-domain data were used to capture in-domain variation and adapt out-domain PLDA parameters to improve the system perfor-mance. However there has not been any detail investigation on in-domain data requirement for PLDA training instead of using huge out-domain data for training.
In this paper, we closely investigated the performance of LDA-projected, length-normalized, Gaussian PLDA (GPLDA) speaker verification systems when trained on limited in-domain training data. We analysed the performance of the system while decreasing the number of speakers, sessions/speaker and active length of sessions of the development data. We tried to deter-mine the minimum in-domain PLDA training data requirement to beat the out-domain baseline PLDA speaker verification per-formance.
This paper is structured as follows: Section 2 gives a brief overview of limited data development. Section 3 and 4 details i-vector feature extraction techniques, LDA and length normal-ized GPLDA system correspondingly. The experimental setup and corresponding results are given in Section 5, Section 6 and Section 7 respectively. Finally, Section 8 concludes the paper.
2. Limited Data Development
2.1. BackgroundRecent advances in speaker verification systems like i-vector based PLDA modelling has shown much promise that resulted in very low error rates. However, we definitely need a huge amount of speech data for faithful speaker verification. But in practical application it is not easy to find such dataset for sys-tem development. Also, we may have to use speaker verifica-tion system in adverse environmental condiverifica-tions, of which we may have very small amount of data. So, we can not hope to
Development
!!!!!Datasets! Extraction Feature
GMM Training (!,Σ) Total Variability Training LDA Training PLDA Modelling (!!,!,!) SWB!development! Limited!NIST!development!
Figure 1:Development phase of speaker verification system.
produce good verification result in every condition. One of the way to deal with this problem is to investigate the effect of lim-ited development data in the target domain. Limiting develop-ment data does not only mean limiting the number of speakers but also sessions per speaker and active length of each session. So, we need to find out the minimum number speaker as well as sessions per speaker and active length of each session required for marginal or faithful verification.
2.2. Reducing Data Requirements
It is obvious that system performance is going to degrade if we limit the amount of speech data available for development. So it is necessary to match the limited in-domain data with data-enriched out-domain data to produce best performance. One of the ways is to capture the in-domain and out-domain data variation in i-vector space and train PLDA parameters with out-domain data [13, 14, 17]. Another way is to adapt out-out-domain PLDA parameters with in-domain PLDA parameters [4, 5]. But instead of using huge amount of out-domain data for PLDA training, limited amount of in-domain data can be used to per-form better than out-domain PLDA speaker verification system. It is necessary to learn the minimum amount of in-domain data required for PLDA training that performs as well as out-domain PLDA system. The basic block diagram of this system is shown in Figure 1.
3. I-vector Feature Extraction
Instead of using two separate subspaces for speaker and chan-nel like JFA, Dehaket al.[18] proposed a single subspace based presentation of GMM super-vector called the i-vector approach. The reason behind this single subspace representation is some speaker discriminant information remain in the channel sub-space which could be useful for speaker modelling. A channel and speaker dependent GMM super-vector can be represented as follows,
µ=m+Tw (1)
where mis the speaker and session independent background
UBM super-vector, andTis a low rank matrix representing the
directions of variability across all data.w, is the total-variability factors which is normally-distributedN(0,1). A detail expla-nation of trainingT, and i-vector extraction process is described in [19, 18].
4. Linear Discriminant Analysis (LDA)
The LDA transformation finds new spatial axes that minimize the intra-class variance caused by channel effects and maximize0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 500 1000 1500 2000 2500 3000 3500 Number of sessions
Cumulative number of speaker
SWB NIST
Figure 2:Cumulative distribution of speakers of SWB and NIST
dataset.
the variance between speakers. Which are calculated as follows,
Sb = S X s=1 ns( ¯ws−w)( ¯¯ ws−w)¯ T (2) Sw = S X s=1 ns X 1=1 (wsi−ws)(w¯ s i −ws)¯ T (3)
whereS is the total number of speakers. ns is the number of
sessions of speakers.ws¯ is the mean i-vector for each speaker andw¯ is the mean of all speakers are defined by,
¯ ws= 1 ns ns X i=1 wsi (4) ¯ w= 1 N S X s=1 ns X i=1 wis (5)
whereN is the total number of sessions. The transformation
matrixGcan be determined through eigenvalue decomposition
ofSbv=τSwv. Finally, LDA projected i-vector can be
cal-culated as follows,
wLDA=GTw (6)
5. Length-normalized GPLDA System
In this paper we employed the length-normalized GPLDAsys-tem, which is introduced by Garcia-Romero et al.[2]. This
approach transforms the non-Gaussian i-vector behaviour into Gaussian i-vector behaviour. This technique consists of lin-ear whitening and length normalization. A speaker and chan-nel dependent whitened and length-normalized i-vector can be defined as,
wnormLDA−r=wLDAnorm+U1x1+U2x2r+r, (7)
where for given speaker recordings r = 1,2, ...R;
wnormLDA +U1x1is the speaker specific part andU2x2r+ris the channel specific part; The covariance matrix of the speaker part isU1U1Tand the covariance matrix of the channel part is U2U2T+Λ−1.
Scoring in GPLDA is calculated using the batch likelihood
ratio [20] between target i-vectors wLDA−target and test
i-vectorswLDA−test, as follows,
ln P(wLDA−target,wLDA−test|H1)
P(wLDA−target|H0)P(wLDA−test|H0) (8) whereH1: The speakers are same,H0: The speaker are differ-ent.
6. Experimental Setup
6.1. Training Datasets6.1.1. Out-domain Dataset (SWB)
The SWB dataset serves as out-domain dataset throughout our experiments. This dataset consists of standard telephone calls taken from Switchboard-I and Switchboard-II and consists of 1115 male, 1231 female speakers and 22,318 sessions data. Fig-ure 2 shows the cumulative distribution of SWB dataset. In our experiment we used full SWB dataset for out-domain PLDA training.
6.1.2. In-domain Dataset (NIST)
Our in-domain dataset consists of subset of telephone calls taken from NIST 2004, 2005 and 2006 SRE corpora. This dataset consists of 1467 male, 2004 female speakers and 30,317 sessions data. From Figure 2 it can be observed that most of speakers in the dataset have at least 8 sessions and only half of them has more than 8 sessions.
To create the data scarce condition of the PLDA training phase we created several sub-lists from our in-domain dataset. Sub-lists with same amount speakers contain different amount of sessions and sub-lists with same amount of sessions contain different amount speakers. For our experimental purpose we looked at a range of PLDA training speaker counts between 1400 and 100 (per gender) and session counts between 8 and 2 (per speaker).
6.2. System Setup
We extracted 13 dimensional feature-warped MFCCs with ap-pended delta coefficients from raw speech data and used two gender dependent UBMs with 512 Gaussian mixtures in our ex-periments. The UBMs were trained on telephone speech data from SWB. Baum-Welch statistics were calculated using these UBMs before training a gender-dependent total-variability sub-space.
An i-vector extractor withRw = 500is used which is
trained on switchboard data. Prior to GPLDA training we re-duced i-vector dimension to 150 using LDA projection. We used length-normalized i-vector for GPLDA [2] modelling with
i-vector centering and whitening. GPLDA parameters were
trained on subsets from NIST 2004, 2005 and 2006 SRE cor-pora. For GPLDA, best 120 eigenvoices were selected to pro-duce better speaker verification performance. Rather than using full precision matrix,Λ, we used the diagonal. For score nor-malization we used source normalisation (S-nornor-malization) [21] in our experiments and normalization dataset was formed by pooling random utterance from NIST 2004, 2005 and 2006 SRE corpora. For evaluation we used NIST 2010 10sec-10sec evalu-ation condition and the performance is evaluated using the equal error rate (EER).
7. Results and Discussions
10 20 30 40 50 60 70 80 90 100 17 18 19 20 21 22 23 24 25 26 27
Active speech length (sec)
EER(%) Number of speakers = 1400 8 sessionss/speaker 6 sessions/speaker 4 sessions/speaker 2 sessions/speaker SWB dev
Figure 3: Performance comparison of out- and in-domain
PLDA speaker verification systems on the common set of the NIST-2010 10sec-10sec evaluation conditions when out-domain PLDA is trained using standard SWB data and in-domain PLDA is trained using different number of sessions/speaker and active speech, performance are presented with respect to active speech length. 10 20 30 40 50 60 70 80 90 100 16 18 20 22 24 26 28
Active speech length (sec)
EER(%) 8 sessions/speaker 1400 speakers 1000 speakers 500 speakers 300 speakers 100 speakers SWB dev
Figure 4: Performance comparison of out- and in-domain
PLDA speaker verification systems on the common set of the NIST-2010 10sec-10sec evaluation conditions when out-domain PLDA is trained using standard SWB data and in-domain PLDA is trained using different number of speakers and active speech, performance are presented with respect to active speech length. Figure 3 compares the performance of out-domain and in-domain PLDA speaker verification on the common set of the NIST-2010 10sec-10sec evaluation conditions when out-domain PLDA is trained using standard SWB data and in-domain PLDA is trained using different lengths of active speech
2 3 4 5 6 7 8 16 18 20 22 24 26 28 30 32 34 36 Number of sessions/speaker EER(%)
Active speech length = 30 seconds
1400 speakers 1000 speakers 700 speakers 500 speakers 300 speakers 100 speakers SWB dev (a) 200 400 600 800 1000 1200 1400 16 18 20 22 24 26 28 30 32 34 36 Number of speakers EER(%)
Active speech length = 30 seconds
8 sessions/speaker 6 sessions/speaker 4 sessions/speaker 2 sessions/speaker SWB dev (b)
Figure 5: Performance comparison of out- and in-domain PLDA speaker verification on the common set of the NIST-2010
10sec-10sec evaluation condition when out-domain PLDA is trained using standard SWB data and in-domain PLDA is trained using 30 sec
active speech with different number of speakers and sessions/speaker. (a)Performance presented with respect to sessions/speaker,(b)
Performance presented with respect to speakers.
with limited number of sessions/speaker NIST data. It can be observed with the aid of Figure 3 that at least 4 sessions/speaker are required for in-domain PLDA training to perform better than an out-domain system. It was also observed that in-domain PLDA speaker verification achieves best performance when PLDA is trained using 30 seconds of NIST data. Subsequently, we experimented with the in-domain PLDA approach using dif-ferent length of active speech and difdif-ferent numbers of speakers in order to find out minimum utterance length. The performance comparison is shown in Figure 4. It can be observed that we re-quire at least 500 speaker with 30 seconds active speech length for PLDA training to estimate reliable PLDA parameters. Based upon results from Figures 3 and 4 we conclude that short length utterance are adequate to train PLDA though PLDA is trained using different number of sessions/speaker or speakers. In real world environments, though it is hard to collect huge amount of in-domain data, our experiments studies have found that short length (30sec) in-domain data is adequate for PLDA training.
Finally, the performance of out-domain PLDA speaker ver-ification is compared against in-domain PLDA approach when in-domain PLDA is trained using 30 sec NIST data with differ-ent number of speakers and sessions/speaker. Figure 5(a) illus-trates the performance comparison with respect to the number of sessions/speaker and Figure 5(b) illustrates the performance comparison of the same result with respect to number of speak-ers. It can be clearly observed that system performance goes down with limiting the number of sessions/speaker (Figure 5(a)) and limiting the number of speakers (Figure 5(b)) for in-domain PLDA training. It was also found that at least 800 speakers should be available for 4 sessions/speaker development, at least 500 speakers should be available for 6 sessions/speaker devel-opment and at least 500 speakers should be available for 8 ses-sions/speaker development. It was also observed that when PLDA is trained using 2 sessions/speaker, system achieves the worst performance as at least 4 sessions/ speaker data with at least 800 speakers is required for reliable PLDA parameter es-timation.
8. Conclusions
In this paper, we presented the results of in-domain data require-ment for faithful PLDA training. We performed experirequire-ments to find out how the PLDA system performance varies when PLDA is training using short utterance development data, and find out the minimum number of in-domain speakers as well as number of sessions/speaker required to produce acceptable speaker ver-ification performance. We found through experimental studies that 30 second utterances are adequate to train in-domain PLDA approach when evaluated on 10 second utterances. It was also found that for in-domain PLDA training, at least 800 speakers were required for 4 sessions/speaker development, and at least 500 speakers were required for 6 and 8 sessions/speaker devel-opment.
9. Acknowledgements
This project was supported by an Australian Research Council (ARC) Linkage grant LP130100110.
10. References
[1] S. J. Prince and J. H. Elder, “Probabilistic linear discriminant analysis for inferences about identity,” inComputer Vision, 2007. ICCV 2007. IEEE 11th International Conference on. IEEE, 2007, pp. 1–8.
[2] D. Garcia-Romero and C. Y. Espy-Wilson, “Analysis of i-vector length normalization in speaker recognition systems.” in Inter-speech, 2011, pp. 249–252.
[3] T. Kinnunen and H. Li, “An overview of text-independent speaker recognition: from features to supervectors,”Speech communica-tion, vol. 52, no. 1, pp. 12–40, 2010.
[4] D. Garcia-Romero and A. McCree, “Supervised domain adapta-tion for i-vector based speaker recogniadapta-tion,” inAcoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Con-ference on. IEEE, 2014, pp. 4047–4051.
[5] D. Garcia-Romero, A. McCree, S. Shum, N. Br¨ummer, and C. Va-quero, “Unsupervised domain adaptation for i-vector speaker
recognition,” Speaker and Language Recognition Workshop (IEEE Odyssey), pp. 260–264, 2014.
[6] R. J. Vogt, B. J. Baker, and S. Sridharan, “Factor analysis sub-space estimation for speaker verification with short utterances,” in INTERSPEECH, 2008, pp. 853–856.
[7] M. McLaren, R. Vogt, B. Baker, S. Sridharan, and S. Sridharan, “Experiments in SVM-based speaker verification using short ut-terances,” inProc. Odyssey Workshop, 2010, pp. 83–90. [8] A. Kanagasundaram, R. J. Vogt, D. B. Dean, and S. Sridharan,
“PLDA based speaker recognition on short utterances,” inThe Speaker and Language Recognition Workshop (Odyssey 2012). ISCA, 2012.
[9] A. Kanagasundaram, D. Dean, and S. Sridharan, “Short utterance PLDA speaker verification using SN-WLDA and variance mod-elling techniques,”In SST 2014 15th Australasian International Conference on Speech Science and Technology, 3 - 5 December 2014, Christchurch, New Zealand, 2014.
[10] A. Kanagasundaram, D. Dean, and S. Sridharan, “Improving PLDA speaker verification with limited development data,” in Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. IEEE, 2014.
[11] A. Kanagasundaram, D. Dean, J. Gonzalez-Dominguez, S. Srid-haran, D. Ramos, and J. Gonzalez-Rodriguez, “Improving the PLDA based speaker verification in limited microphone data con-ditions,” inINTERSPEECH, 2013, pp. 3674–3678.
[12] Domain Adaptation Challenge 2013, Johns Hopkins Uni-versity, 2013 speaker recognition workshop. Available on-line: http://www.clsp.jhu.edu/workshops/archive/ws13-summer-workshop/groups/spk-13/.
[13] H. Aronowitz, “Compensating inter-dataset variability in PLDA hyper-parameters for robust speaker recognition,”Speaker and Language Recognition Workshop (IEEE Odyssey), pp. 282–286, 2014.
[14] H. Aronowitz, “Inter dataset variability compensation for speaker recognition,” in Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. IEEE, 2014, pp. 4002–4006, 2014.
[15] O. Glembek, J. Ma, P. Matejka, B. Zhang, O. Plchot, L. Burget, and S. Matsoukas, “Domain adaptation via within-class covari-ance correction in i-vector based speaker recognition systems,” in Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. IEEE, 2014, pp. 4032–4036. [16] J. Villalba and E. Lleida, “Bayesian adaptation of PLDA based
speaker recognition to domains with scarce development data,” in Odyssey 2012-The Speaker and Language Recognition Workshop, 2012.
[17] A. Kanagasundaram, D. Dean, and S. Sridharan, “Improving out-domain PLDA speaker verification using unsupervised inter-dataset variability compensation approach,” Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Con-ference on, 2015.
[18] N. Dehak, P. Kenny, R. Dehak, P. Dumouchel, and P. Ouel-let, “Front-end factor analysis for speaker verification,”Audio, Speech, and Language Processing, IEEE Transactions on, vol. 19, no. 4, pp. 788–798, 2011.
[19] P. Kenny, P. Ouellet, N. Dehak, V. Gupta, and P. Dumouchel, “A study of interspeaker variability in speaker verification,”Audio, Speech, and Language Processing, IEEE Transactions on, vol. 16, no. 5, pp. 980–988, 2008.
[20] P. Kenny, “Bayesian speaker verification with heavy tailed priors,” inSpeaker and Language Recognition Workshop (IEEE Odyssey), 2010.
[21] S. Shum, N. Dehak, R. Dehak, and J. R. Glass, “Unsuper-vised speaker adaptation based on the cosine similarity for text-independent speaker verification.” inOdyssey, 2010, p. 16.
