The relative duration has a value between 0 (no contribution) and 1 (entire frame covered by one speech interval of that speaker), and can be used as a confidence score for the a/p value obtained for that frame and speaker: if a speaker's relative duration is low, as a result of minimal contribution, such as a single one syllable back-channeling utterance, it is possible to obtain extremely high or low values for some features. The thresholds depend on the frame length, as longer frame lengths reduce the variance more than shorter frame length. In such cases, points can be removed and replaced by either the overall mean or a linearly interpolated value. Interpolation is justified in this case as each point represents an entire frame rather than a single utterance and thus a linear model
can be fitted locally for frame averages (if the a/p feature can be assumed to have a normal
distribution, see section 7.4.1).
In a preliminary study based on three 30-minute long unconstrained dialogues (Kousidis et al. 2008), accommodation was evaluated by visual inspection of the plots for all four a/p features studied (pitch, pitch range, intensity, speech rate). The overall picture was that the two speakers were consistently following each other's prosodic variations over progressively longer time frames (20, 30 and 60 seconds), in all three dialogues. Some dialogue portions, such as the approximately 8
RelativeDuration=
∑
diminute-long extract shown in Figure 7.2, showed accurate “tracking” among the two speakers. Several instances of deviation from this behaviour were also found. A careful inspection of these frames showed that the deviation could be attributed to specific causes such as (a) non-standard speech style, such as laughing speech or extreme expressions of enthusiasm (e.g. “wow”), or (b) inaccurate measurements due to low relative duration. While (b) can be dealt with by increasing the frame length, with the consequences discussed in the previous paragraph, (a) is a natural occurrence in human dialogues and it should not be considered as an error. This means that speakers are not
obliged to converge (accommodate) in their a/p features, rather they do so spontaneously most of
the time.
The results in (Kousidis et al. 2008) showed that the TAMA method can capture accommodation of a/p features in spoken dialogues, in a continuous representation. In order to formally evaluate this, a statistical validation was sought, as described in the next section.
7.4 Statistical evaluation
As previously mentioned, the statistical method employed to evaluate inter-speaker accommodation was bi-variate time series analysis (Chatfield 1996). This type of analysis considers two time series and is mostly useful when there is indication that the values of one series are in some way dependent on the values of the other series. Time series is perhaps most popular in economics, but has a wide range of applications in such areas as biology, medicine, demographics, as well as engineering (Chatfield 1996).
In the special case of bi-variate time series analysis, one of the two series is considered as the predictor (or independent) variable, while the second series is called the predicted (or dependent) variable. For example, a raise of salaries among a population can be used as the independent variable to predict a raise in household spending. This is a classical example of an open loop system: the predicted variable cannot affect the predictor variable in any way. If however both variables are “equal”, one of the series is used as the predictor variable by convention. If the predicted variable is found to have an effect on the predictor variable, then feedback is present in the process, and the system is called close loop. The presence or absence of feedback can be assessed by means of bi-variate time series analysis. In the case of prosodic accommodation, one would expect an open-loop system for uni-directional accommodation (only one of the speakers converges towards the other), or a closed-loop system for bi-directional accommodation (both speakers converge).
7.4.1 Assumptions
Time series analysis considers stochastic processes. These processes have the property that they include a random, non-deterministic component. The purpose of the analysis is to de-compose the series into a deterministic and a non-deterministic component. For example, a simple random walk is a purely stochastic process: the series starts at zero and at every step it increases or decreases by 1 with equal probability. An added noise model is a stochastic process in which each observation is equal to the previous observation plus a random, uncorrelated noise component, εi, i.e. xi = xi-1 +
εi. Stochastic processes describe a wide range of phenomena in which the deterministic component
explains some of the variation in the observations, while the variation due to unknown factors is considered as the “random” component. In this study, the time series of a/p frame averages are considered as stochastic processes.
One of the basic assumptions in time series analysis is that of stationarity. In its strict form, stationarity requires that the joint probability distribution F( x1, … xN ) of the observations xi , of a
time series X = [ x1, x2, … xN ] is constant over time, i.e. F( x1, … xN ) = F ( x1+τ, … xN +τ).
However, since this assumption can rarely be satisfied in real applications, the assumption of weak stationarity (or second order stationarity) is more often used (Chatfield 1996). The latter only requires that the mean and variance of the observations xi need to be constant over time, so that the
correlation between two observations of a time series only depends on their time distance, τ. The frame averages can satisfy this assumption, if the true mean and variance of an a/p feature are considered as inherent to the speaker. However, a realization of the process of a/p feature variation during a dialogue does not necessarily exhibit a stationary form. In this case, it is required to transform a series to stationary by using standard techniques such as differencing or fitting a model to the series.
A second assumption is that of ergodicity. A stochastic process is said to be ergodic if its statistical properties (mean, variance) can be estimated from a single realization of the process: the observed time series is only one possible realization from the probability space comprising all possible realizations of the same underlying process. If the realization is sufficiently long, then the mean and variance of the observed variable can be deduced from this single series of observations. A TAMA a/p feature time series can be considered as a realization of a probability space comprising all dialogues among the two speakers that have the same content (utterance-wise): if the dialogues are sufficiently long, reasonable estimates of the speaker's mean value for a/p features can be obtained.. Conclusively, the underlying assumptions for the individual series of each speaker imply a decomposition of the observed a/p frame averages into a deterministic component (inherent to the
speaker) and a random component which encompasses all other causes of variation: utterance type, mood/emotion, and – most importantly – influence of other speaker. Inclusion of the latter cause of variation as a deterministic component is the purpose of bi-variate analysis.
7.4.2 Time series analysis
The first step in time-series analysis is plotting the data, as useful information can be inferred from the time series plots. Two such plots are shown in Figure 7.3. These plots have been obtained from a dialogue recording obtained with the “shipwrecked” experimental scenario (see section 6.4.3). The plots represent the entire duration of the dialogue (approximately 7 minutes). Considering both series in each plot, there is an indication that the two series are correlated, which implies the presence of inter-speaker influence (accommodation). Theoretically, there could be a third, underlying cause that affects both speakers in a similar way, but the only input to the process of dialogue is the two speakers themselves, and no other external factors exist. Thus, the only reasonable conclusion is that the similar movement is the result of influence from at least one of the series on the other.
In addition, when considering each series individually, there appears to be a certain degree of
autocorrelation: consecutive values in the (individual) series are dependent on preceding values of
the same series. This is partly a result of the moving average filtering introduced by the TAMA method: each point represents a frame 20 seconds long, 50% of which is shared with the immediately preceding frame. The second underlying cause of autocorrelation is that a speaker's a/p feature average is to an extent dependent on the past values, even if there is no overlap between frames: speakers may well maintain their speaking style over several frames, as is the case of points 3-6 in Figure 7.3a, or exhibit smooth transition from a low to a high value, which indicates that the values are dependent on the preceding values. A final indication of this autoregressive structure of the individual series, is that a value above the mean tends to be followed by another value above the mean. The mean in this case is equal to 1, as a result of the normalization method (see section 7.3.1).
Another observation that can be made particularly for the intensity plot (Figure 7.3b) is that the values appear to decline over time. This is an indication of a global decreasing trend in the series. A series exhibiting such a trend is not stationary, as the mean value changes over time. A simple method to transform this series to a stationary one is to use differencing, i.e. subtracting the preceding value from the current value in the series, creating a new series Y, with yi = xi -xi-1. In some cases, differencing more than once is required. If differencing d times is required in order to
achieve stationarity, then the series is called integrated of order d, denoted I(d).
(a)
(b)
Figure 7.3: Time series plots of (a) pitch and (b) intensity for two speakers (A,B). Normalized feature averages over 20 second frames with 50% overlap
A useful way of extracting information on individual series, is the sample autocorrelation function (ac.f), a good estimate of which is the correlogram (Chatfield 1996). A correlogram is a plot of correlation coefficients over a number of lags. A lag of 1 denotes the immediately preceding value of the series, a lag of 2 denotes the value before that, etc. The sample autocorrelation coefficient, rk at lag k is given by: