Assessing the quality of VoIP transmission
affected by playout buffer scheme
and encoding scheme
Communications Network Research Institute
Dublin Institute of Technology
Wireless Cluster Workshop, Maynooth
Application mechanisms used to improve QoS
Network Factors Network Packet Loss Congestion Network Variable delay Network Fixed delay Jitter buffer End-to-end Packet Loss End-to-end Delay Codec Perceived Quality Distortion Delay Echo path & control Terminal Factors Echo Topology Clockresponsibility for packet loss, delay and jitter is delegated to the end-points
as long as the problems in the network remain below a certain level, actions at
the end-systems can greatly mitigate their effect:
ow bit-rate codecs
Packet loss concealment - PLC
echo cancellation
Application mechanisms used to improve QoS
Network Factors Network Packet Loss Congestion Network Variable delay Network Fixed delay Jitter buffer End-to-end Packet Loss End-to-end Delay Codec Perceived Quality Distortion Delay Echo path & control Terminal Factors Echo Topology Clock
transmission impairments (delay, loss, jitter) affect conversational speech
actions at the end-systems can greatly mitigate their effect:
low bit-rate codecs
Packet loss concealment - PLC
echo cancellation
Application mechanisms used to improve QoS
Network Factors Network Packet Loss Congestion Network Variable delay Network Fixed delay Jitter buffer End-to-end Packet Loss End-to-end Delay Codec Perceived Quality Distortion Delay Echo path & control Terminal Factors Echo Topology Clock
transmission impairments (delay, loss, jitter) affect conversational speech
actions at the end-systems can greatly mitigate their effect:
Application mechanisms used to improve QoS
Network Factors Network Packet Loss Congestion Network Variable delay Network Fixed delay Jitter buffer End-to-end Packet Loss End-to-end Delay Codec Perceived Quality Distortion Delay Echo path & control Terminal Factors Echo Topology Clock
transmission impairments (delay, loss, jitter) affect conversational speech
actions at the end-systems can greatly mitigate their effect:
Application mechanisms used to improve QoS
Network Factors Network Packet Loss Congestion Network Variable delay Network Fixed delay Jitter buffer End-to-end Packet Loss End-to-end Delay Codec Perceived Quality Distortion Delay Echo path & control Terminal Factors Echo Topology Clock
transmission impairments (delay, loss, jitter) affect conversational speech
actions at the end-systems can greatly mitigate their effect:
low bit-rate codecs
Application mechanisms used to improve QoS
Network Factors Network Packet Loss Congestion Network Variable delay Network Fixed delay Jitter buffer End-to-end Packet Loss End-to-end Delay Codec Perceived Quality Distortion Delay Echo path & control Terminal Factors Echo Topology Clock
transmission impairments (delay, loss, jitter) affect conversational speech
actions at the end-systems can greatly mitigate their effect:
low bit-rate codecs
packet loss concealment - PLC
Application mechanisms used to improve QoS
Network Factors Network Packet Loss Congestion Network Variable delay Network Fixed delay Jitter buffer End-to-end Packet Loss End-to-end Delay Codec Perceived Quality Distortion Delay Echo path & control Terminal Factors Echo Topology Clock
transmission impairments (delay, loss, jitter) affect conversational speech
actions at the end-systems can greatly mitigate their effect:
low bit-rate codecs
packet loss concealment - PLC
echo cancellation
Application mechanisms used to improve QoS
Network Factors Network Packet Loss Congestion Network Variable delay Network Fixed delay Jitter buffer End-to-end Packet Loss End-to-end Delay Codec Perceived Quality Distortion Delay Echo path & control Terminal Factors Echo Topology Clock
transmission impairments (delay, loss, jitter) affect conversational speech
actions at the end-systems can greatly mitigate their effect:
low bit-rate codecs
packet loss concealment - PLC
Application mechanisms used to improve QoS
Network Factors Network Packet Loss Congestion Network Variable delay Network Fixed delay Jitter buffer End-to-end Packet Loss End-to-end Delay Codec Perceived Quality Distortion Delay Echo path & control Terminal Factors Echo Topology Clock
transmission impairments (delay, loss, jitter) affect conversational speech
actions at the end-systems can greatly mitigate their effect:
low bit-rate codecs
packet loss concealment - PLC
echo cancellation
Size of de-jitter buffer at VoIP receiver is critical!
s
y
Many adaptive playout algorithms:
should achieve the best possible trade-off
between loss delay
Size of de-jitter buffer at VoIP receiver is critical!
Too small
-> decreases e2e delay but increases late packet loss
Too large
-> decreases late packet loss but increases e2e delay
Many adaptive playout algorithms:
should achieve the best possible trade-off
between loss delay
should
maximize user satisfaction
Size of de-jitter buffer at VoIP receiver is critical!
Too small
-> decreases e2e delay but increases late packet loss
Too large
-> decreases late packet loss but increases e2e delay
Many adaptive playout algorithms:
should achieve the best possible trade-off
between loss delay
Size of de-jitter buffer at VoIP receiver is critical!
Too small
-> decreases e2e delay but increases late packet loss
Too large
-> decreases late packet loss but increases e2e delay
Many adaptive playout algorithms!
should achieve the best possible trade-off
WE NEED A METHOD to evaluate them!
should
maximize user satisfaction
Playout delay adjustment (when?)
1. fixed throughout the whole
session;
2. adjusted during silence
periods (per talkspurt)
[Ramjee ‘94, Moon ’98, …]
;
3. adjusted within talkspurts
(per packet) using
packet scaling (TSM
Playout delay adjustment (how?)
Reactive algorithm
[Ramjee ‘94]
Histogram-based algorithm
[Moon ’98]
1
1
:
: network delay of a current packet
_
_
(1
)
_
_
(1
)
_
:
_
_
: playout delay of a current talkspurt
i
i i i
i i i i
i i i
delay
d est
d est
delay
v est
v est
d est
dela
INPUT
OUTP
y
p
d est
v est
UT
α
α
α
α
β
− −= ⋅
+ −
⋅
= ⋅
+ −
⋅
−
=
+
d
a: Playout delay
•
Past 10 – 1000 delays are stored
in a circular array, both the array
and the histogram get updated
once a new packet is received.
•
Loss rate specified by user.
•
Playout time chosen from the
histogram according to the desired
loss rate.
Problem to solve
management of the playout buffer is not specified by any standard and
is vendor specific
many fixed and adaptive playout schemes exist, each with a different
parameter set
information on the implementation of the playout buffer in commercial
applications is practically nonexistent (it has a strategic value from
vendor perspective)
WE NEED A METHOD TO EVALUATE THEM FROM AN END USER
How?
“Listening-only” tests do not take into account delay impairments
PESQ do not take into account end-to-end one-way delay in its rating
-> NOT RECOMMENDED to asses the effect of CONVERSATIONAL delay
E-model relies on static transmission parameters and do not take into
account dynamics of changing transmission impairments (delay,loss)
PESQ+E-model hybrid solution requires a reference speech signal and
does not work in real time
old statistical methods focus on “late packet loss vs. buffering delay”
trade-off only and do not provide a direct link to conversational speech
quality
Trade-off: late packet loss/buffering delay
Trade-off: late packet loss/buffering delay
buffering delay
Trade-off: late packet loss/buffering delay
late loss
buffering delay
Trade-off: late packet loss/buffering delay
late loss
buffering delay
Fixed
LO SS BUFFERING DELAYLOSS-DELAY TRADE-OFF
Trade-off: late packet loss/buffering delay
late loss
buffering delay
Fixed
LO SS BUFFERING DELAYLOSS-DELAY TRADE-OFF
Trade-off: late packet loss/buffering delay
late loss
buffering delay
Adaptive
Fixed
LO SS BUFFERING DELAYLOSS-DELAY TRADE-OFF
Trade-off: late packet loss/buffering delay
late loss
buffering delay
Adaptive
Fixed
LO SS BUFFERING DELAYLOSS-DELAY TRADE-OFF
Trade-off: late packet loss/buffering delay
late loss
buffering delay
late loss
Adaptive
Fixed
LO SS BUFFERING DELAYLOSS-DELAY TRADE-OFF
Trade-off: late packet loss/buffering delay
late loss
buffering delay
late loss
Adaptive
Fixed
LO SS BUFFERING DELAYLOSS-DELAY TRADE-OFF
Trade-off: late packet loss/buffering delay
late loss
buffering delay
late loss
Adaptive
Fixed
LO SS BUFFERING DELAYLOSS-DELAY TRADE-OFF
Trade-off: late packet loss/buffering delay
late loss
buffering delay
late loss
Adaptive
Fixed
LO SS BUFFERING DELAYLOSS-DELAY TRADE-OFF
good algorithm SHOULD:
•
achieve the best possible
loss-delay trade-off
Trade-off: late packet loss/buffering delay
late loss
buffering delay
late loss
Adaptive
Fixed
LO SS BUFFERING DELAYLOSS-DELAY TRADE-OFF
good algorithm SHOULD:
•
achieve the best possible
loss-delay trade-off
•
be well tuned to
maximize user
satisfaction
E-model, speech transmission categories,
user satisfaction
0 5 10 15 20 0 10 20 30 40 5060 Equipment Impairment Ie vs. Packet Loss
packet loss [%] Ie G.711 w/o PLC G.723.1 GSM G.729A G.711 Bursty Loss w. PLC G.711 w. PLC Random Loss
Transmission rating factor R = (R
0– I
s)– I
d(echo,delay)– I
e(codec, loss)+ A
Reduced to transmission impairments: R = 94.15 - I
d(echo,delay)– I
e(codec, loss)Poor Low Medium High Best Speech transmission 60-50 70-60 80-70 90-80 94.15 -90 R-value 0 100 200 300 400 500 0 10 20 30 40 50 60 70
80 Delay Impairment Id vs. Delay
m2e delay [ms] Id TELR=45dB TELR=50dB TELR=55dB TELR=60dB TELR=65dB
Predicting time varying speech transmission quality
800 820 840 860 880 900 200 250 300 pl a yo ut d el a y [ ms ] PLAYOUT DELAY 8000 820 840 860 880 900 10 20 lo ss [ %] time [s] PACKET LOSS late lost packet delays playout delayaverage playout delay
8000 820 840 860 880 900 10 20 Id DELAY IMPAIRMENT 8000 820 840 860 880 900 50 Ie EQUIPMENT IMPAIRMENT 60 80 100 R QUALITY vs. TIME
1. Playout buffer module calculates
playout delays (m2e), average
playout delays…
2. … and resulting average packet
loss values for fixed time
windows
3. E-model simulator calculates
corresponding delay
impairments Id …
4. ... and equipment impairments
Ie…
Predicting time varying speech transmission quality
800 820 840 860 880 900 200 250 300 pl a yo ut d el a y [ ms ] PLAYOUT DELAY 8000 820 840 860 880 900 10 20 lo ss [ %] time [s] PACKET LOSS late lost packet delays playout delayaverage playout delay
8000 820 840 860 880 900 10 20 Id DELAY IMPAIRMENT 8000 820 840 860 880 900 50 Ie EQUIPMENT IMPAIRMENT 100 QUALITY vs. TIME
1. Playout buffer module calculates
playout delays (m2e), average
playout delays…
2. … and resulting average packet
loss values for fixed time
windows
3. E-model simulator calculates
corresponding delay
impairments Id …
4. ... and equipment impairments
Ie…
Predicting time varying speech transmission quality
800 820 840 860 880 900 200 250 300 pl a yo ut d el a y [ ms ] PLAYOUT DELAY 8000 820 840 860 880 900 10 20 lo ss [ %] time [s] PACKET LOSS late lost packet delays playout delayaverage playout delay
8000 820 840 860 880 900 10 20 Id DELAY IMPAIRMENT 8000 820 840 860 880 900 50 Ie EQUIPMENT IMPAIRMENT 60 80 100 R QUALITY vs. TIME
1. Playout buffer module calculates
playout delays (m2e), average
playout delays…
2. … and resulting average packet
loss values for fixed time
windows
3. E-model simulator calculates
corresponding delay
impairments Id …
4. ... and equipment impairments
Ie…
Predicting time varying speech transmission quality
800 820 840 860 880 900 200 250 300 pl a yo ut d el a y [ ms ] PLAYOUT DELAY 8000 820 840 860 880 900 10 20 lo ss [ %] time [s] PACKET LOSS late lost packet delays playout delayaverage playout delay
8000 820 840 860 880 900 10 20 Id DELAY IMPAIRMENT 8000 820 840 860 880 900 50 Ie EQUIPMENT IMPAIRMENT 100 QUALITY vs. TIME
1. Playout buffer module calculates
playout delays (m2e), average
playout delays…
2. … and resulting average packet
loss values for fixed time
windows
3. E-model simulator calculates
corresponding delay
impairments Id …
4. ... and equipment impairments
Ie…
Predicting time varying speech transmission quality
800 820 840 860 880 900 200 250 300 pl a yo ut d el a y [ ms ] PLAYOUT DELAY 8000 820 840 860 880 900 10 20 lo ss [ %] time [s] PACKET LOSS late lost packet delays playout delayaverage playout delay
8000 820 840 860 880 900 10 20 Id DELAY IMPAIRMENT 8000 820 840 860 880 900 50 Ie EQUIPMENT IMPAIRMENT 60 80 100 R QUALITY vs. TIME
1. Playout buffer module calculates
playout delays (m2e), average
playout delays…
2. … and resulting average packet
loss values for fixed time
windows
3. E-model simulator calculates
corresponding delay
impairments Id …
4. ... and equipment impairments
Ie…
Predicting time varying speech transmission quality
800 820 840 860 880 900 200 250 300 pl a yo ut d el a y [ ms ] PLAYOUT DELAY 8000 820 840 860 880 900 10 20 lo ss [ %] time [s] PACKET LOSS late lost packet delays playout delayaverage playout delay
8000 820 840 860 880 900 10 20 Id DELAY IMPAIRMENT 8000 820 840 860 880 900 50 Ie EQUIPMENT IMPAIRMENT 100 QUALITY vs. TIME
1. Playout buffer module calculates
playout delays (m2e), average
playout delays…
2. … and resulting average packet
loss values for fixed time
windows
3. E-model simulator calculates
corresponding delay
impairments Id …
4. ... and equipment impairments
Ie…
E-model, speech transmission categories,
user satisfaction
0 5 10 15 20 0 10 20 30 40 5060 Equipment Impairment Ie vs. Packet Loss
packet loss [%] Ie G.711 w/o PLC G.723.1 GSM G.729A G.711 Bursty Loss w. PLC G.711 w. PLC Random Loss
Transmission rating factor R = (R
0– I
s)– I
d(echo,delay)– I
e(codec, loss)+ A
Reduced to transmission impairments: R = 94.15 - I
d(echo,delay)– I
e(codec, loss)Poor Low Medium High Best Speech transmission quality almost all dissatisfied many very dissatisfied some very dissatisfied satisfied very satisfied User satisfaction 60-50 70-60 80-70 90-80 94.15 -90 R-value 0 100 200 300 400 500 0 10 20 30 40 50 60 70
80 Delay Impairment Id vs. Delay
m2e delay [ms] Id TELR=45dB TELR=50dB TELR=55dB TELR=60dB TELR=65dB
Quality contours for G.711 with PLC (random loss)
0
200
400
0
5
10
15
pa ck et lo ss [ % ]TELR = 65 dB
0
200
400
0
5
10
15
pa ck e t l o ss [% ]TELR = 60 dB
5
10
15
p a cke t l o ss [% ]TELR = 55 dB
0
200
400
0
5
10
15
pa ck et lo ss [ % ]TELR = 50 dB
0
200
400
0
5
10
15
pack
e
t l
o
ss
[%
]
playout delay [ms]
TELR = 45 dB
not recommendedalmost all users dissatisfied many users dissatisfied some users dissatisfied satisfied
Quality contours for G.711 with PLC (bursty loss)
0
200
400
0
5
10
15
pa ck et lo ss [ % ]TELR = 65 dB
0
200
400
0
5
10
15
pa ck e t l o ss [% ]TELR = 60 dB
0
200
400
0
5
10
15
p a cke t l o ss [% ]TELR = 55 dB
playout delay [ms]
0
200
400
0
5
10
15
pa ck et lo ss [ % ]TELR = 50 dB
0
200
400
0
5
10
15
pack
e
t l
o
ss
[%
]
playout delay [ms]
TELR = 45 dB
User Satisfaction Categories
not recommended
almost all users dissatisfied many users dissatisfied some users dissatisfied satisfied
Quality contours for G.729A
0
200
400
0
5
10
15
pa ck et lo ss [ % ]TELR = 65 dB
0
200
400
0
5
10
15
pa ck e t l o ss [% ]TELR = 60 dB
5
10
15
p a cke t l o ss [% ]TELR = 55 dB
0
200
400
0
5
10
15
pa ck et lo ss [ % ]TELR = 50 dB
0
200
400
0
5
10
15
pack
e
t l
o
ss
[%
]
playout delay [ms]
TELR = 45 dB
not recommendedalmost all users dissatisfied many users dissatisfied some users dissatisfied satisfied
Quality contours for GSM
0
200
400
0
5
10
15
pa ck et lo ss [ % ]TELR = 65 dB
0
200
400
0
5
10
15
pa ck e t l o ss [% ]TELR = 60 dB
0
200
400
0
5
10
15
p a cke t l o ss [% ]TELR = 55 dB
playout delay [ms]
0
200
400
0
5
10
15
pa ck et lo ss [ % ]TELR = 50 dB
0
200
400
0
5
10
15
pack
e
t l
o
ss
[%
]
playout delay [ms]
TELR = 45 dB
User Satisfaction Categories
not recommended
almost all users dissatisfied many users dissatisfied some users dissatisfied satisfied
Quality contours for G.723.1 (6.4kbps)
0
200
400
0
5
10
15
pa ck et lo ss [ % ]TELR = 65 dB
0
200
400
0
5
10
15
pa ck e t l o ss [% ]TELR = 60 dB
5
10
15
p a cke t l o ss [% ]TELR = 55 dB
0
200
400
0
5
10
15
pa ck et lo ss [ % ]TELR = 50 dB
0
200
400
0
5
10
15
pack
e
t l
o
ss
[%
]
playout delay [ms]
TELR = 45 dB
not recommendedalmost all users dissatisfied many users dissatisfied some users dissatisfied satisfied
Quality contours for G.711 w/o PLC
0
200
400
0
5
10
15
pa ck et lo ss [ % ]TELR = 65 dB
0
200
400
0
5
10
15
pa ck e t l o ss [% ]TELR = 60 dB
0
200
400
0
5
10
15
p a cke t l o ss [% ]TELR = 55 dB
playout delay [ms]
0
200
400
0
5
10
15
pa ck et lo ss [ % ]TELR = 50 dB
0
200
400
0
5
10
15
pack
e
t l
o
ss
[%
]
playout delay [ms]
TELR = 45 dB
User Satisfaction Categories
not recommended
almost all users dissatisfied many users dissatisfied some users dissatisfied satisfied
Predicting user satisfaction (proposed solution)
0 200 400 0 5 10 15 20 pa ck et l o ss [ %] playout delay [ms] DELAY/LOSS DISTRIBUTION 0 200 400 0 5 10 15 pack et l oss [ %] playout delay [ms] QUALITY CONTOURS not recommended almost all dissatisfied many dissatisfied some dissatisfied satisfied very satisfied 0 200 400 0 5 10 15 pa ck et l o ss [ %] playout delay [ms] DELAY/LOSS DISTRIBUTION not recommended almost all dissatisfied many dissatisfied some dissatisfied satisfied very satisfied 9% 27% 9% 27% 27% USER SATISFACTIONPredicting user satisfaction (our solution)
0 200 400 0 5 10 15 20 pa ck et l o ss [ %] playout delay [ms] DELAY/LOSS DISTRIBUTION 0 200 400 0 5 10 15 pack et l oss [ %] playout delay [ms] QUALITY CONTOURS not recommended almost all dissatisfied many dissatisfied some dissatisfied satisfied very satisfied 0 200 400 0 5 10 15 pa ck et l o ss [ %] playout delay [ms] DELAY/LOSS DISTRIBUTION not recommended almost all dissatisfied many dissatisfied some dissatisfied satisfied very satisfied 9% 27% 9% 27% 27% USER SATISFACTIONPredicting user satisfaction (our solution)
0 200 400 0 5 10 15 20 pa ck et l o ss [ %] playout delay [ms] DELAY/LOSS DISTRIBUTION 0 200 400 0 5 10 15 pack et l oss [ %] playout delay [ms] QUALITY CONTOURS not recommended almost all dissatisfied many dissatisfied some dissatisfied satisfied very satisfied 0 200 400 0 5 10 15 pa ck et l o ss [ %] playout delay [ms] DELAY/LOSS DISTRIBUTION not recommended almost all dissatisfied many dissatisfied some dissatisfied satisfied very satisfied 9% 27% 9% 27% 27% USER SATISFACTIONPredicting user satisfaction (our solution)
0 200 400 0 5 10 15 20 pa ck et l o ss [ %] playout delay [ms] DELAY/LOSS DISTRIBUTION 0 200 400 0 5 10 15 pack et l oss [ %] playout delay [ms] QUALITY CONTOURS not recommended almost all dissatisfied many dissatisfied some dissatisfied satisfied very satisfied 0 200 400 0 5 10 15 pa ck et l o ss [ %] playout delay [ms] DELAY/LOSS DISTRIBUTION not recommended almost all dissatisfied many dissatisfied some dissatisfied satisfied very satisfied 9% 27% 9% 27% 27% USER SATISFACTIONPredicting user satisfaction (our solution)
0 200 400 0 5 10 15 20 pa ck et l o ss [ %] playout delay [ms] DELAY/LOSS DISTRIBUTION 0 200 400 0 5 10 15 pack et l oss [ %] playout delay [ms] QUALITY CONTOURS not recommended almost all dissatisfied many dissatisfied some dissatisfied satisfied very satisfied 0 200 400 0 5 10 15 pa ck et l o ss [ %] playout delay [ms] DELAY/LOSS DISTRIBUTION not recommended almost all dissatisfied many dissatisfied some dissatisfied satisfied very satisfied 9% 27% 9% 27% 27% USER SATISFACTIONAlgorithms evaluation – testing environment
0 100 200 300 400 0 5 10 15 pa cke t l oss [% ] playout delay [ms]PLAYOUT DELAYS AND PACKETS LOSS DISTRIBUTION not recommended almost all dissatisfied many dissatisfied some dissatisfied satisfied very satisfied 3% 4% 7% 18% 44% 24% USER SATISFACTION User satisfaction for G.711 w. PLC (random loss) & TELR = 65
α αα α=0.998002, ββββ=4.00 0 500 1000 1500 2000 2500 3000 3500 50 60 70 80 90 100 R time [s] QUALITY vs. TIME 2003/07/01 11:00 30 30 1 65 35 0 99 39 0 120 0
playout
buffer
simulator
parameters
E-model
simulator
parameters: encoder, echo
packet loss
playout delays
prerecorded
network delays,
marker bits
user satisfaction
time varying quality of the call
10 20 30 40 50 60
0 5 10
15 LOSS/DELAY TRADE-OFF: histogram b. algs. # =100 # =200 # =400 # =1000 la te p a ck e ts lo ss [ % ]
av. buffering delay [ms]
concord alg. dynamic α Moon's alg.
loss/delay trade-off
Experiment in the wireless LAN
Influence of the background traffic
Measurement setup
Loss/delay trade-off (various algorithms)
0 20 40 60 80 100 120 2 4 6 8 10 12 14 α=0.998002 α=0.99 α=0.98 α=0.95LOSS vs BUFFERING DELAY: reactive alg.
la te p ack et l oss [ %]
av. buffering delay [ms]
dynamic α alg. Bolot's alg. Ramjee's alg. 0 20 40 60 80 100 120 2 4 6 8 10 12 14 #=50 #=100 #=200 #=400 #=1000
LOSS vs BUFFERING DELAY: histogram b. alg.
av. buffering delay [ms] la te p ack et l oss [ %] dynamic α alg. concord alg. Moon's alg. 60 80 100 120 140 160 180 200 2 4 6 8 10 12 14 la te p ack et l oss [ %]
LOSS vs PLAYOUT DELAY on QUALITY CONTOURS: reactive alg.
α=0.998002 α=0.99 α=0.98 α=0.95 dynamic α alg. Bolot's alg. Ramjee's alg. not recommended
almost all users dissatisfied many users dissatisfied some users dissatisfied satisfied very satisfied 60 80 100 120 140 160 180 200 2 4 6 8 10 12 14 la te p ack et l oss [ %]
LOSS vs PLAYOUT DELAY on QUALITY CONTOURS: hist. b. alg.
#=50 #=100 #=200 #=400 #=1000 dynamic α alg. concord alg. Moon's alg. not recommended
almost all users dissatisfied many users dissatisfied some users dissatisfied satisfied
Transmission quality vs. time and user satisfaction
(Ramjee’s: 0.95, Bolot’s, “dynamic
α
”, Moon’s: #200/1%
500 1000 1500 2000 2500 3000 3500 60 80 100 R time [s] QUALITY vs. TIME 0 100 200 300 400 0 5 10 15 pa ck et l o ss [ %] playout delay [ms] DELAY/LOSS DISTR. not recommended almost all dissatisfied many dissatisfied some dissatisfied satisfied very satisfied 12% 8% 6% 6% 52% 16% USER SATISFACTION 500 1000 1500 2000 2500 3000 3500 60 80 100 R time [s] QUALITY vs. TIME 0 100 200 300 400 0 5 10 15 pa ck et l o ss [ %] playout delay [ms] DELAY/LOSS DISTR. not recommended almost all dissatisfied many dissatisfied some dissatisfied satisfied very satisfied 12% 3% 3% 1% 25% 56% USER SATISFACTION 500 1000 1500 2000 2500 3000 3500 60 80 100 R time [s] QUALITY vs. TIME 10 15 o ss [ %] DELAY/LOSS DISTR. not recommended almost all dissatisfied many dissatisfied some dissatisfied satisfied 9% 1% < 1% 24% 500 1000 1500 2000 2500 3000 3500 60 80 100 R time [s] QUALITY vs. TIME 10 15 o ss [ %] DELAY/LOSS DISTR. not recommended almost all dissatisfied many dissatisfied some dissatisfied satisfied 9% 3% 3% 2% 32%
0 64 25 2 1 9 dynamic αalg. 0 51 30 4 2 13 Bolot’s alg. 0 37 49 4 1 9 Moon’s alg. 0 1 6 3 1 89 Concord alg. 0 10 50 19 6 15 Ramjee’s alg. α=0.9 0 36 45 4 3 12 Ramjee’s alg. α=0.9980 G.729A 0 0 85 4 2 9 dynamic α 0 0 80 8 4 14 Bolot’s alg. 0 0 80 8 2 10 Moon’s alg. 0 0 5 2 2 90 Concord alg. 0 0 37 35 10 18 Ramjee’s alg. α=0.9 0 0 72 11 2 15 Ramjee’s alg. α=0.9980 G.723.1 65 24 2 0 0 9 dynamic ααααalg. 56 25 1 3 3 12 Bolot’s alg. 32 51 2 3 3 9 Moon’s alg. 1 6 3 1 0 89 Concord alg. 11 60 11 6 3 10 Ramjee’s alg. α=0.9 40 42 3 2 1 11 Ramjee’s alg. α=0.9980 G.711 very satisfied [% time] satisfied [% time] some dissatisfied [% time] many dissatisfied [% time] almost all dissatisfied [% time] not recommended [% time] PLAYOUT MECHANISM CODEC
Summary
We need a method to evaluate both playout buffer algorithms and
encoding scheme in end-to-end VoIP system
statistical loss/delay metrics (average delay/average loss) give little
information on user satisfaction
“listening-only” tests and PESQ don’t take into account interactivity
E-model relies on static transmission impairments that do not
correspond to dynamics of de-jitter buffering
We proposed extended version of the E-model (short-time version)
It provides direct link to speech transmission quality by estimating user
satisfaction. User satisfaction is estimated from varying transmission
impairments (playout delay, resulting packet loss) and takes into
account encoding scheme
Pictorial representation of playout delays and packet loss gives more
Conclusions
We compared the performance of three audio codecs (ITU-T G.711,
G.723.1, and G.729A) and various buffering schemes in a WLAN
environment under varying load conditions using the extended version
of the ITU-T E-model methodology.
Results show that the use of the G.711 audio codec in conjunction with
“dynamic alpha” adaptive playout scheme gives the highest user
satisfaction of the Voice over WLAN schemes considered.
0 100 200 300 400 0 2 4 6 8 10 12 14 16 18 p acke t l oss [% ] playout delay [ms] DELAY/LOSS DISTRIBUTION not recommended almost all dissatisfied many dissatisfied some dissatisfied satisfied very satisfied 3%2% 7% 44% 44% USER SATISFACTION 0 2 4 6 8 10 12 14 16 18 p a cke t l oss [ % ] DELAY/LOSS DISTRIBUTION not recommended almost all dissatisfied many dissatisfied some dissatisfied satisfied very satisfied 4%1%2% 27% 50% 15% USER SATISFACTION
