Performance Issues and Error Analysis in an Open Domain Question Answering System

(1)

!"$#%'&$(&)*,+-.%/1023&)

4 65'&$7!98:&);=<>#%?5'

@BA'C(DFEHGJIE6K'ACLMDNA'OQPJRTSVUAWSYXZA[L3\]ACTIA_^`A'OaA'bA'cHPdRfe1g'hiDFPdjAP%\[ROQIk?AC[R

lemg'nZoe1nZprq7sZtvu'owxpzy{q7sZy|uHsZyxeQwx}~smg

em~~emY6pem

tvsZ~h's1emg[temy|}o'Yxemghet}em}d1~emg'nZoe1nZpYzsZtuowxpzyYYsZt

bSOaAXm

?76' ?$¡¢1£$%¢`¢Z¤J¥?£$¦¢?§¨¤

©

$ ª«¬$£$£¡x¤ª«®¤£$x¤J)£{¯±°¡£$¨ªa¢²3¢¤

$³T)®¢´

©

¡£¡µV¶¸·m¹)§$ºx¢?)ªa»)

x¼µ½¢?¥¾À¿ÂÁÃ£$'$«~ª)µ½¢6ºxª«º)

µ½ªm¥?°6§¨7®¢V½¡)§'Ä9¡§¢±

©

¡£¡µVÅÆ¿ÈÇaÃ

£$?%µ6ºx£ ª«]«~¥Ä6º)Ém7¢?¥v£$%¢?$¡¤

£$ªa¢ª«ÀÊ§¨ª´aº»?$ªz¹ÅH¢?¥Ë¿ÈÌaÃ7£$?®µ¤

9ºx£{ª«M¹)¨ªa°?{§¨x¼º§$$ªa°)ºx¶v

µr¢ºxªa¢?º§°?$ªa¢»®[£$?££$Àª¹)§®§'¡¤

«~ª)µ½¢6ºx3¥'¢?¥?ªa¢½£$7¥?£$½ª«¢6£Â¤

°?)§?§¢´a°6´ ?$ªZºx$$®¢´>$¡ªa°?)ºxT¢6¥

£$? £¡ª1ªa§3°?¡¥Ê«~ª ¢?¡³ Í9¢?¥?¢´¶

Î Ï CÆOaEIRXmaPJEHC

²7µ½¢´±£$£$°Ð¢?¢´3Ä?)¨«6¢6¡³TÐ¢»$¡'ªa¢?¡£¡ª

¢?£$°Ð§§¢´a°?´%Ñm°¡£$ªa¢?ÅZª'¢Z¤J¥ªaµ½¢v¯ °Â¤

£$¨ªa¢²3¢?$³T)®¢´`¿¯±² Ã

©

¡£¡µ½%$?$¡¢1£{¢¥Z¤

¹¢?ºx¥v?9§º£$¨ªa¢Êª«¢?£$°Ð§§¢´a°?´%?$ªZºx$Â¤

¢´¶Ê´a§¨ªÄ6§Àµ£¡)®º°?¡¥Ë¢£$¯±²£¡)º$É

¹§°?£$¨ªa¢?¦ª«Ê£$ÒÆx¼Z£`Ó Ô£¡)¨¹§rÕªa¢«~$¢?ºx

¿ÖÓ3ÔÕÃr¿~×ª1ª)?ÅÁØØØaÃ»§§¨ªz³!«~ª{£$.ª¹Ð§§

$¡)$µ¢1£>ª«À£$¬¯±²2

©

$£¡µi'$«~ª)µ½¢?ºx¶½²7

6$£.ª«{$§¨¢1£$§¨$ÊÑm°¡£Ê£¡ªËµ?)ª¹B¯±²i

©

Â¤

£¡µ½Å7¨£V.¢ºx$$

©

£¡ªÒµ$°)Ù¢ª£Êªa¢?§

©

£$

´a§¨ªÄ6§6'$«ªÐµ½¢?ºxÅaÄ6°?£§¡ª>£$? '$«~ª)µ½¢?ºxª«

º)¢?¥?¹m¥6°?§µªZ¥?°?§¨»¢?¥¦ª£$%Ðº)?¨£¡ºx£$°?)§

«~£$°$¶»²Ú¥£$®§¨¥Ù)«ª)µr¢?ºx»¢6§

©

)±¢?¥?Û¤

º£¡¢ª£Æªa¢?§

©

£$?£]£$T

©

¡£¡µÜ«~®§£¡ª7?$ªz¹m¥?¢

¢?¡³ Ä6°£±³

©

£$?½

©

$£¡µÝ«~§¥¶{?»'$«~ª¡¤

µ½¢?ºxM¢?§

©

$®H]°?¡«Ö°?§1£¡ª7

©

¡£¡µ(¥)¨´a¢)H³ ª

³À¢Q£Æ£¡ª ¥¢1£$¨«

©

$$ª]¡ªa°)ºxÆ¢?¥ ³É µªm¥6°?§¨¶

?Ò¯ ²=§¨£¡Ð£$°$«$ªaµÞ£$?Ò§¡£¦«³

©

Ð

$'ª$£$vªa¢ß´a§¨ªÄ9§>'$«~ª)µ½¢?ºxBª«¹)¨ªa°6v

©

Â¤

£¡µ½à¿Ö² Ä6¢? £*§È¶¨ÅáÇâââZã ä3ª¹ £*§È¶¨Å

ÇââÁÃÐ¶Àå±¢)§¹§°?£$¨ªa¢Êµ£¡ÐºM$>¥?)º°?$¡¥

¢¿~×ª1ª)?7¢6¥vºxÅÇâââQÃ3¢?¥¿Öæ$º)Év£±§È¶¨Å

ÇâââQÃÄ9°£Å³ £$ç«³Fx¼ºx?£$¨ªa¢6v¿~èd£¡£

©

ºÐ)é£

§È¶¨ÅÇââÁÃÐÅa§¨£¡£$§¨H$¥ Ä'ªa°£H¢¤J¥?£$±$$ªÆ¢?§Û¤

©

$M¢v¯±²ê

©

¡£¡µ½¶

·Z®¢?ºxµªa$£¸¯±²ë

©

¡£¡µrÙºxªa¢6$¡£ª«rµªZ¥?°?§¨

£$?£v)`º)?®¢¥ß¡Ð§§

©

¿Ö²3Ä6¢

©

£v§¶¨Å%ÇâââZã

ì Ð´>£%§È¶¨Å[ÇâââQÃÐÅ]£$?ª¹)§§)«ª)µr¢?ºx½

ºxªa¢1£¡$ªa§§¨¥Ä

©

£$¨³É¡£T§®¢É¶HèÂ¢£$?º¡3£$

$$ªí¢6§

©

)ÊV¡£¡)¨´a1£¡«~ª$³Ð¥¶Nî °í

©

¡£¡µ

)ºÐ?¨£¡ºx£$°$Ê°?¡»¡¹)§«~¥Ä6º)Ém³ 6º)ºxªaµ¤

6§®º£¡$¨´a¢?Í6º¢Q£$§

©

£$>$)ª ¢?§

©

$¶

6À6'?$$¢Q£$3¢v¢Z¤J¥??£$V'$«~ª)µ½¢?ºx

¢?§

©

$ª«Z ¡£$£¡x¤ª«®¤£$?x¤J$£¯±²ç

©

$£¡µV¶·Z¹)§

ºxªa¢Í?´a°?)£$¨ªa¢?V$¸x¼µ½¢?¥ã>Í?)$£Å3£$¸'$«~ª¡¤

µ½¢?ºx{ª«º)¦µªZ¥?°?§¨>®¢í½Ä6$§¢%ºÐ?¢¥¡¤

º)6¨£¡ºx£$°$Å£$¢Åz£$µ½6ºx£ª«Z«¥?Ä6º$ÉZH¢?¥%£$

¢?$$£$¨ªa¢Vª«¢³¥¹Y¢6ºx¥µªZ¥?°?§Å?¢?¥íÍ9¢?§§

©

Å

£$±µ6ºx£Tª«¹)¨ªa°?§¨x¼º§6)¡ªa°)ºx¶Tî °À¯±²

©

¡£¡µï³ÀÀ)¢É¥Ê?¨´a.¢.£$ §¡££$$ Ó3ÔÕ

¯±²,£¡)º$ÉN¹§°?£$ªa¢?ð¿Öºx«d¶Ù¿~×ª1ª)?ÅÁØØØaÃ¡ÃÐ¶

)«ª$ £$ $)°?§¨£$) $?)¡¢Q£$£$¹ £¡ª»ª£$

¯±² $)§])º)?£¡ºx£$°$7³ ªa¡¢1£¡)¢?§]µªZ¥?°?§¨

'$«~ª)µ(Ñm°?¨¹§¨¢Q££$¡Ém±¿Ö²3Ä6¢

©

£T§¶¨ÅmÇâââQÃª

µ6§ª

©

$µ½§®]§¨x¼º§Z$$ªa°)ºx¢?¥½£¡ª1ªa§3¿Öä ªz¹

©

£ §È¶¨ÅÇââÁã ì Ð´ £ §È¶¨Å6ÇâââQÃÐ¶

ñ ò

AóEHCEHôçõ

7'$«~ª)µ½¢?ºx3ª«H½¯±²ö

©

¡£¡µNT£$¨´aQ£$§

©

ºxªa°Z¤

6§¨¥½³ ¨£$£$7ºxªaµ6§¨x¼¨£

©

ª«Ñm°¡£$¨ªa¢6¡É¥.¢?¥

£$í¥?÷rº°?§¨£

©

ª« ¢?¡³½x¼Z£¡)ºx£$¨ªa¢¶éø?ªrx¼µ¤

6§¨ÅZ¢ÊÓ3ÔÕéµ½¢

©

¡£¡µ½³T)±Ñm°?¨£¡±$°?ººx)Â¤

«Ö°?§£?$ªz¹m®¥?¢´±ºxª$$ºx£¢6¡³TÐ£¡ª%$®µ6§¨Å«Öºx£Â¤

¡ÉZ¢´±Ñ1°?¡£$¨ªa¢?ÅÄ6°?£Æ«Ö§¨¥{£¡ª ¢?¡³Ñ1°$£$¨ªa¢?

£$?£Ê$Ñm°?¨$¥é$¡ªa¢?®¢´ªV¥¹¢?ºx¥ö§®¢´a°?¡£$®º

(2)

©

¡£ª«[ÁxùQúâ ¹§°?£$¨ªa¢Ñm°¡£$¨ªa¢6Å1ûYâQüª«6£$?M6¡¤

£$º¨9£$¢´Ë

©

¡£¡µ½í¢?$³T$¥Ü$°?ººx$$«~°?§®§

©

Ñm°Â¤

£$¨ªa¢?±§É½¯ÁâÁÌm¾çýzþÊÿ ~ÿÐÅHÄ6°£±¢ªa¢

ºxªa°?§¥{Í6¢?¥»±ºxª$$ºx£¢?¡³£¡ª ºxªaµ6§¨x¼»Ñ1°$£$¨ªa¢?

$°?ºÐv±¯ÁÁúm¾ÙýzþÊÿ~ÿ !"#%$"

&' (*)+,-(%%)./021 ' 34%)5,-%*)6Ð¶

·Z®¢?ºxM'$«ªÐµ½¢?ºx 87ºx£¡¥¬Ä

©

£$? ºxªaµ6§¨x¼m¤

¨£

©

ª«¬Ñm°¡£$ªa¢Ü6$ªmºx)$¢´Å»³TçÍ?Ð¡£6$ª¹Z¥Ë

Ä?$ªa¥í£$Y¼mªa¢ªaµ

©

ª«¯±²Ü

©

¡£¡µr¶

9;:*< =?>8@ACB>@D

£$Y¼Zªa¢ªaµ

©

±Ä6¡¥¦ªa¢B$¹)§ºx)¨£¡Ð.£$?£

6§

©

¢vµ'ª$£$¢1£)ªa§¨>¢ÊÄ6°?®§¥?¢´¯±²

©

$£¡µ½¾

¿ÂÁÃ]§¢?´a°?¡£$º¢?¥Ém¢?ª³ §¨¥?´)¡ªa°)ºxÅ¿ÈÇaÃ]¢?£$°Z¤

)§[§¢´a°6´?$ªZºx$$¢´Ê¢1¹ªa§¨¹¥Å¿ÈÌaÃ7¥ªZº°?µ¢1£

?$ªZºx$$®¢´Å¿~ù1Ã)¡ªa¢?¢´Vµ£$?ªm¥?Å¿*aÃ3³3£$

ªT¢?ª£¢?¡³T[x¼Z6§º£$§

©

¡£$£¡¥.¢>¥ªZº°?µ¢1£Å

¿ÈúaÃ³ ?£$ª3¢?ª£¢?¡³«~°6$¨ªa¢V¢?ºx$$

©

¶

9;:E9 =/FGD.H#H#BH2IKJLNMBHAO@GIPH

QSR

8"T(UWV

&

,XOYZ/" [64!

R

5)3\[0)# " G^]/\(_

%`. Rba `.,%*)6

${

©

¡£¡µ½7x¼Z£¡)ºx£ ¢?¡³)±3£¡x¼Z£±$¢?¨?'£$

«~$ªaµ ªa¢`ªµª$¥ªmº°6µ¢Q£$¶ î7«~£¡¢ö£$¢Z¤

¡³ 7«ªa°?¢6¥v¹$Ä9£$µÝ¢¦r£¡x¼Z£ ª .)µ6§¨

µª$9ªa§¨ª´aº§]¹)£$¨ªa¢¶{

©

6º§§

©

£$¢?¡³)

$¦x¼Z£¡)ºx£¡¥ð°?$®¢´µ½6¨)º§3µ£$ªZ¥?Ê$§

©

¢´

ªa¢VÉ

©

³TªÐ¥vµ½¢?6°?§£$¨ªa¢6¶

QSR

8"/c^UdV

&

X8YZ/04!

R

G^]5,Z[

R

/8)._

G^]+Z2"xÿ.E,Z/

Êº)6)ºx£¡)¡£$®º¬ª« £$?»º§${£$?£¢?¡³)

$M«~ªa°?¢?¥¢)¢?¨?'£$]ª«9£¡x¼Z£ÅQÄ9°£°?¢?§É¢rÕM§$

ÁÅT®¢«)¢?ºx.®»¢?ºx$$

©

£¡ªÙ)§£¡V£$VÑ1°?¡£$¨ªa¢

³ ¨£$v£$¢?¡³¶fevª$§Ä'ª)£¡¢?¡³ ¥£¡º|¤

£$¨ªa¢Bµ½£$ªm¥6 $°?º)`>ªa¢1£¡ªa§¨ª´a¨>ª{ºxªm¥?Í6º£$¨ªa¢

ª«?)´aµ½£$ºÉZ¢ª³3§¨¥´$M¢ºx$)

©

¶T·mµ½¢1£$º

§¨£¡)¢6£$¨ªa¢?Å³ª)§¥¦ÉZ¢ªz³ §¨¥´½Y¼¨ªaµ½ ¢?¥`$µ¤

6§¨ )¡ªa¢?¢´µ£$?ªm¥? $%¢ºx$)

©

¶T²7¢Vx¼µ¤

6§¨Vr¯ÁØgm¾ ýGh?)$i*kj)#Y,l*";!³3$

-]?Æ£¡ª3Ä'T§®¢É¥>³3¨£$d G6mOG^]n[));?$o;x¶

p

ª)¥.q3£¢?¥.¨£$x¼m£¡¢6$¨ªa¢?À$±¡ªaµ£$µ½M°?¡¥

$ªa°)ºxMª«[³ª)§¥ÊÉZ¢ª³ §¥´¶

QSR

8"r4U/V

&

X8,YZ?f" [^!

R

)3\6 $b\,` *)

\,3)Zs0t()C`ZKG

èÂ¢Ò£$?¬º§$¬£$v9$£$§ ¢6¡³TÊ¢«~ª)µ½£$ªa¢Ò

$º£¡£¡$¥½£$$ªa°´aªa°?£[$¹)§?¥ªZº°?µ¢1£$[¢?¥¢Z¤

¡³½«Ö°?$¨ªa¢Ër¢ºx$$ ¶:íºxªaµ9§¨x¼Z£ $

ºxªaµ6§x¼ÚÑ1°?¡£$¨ªa¢?ç§¨É_$ºx)?£Ñm°¡£$¨ªa¢6ÅB¿~¶´¶

ýGh?)$u()wv+"Z!

R

xk!%#X

R

6¡ÃÐÅª£¡µ9§£¡x¤

§¨É¬Ñ1°$£$¨ªa¢?Ê¿ZýþÊÿ(fZ0#]^ZK,`6 " *)6

)# ` xyvz

'

Gl~ÿ6{[8,tX4^$ÃÐ¶

QSR

8"|KU{v KY(C%}dV

&

X8,YZ?

$Ê

©

¡£¡µr$VÄ6§¨Ê£¡ª¸¢?¡³½Ñ1°$£$¨ªa¢?¢

£$Òºxªa¢1£¡x¼m£ª«¬6$¹mªa°?¸¢1£¡)ºx£$¨ªa¢?¸³3¨£$ê£$

°?¡¶B²3$'ª$£¡¥ç¢_¿Öä7)Ä6´a°ç£½§È¶¨ÅÇââÁÃÐÅ

?$ªZºx$$®¢´ËÒ§¡£Êª«Ñ1°?¡£$¨ªa¢?Ê'ªa¡¥ð®¢öËºxªa¢Z¤

£¡x¼Z£±®¢Q¹ªa§¨¹±ºxªaµ9§¨x¼í$«~$¢?ºx»$¡ªa§°£$ªa¢¶~7¢Z¤

§¨É%£

©

6º§)«$¢6ºx%$¡ªa§®°£$¨ªa¢v§´ª)¨£$?µ½£$?£

$¡ªZº£¡¢?9ª$À³ ¨£$7$«~$¢Q£ÅQ£$À$«)¢?ºx

µ'ªa¡¥.Ä

©

ºxªa¢1£¡x¼m£7Ñ1°?¡£$¨ªa¢?$Ñm°?¨$£$>)¡ª¤

º£$¨ªa¢Òª«±¢é¢?6ª)¦«~$ªaµë£$?vº°$$¢1£½Ñm°Â¤

£$¨ªa¢³ ¨£$¨£$ªa¢Êª«3£$Ê?$¹Z¨ªa°?»Ñ1°$£$¨ªa¢?Å

¢?¡³)Mª £$¨ ¢?9ª)Z¶

QSR

8"{(UV

&

X8YZ/n [4!

R

)3\n0

R

)](%

R

_

).]

±º)6)ºx£¡)¡£$®º3ª«H£$¡

©

¡£¡µ½ÀT£$¨ÀÄ6§Û¤

¨£

©

£¡ªr¢6¡³T7¡'º°?§£$¨¹ Ñm°¡£$¨ªa¢6$µ½®§À£¡ª¾

ýGv"~ÿf1b]4)G^]d-)3?GKY,G?~ÿG"(

Z"#%^]NÐã3ýGv"n~ÿSjd)`b)3\" , *)|ã7ý%v"~ÿ6

G

R

G?`,%,X+G%)`K!

R

(Ð¶

·Z®¢?ºx µªa¡£?$ªÄ9Ä6§

©

£$7¢6¡³T£¡ª$°?º).Ñm°Â¤

£$¨ªa¢?M®¢?ª£Mx¼m6§®º¨£$§

©

¡£$£¡¥v¢Ê¥ªmº°6µ¢Q£$Å?$µ¤

6§

©

Ä'º°?¡ ¹¢1£$ µ½

©

¢?ª£?¹%6?'¢¥

©

£Å

¯±²

©

$£¡µ½«~$ªaµê£$?[º§$¥ºxªaµ'ªa¡À£$Ñm°Â¤

£$¨ªa¢.¢1£¡ª{Ñm°)¨£$?£x¼Z£¡)ºx£À6¨ºxTª«¹m®¥¢?ºxÅ

«~£¡]³3?º)%¢?¡³]H«ªÐµ{°?§®£¡¥{°?)¢´$$ªa¢?¢´

Ä

©

¢6§¨ª´

©

¶$$ªa°)ºxH®¢?º§°?¥¥Z¤J?ªmºÉZ¢ªz³ §Û¤

¥´Ä6¡Ù´¢)£¡¥«~$ªaµÞµ½¢?®¢´Ë£¡x¼m£¦¥ªZº°Z¤

µ¢1£$íº§°?$£¡$¥öÄ

©

£$`Ñm°¡£$ªa¢_£¡ª6º¶N²3)¡ª¤

º£¡¥ö³3¨£$:£$¡¸ÉZ¢ª³ §¥´¸¡ªa°)ºxí)Bº¡x¤

Ä6¡¥$¡ªa¢6¢´v£¡ºÐ?¢?Ñm°»»³T§§µ£$ªZ¥?

«~ª£¡µ'ª)§{$¡ªa¢6¢´Å%¡9£$§%)¡ªa¢?¢´:¢?¥

¹Z¥¢1£$§'$¡ªa¢?®¢´¶

]Ä6§¨½Á¾7®¡£¡)¨Ä6°?£$¨ªa¢rª«Ó3ÔÕ:Ñm°¡£$¨ªa¢6

©

' q3°6µ%Ä'%¿ü½Ã

ÕÀ§$ Á¬¿~«~ºx£$°?§~Ã Øgv¿Èúaû1¶aürÃ

ÕÀ§$Çv¿Ö$µ6§x¤$¡ªa¢?¢?´1Ã ùaâ(gv¿ÈÇaû1¶ØaürÃ

ÕÀ§$Ìv¿~«~°?)¨ªa¢.¤§¡£ÐÃ Çí¿ÂÁ¶ûürÃ

ÕÀ§$ÀùÙ¿Ö¢Q£¡Ðºx£$¨¹ ¤Tºxªa¢1£¡x¼m£ÐÃ ùQÇí¿ÈÇm¶ØaürÃ

ÕÀ§${v¿Ö¡'º°?§£$¨¹zÃ âv¿ÖâZ¶âQü½Ã

(3)

Volkswagen AND bug

60 passages

2 passages

"in 1966"

USD 520

$1

USD 520

"rent a Volkswagen

bug for $1 a day"

of expected

answer type

selection

Keyword

expansion

post−filtering

Passage

of candidate

Identification

Answer

formulation

ranking

Answer

Question

Country

Volkswagen

bug

rent

built − build

Examples

System

answers

and passages

of documents

Modules

Niagra − Niagara

Volkswagen

Volkswangen −

Money

Person

invented − inventor

rented − rent

pre−processing

(split/bind/spell)

Keyword

of question

representation

Derivation

Construction

Retrieval

How much

rent

1966

bug

Volkswagen

M1

M2

M3

M4

M5

M6

M7

M8

M9

M10

ø]´a°$½Á¾²3)º)6¨£¡ºx£$°$ ª«[Ä6¡§®¢ ¡)§

©

¡£¡µ ¿Ö¢ª«~¥Ä6º$ÉZÐÃ

Ñm°¡£$¨ªa¢?¦¢1£¡ª_£$ËÑ1°?¡£$¨ªa¢(º§$$¶=èÂ¢ê¥?¥?Û¤

£$¨ªa¢V£¡ªÙÁÌØÌ½µ½®¢Z¤£$¡ÉVÑ1°?¡£$¨ªa¢?Mºxªa§§ºx£¡¥í«~$ªaµ

Ó3ÔÕT¤gmÅ7Ó ÔÀÕT¤dØ¢?¥_Ó ÔÀÕT¤dÇââÁÅ±£$$¸$

Ç¬§¡£ Ñm°¡£$ªa¢?{¿~¶´¶¨ÅýG?Zc")`K%,-,/~ÿ

[0)#`. ?")"CU$Ã%¢6¥ùQÇ>ºxªa¢1£¡x¼m£ÀÑm°¡£$¨ªa¢?3¿~¶´¶¨Å

ýGh?)$ R)^]d$8~ÿl. X(#]NÐã»ýGh?)$$o*46¡ÃÐ¶

\]k?OQPJAG3SzõSzk9ô A'OQXQjTPÈk6XmaROak

?®»$ºx£$¨ªa¢Ò¢1£¡$ªZ¥?°?ºx»£$í¡)®§¥ç)º)6¨£¡º|¤

£$°$Ùª«>ªa°v¯±²

©

$£¡µ*®¢é³ 6º) £$$¦$¸¢ª

«~¥Ä6º$ÉZ¶:íºxªaµ6§£¡v)º)6¨£¡ºx£$°$í³3¨£$Ë§§

£$.«¥?Ä6º$ÉZ>?)¡¢Q£¡¥ç¢§®£¡»¡ºx£$ªa¢ª«

£$6'¶{²3>)ª³ ¢¦¢¦ø[¨´a°)ÊÁÅH£$½)º)6¨£¡º|¤

£$°$Êºxªa¢?$®¡£$»ª«{Áâ¦µªZ¥?°?§¨%)«ª)µr¢´$¹)§

¢?£$°Ð§§¢´a°?´>6$ªmºx)$¢´£$¡ÉZ¶

?7Í6)¡£ÀÍ?¹>µªZ¥?°?§¨Àºxª$)¡'ªa¢?¥.£¡ªrÑ1°?¡£$¨ªa¢

?$ªZºx$$®¢´Åa£$ ¢?x¼m££d³Tª»µªZ¥?°?§¨['$«~ª)µ(¥ªZº°Z¤

µ¢1£¢6¥¬9$$´7?$ªZºx$$¢´ÅZ¢?¥¬£$±§¡££$$

µªZ¥?°?§¨À'$«~ª)µN¢?$³T6$ªmºx)$¢´¶

eÁ ¬¢6¥?¨¹Z¥?°?§[Ñm°¡£$¨ªa¢`³Tª)¥6{$Ê¡'§§Û¤

º)?º$É¥¶ p ª)¥?Æ§¨É2.) RmC,$^]±¢?¥*#](3T$

x¼Z6¢?¥¥ ¢Q£¡ª£$¨.¡'§§¢´¦¹)¢1£$.)

R

mO $_

]^¢?¥*#]43¶öè«±¢ºx$$

©

ÅÑm°¡£$ªa¢?½$°?ºÐ

`¯ggm¾ýG{)-,X^](;" $Z2(4!X

$[ÿ?")Z[KX(!$¬)6)¡¥ç¢1£¡ª¦Ù¢?ª)µ½§Û¤

¥Ê«~ª)µï³ ) £$ ³ Z¤³ª)¥¸¿$[ÿ®Ã?')M£

£$ Ä'´a¢?¢6¢´Åm¶´¶výþÊÿt )Z[.X5$?)-,X

]G/" Z2(4!X(Ð¶

eÙÇ Þ¢6°?£*Ñm°¡£$¨ªa¢ *6Ð¡¥ ¢?¥

£¡)¢?$«ª)µ½¥ ¢Q£¡ª ¢ ®¢Q£¡)¢6§ )?$¡¢1£$Y¤

£$¨ªa¢Ù¿Öä3)Ä9´a°r£À§È¶¨ÅÇâââQÃTº?£$°)¢?´%Ñ1°?¡£$¨ªa¢

ºxªa¢?ºx?£$ë¢6¥ Ä6®¢?

©

¥'¢?¥¢?º¨Ä'£J³¢

£$éºxªa¢6ºx?£$¶ ·m£¡ª!³ª)¥? ¿~¶´¶¨Å¬?)ªa)¨£$¨ªa¢?

ªF¥£¡)µr¢)ÐÃ $ ¥¢1£$¨Í?¥*¢?¥*$µªz¹¥

«~$ªaµ£$¸$6$¡¢1£$£$¨ªa¢¶øªÙ§§°6¡£¡)£$¨ªa¢Å3£$

$?)¡¢Q£$£$ªa¢v«~ª ¯ âÁÌm¾BýGh?)$Z`6ÿw")` RX()`

KK)

R

mO $#]!`(]\)+GT(4(;º?£$°?$

£$ Ä6¢?

©

¥'¢?¥¢6º

©

Ä'£J³¢Ú£$ ºxªa¢?ºx6£$

K¢6¥T(4(¶

eÙÌ ¬µr?6¢´ª« ºx$£$¢çÑm°¡£$¨ªa¢¥'¢Z¤

¥¢?º%ªa¢ p ªÐ¥.q £Â¤Ä9¡¥ç¢?¡³{£

©

'Ê?¨¡¤

)ºÐ

©

¥6$µ{Ä6¨´a°?£¡v£$¡µr¢Q£$ºº£¡´ª

©

ª«

£$:x¼m'ºx£¡¥ï¢?¡³)_¿ ì .$º(¢?¥2ä3)Ä9´a°Å

ÇââÁÃÐ¶ øªVx¼µ6§¨Å £$Ù¥?¢6¥¢?º

©

Ä'£J³¢

h?)$ Z`.xÿ{¢?¥K«~ª3¯±âÁÌTx¼m9§¨ªa¨£¡¥.£¡ª½¥x¤

)¨¹»£$x¼Zºx£¡¥¸¢6¡³T±£

©

'

);X¶>¢Z¤

¡³£

©

'.»6)¡¥£¡ª¦)°Ä6¡Ñm°¢1£»µªZ¥?°?§%«~ª

£$>¥?¢Q£$¨Í9º£$¨ªa¢Êª«Æ'ªa$)¨Ä6§¨±¢?¡³)%¿Ö§§Hµªa¢x¤

£$

©

¹§°ÐÃÐ¶

evù æ$¥¸µ½¢?§

©

ªa¢¦6)£3ª«$ºÐ¸¢«~ª)µ½Y¤

£$¨ªa¢ÅT¸$°Ä6¡£ª«3£$íÑ1°?¡£$¨ªa¢Ëºxªa¢?ºx?£$r$V¡x¤

§¨ºx£¡¥¦7É

©

³ª)¥?7«~ª7ººx)$¢´.£$°?¢?¥)§

©

¢?´

¥ªZº°?µ¢1£Mºxªa§§¨ºx£$¨ªa¢¶²ß6$$´ )£¡)¨¹§9¢?´a¢

ººx?£$>æªmªa§¨¢¦Ñ1°?)¨7Ä6°?§£«)ªaµ £$?»¡§ºx£¡¥

É

©

³Tª)¥6Å¶´¶¡K) RmO $#]B²q¢!`]a¶B.$x¤

£¡)¨¹§¢´a®¢½$£$°Ð¢?±6$$´{£$?£»ºxªa¢Q£$¢§§

É

©

³Tª)¥6¡'º¨Í?¥¢£$? æª1ªa§¨¢rÑ1°

©

¶[ $x¤

«~ª$>É

©

³Tª)¥¡§¨ºx£$¨ªa¢M½¡¢6$¨£$¨¹ £$¡É¶èd«]£$

³$ªa¢?´ÒÑ1°$£$¨ªa¢_³TªÐ¥ ¿~¶´¶£Z`6ÿÃÊV®¢?º§°?¥¥

¢ç£$æTªmªa§¨¢ Ñ1° ¿Z`.xÿ¦²q .)

R

(4)

£${6$$´7ºxªa¢Q£$®¢?¢´½£$?{ºxª$$ºx£±¢?¡³)7$

µ½)¡¥¶

e æ«~ª$T£$?Tºxªa¢?¡£¡Ð°?ºx£$¨ªa¢{ª«?æTªmªa§¨¢»Ñ1°?)¨

«~ªíºx£$°?§±$£¡)¨¹§ÈÅ3£$¸¡§¨ºx£¡¥_É

©

³Tª)¥6í$

x¼Z6¢?¥¥Ê³ £$.µ½ª$6ªa§¨ª´a®º§ÈÅm§x¼Zº§ª $µ½¢Z¤

£$º7§¨£¡)¢?£$ªa¢?¶7§¨£¡)¢?£$ªa¢?Àºxª$$¡'ªa¢?¥¬£¡ª

ª£$À«~ª)µ½¢½³ 6º)¬£$±Ñ1°$£$¨ªa¢¬ºxªa¢6ºx?£$Àµ½

©

ªZºº°M¢r£$?7¢?$³T)¶øªMx¼Zµ½6§¨Å;KY±®x¼m¤

6¢?¥?¥V¢Q£¡ªxKÖ¶

eÙú ?Ë$£¡)¨¹§¢´a¢?Ë$£$°)¢?B£$é¥ªZº°Z¤

µ¢1£$rºxªa¢1£$¢?¢?´B§§MÉ

©

³Tª)¥6r¡'º¨Í?¥Ë®¢Ë£$

æª1ªa§¢Ñm°)¨¶í¬¥?ªmº°?µ½¢Q£$»$r£$?¢`«Ö°¡¤

£$V$$£¡)ºx£¡¥:£¡ªÒ$µ½§§¬£¡x¼Z£V6$$´Ê³3$

§§9É

©

³ª)¥?M)±§ªmº£¡¥V¢¬£$±?$ª¼Z®µ½¨£

©

ª«Æªa¢

¢ª£$¶rÔºÐB$£¡)¹¥B6$)´r¢?º§°6¥±¥?¥?Û¤

£$¨ªa¢?§£¡x¼m£Ê¿~x¼m£¡)§¢|Ã3Ä«~ª$½£$?½)§¨$£%¢?¥

«~£¡T£$3§£¡¡£É

©

³ª)¥¬µr£$º)¶øªÀ§§®°?¡£¡)£$¨ªa¢Å

ºxªa¢?$®¥±¯±ââ(m¾¸ýzþÊÿ?~ÿ/Z2)3\/~ÿZ2._

#](^]/G"#-))3\

&

[0,*")

Q

)Z[0`6Y%¢?¥»£$ÀÂ¤

¡ªZº£¡¥.æTªmªa§¨¢ÊÑm°

©

&

[ *")'²q

Q

)Z[0`6Y

²q¤G#-)Ð¶ $§¨¹¢1£M£¡x¼m£«~)´aµ¢1£M«~$ªaµ

£$Ò¥ªZº°?µ¢1£¸ºxªa§§¨ºx£$¨ªa¢!NýG¥k ¦YWh?),,§

Z20#](]wG#-))3\

&

[0,*")

Q

)Z¦[`Y"^Ð¶~7¢Z¤

§¨$±¥?¥6¨£$¨ªa¢?§Æ£¡x¼m£>® ¢?º§°?¥?¥v¢Ù£$6$)´Å

£$±ºx£$°?§¢?¡³¦ ¦#Y{h),³ªa°?§¥¬Ä'3µ½®$¡¥

Ä'º°?¡½¨£±ªZºº°) Ä'«~ª$§§µ½£$ºÐ¥¸É

©

³Tª)¥6Å

¢?µ§

©

G#-)ÐÅ & [0,*")Æ¢?¥ Q )Z[0`6Y|¶

e¦û $£¡Ð¨¹¥.6$$´À$7«~°?$£$)Í6¢¥

«~ªV¢??¢?ºx¥:?$º®$¨ªa¢¶ ì $$´í£$?£í¥ªË¢ª£

$£$$«

©

£$»¡µ½¢1£$º%ºxªa¢6¡£¡)¢1£$±¡'º¨Í?¥í¢v£$

Ñm°¡£$¨ªa¢ $v¥?)º)¥¥¶_øª.x¼Zµ9§¨ÅM¡ªaµvª«

£$í9$$´½)£¡)¨¹¥Ë«~ªÊ¯±âÁÌB¥ª`¢ª£½$£$®¡«

©

£$M¥?£¡Mºxªa¢?¡£¡Ð¢Q£nT(44Y¶î±°£]ª«?£$úâ76$)´

$£$°Ð¢¥Ä

©

£$M$£¡Ð¨¹Y§?¢´a¢À«~ª¯±âÁÌmÅmÇ 6Â¤

$´3$>$£$¢?¥í«£¡36$$´%ªa$£Â¤Í6§¨£¡)¢?´¶

eg ¡Ðº)2«ªË¢?$³T)³ ¨£$6¢(£$:$x¤

£¡)¨¹¥»6)$´Æ$¡£¡)ºx£¡¥»£¡ª±£$ªa¡Àº¢?¥?®¥?£¡

ºxª$$$ªa¢6¥?¢´í£¡ªÙ£$.x¼m'ºx£¡¥ç¢?¡³»£

©

¶èd«

£$Mx¼Z'ºx£¡¥½¢?¡³£

©

' ¢?µ¥¢1£$¨£

©

$°?ºÐ

d¨©bªN«¬Å£$vº¢6¥?¥?£¡Ù¿4TÅbj¥®cÃr$

¥¢1£$¨Í?¥Ê³3¨£$V½¢?µ¥í¢Q£$¨£

©

$ºxª´a¢?¶ÀÕªa¢Z¤

¹)¡§

©

ÅZ«£$ ¢?¡³À£

©

2¯«.°±ª²±G³±G©bª[Å¶´¶

¯ ØâaÌm¾ýzþÊÿ/w`6%E,ZlÐÅ3£$Ùº¢6¥?¥?£¡.$

ªÄ?£$¢?¥BÄ

©

µr£$º)?¢?´vv¡£{ª«¢?¡³%6£¡£¡Ð¢?

ªa¢V£$%6$$´¶

eÙØ ÔºÐ!º¢?¥?®¥?£¡Ò¢?¡³¸$ºx¨¹_$§¨x¤

¹¢?ºxV$ºxª$íººxª)¥?®¢´¦£¡ª¦§x¼Zº§À¢?¥?)ª¼µ½¨£

©

£$ÊªZºº°$$¢?ºxVª«3£$Vº¢?¥?¥6£¡Ê¢6¡³T³3¨£$?¢

¢í?'ªa$£$¨ªa¢¶[>º¢?¥?®¥?£¡M$>¡ª$£¡¥v¢V¥x¤

ºx$$®¢´ª)¥ª«[£$$ºxª$¶

eÁâ ?

©

¡£¡µß¡§ºx£$H£$Tº¢?¥?¥?£¡T¢?¡³)

³ ¨£$í£$%?¨´a?¡£M$§¨¹¢?ºx{)ºxª$¶À>Í6¢6§¢Z¤

¡³)]$¨£$?H«~)´aµ¢1£$Hª«£¡x¼Z£Æx¼Z£¡)ºx£¡¥»«~$ªaµ

£$À6$)´$ªa°?¢?¥»£$ÀÄ'¡£º¢?¥?®¥?£¡M¢?¡³)Å

ª£$

©

${¢Q£¡Ð¢?§§

©

´¢?)£¡¥¶

´ µ OQOaEOvACA'GõSYPdS·¶ÂEOajkçbASYk6GJPdCk

Szk6OQPAG7SõSzk6ô

¸N:*< ¹B>OJIK>ºD0P»(BB^¼N½B^>@%ºBPACH

©

$£¡µ ³À¬£¡¡£¡¥:ªa¢ÜÁxùQúâÑ1°?¡£$¨ªa¢?.ºxªa§Û¤

§¨ºx£¡¥é«$ªaµë Ó ÔÕ¤gmÅÀØ¸¢6¥ËÓ ÔÀÕT¤dÇââÁ¶ ²7¢Z¤

¡³)³$x¼Z£¡)ºx£¡¥¬«$ªaµ!%Ì%å7Ä

©

£¡ £¡x¼m£ºxªa§§¨º|¤

£$¨ªa¢.ºxªa¢Q£$®¢?¢´»Äªa°?£3Á7µ½§§ªa¢¥ªmº°6µ¢Q£$T«~$ªaµ

¡ªa°ÐºxÙ)°?º)l¾ªa¦²3¢?´§¨¦µÙ¢?¥ p §§

·m£¡$£¿ªa°)¢6§È¶Ôº)¢?¡³ ?¦ârÄ

©

£¡¶

?7ºº°Ðº

©

³ÀMµ$°$¥.Ä

©

£$eí¢VÓ x¤

º¨?)ªmº§MÓ3£¡¿*eÓ Ó7Ã½µ£¡)ºV°6¡¥ËÄ

©

q3è¡·F¢

£$:Ó ÔÀÕ ¯±²f¹§°?£$¨ªa¢? ¿~×ª1ª)?Å¸ÁØØØaÃÐ¶

)º¨?$ªZº§)¢ÉZ¢´Ä6$º§®§

©

$)¨´a¢?M¢m°?µ¤

Ä'VÑ1°6§±£¡ªðÁCÀYÓ³ ?$¸ÓÊ£$¸Ð¢Ééª«»£$

ºxª$$ºx£¢6¡³T¶î±¢6§

©

£$ÀÍ?)¡£y7¢?¡³)[$Mºxªa¢Z¤

$¥?$¥Å£$1°?7Ó 7§¨$3ª±Ñ1°6§Æ£¡ª·m¶

p

¢Ù£$

©

¡£¡µ¥ª1]¢ª£Æ$£$°)¢{ ºxª$$ºx£[¢?¡³[¢>£¡ªmÅ

£$À?$º®$¨ªa¢»$ºxª$M«~ª]£$6£Ñ1°$£$¨ªa¢S$ª¶

ª¹Ð§§

©

¡£¡µï?$º)¨ªa¢r£$ µ¢Êª«H£$ ¢?¥?Û¤

¹Z¥?°?§M$ºxª)¶Ü·

©

¡£¡µ=¢?$³T)¬³$µ)°$¥

´a¢?$£ ºxª$$ºx£ ¢6¡³TÐM?$ªz¹m¥?¥ÊÄ

©

q3è¡·Z>¶

¸N:E9 ÁÂI²ÃyMSFBlB>O>8IK>8H

ö¢6¡'ºx£$¨ªa¢!ª«¦¢1£¡)¢?§.£¡)ºxÅÙ£Ë¹YÐ¨ªa°?

º)?º$Émªa®¢Q£$Æ¢?¡$£¡¥%«£¡Æº)»µªm¥6°?§¨[«)ªaµßø[¨´¤

°$ÁÅ$¹§®M£$

©

¡£¡µN$$ªÐÀ«ªMºÐí¹§°?Y¤

£$¨ªa¢íÑm°¡£$ªa¢¶±´ªa§H®¢Ê£$?Àx¼ZÐµ¢1£À£¡ª

¥¢1£$¨«

©

£$?r)§¡£{µªZ¥?°?§½¢B£$.º)?®¢ ¿~«~$ªaµ

§¨«~£½£¡ªB)´aQ£ÐÃ»£$6£r?$¹¢1£$½£$?

©

¡£¡µ £¡ª`Í6¢?¥

£$>Ð¨´aQ£¢6¡³TÅ?È¶¶º°?¡£$>$$ª¶

²7 $?ª³ ¢¢í]Ä6§¨{ÇmÅÑ1°$£$¨ªa¢V?$x¤6$ªmºx)$¢´

3$¡'ªa¢?$Ä6§¨±«ª>û1¶Áü ª«£$»$$ª)7¥?$£¡)¨Ä6°£¡¥

µªa¢´>µ½ªm¥?°6§¨teÁ>¿ÂÁ¶Øaü½Ã[¢?¥eÙÇ¿*m¶Ìaü½ÃÐ¶evªa¡£

$$ªÐ]®¢{µªZ¥?°?§eÇ7$M¥?°?£¡ª ¢?ºxª$)ºx£Æ6Ð$¢´

¿~ù¶aü½ÃÐ¶.M³TªVª«À£$£¡¢µªZ¥?°?§¨¿*eÙÌV¢?¥eaÃ

ººxªa°?¢1£«ª½µ½ª$Ê£$?¢Ë?§¨«ª«3£$Ê$$ª)¶B

(5)

©

evªZ¥?°?§¨ evªm¥?°6§¨ ¥Í6¢?¨£$ªa¢ Ô$$ª)>¿ü½Ã

¿*eÁÃ Ä

©

³TªÐ¥V?$x¤?)ªmºx$)¢´í¿Ö¡6§£ ÀzÄ6¢?¥0ÀY$§®§?º)º)ÉÃ Á¶Ø

¿*eÙÇaÃ Õªa¢?$£¡)°?ºx£$¨ªa¢Êª«¢1£¡)¢?§Ñm°¡£$¨ªa¢Ê)?$¡¢1£$£$¨ªa¢ m¶Ç

¿*eÙÌaÃ 7)¨¹£$¨ªa¢Vª«]x¼m'ºx£¡¥¢?¡³£

©

' Ìúm¶ù

¿*evù1Ã Ä

©

³TªÐ¥í¡§¨ºx£$¨ªa¢¿Ö¢?ºxª$$ºx£$§

©

¥?¥?¥Vªx¼º§°?¥¥9Ã gm¶Ø

¿*eaÃ Ä

©

³TªÐ¥Vx¼m9¢?$¨ªa¢V¥)¨)Ä6§¨±Ä6°?£Àµr$$¢?´ Çm¶û

¿*eÙúaÃ ²7ºx£$°?§$£¡)¹Y§¿Ö§®µ½¨£ªa¢í6)$´%¢m°?µ%Ä'Mª )zÃ Á¶ú

¿*e¦ûÃ ì $$´{ªa$£Â¤Í6§¨£¡)¢?´V¿Ö®¢?ºxª$$ºx£$§

©

¥?$ºÐ¥¥9Ã Á¶ú

¿*egaÃ èÂ¥¢Q£$Í6º£$¨ªa¢Êª«º¢?¥?¥6£¡>¢?¡³) gm¶â

¿*eÙØaÃ ²7¢?¡³)¢ÉZ¢´ úm¶Ì

¿*eÁâQÃ ²7¢?¡³«ªÐµ{°?§®£$¨ªa¢ ù¶ù

$¨Ä9§¨zÃ7«~ª)°Ä6¡Ñm°¢1£%µªZ¥?°?§¨%£¡ªÙ'$«~ª)µ£$¨

£$¡É¶

p

¢¹»£$½¥)¨¹£$¨ªa¢¦ª«À£$½x¼Z'ºx£¡¥

¢?¡³£

©

¿ÖµªZ¥?°?§¨eÙÌaÃ«~§®ÅQ£$7¡£ª«Æº¢?¥?Û¤

¥?£¡7¢?¡³)T®¥¢Q£$Í?¥¢½£$? $£¡)¹¥½6$)´

M£$µ?£

©

¢íÇgm¶ÇaüNª«[£$>º$»¿~³ ?¢V£$

¢?¡³£

©

'% °?¢ÉZ¢ª³ ¢Ãª7ºxªa¢Q£$®¢?£$ ³ $ªa¢´

¢1£$¨£$¨«~ªgm¶ÇaüF¿~³3¢£$ ¢?$³T£

©

'®¢?ºxª¡¤

$ºx£ÐÃÐ¶è «H£$? É

©

³ª)¥?M°?$¥Ê«ªM6)$´ $£¡)¨¹§

$¢ª£7x¼m6¢6¥¥í³ £$v£$»¡µ½¢1£$º§§

©

)§£¡¥

«~ª)µ½ªZºº°$)®¢´¢±£$¢6¡³TÐT¿ÖµªZ¥?°?§¨eaÃÐÅ£$

$§¨¹¢1£6$$´3$%µ½®$¡¥¶

?v¡§¨ºx£$¨ªa¢éª« É

©

³ª)¥?¬«~$ªaµ £$Ù¢1£¡)¢?§

Ñm°¡£$¨ªa¢()?$¡¢1£$£$¨ªa¢¿ÖµªZ¥?°?§evù1Ã¸ºxªa°?6§¨¥

³ ¨£$¦£$?É

©

³ª)¥Bx¼Z6¢?)¨ªa¢Ë¿Öµªm¥6°?§¨2ewaÃ±´¢Z¤

)£¡VÌYù¶úaüª«£$?r$$ªÐ¶VæTª£$£$¡.µªZ¥?°?§¨

87'ºx£>£$ªa°£¡6°£±ª«T6$$´$£¡Ð¨¹Y§Å')¢?ºx£$

¡£]ª«?$£¡)¹¥{9$$´[¥'¢?¥?Hªa¢{£$?æªmªa§¨¢

Ñm°)¨ Ä6°?§¨£3¢?¥v$°?Ä6µ½¨£¡£¡¥v£¡ª.£$?%$£¡)¹Y§Æ¢Z¤

´a¢ Ä

©

£$»¯±²

©

¡£¡µV¶

evªm¥6°?§¨eúÙ¢?¥eÙû$Ê$$ªa¢6$¨Ä6§¨«~ª£$

$£¡)¹Y§1ª«6$$´]³ ?$T¢?$³T)]µ½

©

ºx£$°?§§

©

ªZºº°¶Æ ¨ºxªaµ{Ä6¢¥±$)ª)HÌm¶ÇaüÊ¶ÆèÂ¢>µªZ¥?°?§¨

eÙú£$$%)>6)µ£¡Ð£¡ªrºxªa¢1£¡$ªa§£$?%¢1°6µ%Ä'

ª«$£¡)¹¥v¥ªZº°?µ¢1£$ ¢?¥v6$$´Å ³T§§Æ

£$%) ª«ÆºÐv9$$´¶

²7¢?¡³ç?$ªZºx$$¢´¥ªa¢_¢Úµ½ªm¥?°6§¨Åeg

£$$ªa°?´aleÁâZ¶ p ?¢¦£$x¼m'ºx£¡¥B¢?$³T £

©

'

íºxª)$ºx£$§

©

¥£¡ºx£¡¥Å%£$¥?¢Q£$¨Í9º£$¨ªa¢_ª«£$

º¢?¥?®¥?£¡»¢6¡³TÐ¿Öµ½ªm¥?°6§¨/egaÃ ?$ªZ¥?°?ºxfgm¶âQü

$$ªÐ¶áÌm¶Áü $)ª)¦$¥6°`£¡ªö¢?µ¥Ü¢1£$¨£

©

$ºxª´a¢?£$¨ªa¢ð¿Ö¢?ºxªaµ9§¨£¡V¥?ºx£$¨ªa¢?Ð¨ÐÃ ¢6¥çù¶Øaü

$¸¥?°Ù£¡ªË¡6°Ð¨ªa°?r¢6¡³TV9£¡£¡)¢öµ½£$º)6¢´¶

evªm¥?°6§¨2eØB¢?¥eÁâB«Ö§À£¡ªB)¢É£$vºxª)$ºx£

¢?¡³.³ ¨£$?¢Ò£$£¡ª`$£$°)¢¥ ¢ßÁâZ¶ûü ª«

£$¦º¡¶ÆeíªZ¥?°?§eÙØB«Ö§¬¨« £$¦ºxª$$ºx£V¢Z¤

¡³{º¢?¥6¥?£¡r{¢ª£>)¢?É¥B³ ¨£$6¢¦£$r£¡ªmÅ

³ )eÁâr«Ö§3¨«[£$»$£$°)¢¥¢?¡³7$£¡)¢´

{¢?ºxªaµ6§£¡ÅÆ¢?µ§

©

£%¥ªm%¢ª£{Í?£>³ ¨£$6¢kâ

Ä

©

£¡¶

¸N:ÈÇ É+BH#I0Mb>8»(BlB^>8>OI0>OH

¡ºxªa¢?¥é$£rª«±x¼m')®µ¢Q£$rºxªa¢?$¡£$rª« ¥?Â¤

Ä6§®¢´3£$? µ½¢¢?£$°?)§§¢´a°?´ $¡ªa°)ºx°?¡¥

¢Ù£$r¯±²Ú

©

$£¡µVÅH¢?µ§

©

£$½ººx$ £¡ª p ªÐ¥Z¤

q3£¢?¥!£$é¢?µ¥!¢Q£$£

©

$ºxª´a¢?ÈÅr£¡ªÂ¤

¡$[£$?¨]®µ6ºx£]ªa¢£$Àª¹)§§¢6¡³Tºº°Ðº

©

¶

q3ª£¡.£$6£%£$.6)¡{¢¢1£¡´)§T6$£{ª«Mªa°

Ñm°¡£$¨ªa¢{?$ªZºx$$¢?´3µªZ¥§m¢?¥%£$?$«ª)¨£[]µ¤

?)ºx£$®º§£¡ªr¥?)Ä6§¨ ¨£¶

7¢ª£¡ö³ £$Ê!£$:Ä6¡§®¢

©

¡£¡µ '$«~ª¡¤

µ½¢?ºx±³ ?¢.§®§6$$ªa°)ºxM$±¢?Ä6§¥¶7?$x¤

º$ªa¢`$ºxª$¸¿*eÓ Ó7Ã%¥$ª9>£¡ª(U(^!¨« p ª)¥.q3£

{¥?$Ä6§¥¶. ¬¥Ð¨¹Y£$ªa¢¸ª«£$.¢?¡³{£

©

'

¿ÖµªZ¥?°?§¨xeÙÌaÃr¢?¥ÒÉ

©

³TªÐ¥éx¼Z6¢?$ªa¢ß¿ÖµªZ¥?°?§¨

eaÃ.«)ªaµø]´a°$ËÁ¸$¸£$¸£d³Tªéµ½ªm¥?°6§¨.£$?£

$µªa¡£®¢6Ë6°¢?ºx¥_Ä

©

p

ª)¥.q3£¶øªÙx¼µ¤

6§¨Å'£$ p ª)¥Kq £>¢ªa°6¢Ù6¨))ºÐ?¨±¡'º¨«

©

£$?£

£$Vºxªa¢?ºx?£/[R)À¦¡'º§£$¨ªa¢ª«/}C*-)|Å

³ ?®º)B¢B£$°)¢B{íÉZ¢?¥¦ª«t[6")?¶í?r¢?¡³

£

©

'«ª%¯±âaÌaû1¾çýþÊÿn$82~ÿ0Zd)3\/~ÿ6WSj

ÿ R*") [;Y[0R)Ðÿ)()$o)}+)~ÿÌ)";

· ¦")?¶! Ù

©

¡£¡µº¢6¢ª£V¥)¨¹Ù£$¸¢Z¤

¡³£

©

'Mºxª$$ºx£$§

©

°?¢?§¨)]£?ººx$£¡ª p ªÐ¥Z¤

q3£?))º)6¨Ä'º°?$ £$±µ%Ä9¨´a°ªa°?Ñ1°?¡£$¨ªa¢

¡£¡µ þVÿ?§¨ªa¢?À¥?ª1¢?ª£?$ª¹Z¥M¢

©

º§®°M£¡ª

(6)

D

N

P

N

P

N

P

ÍÍÍ ÍÍÍ ÍÍÍ ÍÍÍ ÍÍÍ ÍÍÍ ÍÍÍ ÍÍÍ ÍÍÍ ÍÍÍ ÍÍÍ ÎÎ ÎÎ ÎÎ ÎÎ ÎÎ ÎÎ ÎÎ ÎÎ ÎÎ ÎÎ ÎÎ ÏÏ ÏÏ ÏÏ ÏÏ ÏÏ ÏÏ ÏÏ ÏÏ ÏÏ ÏÏ ÏÏ ÏÏ ÏÏ ÏÏ ÏÏ ÏÏ ÏÏ ÏÏ ÏÏ ÏÏ ÏÏ ÏÏ ÏÏ ÐÐ ÐÐ ÐÐ ÐÐ ÐÐ ÐÐ ÐÐ ÐÐ ÐÐ ÐÐ ÐÐ ÐÐ ÐÐ ÐÐ ÐÐ ÐÐ ÐÐ ÐÐ ÐÐ ÐÐ ÐÐ ÐÐ ÑÑÑ ÑÑÑ ÑÑÑ ÑÑÑ ÑÑÑ ÒÒÒ ÒÒÒ ÒÒÒ ÒÒÒ ÒÒÒ ÓÓ ÓÓ ÓÓ ÓÓ ÓÓ ÓÓ ÓÓ ÔÔ ÔÔ ÔÔ ÔÔ ÔÔ ÔÔ ÔÔ ÕÕ ÕÕ ÕÕ ÕÕ ÕÕ ÕÕ ÕÕ ÕÕ ÖÖ ÖÖ ÖÖ ÖÖ ÖÖ ÖÖ ÖÖ ÖÖ ××× ××× ××× ××× ××× ××× ××× ××× ××× ××× ××× ××× ××× ××× ××× ××× ØØØ ØØØ ØØØ ØØØ ØØØ ØØØ ØØØ ØØØ ØØØ ØØØ ØØØ ØØØ ØØØ ØØØ ØØØ ØØØ Ù Ù Ú Ú Û Û Ü Ü

50

20

200

0.421

0.419

0.414

0.36

0.37

0.38

0.39

0.40

0.41

0.42

0.369

0.374

0.376

0.400

0.392

Precision (MRR)

0.400

= 50

= 200

= 500

$¨´a¢6¨Í6º¢1£]«~ª£$íþVÿ?Ñm°¡£$¨ªa¢6¶ p ¢ p ªÐ¥Z¤

q3£]Æ¥?$Ä9§¨¥Å£$evÓ3Ó«ª]£$¬þVÿmÑ1°$£$¨ªa¢?

¥$ª9À£¡ªUr.ÝO! ºxªaµ6$¥v£¡ªwUG(4! «ªM£$?>¢Z¤

£$¨$ ¡£¶T?T$$°?§£À®¢?¥?º£¡£$?£M£$ ¹Y§®Ä6§Û¤ ¨£ © ª«9§x¼Zºxª¤J$µ½¢Q£$®º®¢«ªÐµ½£$¨ªa¢{Äºxªaµ½µª$ µ'ª$£$¢1£M«~ª ¥?¨÷rº°6§¨£Ñm°¡£$ªa¢?¶ æ © ¥?)Ä6§¢´±£$±¢?µ¥r¢Q£$£ © $ºxª´a¢6ÅZ£$ ¢?¡³.?$ªZºx$$¢´§º$ÉZr£$?¡µ½¢1£$ºÙ¢«~ª)µ½Y¤ £$¨ªa¢2¢ºx$$ © £¡ªê¥¢1£$¨« © º¢?¥6¥?£¡_¢?¡³)¶ ¾ªmªa¡ 6?$ª¼µ½£$ªa¢?[«ª£$º¢?¥?¥6£¡¢?¡³) $Mºxªaµ6°?£¡¥{Ä9¡¥¡£¡)ºx£$§ © ªa¢{É © ³ª)¥?µ½£$ºÐZ¤ ¢´¶èÂ¢V£$?®Mº¡>£$>6$º$¨ªa¢Ê¥)ª6M£¡ªUrc!¶ Þ Ï ôßAXmvEN¶{SõSzk6ô ßTAOQA'ô:kk?O1S Ñm°?¢1£$¨£$£$¨¹3)«ª)µr¢?ºxMª«9£$?7¯±²_ © ¡£¡µ {§$´§ © ¥'¢?¥¢1£ ªa¢£$¬µªa°6¢Q£{ª«M£¡x¼m£»$x¤ £¡)¨¹¥«~$ªaµ£$?»¥ªZº°?µ¢1£3ºxªa§®§¨ºx£$¨ªa¢ã£$µª$ £¡x¼Z£$£¡)¨¹¥.£$3Ä'£¡£¡T£$±º)6¢?ºx3ª«Í9¢?¥?¢´ £$¢?¡³¶]ä3ª³¹Åa6)ºx£$º§¯±²Ë © ¡£¡µ½]º¢Z¤ ¢ª£%87'ª)¥`£¡ªv?6§ ©

£$µ½ºxªa¢6$°?µ½¢?´·q¾ ì £¡ºÐZ¤

¢?Ñm°¿~¡'º§®§

©

6Ð$¢´1Ã6£¡ª¹

©

§)´µªa°?¢1£$

ª«Z£¡x¼m£¶Ó £¡)¨¹§6)µ£¡ÐH$°6¡¥ £¡ª 6$ª¹Z¥

£¡)¥x¤ª7ÙÄ'£J³¢ß£$µªa°?¢1£vª«£¡x¼Z£v6)¡¥ ¢1£¡ª3£$?¢?$³T[?$ªZºx$$®¢´3µªZ¥?°?§¢?¥»£$Àºº°Z¤ )º © ª«£$»x¼m£¡Ðºx£¡¥B¢?¡³)¶>?¯±²Ú © ¡£¡µ

?M£$?>«ªa§®§¨ª³ ®¢´5#% -} R [ZY"|¾

àWáâ

¤7£$¬µ½Y¼µ»°?µ¢m°?µ{Ä'%ª«¥ªZº°?µ¢1£$

)£¡)¨¹¥f«~$ªaµ ë$°Ä¤Jºxªa§®§¨ºx£$¨ªa¢ ¿Ö¥«Ö°?§¨£

¹§°%Çââ½«ªºÐvª«MÁÇ)°Ä¤Jºxªa§§¨ºx£$ªa¢?ÐÃÐãã

ä-å;æOç0è8é ê*ê-é"ë,ç0ì-çí*ì-îçï,é"ðç3ñ8ë îòñ8çKó{é ñé"ë,ç3ê4í-æ8çKå;ôbõö÷Oøùú

ónçñ#í;ù3ø,ðòðòç3ùí-îòø,ñé"êétê*çíø ü²ýþê*ç3èéìYé"í-çê úEù3ø,ðòðòç3ùí-îòø,ñOê

265

S

P

Precision (MRR)

3

6

10

20

40

110

59

43

32

Exec time (sec)

0.400

0.411

0.421

0.401

0.387

(nr. extra lines)

precision

time

ø[¨´a°$¦Ìm¾`èÂµ6ºx£.ª«>6$)´¦)ªa¢:?)º$¨ªa¢ ¢?¥Vx¼Zº°£$¨ªa¢í£$µ àWá ¤£$µrY¼Zµ»°?µÜ¢1°?µ{Ä'Hª«6)$´]?$ª¤ ºx)¡¥ß£¡ªö¥?¢Q£$¨« © º¢?¥?¥?£¡¢?¡³)ç¿Ö¥x¤ «Ö°?§¨£À¹§°ââQÃÐ¶ à

¤F£$?=$ §§ª³À¢?ºx «~ªiº) $x¤

£¡Ð¨¹¥6$$´B¿Ö¥«Ö°?§¨£»¹Y§®°ÙÁâ¦§®¢>Ä'x¤ «~ª$£$?)§¨$£¢?¥%«~£¡H£$T§£¡¡£É © ³TªÐ¥ µr£$º)9ÃÐã p ¢ áâ ¢?¥ á $r¡£%£¡ª$µ½§®§¨ ¹Y§®°Å £$Êx¼Zº°£$¨ªa¢ç£$µV§¨ªz³TrÄ6°£»$§¨¹¢Q£¥ªZº°Z¤ µ¢1£$¬¢6¥Ò6$)´¬µ½ © Ä'vµ½$$¥¶:ø[¨á°?$Ç §§®°?¡£¡)£¡[£$®µ6ºx£ª«'£$M6)µ½£¡) áâ ¢?¥ á ªa¢>£$T?$º$ªa¢>ºxªaµ6°£¡¥%ª¹]£$¢1£$¨$T¡£ ª«±ÁxùQúâí£¡¡£%Ñm°¡£$ªa¢?¶?¨á? £$½¢1°6µ%Ä' ª«H¥ªZº°?µ¢1£$$£¡)¨¹¥Åm£$?7?áT£$?3?)º$¨ªa¢ $ºxª$¶çè £½?6$¢1££$?£ á 6½¦$§£$¨¹§ © $µ½§®§¨3µ½6ºx£ ªa¢£$»?$º$ªa¢í£$?¢ á/â ¶7?

¥6°M£¡ª%£$ «~ºx£T£$?££$ $£¡)¨¹¥r6$$´$

$x¤ª)¥?$¥>Ä6$¥%ªa¢{ ¡£]ª«?§¨x¼º§«~£$°$Å$°?ºÐ

£$?£3£${¥?¢Q£$¨Í9º£$¨ªa¢Vª«£$»º¢?¥?¥?£¡{¢?¡³)

['$«~ª)µ¥»ªa¢£$M£¡ª

á

$x¤ª)¥?$¥6$)´¶

ø[¨´a°)±Ì»$ªz³ À£$7'ªa$$¨Ä6§ £¡)¥x¤ª7ÙÄ'£J³¢

ª¹Ð§§6$º$¨ªa¢¸¢6¥Bx¼Zº°£$¨ªa¢£$µr»í«Ö°?¢?º|¤ £$¨ªa¢çª«±£$V6)$´v$ ¶ËèÂ¢Q£¡)¡£$¢´a§ © ÅT£$ ?¨´a?¡£H?)º$¨ªa¢>)ºxª$TªZºº°)]«~ªÆ£$?T¥«Ö°?§¨£]¡£Â¤

£$¢´ÅÁâZ¶ p ?¢

(7)

©

£$>£¡ª$£$°)¢?¥¶

Ï

ôßAXmvEN¶ k?k6IbAXS

.$$°6§¨£$%6$¡¢1£¡¥®¢`?)¹m¨ªa°6{¡ºx£$ªa¢?ºxª¡¤

$¡'ªa¢?¥:£¡ªç£$B¡Ð§¥ Ä6¡§®¢ÙÐº)?¨£¡ºx£$°?$

«~$ªaµ ø[¨´a°?$Á¶ê?£Ê)ºÐ?¨£¡ºx£$°$¦.¢ «~ºx£V

$µ½6§¨Í?¥r¹)$¨ªa¢íª«Æªa°

©

¡£¡µF³ ?ºÐV°?¡M¡¹1¤

)§9«¥?Ä6º$ÉZÀ£¡ª{Ä'ªmªa¡££$?3ªz¹)§§'$«ªÐµ½¢?ºx¶

²7$ª³3¢®¢ø[¨´a°)¬ùÅ£$?ÊÐº)?¨£¡ºx£$°?$Ê³ ¨£$

«~¥Ä6º$ÉZ x¼Z£¡¢?¥? £$?½¡)§È¥ÙÐº)?¨£¡ºx£$°?$½¢

¡¹)§Æ³À

©

¶¦Ä

©

³TªÐ¥íx¼Z6¢?$¨ªa¢¿Öµªm¥6°?§¨?eaÃ

%¢??¢?ºx¥£¡ª¸¢?º§°?¥?½§¨x¼Z®ºxª¤J¡µ½¢1£$º¬§£¡)¢?Y¤

£$¨ªa¢?r«)ªaµ

p

ª)¥.q3£¶ð² ¢³µªZ¥?°?§V«ª.§¨ª´aº

?$ªz¹m®¢´¢6¥$°?¡£$Í6º£$¨ªa¢ ª«>£$¸¢?¡³)ÊÊ¢Z¤

¡$£¡¥¬Ä'«ª) ¢?¡³TÐ¢Ém®¢´¶ÆèÂ¢r¥6¥?¨£$¨ªa¢ÅQ£$$

§¨ªmª6íÄ'ºxªaµç¢¢Q£¡´Ð§%6)£vª«£$ç

©

¡£¡µV¾

£$Ê6$)´V$£¡)¨¹§M§¨ªmªö¿Ö§ª1ªöÁÃÐãM£$V§¨x¼ºxª¤

¡µ½¢1£$º¦§¨ªmª ¿Ö§¨ª1ªðÇaÃÐã%¢6¥ £$?Ù§¨ª´a®º6$ª¹Z¢´

§¨ªmª`¿Ö§¨ªmªÌaÃÐ¶

²7Ê6$£.ª«{§¨ªmª(ÁÅ £$`¯ÀY²

©

¡£¡µ¥$°?¡£$

æª1ªa§¢`Ñm°)¨±Ä'«~ª$6$$®¢´Ê£$µi£¡ªí£$?½$x¤

£¡)¨¹§¢´a®¢¶½è «T£$rªa°£¡6°£>«~$ªaµ £$?½$£¡)¨¹§

¢´a¢?.{£¡ªmª¦)µ½§§ÈÅÉ

©

³ª)¥Ë»¥$ª?'¥¢?¥

$£¡)¹Y§7$$°6µ¥¶!è «%£$?Ùªa°?£¡6°£VÊ£¡ªmªÒ§$´Å

»É

©

³ª)¥V¥?¥¥Ê¢?¥Ê»¢³ö¨£¡)£$¨ªa¢Ê¡£$)£¡¥Å

°?¢1£$§m£$Mªa°£¡6°?£$M¢¨£$?£¡ª1ª%§®$´ÅQ¢ª£¡ªmª

$µ½§®§È¶ p ¢§¨x¼Z®ºxª¤J¡µ½¢1£$ºÊºxªa¢?¢ºx£$¨ªa¢?{«~$ªaµ

£$ÙÑ1°$£$¨ªa¢Ë£¡ª`£$?v$£¡)¹¥Ò6$)´¬$Ù¢ª£

'ªa$$¨Ä9§¨ÅQ§¨ªmªÊÇ»T£¡)¨´´)¥¶T¯±°¡£$ªa¢rÉ

©

³TªÐ¥?

$¸$9§ºx¥_³ ¨£$

p

ª)¥Kq £Â¤Ä6$¥ß§¨£¡)¢6£$¨ªa¢?

¢?¥»$£¡)¹Y§]$)°?µ¥¶S¾ªmªrÌ )§¨]ªa¢±§¨ª´aº

?$ªz¹M£$?£À¹)¨Í?£$ °?¢6¨Í6º£$¨ªa¢?TÄ'£J³¢Ê£$

Ñm°¡£$¨ªa¢½¢6¥r§¨ª´aºÀ«~ª)µ½¶ p ¢r£$ °?¢?Í6º£$¨ªa¢?

«Ö§ÈÅ£$VÉ

©

³ª)¥?½$Vx¼Z6¢?¥?¥³3¨£$Ë¡µ½¢1£$Û¤

º§§

©

$§£¡¥í§£¡)¢?£$¨ªa¢? ¢?¥Ê$£¡)¨¹§'$$°6µ¶

]Ä6§¨%Ìm¾èÂµ6ºx£ ª«Æ«~¥Ä6º)Ém ªa¢í?$º®$¨ªa¢

ø¥Ä6º)É ì $º)¨ªa¢ èÂ¢?ºx$µ¢1£$§

¥?¥¥ ¿*evÓ3Ó7Ã ¢??¢?ºxµ¢1£

¢ªa¢ âZ¶ùQÇZÁ âQü

ì $$´%)£¡)¨¹§ âZ¶ùQúg

ã

ÁÁü

¿Ö§¨ª1ªÁÃ

¾x¼ºxª¤J¡µr¢Q£$º âZ¶YùQÇ

ã

Á#aü

¿Ö§¨ª1ªÙÇaÃ

ì

$ªz¹m¢?´v¿Ö§¨ª1ªÙÌaÃ âZ¶aûÇ aü

§¨ªmª6»ªa¢ç£$í¢?¡³½ºº°)º

©

¶çÊÉZ¢ª³3§¨¥´

Ä?$ªa°?´aQ£7¢Q£¡ª¬£$Ñ1°$£$¨ªa¢v¢?$³T)®¢´½?$ªZºx$3Ä

©

§¨x¼ºxª¤J¡µ½¢1£$º§¨£¡Ð¢?£$¨ªa¢?±?±£$?´a¡£ ¢Z¤

¥?¨¹Z¥?°6§7ºxªa¢1£¡)¨Ä9°£$¨ªa¢Å7«ªa§®§¨ª³¥ðÄ

©

£$`µºÐ?Y¤

¢?)µïª«]¥?¥6¢´^ÀY¥$ª66¢´{É

©

³ª)¥?¶

?¢?¡$£$ªa¢±ª«m£$?T§¨ª´aº[6$ª¹Z¢´µªZ¥?°?§¥?¥?

7¢³ºxªaµ½6§¨x¼¨£

©

§®

©

[£¡ª ¢?¡³Æ?)ªmºx$)¢´Å¢Z¤

Ä6§®¢´½µª${£¡)¥x¤ª7 Ä'£J³¢Ù?$ªZºx$$¢?´½ºxªaµ¤

6§¨x¼¨£

©

¢?¥Ê¢?¡³ ºº°Ðº

©

¶T]Ä6§¨±ù½$ªz³ À£$

ª¹Ð§§?)º$¨ªa¢V«~ª «~ªa° ¥67)¢Q£ $£¡£$¢´a¶

Í?)$£¡£¡£$¢´ÅC4%3(C%%)?Åaºxª$$¡'ªa¢?¥6'£¡ª£$

$µ½6§¨¡£M¯±²ð

©

¡£¡µï£$?£M¥ªmÀ¢ª£M°?¡±¢

©

q¾ ì

£¡º)6¢?Ñm°Êªí$¡ªa°?)ºx¶ï¸¢?¡³)V$¸x¼m¤

£¡)ºx£¡¥`«$ªaµi£$r¡£$$£%ª«TºÐ`6)$´Å]¢6¥¸$x¤

£$°)¢?¥Ù®¢v£$ªÐ¥ ¢Ù³ ?ºÐ£$?»9$$´±³$

$£¡)¹¥¶öí?)º$¨ªa¢Ërªa¢?§

©

âZ¶âaÇgm¶ p ¢

£$xq¾

ì

£¡º)6¢?Ñm°r$¢?Ä6§¨¥ÅT³ £$Ë£$íx¼m¤

ºx?£$¨ªa¢¬ª«£$7¥)¨¹£$¨ªa¢ª«£$3x¼m'ºx£¡¥.¢?¡³

£

©

'Å£$T6$º$¨ªa¢{µ?)ª¹Æ«$ªaµâZ¶âaÇg3£¡ª âZ¶Á#âZ¶

±¢?¡³Mºº°)º

©

¡£$§§9§µ½¨£¡¥rÄ'º°?¡±£$

º¢?¥?®¥?£¡3¢?$³T)Àº¢?¢?ª£TÄ' 6$ª')§

©

¥¢1£$¨Í?¥

³ ¨£$?ªa°£%ÉZ¢ª³3¢´v£$?¨»$µ½¢Q£$®º¬º£¡´ª

©

¿~'¡¤

¡ªa¢?Åº¨£$¢?¥Ò¡ªB«~ª$£$9ÃÐ¶Ëè «7£$v¥)¹Y£$¨ªa¢çª«

£$Ùx¼m'ºx£¡¥_¢?¡³Ê£

©

'Ù.§¡ª¢?Ä6§¥ÅM£$

?$º®$¨ªa¢Ù$ºxª$º)?¢?´ £¡ªíâZ¶ùQúgm¶»ø[¢6§§

©

Å³ ¢

§§m«~¥Ä6º$ÉZ$À¢?Ä6§¥{£$?À6¨´a¡£]ª¹)§®§Z?$x¤

º$ªa¢¸ª«MâZ¶aûÇí>ºÐ?¨¹¥¶íÕªaµ9)£$¨¹§

©

Å]£$

¢?¡³6$ªmºx)$¢´%µ½ªm¥?°6§¨[ª«ª£$¯±²_

©

$£¡µ½

°?$°6§§

©

¡9¢Vª¹7§¨¹§ Ç¢6¥vÌ«~$ªaµF]Ä6§¨ ù¶

]Ä6§¨ ù¾ ì $«~ª)µ½¢?ºx%ª«]¢?$³T3?$ªZºx$$¢´

²7¢?¡³?$ªZºx$$¢?´ eíªZ¥?°?§¨ ì $º$ªa¢

ºxªaµ6§x¼Z¨£

©

§¹§ °?¡¥ ¿*evÓ3Ó7Ã

¿ÂÁÃ±¨$ºx£ x¼m£¡)ºx£$ªa¢ e`Á|¤eÙúmÅ âZ¶âaÇg

e`Áâ

¿ÈÇaÃ¾Hx¼Zº§Hµ½£$º)?®¢´ e`Á|¤e¦û1Å âZ¶Á#â

eØz¤eÁâ

¿ÈÌaÃ·mµr¢Q£$º%µ½£$º)?®¢´ e`Á|¤eÁâ âZ¶ùQúg

¿~ù1ÃÀø¥?Ä6º$ÉZ ¢6Ä6§¨¥ §§ âZ¶aûÇ

?Í6¢?§?)º$¨ªa¢ $ºxª)H«~ªÆÓ3ÔÕT¤gmÅÓ3ÔÕT¤

Ø»¢?¥VÓ3ÔÕT¤dÇââÁ $ )¡'ºx£$¨¹§

©

âZ¶mÅâZ¶gâZÅ

¢?¥íâZ¶aûYâZ¶À)«ª$%£$>?$º®$¨ªa¢Ê¥?¥V¢ª£ ¹Y

©

µ»°?º)B®¢Ù$6¨£¡ª«£$½6¨´a ¥´$ª«À¥?¨÷rº°6§¨£

©

(8)

Question

+ M3 + M4

M1 + M2

M5

alternations

+ lexico−sem

M6

M7 + M8

M9 + M10

Answer

Loop 1

Loop 2

proving

Logic

Loop 3

ø[¨´a°) ù¾² Ðº)?¨£¡ºx£$°?$>³ ¨£$Ê«~¥Ä6º)Ém

á°?´{?$ªZºx$$¢´¢Êªa°? © ¡£¡µV¶ EHCXGdRSPEÆCS Òµr¢(ºxªa¢?º§°?)¨ªa¢(¸£$?£`£$Òª¹Ð§§½'¡¤ «~ª)µ½¢?ºxª«»¯±²Ý © ¡£¡µ½.¬¥6¨$ºx£$§ © $§£¡¥ £¡ª £$:¥?£$Úª«í¢?£$°)§¬§¢á°?´:?$ªZºx$$¢?´ð$x¤ ¡ªa°Ðºxr¢?¥é£$í£¡ªmªa§r°?$¥Ò«~ª¬¢?¡³rÍ9¢?¥?¢´¶ ²7 $ª³3¢Ù®¢]Ä6§¨»ùÅ£$'$«ªÐµ½¢?ºx»ª«T¢«~ª¡¤ µ½£$¨ªa¢:$£¡Ð¨¹Y§3£¡º)6¢?Ñm°¬®¬$á¢?¨Í6º¢1£$§ © ¢Z¤ ?¢?ºx¥>³ ¢>§¨x¼ºxª¤J¡µr¢Q£$º¢«~ª)µ½£$¨ªa¢ «~°?§®§ © x¼Z6§¨ªa¨£¡¥ £$?$ªa°áªa°££$?¢?¡³HÍ6¢?¥6¢´?$ªZºx$¶ ]Ä6§¨Çr§®§°?¡£¡)£¡ £$?£±£$»)«ª)µr¢?ºx%Ä'ª£Â¤ £$§¨¢º)Ém ª«Æªa°?7¯±² © $£¡µ$>¥?°? £¡ª½£J³ªrµªZ¥Z¤ °?§¨Å¢?µ½§ © £$¥)¨¹£$¨ªa¢íª«£${x¼Z'ºx£¡¥¸¢Z¤ ¡³T£ © '7¢?¥r£$? É ©

³Tª)¥.x¼Z6¢?$ªa¢¶] Ä'ª£Â¤

£$§¨¢º)Ém%$r¢ª£>¡'º¨Í9º{£¡ªVªa°?»¯±²2 © $£¡µ Ä6°£ $Ë?ºx££$?í§µ½¨£$£$ªa¢?%ª«±º°$)¢Q£r¯±² £¡º)?¢?ªa§¨ª¤ á¨¶r¯±°?¡£$¨ªa¢¦¢?¡³)¢´V © ¡£¡µ½>)«ª)µ Ä'£Â¤ £¡r³3¢ç£$í$§¹Y¢1£6$$´¬¢?¥Ë£$º¢?¥?Û¤ ¥?£¡Ë¢?¡³)¸$Ëº§¨Ð§ © ¥Í6¢¥ê¢Ü£$ËÑm°Â¤ £$¨ªa¢?¶[ µ½¢¬?$ªÄ6§¨µ2£$ §º)Érª«'ª³$«Ö°?§ $ºÐµÊ¢?¥:§¨´ª)¨£$?µr½«~ª.µªZ¥§®¢´Bºxªaµ6§¨x¼ Ñm°¡£$¨ªa¢?[¢{ª)¥[£¡ª>¥Ð¨¹µ»°?ºÐ¢«~ª)µ½£$¨ªa¢ Ù'ªa$$¨Ä6§Å±¢6¥ß«~ª¦)«ª)µr¢´Ò ³T§§¨¤á°?¥¥ ¡)ºÐí£$)ªa°áV£$ªa°?)¢?¥?Àª«[£¡x¼m£3¥ªmº°6µ¢Q£$¶ ?½§¨x¼Z®ºxª¤J¡µ½¢1£$º½¢«~ª)µ½£$ªa¢¸µ'ª$£¡¥¦¢ £$¬¯±²2 © ¡£¡µi£$?$ªa°á¦£$½)£¡)¨¹§[«¥Ä9º$ÉZ Ä?)®¢á3ºxªa¢6$¡£¡¢1£ µ?$ªz¹µ¢1£$±ª¹%¡Ð§]?$ª¤

ºx$$®¢´¶ ì ¡¤Jºxªaµ½ªa¢?¢Q£½)$ª)$í¡6$¥ç°?¢?Û¤

«~ª)µ½§ © ª¹[£$Í6)¡£«~ªa°Æº§)¡ª«?Ñm°¡£$¨ªa¢%ºxªaµ¤ 6§¨x¼¨£$¨ÅÀ®¢?¥?º£$¢?´`ª³Ýªa°Ê © ¡£¡µfµ?)ª¹¥ ª¹7£$ © )¶ ¦k.¶Âk?Oak6CTXZk6S ÿ ñ8ç! "$# öø,ðòðòîòñ8ê%"Sé ñ8÷ &' îòñ8ë æé"ð þ)()(( * ñ8ê,+çì

ç!-#í*ìYé ùí-îòø ñ

/. ñ102,3547686:9);=<>)?@3,ACBED6@FGBEDH'IIKJ;L6:9@M/N)BPO2:NJ Q N)<>)ON%>6R0'2:354S67?8?S;E<>UTV3G<5A5672:67<K4S6@W=HXM Q 0$Y[Z\]\]\S^_"8èé"ë,ç3ê þ)`)a_bc)(Cý)"

çéí*í-ðç)"edé ê*æ8îòñOë í-ø,ñ

õ gf ì-ç3ù7hK"gi gf û ì-ë çì5"kj gl çì*ì-øe"gj nm îì-ê*ùYæ8ó{é"ño"np nm ø û ê*ç" #

j4îë æí5".é ñ8÷

.7 #fé ñ8î

þ)(()(

Um

ø_+kí-øç3ï é ð é"í-çq ø ì

rû

ç3ê%í-îòø,ñé ñ8ê,+çì-îòñ8ëê, Cê%í-ç3óç3ï,çì: ?÷8é%

ss

é"ñ÷ê%í-îðòð;ë çí

ì-çé ðn+;ø"ì8hf÷Oø ñ8ç

. ñ02,3547686:9);=<>)?t3,ACBEDe6&Z]<K9TV3)<%A5672867<u476 3)< Q N)<>)OeN5>6Xv$67?73)O2:4S67?wNG<K9/x'yGNJOeN)BP;[3)<W Q vx/TzY[Z\]\\7^5" í-æ8ç3ñOê%"u{bì-çç3ù3ç em éìYé ÿ é ë,î û "# |

ée}ê*ùé"é ñ8÷

#fé îòø"ìYé ñ8ø

þG(()( õk-Cèçì*ú îòónç3ñí-ê'+Nîòí-æø,èç3ñCú÷Oø,ó{é"îòñí-ç!-#í û é"ð r#û ç3ê%í-îòø,ñé ñ8ê,+çì-îòñ8ë . ñ~0'2:354S686:9);=<>)?3,ABED6)BEDS<eB6727<uN)BP;[3)<uNJTV3)<%A5672867<u476 3)<TV3)RIuOBLN)BP;P3)<KNJ Q ;=<>)O;E?SBP;P4!?qWST Q :MYLZ\]\]\7^5" é é"ì*ú ÿ ì û ù7hçño"e{Sçì-ó{é"ñ m éìYé ÿ é ë,î û "p

#ø,ð÷Cø"ï é ñn"#

/| é}ê*ù3é"# C û ìY÷Oçé"ñ û " ô #îæ8é ðòù3çé"ô {ìP û " ô û ê%" l jé"ù]é"í û }ê û | #ø ì_éì-ç3ê*ù û "é ñ8÷ô f û ñOç3ê*ù û þ)((#ý ñOê,+;çì-îòñOëùø,óú

è8ðòç!-"Nðòîòê%íé ñ8÷dù3ø ñ#í-ç7-Cí

rû

çê%í-îø ñ8ê@+Nîí-æðùùê

rû çê%í-îø ñOú é ñOê,+;çì-îòñ8ëxê*çì-ïçì . ñ0'2:354S68689G;=<>)?3,ABED6\GBEDV6:)B vxBP2S;L67yGNJTV3)<5A5672:67<K4S6WSovxwTzYLZ\\]7^5"{yé"îí-æ8çì-ê ÿOû ì-ëe"

#fé"ì: Cðé ñ8÷

eR.: å õ m øï "j {Sçì ÿ çì5" @m çì-ów%é)hø ÿ "ö t j^îòño"/é ñ8÷ p ôbéïCîòùYæé"ñ÷CìYé"ñ þ)()(Cý åø5+²éìY÷ê*ç3ó{é"ñí-îùê%ú ÿ é ê*ç÷é"ñOú ê,+;çìèOîñOèø,îòñí-îòñ8ë . ñ02,3547686:9);=<>)?3,ABED6twON)< Q N)<Y >)OeN5>66:4SD<K3Js38>) TV3)<5A%672867<K4S6W= Q kYLZ\]\]7^5" é"ñpSîçë,øe"

ö;é ðòîüø"ì-ñ8îé

&¡.

í*í OùYæ8çì-îé"æo"#

¢l ìYé ñe£"d ¤ æ û "é ñ÷ C ôbé"í-ñ8é"ú èéì8hCæOî þG((#ý ¦¥ û ç3ê%í-îòø,ñ+é ñOê,+;çì-îòñ8ë û ê*îòñ8ë?ó{éG-Cîòó û óú

ç3ñí*ì-ø,è ¦ù3ø,ónèø ñ8ç3ñí-ê

. ñU0'2:354S686:9);=<>)?3§ACBED6CZ]<K9¨6867B[Y ;=<>q3,AXBED6RM/3)2SBED@HX@672S;[48N)<TgDeN7IB672C3,ARBED6Hw?8?7354!;[N)BP;P3)< A%3)2TV3)RIOBLNGBP;[3)<KN)J Q ;=<>)O;=?8BP;[4!?W=MHXH&T Q Y[Z\]\]S^_" | îí*í-ê%ú ÿ8û ì-ë,æo" | ç3ñOñ8ê, Cðòï,é"ñ8îé # |

é}ê*ùé{é ñ÷

um é"ìYé

ÿ

é ë î

û

þ)()(Cý

å;æOçîñCüø"ì-ó{é"í-îòïçoì-ø,ðòç

ø üdø ìY÷

çíîñø,èçñOú÷Oø ó{é îòñ

rû ç3ê%í-îòø,ñ+é ñ8ê,+çì-îòñ8ë . ñ 02,354S6S6:9);E<>)?R3,AXBED6Z<K9/¨6S67BP;E<>3,ABED6$M/3)27BEDHX@672S;[48N)< TgDeN7IuB6

Performance Issues and Error Analysis in an Open Domain Question Answering System

9;:E9 =/FGD.H#H#BH2IKJLNMBHAO@GIPH

Volkswagen AND bug

built − build

 )º¨?$ªZº§)¢ÉZ¢´Ä6$º§®§

Precision (MRR)

«Ö§ÈÅ£$VÉ

Question

)º¨?$ªZº§)¢ÉZ¢´Ä6$º§®§

«Ö§ÈÅ£$VÉ