Automatic Extraction of Semantic Similarity of Words from Raw Technical Texts

$XWRPDWLF([WUDFWLRQRI6HPDQWLF6LPLODULW\RI:RUGVIURP5DZ7HFKQLFDO7H[WV

$ULVWRPHQLV7KDQRSRXORV1LNRV)DNRWDNLV*HRUJH.RNNLQDNLV

:LUH&RPPXQLFDWLRQV/DERUDWRU\

(OHFWULFDO &RPSXWHU(QJLQHHULQJ'HSW8QLYHUVLW\RI3DWUDV 5LRQ3DWUDV*UHHFH

^DULVWRPIDNRWDNLJNRNNLQ`#ZFOHHXSDWUDVJU

$EVWUDFW

,QWKLVSDSHUZHDGGUHVVWKHSUREOHPRIH[WUDFWLQJVHPDQWLFVLPLODULW\UHODWLRQVEHWZHHQOH[LFDOHQWLWLHVEDVHGRQFRQWH[WVLPLODULWLHVDV WKH\DSSHDULQVSHFLDOL]HGWH[WFRUSRUD2QO\JHQHUDOSXUSRVHOLQJXLVWLFWRROVDUHXWLOL]HGLQRUGHUWRDFKLHYHSRUWDELOLW\DFURVVGRPDLQV DQGODQJXDJHV/H[LFDOFRQWH[WLVH[WHQGHGEH\RQGLPPHGLDWHDGMDFHQF\EXWLVVWLOOFRQILQHGE\FODXVHERXQGDULHV0RUIRORJLFDODQG FROORFDWLRQDOLQIRUPDWLRQDUHHPSOR\HGLQRUGHUWRH[SORLWWKHPRVWRIWKHFRQWH[WXDOGDWD7KHH[WUDFWHGVHPDQWLFVLPLODULW\UHODWLRQV DUHWUDQVIRUPHGWRVHPDQWLFFOXVWHUVZKLFKLVDSULPDOIRUPRIDGRPDLQVSHFLILFWHUPWKHVDXUXV

,QWURGXFWLRQ

7KH XQFHDVLQJ H[SDQVLRQ RI KXPDQ DFWLYLW\ WR QHZ WKHPDWLF DUHDV LQ VFLHQFH VRFLHW\ DQG FXOWXUH UHTXLUHV D FRUUHVSRQGLQJ H[SDQVLRQ RI WKH QHFHVVDU\ OLQJXLVWLF UHVRXUFHVVRWKDWERWKKXPDQVDQGPDFKLQHVDUHDEOHWR SURFHVVHIIHFWLYHO\UHOHYDQWLQIRUPDWLRQ,QWKHFRQWH[WRI OH[LFDO NQRZOHGJH DFTXLVLWLRQ OH[LFDO VHPDQWLFV H[WUDFWLRQ LV QRW RQO\ WKH PRVW GHPDQGLQJ WDVN EXW DOVR FUXFLDO IRU PDQ\ DSSOLFDWLRQV VXFK DV /DQJXDJH 0RGHOLQJ ,QIRUPDWLRQ 5HWULHYDO ,QIRUPDWLRQ ([WUDFWLRQ 0DFKLQH 7UDQVODWLRQ :RUG 6HQVH 'LVDPELJXDWLRQ 7KHVDXUL FRQVWUXFWLRQ HWF 7KH DEXQGDQFH RI HOHFWURQLF WH[W LQ VHYHUDO QDWXUDO ODQJXDJHV DQG PDQ\ WKHPDWLF GRPDLQVDOORZVXVWRHPSOR\DXWRPDWLFFRUSXVSURFHVVLQJ PHWKRGV LQ RUGHU WR H[WUDFW OH[LFDO VHPDQWLFV 7KH GHYHORSPHQW RI DXWRPDWLF DSSURDFKHV LV LPSRUWDQW VLQFH WKH\ RIIHU UDSLG DFTXLVLWLRQ RU PRGLILFDWLRQ RI OH[LFDO UHVRXUFHV LQ QHZ RU G\QDPLFDOO\ HYROYHG VXEODQJXDJHV 0RUHRYHU ZRUG PHDQLQJV DUH GRPDLQGHSHQGHQW LQ JHQHUDODQGKXPDQVGRQRWDOZD\VVXFFHHGLQFODVVLI\LQJ VSHFLDO PHDQLQJV WKDW HYHU\GD\ ZRUGV DSSHDU LQ VSHFLILF WKHPDWLFFRQWH[WV

7KH PDLQ DSSURDFKHV SURSRVHG IRU WKH H[WUDFWLRQ RI OH[LFDOOHYHOVHPDQWLFNQRZOHGJHDUHV\QWD[EDVHGDQGQ JUDPEDVHG 6\QWD[EDVHG PHWKRGV 3HUHLUD 7KLVKE\ *UHIHQVWHWWH /L $EH UHSUHVHQWWKH ZRUGV DV YHFWRUV FRQWDLQLQJ VWDWLVWLF YDOXHV RI WKHLU V\QWDFWLFSURSHUWLHVLQUHODWLRQWRDJLYHQVHWRIZRUGVHJ VWDWLVWLFV RIREMHFW V\QWD[ UHODWLRQV UHIHUULQJ WR D VHW RI YHUEV DQG FOXVWHU WKH GHULYHG YHFWRUV DFFRUGLQJ WR WKH FRPPRQDOLW\ RI WKHVH SURSHUWLHV 0HWKRGV WKDW XVH ELJUDPV %URZQ HW DO RU WULJUDPV 0DUWLQ HW DO FOXVWHU ZRUGV FRQVLGHULQJ DV D ZRUG¶V FRQWH[W WKH RQH RU WZR LPPHGLDWHO\ DGMDFHQW ZRUGV DQG HPSOR\ DV FOXVWHULQJ FULWHULD WKH PLQLPDO ORVV RI DYHUDJH PXWXDO LQIRUPDWLRQDQGWKHSHUSOH[LW\LPSURYHPHQWUHVSHFWLYHO\ 6XFKPHWKRGVDUHRULHQWHGWRODQJXDJHPRGHOLQJDQGDLP SULPDULO\DWURXJKFOXVWHULQJRIODUJHYRFDEXODULHV

3UREOHP6HWWLQJ

,Q WKLV SDSHU ZH DGGUHVV WKH SUREOHP RI DXWRPDWLF H[WUDFWLRQRIVHPDQWLFVLPLODULW\UHODWLRQVEHWZHHQOH[LFDO LWHPV LQ UHODWLRQDO IRUP IURP ZKLFK ILQHJUDLQHG KLHUDUFKLFDO FOXVWHUV DUH REWDLQHG ,Q RUGHU WR UHVWULFW WKH YRFDEXODU\ DQG WKH ZRUG DPELJXLW\ DQG WR XWLOL]H LQIRUPDWLRQULFKWHFKQLFDOWH[WVSURFHVVLQJLVFRQILQHGWR

FRUSRUD IURP VSHFLILF GRPDLQV 7KLV UHVWULFWLRQ LV DFFHSWDEOH LQ WKH IUDPHZRUN RI 1/3 V\VWHPV ZKLFK DUH XVXDOO\RSHUDWLQJRQVXEODQJXDJHVDQGLQWHUHVWHGRQO\LQ GRPDLQVSHFLILFZRUGPHDQLQJV+XPDQUHDGDEOHWKHVDXUL VKRXOG SURYLGH DV ZHOO VHPDQWLF UHODWLRQV RI ZRUGV LQ UHOHYDQFH WR WKHPDWLF GRPDLQV 7KHUHIRUH ZH DLP DW GHYHORSLQJ D PHWKRG DSSOLFDEOH WR HYHU\ GRPDLQ IRU ZKLFK VSHFLILF FRUSRUD DUH DYDLODEOH LQ RUGHU WR H[WUDFW GRPDLQGHSHQGHQWZRUGPHDQLQJUHODWLRQV

,Q RUGHU WR DFKLHYH SRUWDELOLW\ ZH FKRVH WR DSSURDFK WKH SUREOHPIURPDNQRZOHGJHSRRUSHUVSHFWLYH1JUDP PHWKRGVZKLFKVKDUHWKHVDPHSHUVSHFWLYHIRFXVRQIDVW SURFHVVLQJ RI ODUJH FRUSRUD DQG FRQVLGHU DVFRQWH[WRQO\ WKH LPPHGLDWHO\ DGMDFHQW ZRUGV ZLWKRXW H[SORLWLQJ PHGLXPGLVWDQFHZRUGGHSHQGHQFLHV6LQFHODUJHFRUSRUD DUH DYDLODEOH RQO\ IRU IHZ GRPDLQV ZH DLPHG DW GHYHORSLQJ D PHWKRG IRU SURFHVVLQJ VPDOO RU PHGLXP VL]HG FRUSRUD H[SORLWLQJ DV PXFK DV SRVVLEOH FRQWH[WXDO LQIRUPDWLRQ ULFK LQ VHPDQWLF UHVWULFWLRQV 2XU DSSURDFK ZDVGULYHQE\WKHREVHUYDWLRQWKDWLQFRQVWUDLQHGGRPDLQ FRUSRUD WKH YRFDEXODU\ DQG WKH V\QWDFWLF VWUXFWXUHV DUH OLPLWHGDQGWKDWVPDOORUPHGLXPGLVWDQFHZRUGRUSKUDVH SDWWHUQVDUHRIWHQXVHGWRH[SUHVVVLPLODUIDFWV

6WRFNPDUNHWILQDQFLDOQHZVDQG0RGHUQ*UHHNZHUH XVHG DV GRPDLQ DQG ODQJXDJH WHVW FDVH UHVSHFWLYHO\ 7KURXJKRXW WKH SDSHU H[DPSOHV WDNHQ IURP (QJOLVK FRUSRUDDUHXVHGDVZHOO

([WUDFWLRQRI6HPDQWLF&ODVVHV

$ IXQGDPHQWDO LVVXH RQ FRQWH[WEDVHG GLVWULEXWLRQDO ZRUGFOXVWHULQJPHWKRGVLVWKHGHILQLWLRQRIWKHHIIHFWLYH VFRSH RI FRQWH[W $OWKRXJK PRVW DSSURDFKHV XVH ORFDO FRQWH[W RI RQH RU WZR ZRUGV ZH HPSOR\ D PRUH IOH[LEOH QRWLRQ RI FRQWH[W :H FRQVLGHU WKDW V\QWDFWLF UHODWLRQV ZKLFK LPSRVH VHPDQWLF FRQVWUDLQWV UDUHO\ H[FHHG FODXVH ERXQGDULHV (YHQ ZKHQ WKH\ GR DGYDQFHG OLQJXLVWLF WHFKQLTXHV DV HOOLSVLV DQG DQDSKRUD UHVROXWLRQ DUH UHTXLUHG WR GHWHFW WKHP 7KHUHIRUH ZH XWLOL]H FRQWH[WXDO GDWDH[WHQGHGXSWRFODXVHERXQGDULHV

:H GHWHFW FODXVH ERXQGDULHV DV D 6HQWHQFH ERXQGDULHV XVLQJ D VHQWHQFH VHJPHQWDWLRQ WRRO 6WDPDWDWRV HW DO E ,QWUDVHQWHQWLDO FODXVH ERXQGDULHV EDVHG RQ GHWHFWLRQ RI VSHFLILF FRQMXQFWLRQV ZKLFKDQDPELJXRXVO\LQWURGXFHWKHP

)LJXUH&RPSDUDWLYHUHVXOWVRIWKHHPSOR\HGVHPDQWLFVLPLODULW\H[WUDFWLRQDOJRULWKPV

1XPEHURIEHVWKLWV 3

UH FL VL RQ

&96(

9&96(

30

30QGLWHU

6DPSOH&OXVWHUVDIWHUVW_{LWHUDWLRQ 6DPSOH&OXVWHUVDIWHU}QG_LWHUDWLRQ

IHOO DXJPHQWHG

IUIFKILWOGHPJESMS\ IUIFKILWOGHPJESMS\XVG '$;&$& )76(GRZBMRQHV

'$;&$& GROODU)76(

FORVHGDW IOXFWXDWHV FRPPRQ SUHIHUHQFHVKDUHV

ULVH ORVVHV FRPSDQ\ JURXS PDQDJHPHQW XQLW

XSZDUG GRZQZDUG VKDUHKROGHUV

7DEOH6DPSOHFOXVWHUVGHULYHGDIWHUVWDQGQG_LWHUDWLRQ /H[LFDO HQWLWLHV RFFXUULQJ OHVV IUHTXHQWO\ RU DSSHDULQJ

YDULRXV OH[LFDO IXQFWLRQV LQ WKH FRUSXV GR QRW DOZD\V REWDLQ SURSHU VHPDQWLF FRQQHFWLRQV ,Q RUGHU WR XWLOL]H PRUHFRQWH[WXDOGDWDIRUVXFKZRUGVZHLQLWLDOO\DSSO\WKH VLPLODULW\ HVWLPDWLRQ DOJRULWKP DQG PDNH XVH RQO\ RI WKH WRSVFRULQJUHODWLRQV7KHFOXVWHULQJDOJRULWKPRSHUDWHVRQ D SRUWLRQ RI WKH VHPDQWLF HQWLWLHV DQG SURGXFHV FOXVWHUV ZLWK YHU\ KLJK SUHFLVLRQ 7KH SUHYLRXVO\ GHVFULEHG SURFHGXUHV VLPLODULW\ HVWLPDWLRQ KLHUDUFKLFDO FOXVWHULQJ DUH DSSOLHG LWHUDWLYHO\ XVLQJ WKH WDJV RI WKH SUHYLRXVO\H[WUDFWHGVHPDQWLFFOXVWHUVDVVHPDQWLFWRNHQV +DYLQJ UHGXFHG WKH QXPEHU RI SDUDPHWHUV PRUH GHQVH FRQWH[WXDOGDWDDUHH[SORLWHGDQGPRUHDFFXUDWHVHPDQWLF UHODWLRQVDQGFOXVWHUVDUHH[WUDFWHG6LQFHLQLWLDOFOXVWHULQJ HUURUV DUH SURSDJDWHG WR WKH QH[W LWHUDWLRQV GHJUDGLQJ VHYHUHO\WKHILQDORXWFRPHVSHFLDOFDUHPXVWEHWDNHQLQ RUGHUWRDYRLGWKHPHLWKHUE\FKRRVLQJDUDWKHUKLJKFXW RII VLPLODULW\ VFRUH RU E\ PDQXDOO\ GLVFDUGLQJ HUURQHRXV VHPDQWLFUHODWLRQV

([SHULPHQWDO5HVXOWV

7KH UHSRUWHG H[SHULPHQWV KDYH EHHQ FDUULHG RXW RQ D ZRUGV FRUSXV FRPSULVHG RI ILQDQFLDO QHZV RI ZKLFK ZDV FRQVWUXFWHG LQ WKH IUDPHZRUN RI D FXUUHQWO\ FDUULHG RXW 5 ' SURMHFW IRU ,QIRUPDWLRQ ([WUDFWLRQ IURP UDZ WH[W _{7KH HYDOXDWLRQ RI WKH UHVXOWV} ZDV SHUIRUPHG E\ FKHFNLQJ PDQXDOO\ WKH YDOLGLW\ RI H[WUDFWHG VLPLODULW\ UHODWLRQV DQG FOXVWHUV 7KH VHPDQWLF VLPLODULW\HVWLPDWLRQDOJRULWKPVGHVFULEHGLQVHFWLRQ ZHUHWHVWHGDQGWKHLUSUHFLVLRQZDVPHDVXUHG7KHUHVXOWV

_{3URMHFW³0,726´RIWKH*UHHN*HQHUDO6HFUHWDULDWIRU} 5HVHDUFKDQG7HFKQRORJ\

LQGLFDWHWKDWFRQWH[WVLPLODULW\GHWHFWLRQEDVHGRQSDWWHUQ PDWFKLQJ \LHOGV PRUH UHOLDEOH UHVXOWV WKDQ WKH YHFWRU VLPLODULW\ PHWKRG 7KLV GHPRQVWUDWHV WKH LPSRUWDQFH RI WKH FURVVFRUUHODWLRQ SURFHGXUH ZKLFK LV WKH RQO\ FRPSXWDWLRQDOO\ IHDVLEOH PHWKRG IRU SDWWHUQ VLPLODULW\ GHWHFWLRQ

2QH LWHUDWLYH ORRS ZDV SHUIRUPHG XVLQJ WKH 30 DOJRULWKP6DPSOHFOXVWHUVDIWHUILUVWDQGVHFRQGLWHUDWLRQ DUHVKRZQRQ7DEOH

7KH SUHFLVLRQ RI DOO WHVWHG DOJRULWKPV LQ GHWHFWLQJ VHPDQWLF VLPLODULW\ UHODWLRQV LQ FRQQHFWLRQ ZLWK WKH QXPEHU RI WKH EHVW VFRUHG UHODWLRQV PDLQWDLQHG FDQ EH VHHQRQ)LJXUH

5HJDUGLQJWKHFOXVWHULQJSURFHGXUHDVHWRIDERXW )6(V IURP WKH VXEMHFWHG WR FOXVWHULQJ ZDV SDUWLWLRQHG WR FOXVWHUV HDFK RQH KLHUDUFKLFDOO\ VWUXFWXUHG&RQVLGHULQJDW\SLFDOFOXVWHUIRUPHG)LJXUH ZH QRWH WKDW IURP WKH VHPDQWLF WRNHQV WKDW FRQVWLWXWH WKHFOXVWHUDOOEXWWZRGHQRWHGZLWKUHIHUWRIRUPVRI PRQH\ LQYHVWPHQW RU SURILW )URP WKH YRFDEXODU\ RI VHPDQWLF WRNHQV VXEMHFW WR FOXVWHULQJ WRNHQV EHORQJLQJ WR WKH VDPH FODVV ZHUH QRW GHWHFWHG WKHUHIRUH DFFXUDF\ DQG UHFDOO IRU WKH VSHFLILF FOXVWHU DUH DQG UHVSHFWLYHO\

'LVFXVVLRQ

SDWWHUQEDVHG QRWLRQ RI FRQWH[WLQVWHDGRIVLPSOHFRQWH[W DGMDFHQF\ VWDWLVWLFV LQIRUPDWLRQULFKHU VWUXFWXUHV RI QDWXUDO ODQJXDJH DUH H[SORLWHG ZLWKRXW WKH QHHG RI VSHFLDOL]HG OLQJXLVWLF WRROV WKDW V\QWD[EDVHG DSSURDFKHV UHTXLUH 0RUHRYHU QHZ FRUSRUD FDQ EH XVHG GLUHFWO\ WR LPSRUYH WKH RXWFRPH ZLWKRXW SURFHVVLQJ DQHZ WKH XSGDWHGVWDWLVWLFDOGDWD

+RZHYHUWKH9&96($OJRULWKPRIIHUVWKHSRVVLELOLW\ WRH[SUHVVWKHVHPDQWLFVLPLODULW\HVWLPDWLRQDVDIXQFWLRQ RI LQGLYLGXDO FRQWH[W SDUDPHWHUV DQG WKHUHIRUH WR EDVH VLPLODULW\ HVWLPDWLRQ WR YDULHW\ RI FRQWH[W VLPLODULW\ &96( $OJRULWKP UHTXLUHV H[WUDFWLRQ RI FRQWH[WXDO GDWD RQO\ RQFH IURP WKH FRUSXV ,WHUDWLYH DSSOLFDWLRQV UHTXLUH RQO\WKHXSGDWHRIWKHFRQWH[WYHFWRUVVLPSO\E\PHUJLQJ SDUDPHWHUV

&RQFOXVLRQV

,QLWLDWLQJIURPWKHFRQFHSWRIZRUGVHPDQWLFVLPLODULW\ HVWLPDWLRQLQWHUPVRIFRQWH[WVLPLODULW\ZHKDYHSURSRVHG DQ DSSURDFK IRU H[WUDFWLQJ VHPDQWLF VLPLODULW\ UHODWLRQV EHWZHHQ OH[LFDO HQWLWLHV DQG WKHQ VHPDQWLF FOXVWHUV E\ SURFHVVLQJ ZRUG DGMDFHQF\ GDWD REWDLQHG IURP VPDOO RU PHGLXPVL]HGFRUSRUD

7KH LQQRYDWLRQV RI WKH SUHVHQW ZRUN DUH WKH G\QDPLF UHVWULFWLRQRIFRQWH[WSDUDPHWHUVLQVLGHFODXVHERXQGDULHV WKH XVDJH RI ULJLGSKUDVH DQG ZRUGPRUSKRORJ\ EDVHG LQLWLDOVHPDQWLFFDWHJRULHVDQGWKHSDWWHUQPDWFKLQJFURVV FRUUHODWLRQ DOJRULWKP ZKLFK G\QDPLFDOO\ GHWHFWV SDWWHUQ FRQWH[W VLPLODULW\ RIIHULQJ VWURQJ HYLGHQFH IRU VHPDQWLF VLPLODULW\

$OWKRXJKWKHSUHVHQWHGPHWKRGIHDWXUHVODQJXDJHDQG GRPDLQSRUWDELOLW\ZHFRQVLGHUWKDWWKHJHQUHRIWKHWH[WV FRQVLVWLQJ WKH WUDLQLQJ FRUSRUD VKRXOG EH FRQVWUDLQW WR QHZVRUUHSRUWVSUHIHUDEO\ZLWKVPDOOYDULDWLRQUHJDUGLQJ WKHLUVRXUFHLQRUGHUWRDFKLHYHUHOLDEOHUHVXOWV

,Q WKH LPPHGLDWH IXWXUH ZH SODQ WR DSSO\ D VRIW FOXVWHULQJ DOJRULWKP WR WKH H[WUDFWHG UHODWLRQV WR WHVWWKH PHWKRG WR D GLIIHUHQW GRPDLQ DQGRU ODQJXDJH DQG WR SURFHHG WR TXDQWLWDWLYH FRPSDULVRQ ZLWK RWKHU ZRUG FOXVWHULQJPHWKRGV

5HIHUHQFHV

%URZQ3)'HOOD3LHWUD9-'H6RX]D39/DL-&DQG 0HUFHU 5/ &ODVV%DVHG QJUDP 0RGHOV RI 1DWXUDO /DQJXDJH &RPSXWDWLRQDO /LQJXLVWLFV

&KDUQLDN ( 6WDWLVWLFDO /DQJXDJH /HDUQLQJ 7KH 0,73UHVV

*UHIHQVWHWWH * 6(;7$17([WUDFWLQJ6HPDQWLFV IURP5DZ7H[W,PSOHPHQWDWLRQ'HWDLOV7KH-RXUQDORI .QRZOHGJH(QJLQHHULQJ

/L + $EH 1 &OXVWHULQJ :RUGV ZLWK WKH 0'/ 3ULQFLSOH-RXUQDO RI 1DWXUDO /DQJXDJH 3URFHVVLQJ

0DUWLQ 6 /LHUPDQQ - 1H\ + $OJRULWKPV IRU ELJUDP DQG WULJUDP ZRUG FOXVWHULQJ 6SHHFK &RPPXQLFDWLRQ

0F0DKRQ -* 6PLWK )- ,PSURYLQJ 6WDWLVWLFDO /DQJXDJH 0RGHO 3HUIRUPDQFH ZLWK $XWRPDWLFDOO\ *HQHUDWHG :RUG +LHUDUFKLHV &RPSXWDWLRQDO /LQJXLVWLFV

3HUHLUD ) 7LVKE\ 1 'LVWULEXWLRQDO 6LPLODULW\ 3KUDVH 7UDQVLWLRQV DQG +LHUDUFKLFDO &OXVWHULQJ

:RUNLQJ 1RWHV )DOO 6\PSRVLXP 6HULHV $$$,

5HGLQJWRQ ) 0 &KDWHU 1 DQG )LQFK 6 'LVWULEXWLRQDO ,QIRUPDWLRQ DQG WKH $FTXLVLWLRQ RI /LQJXLVWLF &DWHJRULHV $ 6WDWLVWLFDO $SSURDFK 3URFHHGLQJVRIWKH)LIWHHQWK$QQXDO&RQIHUHQFHRIWKH &RJQLWLYH6FLHQFH6RFLHW\

6WDPDWDWRV ( )DNRWDNLV 1 .RNNLQDNLV * $XWRPDWLF ([WUDFWLRQ RI 5XOHV IRU 6HQWHQFH %RXQGDU\ 'LVDPELJXDWLRQ3URFHHFLQJV RI $&$, :RUNVKRS RQ 0DFKLQH /HDUQLQJ LQ +XPDQ /DQJXDJH 7HFKQRORJ\