Squared Error Minimization

4.3.3.1 Training Using the Squared Error Loss

(QT VJG GZGORNCTENCUU VCUM VJG USWCTGF GTTQT NQUU KU FGſPGF CU

YJGTG

KU C VGCEJKPI VCTIGV UKIPCN VJCV KU WUWCNN[ UGV HQT C VTCKPKPI UCORNG

VQ QVJGTYKUG 6JG NQUU KU VJGP TGYTKVVGP CU

*GTG QPG ECP ſPF VJCV VJG DQVVQO NKPG GZRTGUUKQP QH ECP DG VTGCVGF CU C MKPF QH OKUENCUUKſECVKQP OGCUWTG

+V VWTPU QWV VJCV VJG TGFWEVKQP QH VJG USWCTGF GTTQT NQUU TGUWNVU KP VJG TGFWEVKQP QH VJKU OKUENCUUKſECVKQP OGCUWTG CPF CNUQ VJCV VJG OKPKOK\CVKQP QH VJG USWCTGF GTTQT NQUU QH KU GSWKXCNGPV VQ VJG OKPKOK\CVKQP QH C UKORNG NKPGCT NQUU KP YJKEJ VJG

EGJ VKLIWLQJRYHULQSXW VKLIWLQJRYHULQSXW VKLIWLQJRYHULQSXW LQSXWIHDWXUH YHFWRUVHTXHQFH DEVWUDFW OD\HU FODVVDFWLYDWLRQ OD\HU RXWSXWOD\HU FIGURE 4.4

Architecture of time-delay neural network [27].

OKUENCUUKſECVKQP OGCUWTG KU GODGFFGF 6JWU VJG VTCKPKPI WUKPI VJG NQUU KU EGTVCKPN[ FKUETKOKPCVKXG DWV QDXKQWUN[ FKHHGTGPV HTQO VJG CVVGORV VQ CEJKGXG VJG OKPKOWO ENCUUKſECVKQP GTTQT EQPFKVKQP D[ WUKPI VJG UOQQVJGF GTTQT EQWPV NQUU 5GG =? HQT FGVCKNGF FKUEWUUKQPU

4.3.3.2 Time-delay Neural Network

6JG VKOGFGNC[ PGWTCN PGVYQTM 6&00 KU QPG QH VJG ENCUUKE 00 CRRNKECVKQPU VQ URGGEJ RCVVGTP TGEQIPKVKQP =? 6JG VKOGFGNC[ CTEJKVGEVWTG KU KPEQTRQTCVGF VQ EQRG YKVJ VJG URGGEJ UKIPCN F[PCOKEU D[ WUKPI OWNVKRNG HGCVWTG XGEVQTU GCEJ DGKPI IGP GTCVGF D[ UJKHVKPI C VKOG YKPFQY QXGT CP KPRWV URGGEJ RCVVGTP 6JCV KU VJKU FGUKIP KU C UJKHVVQNGTCPV CTEJKVGEVWTG CPF KV YCU C OQFGN HQT VJG FGXGNQROGPV QH VJG UJKHV VQNGTCPV .83 ENCUUKſGT =?

(KIWTG KNNWUVTCVGU C V[RKECN GZCORNG QH VJG 6&00 CTEJKVGEVWTG 6JG 6&00 ENCU UKſGT CUUWOGU CP KPRWV URGGEJ RCVVGTP VQ DG C UGSWGPEG QH HGCVWTG XGEVQTU +V WUGU C NKOKVGFNGPIVJ UVTGCO QH VJG HGCVWTG XGEVQTU CU KVU KPRWV CPF KP QTFGT VQ HGGF HQTYCTF VJG KPHQTOCVKQP QH C UJQTV HQEWUGF UGIOGPV KV EQPUVTCKPVU PGVYQTM EQPPGEVKQPU VQ C UOCNNGT PWODGT QH PQFGU VJCP VJG PWODGT QH HTCOGU KP VJG YJQNG NKOKVGF UVTGCO HQT GZRNCPCVKQP RWTRQUGU NGV WU ECNN VJKU UGV QH NKOKVGF EQPPGEVKQPU C FGNC[ITQWR 6JG ENCUUKſGT VJGP CEEWOWNCVGU VJG KPHQTOCVKQP HGF HQTYCTF HTQO VJG NQYGT PGVYQTM NC[GTU D[ UJKHVKPI VJG FGNC[ITQWR QXGT VJG KPRWV

FKOQWVRWV

c

c

c

c

c

c

FKOKPRWV

FKOTGHGTGPEG

XGEVQT

Schematic description of distance classifier as a single intermediate layer net- work (2-dimensional input, 3 references/class, 3 classes).

VCTIGV KU C XGEVQT QH YJKEJ VJG EQORQPGPV HQT VJG EQTTGEV ENCUU KU QPG CPF QH YJKEJ EQORQPGPVU HQT QVJGT ENCUUGU CTG UGV VQ \GTQ 6JGP VJG NQUU KU FGſPGF DGVYGGP VJG VCTIGV CPF CP QWVRWV QH VJG ENCUUKſGT #EEQTFKPIN[ VJG OKPKOK\CVKQP QH VJKU USWCTGF GTTQT NQUU YQTMU UQ VJCV VJG ENCUUKſGT QWVRWV TGUGODNGU VJG VTCKPKPI VCTIGV +P VJG ENCUUKſECVKQP UVCIG HQT WPMPQYP RCVVGTPU VJG ENCUU JCXKPI VJG NCTIGUV PGVYQTM QWVRWV KU UGNGEVGF CU C ENCUUKſECVKQP TGUWNV

#U UJQYP KP(KIWTG VJG 6&00 ENCUUKſGT JCU OWNVKRNG KPVGTOGFKCVG NC[GTU 6JKU OWNVKKPVGTOGFKCVGNC[GT UVTWEVWTG KU C URGEKCN HGCVWTG QH OWNVKNC[GT PGVYQTMU UWEJ CU /.2 CPF KV KU ENGCTN[ FKUVKPEV HTQO VJG UVTWEVWTG QH VTCFKVKQPCN ENCUUKſGTU UWEJ CU C FKUVCPEG ENCUUKſGT QPN[ WUKPI TGHGTGPEG XGEVQTU CU ENCUU OQFGNU YJKEJ ECP DG EQPUKFGTGF C UKPING KPVGTOGFKCVG NC[GT PGVYQTM (KIWTG +P CFFKVKQP VJG TGCFGT EQWNF TGECNN VJG ECUG QH CP .83 ENCUUKſGT VJCV JCU QPN[ TGHGTGPEG XGEVQTU # RTKP EKRCN HWPEVKQP QH VJG OWNVKRNG KPVGTOGFKCVG NC[GTU KU KPHQTOCVKQP CDUVTCEVKQP +P VJG 6&00 ENCUUKſGT KV KU GZRGEVGF VJCV KPHQTOCVKQP GUUGPVKCN HQT ENCUUKſECVKQP YJKEJ KU CEEQTFKPIN[ UJKHVVQNGTCPV ECP DG GZVTCEVGF VJTQWIJ VJGUG JKFFGP NC[GTU

+P =? JKIJN[ KPƀWGPVKCN GZRGTKOGPVCN TGUWNVU QH WUKPI VJG 6&00 ENCUUKſGT HQT RJQPGOG ENCUUKſECVKQP CTG TGRQTVGF

4.3.3.3 Multi-state Time-delay Neural Network

6Q EQRG YKVJ VJG F[PCOKE PCVWTG QH NQPIGT URGGEJ WPKVU VJG EQPEGRV QH 6&00 YCU FKTGEVN[ GZVGPFGF VQ C OWVNKUVCVG VKOGFGNC[ PGWTCN PGVYQTM /56&00 =? 6JKU

FGXGNQROGPV EQPVTCUVU YKVJ VJCV QH VJG J[DTKF U[UVGOU WUKPI .83 CPF *// +V ECP DG UCKF VJCV /56&00 KPEQTRQTCVGU VJG UVCVG VTCPUKVKQP UVTWEVWTG YJKEJ YCU YKFGN[ WUGF KP ITCRJKECN OQFGNU UWEJ CU *// KP KVU U[UVGO UVTWEVWTG KP RNCEG QH UKORN[ EQODKPKPI GZKUVKPI EQPEGRVU UWEJ CU 6&00 CPF *//

#P /56&00 ENCUUKſGT EQPUKUVU QH C PWODGT QH NQECN 6&00 U[UVGOU GCEJ QH YJKEJ HQTOU C UVCVG CPF KU FGUKIPGF HQT VJG ENCUUKſECVKQP QH RJQPGOGU QT UWDRJQPGOKE WPKVU CPF KV QWVRWVU C ENCUUKſECVKQP TGUWNV HQT VJG GPVKTG URGGEJ KPRWV GI YQTF UGSWGPEG ENCUUKſECVKQP TGUWNV VJTQWIJ VJG &2DCUGF UGCTEJ QH VJG DGUV RJQPGOG UVCVG UGSWGPEG

6TCKPKPI QH VJG ENCUUKſGT KU FQPG YKVJ VJG USWCTGF GTTQT NQUU CU HQT VJG QTKIKPCN 6&00 ENCUUKſGTU *QYGXGT VJG NQUU JGTG KU FGſPGF CV VJG NGXGN QH VJG ſPCN ENCUU QWVRWVU QH VJG VCUM KG YQTF QT YQTF UGSWGPEG ENCUUGU KP RNCEG QH VJG UJQTV RJQPGOG NGXGN CPF CFFKVKQPCNN[ VJG &2DCUGF UGIOGPVCVKQP QXGT VJG GPVKTG VTCKPKPI KPRWV RCVVGTP KU GODGFFGF KP VJG NQUU OKPKOK\CVKQP %NGCTN[ VJG VTCKPKPI QH /56&00 KU UWKVCDNG HQT VJG ENCUUKſECVKQP QH NQPIGT URGGEJ WPKVU +PFGGF CU CP CNVGTPCVKXG VQ VJG UVCP FCTF *//DCUGF ENCUUKſGT VJG /56&00 ENCUUKſGT JCU DGGP WUGF KP XCTKQWU URGGEJ RCVVGTP TGEQIPKVKQP VCUMU =?