The first thing to be said is that the easy compare algorithm provides the best results up until the deep learning algorithm, which has been trained on a GPU for one and a half week. This is really interesting since it means that the naive approach in how to compare two signatures with an engineered feature was strikingly good. Showing that the human mind is capable of constructing an acceptable guess of what is important to look at for a solution.
When it comes to the logistic regression (LR) and the multi-layer perceptron (MLP) algorithms they were mainly used to understand the signature verifica- tion problem and gather an understanding of learning algorithms. The focus on these algorithms (LR and MLP) was never to achieve top performance but rather to obtain the knowledge required for a deep learning algorithm.
When it comes to understanding how the prediction of the di↵erent algo- rithms works, it is much easier to understand the engineered features, compared to the weights of the trained model, which becomes increasingly more complex as the model increases in size.
10
Future Work
This section is intended to provide some suggestions of further work and studies, which could be a continuation of this project.
10.1
Data Collecting
The very first step on the signature verification problem is to be able to separate genuine signatures form each other. The next step will be to move on to other types of forgeries and see how well the final model in this project perform on those data sets.
If it is possible to get an algorithm which will work on all the other types of forgeries and if the accuracy of the algorithm is good enough (i.e. close to ideal),
the data collection could be linked to general identification procedures such as for passport controls and the data could be saved on a chip in the passport. Because the signature is a behavioral biometric parameter, the samples should be replaced after X number of years, which is the general procedure for passports and other identification cards.
In future work it would also be interesting to study how the signatures changes over time, for example a couple of weeks or even years.
In the future development of the algorithm it is then important to mimic the real world as much as possible and collect samples at di↵erent occasions.
10.2
Pre-processing
The normalization could be made in di↵erent ways. If the input data is normal- ized to be symmetric around zero, the gradient in the training step will variate less and this will probably lead to improved training [40]. In this project one type of normalization was used, but other types of normalization are also worth trying in future development.
10.3
Convolution Neural Network with Deep Architec-
ture
It is an engineering task to find out which model selection that gives the best results. In this project there was not enough time to variate the structure of the final model, and analyze this further.
To improve the developed model further, the structure of the network could be utilized better, where a trade o↵ between needed complexity is considered, i.e. the minimum number of parameters that can be used without losing accuracy. The network could perhaps be implemented to work on an adaptable input length and/or be implemented on sub problems.
In Section 1.2.3, the results from the competition showed that the models performed better on the data set, that only contain the coordinates and not the pen orientation and pressure. It would be interesting to customize the final model developed in this project to only work on the coordinates and see how well it performed.
The GPU based, max-pooling, convolutional neural networks has shown great results on image recognition through history, but even better results has been shown by GPU based, ensemble, max-pooling, convolutional neural net- works. To improve this model even more, the model could also use ensemble learning in future development.
11
Conclusion
In this report a deep learning algorithm has been developed to deal with the signature verification problem. In order to solve the problem a code framework was built with a database of signatures, a classifier and an evaluation method. Several di↵erent classifying algorithms were tested before the final convolutional neural network was developed. The result from this final model was a true positive rate (sensitivity) of 96.7 % and a false positive rate (fall-out) of 0.6 %. The suggested solution for the problem shows promise, but future studies are recommended.
References
[1] Deep learning comminity. http://deeplearning.net/tutorial/. Ac- cessed: 2015-07-28.
[2] historyofinformation.com. http://www.historyofinformation.com/ expanded.php?id=2614. Accessed: 2015-09-05.
[3] Tutorial for classification with convolutional neural network on the mnist data set. http://deeplearning.net/tutorial/logreg.html#lenet. Ac- cessed: 2015-07-28.
[4] Tutorial for classification with logistic regression on the mnist data set. http://deeplearning.net/tutorial/logreg.html#logreg. Accessed: 2015-07-28.
[5] Tutorial for classification with multi-layer perception on the mnist data set. http://deeplearning.net/tutorial/logreg.html#mlp. Accessed: 2015-07-28.
[6] Where to find the package, where hid-recorder.c is included. http: //bentiss.github.io/hid-replay-docs/, note = Accessed: 2015-08-16. [7] Dilnya Mahmood Jafar Agnieszka Nabrdalik. Kan biometri ¨oka s¨akerhetsniv˚an p˚a svenska sjukhus? Master’s thesis, Blekinge Tekniska H¨ogskola, Avdelning f¨or Elektroteknik, 2012.
[8] Cyrus Bakhtiari-Haftlan. Arabic online handwritten recognition. Master’s thesis, Lund Institute of Technology, Centre for Mathematical Sciences, 2008.
[9] P. Baldi and Y. Chauvin. Neural networks for fingerprint recognition. Neu- ral Computation, 5(3), 1993.
[10] Yoshua Bengio, Ian J. Goodfellow, and Aaron Courville. Deep learning. Book in preparation for MIT Press, 2015.
[11] Christopher Michael Bishop. Pattern recognition and machine learning. springer, 2006.
[12] Sandra Blakeslee. Behind the veil of thought: Advances in brain research; in brain’s early growth,timetable may be crucial. The New York Times, 1995.
[13] Kumar Chellapilla, Sidd Puri, and Patrice Simard. High performance con- volutional neural networks for document processing. In International Work- shop on Frontiers in Handwriting Recognition, 2006.
[14] D. C. Ciresan, U. Meier, J. Masci, and J. Schmidhuber. A committee of neural networks for traffic sign classification. In International Joint Conference on Neural Networks (IJCNN), pages 1918–1921, 2011.
[15] D. C. Ciresan, U. Meier, J. Masci, and J. Schmidhuber. Multi-column deep neural network for traffic sign classification. Neural Networks, 32:333–338, 2012.
[16] D. C. Ciresan, U. Meier, and J. Schmidhuber. Multi-column deep neu- ral networks for image classification. In IEEE Conference on Com- puter Vision and Pattern Recognition CVPR 2012, 2012. Long preprint arXiv:1202.2745v1 [cs.CV].
[17] Dan Claudiu Ciresan, Alessandro Giusti, Luca Maria Gambardella, and J¨urgen Schmidhuber. Mitosis detection in breast cancer histology images with deep neural networks. In Proc. MICCAI, volume 2, pages 411–418, 2013.
[18] Dan Claudiu Ciresan, Ueli Meier, Luca Maria Gambardella, and J¨urgen Schmidhuber. Deep big simple neural nets excel on handwritten digit recog- nition. CoRR, abs/1003.0358, 2010.
[19] David Danowsky. Cyrillic handwriting recognition using support vector machines. Master’s thesis, Lund Institute of Technology, Centre for Math- ematical Sciences, 2006.
[20] Stefano Rodo`a Data Protection Working Party of the European commis- sion. Working document on biometrics. WP80, 2003.
[21] E. D. Dickmanns, R. Behringer, D. Dickmanns, T. Hildebrandt, M. Maurer, F. Thomanek, and J. Schiehlen. The seeing passenger car ’VaMoRs-P’. In Proc. Int. Symp. on Intelligent Vehicles ’94, Paris, pages 68–73, 1994. [22] C. Petrinin E. Mordini. Ethical and social implications of biometric iden-
tification technology. Annali Dell’Instituto Superiore di Sanita, 43(1):5-11, 2007.
[23] J.L. Elman. Rethinking Innateness: A Connectionist Perspective on De- velopment. A Bradford book. Kluwer, 1998.
[24] Jimmy Engstr¨om. On-line handwriting recognition. Master’s thesis, Lund Institute of Technology, Centre for Mathematical Sciences, 2003.
[25] K. Fukushima. Neural network model for a mechanism of pattern recog- nition una↵ected by shift in position - Neocognitron. Trans. IECE, J62- A(10):658–665, 1979.
[26] K. Fukushima. Neocognitron: A self-organizing neural network for a mech- anism of pattern recognition una↵ected by shift in position. Biological Cybernetics, 36(4):193–202, 1980.
[27] Kunihiko Fukushima. Artificial vision by multi-layered neural networks: Neocognitron and its advances. Neural Networks, 37:103–119, 2013.
[28] C. F. Gauss. Theoria combinationis observationum erroribus minimis ob- noxiae (theory of the combination of observations least subject to error), 1821.
[29] Carl Friedrich Gauss. Theoria motus corporum coelestium in sectionibus conicis solem ambientium, 1809.
[30] D. H. Hubel and T.N. Wiesel. Receptive fields, binocular interaction, and functional architecture in the cat’s visual cortex. 160, 1962.
[31] David H Hubel and Torsten N Wiesel. Receptive fields and functional architecture of monkey striate cortex. 195, 1968.
[32] ICPR 2012 Contest on Mitosis Detection in Breast Cancer Histological Images. IPAL Laboratory and TRIBVN Company and Pitie-Salpetriere Hospital and CIALAB of Ohio State Univ., http://ipal.cnrs.fr/ICPR2012/, 2012.
[33] A.K. Jain, A.A. Ross, and K. Nandakumar. Introduction to Biometrics. SpringerLink : B¨ucher. Springer, 2011.
[34] Anil K. Jain, Arun Ross, and Salil Prabhakar. An introduction to biometric recognition. IEEE Trans. on Circuits and Systems for Video Technology, 14:4–20, 2004.
[35] Alex Krizhevsky, I Sutskever, and G. E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS 2012), page 4, 2012.
[36] O. Abdel-Hamid L. Deng and D. Yu. A deep convolutional neural network using heterogeneous pooling for trading acoustic invariance with phonetic confusion. Technical report, Microsoft Research, One Microsoft Way, Red- mond, WA, USA York University, Toronto, ON Canada, 2013.
[37] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hub- bard, and L. D. Jackel. Handwritten digit recognition with a back- propagation network. In D. S. Touretzky, editor, Advances in Neural In- formation Processing Systems.
[38] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hub- bard, and L. D. Jackel. Back-propagation applied to handwritten zip code recognition. 1(4), 1989.
[39] Y. LeCun, L. Bottou, Y. Bengio, and P. Ha↵ner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278– 2324, November 1998.
[40] Yann Lecun, Leon Bottou, Genevieve B. Orr, and Klaus-Robert M¨uller. Efficient backprop, 1998.
[41] Adrien Marie Legendre. Nouvelles m´ethodes pour la d´etermination des orbites des cometes. F. Didot, 1805.
[42] Fredrik Mattisson. On-line signature verification using a multi-judge strat- egy. Master’s thesis, Lund Institute of Technology, Centre for Mathematical Sciences, 2001.
[43] W. McCulloch and W. Pitts. A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 7:115–133, 1943. [44] Jonas Morwing. Recognition of cursive handwriting. Master’s thesis, Lund
Institute of Technology, Centre for Mathematical Sciences, 2001.
[45] Committee of SVC 2004. The results of the svc 2004. http://www.cse.ust.hk/svc2004/results.html, 2004. Accessed: 2015- 07-28.
[46] M. Ranzato, C. Poultney, S. Chopra, and Y. Lecun. Efficient learning of sparse representations with an energy-based model. In Advances in Neural Information Processing Systems (NIPS 2006), pages 1137–1144, 2006. [47] L. Roux, D. Racoceanu, N. Lomenie, M. Kulikova, H. Irshad, J. Klossa,
F. Capron, C. Genestie, G. Le Naour, and M. N. Gurcan. Mitosis detection in breast cancer histological images - an ICPR 2012 contest. J. Pathol. Inform., 4:8, 2013.
[48] Stuart J. Russell and Peter Norvig. Artificial Intelligence: A Modern Ap- proach. Pearson Education, 3 edition, 2009.
[49] Jorge Saint-Aubyn. Using linguistic knowledge and edit distance to improve recognition or fuzzy cursive handwriting. Master’s thesis, Lund University, Faculty of Engineering, Centre for Mathematical Sciences, 2008.
[50] Jurgen Schmidhuber. Deep learning in neural networks: An overview. 2014. [51] Patrice Y. Simard, Dave Steinkraus, and John C. Platt. J.c.: Best practices for convolutional neural networks applied to visual document analysis. In In: Int’l Conference on Document Analysis and Recognition, pages 958– 963, 2003.
[52] J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. The German traffic sign recognition benchmark: A multi-class classification competition. In International Joint Conference on Neural Networks (IJCNN 2011), pages 1453–1460. IEEE Press, 2011.
[53] J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks, 32:323–332, 2012.
[54] Brian Kingsbury Tara N. Sainath, Abdel-rahman Mohamed and Bhuvana Ramabhadran. Deep convolutional neural networks for lvcsr. 2013.
[55] Theano. Software. http://deeplearning.net/software/theano/. Version: 0.6.0, Accessed: 2015-07-28.
[56] V. Vapnik. The Nature of Statistical Learning Theory. Springer, New York, 1995.
[57] Juyang Weng, Narendra Ahuja, and Thomas S Huang. Cresceptron: a self- organizing neural network which grows adaptively. In International Joint Conference on Neural Networks (IJCNN), volume 1, pages 576–581. IEEE, 1992.
[58] Dit yan Yeung, Hong Chang, Yimin Xiong, Susan George, Ramanujan Kashi, Takashi Matsumoto, and Gerhard Rigoll. Svc2004: First interna- tional signature verification competition. In In Proceedings of the Interna- tional Conference on Biometric Authentication (ICBA), Hong Kong, pages 16–22. Springer, 2004.
Master’s Theses in Mathematical Sciences 2015:E40 ISSN 1404-6342
LUTFMA-3282-2015 Mathematics
Centre for Mathematical Sciences Lund University