Lecture Notes in Electrical Engineering 594

(1)

Lecture Notes in Electrical Engineering 594

Yingmin Jia Junping Du

Weicun Zhang Editors

Proceedings of 2019 Chinese Intelligent Systems Conference

Volume III

(2)

Lecture Notes in Electrical Engineering

Volume 594

Series Editors

Leopoldo Angrisani, Department of Electrical and Information Technologies Engineering, University of Napoli Federico II, Naples, Italy

Marco Arteaga, Departament de Control y Robótica, Universidad Nacional Autónoma de México, Coyoacán, Mexico

Bijaya Ketan Panigrahi, Electrical Engineering, Indian Institute of Technology Delhi, New Delhi, Delhi, India Samarjit Chakraborty, Fakultät für Elektrotechnik und Informationstechnik, TU München, Munich, Germany Jiming Chen, Zhejiang University, Hangzhou, Zhejiang, China

Shanben Chen, Materials Science and Engineering, Shanghai Jiao Tong University, Shanghai, China Tan Kay Chen, Department of Electrical and Computer Engineering, National University of Singapore, Singapore, Singapore

Rüdiger Dillmann, Humanoids and Intelligent Systems Lab, Karlsruhe Institute for Technology, Karlsruhe, Baden-Württemberg, Germany

Haibin Duan, Beijing University of Aeronautics and Astronautics, Beijing, China Gianluigi Ferrari, Università di Parma, Parma, Italy

Manuel Ferre, Centre for Automation and Robotics CAR (UPM-CSIC), Universidad Politécnica de Madrid, Madrid, Spain

Sandra Hirche, Department of Electrical Engineering and Information Science, Technische Universität München, Munich, Germany

Faryar Jabbari, Department of Mechanical and Aerospace Engineering, University of California, Irvine, CA, USA

Limin Jia, State Key Laboratory of Rail Traffic Control and Safety, Beijing Jiaotong University, Beijing, China Janusz Kacprzyk, Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland

Alaa Khamis, German University in Egypt El Tagamoa El Khames, New Cairo City, Egypt Torsten Kroeger, Stanford University, Stanford, CA, USA

Qilian Liang, Department of Electrical Engineering, University of Texas at Arlington, Arlington, TX, USA Ferran Martin, Departament d’Enginyeria Electrònica, Universitat Autònoma de Barcelona, Bellaterra, Barcelona, Spain

Tan Cher Ming, College of Engineering, Nanyang Technological University, Singapore, Singapore Wolfgang Minker, Institute of Information Technology, University of Ulm, Ulm, Germany Pradeep Misra, Department of Electrical Engineering, Wright State University, Dayton, OH, USA Sebastian Möller, Quality and Usability Lab, TU Berlin, Berlin, Germany

Subhas Mukhopadhyay, School of Engineering & Advanced Technology, Massey University, Palmerston North, Manawatu-Wanganui, New Zealand

Cun-Zheng Ning, Electrical Engineering, Arizona State University, Tempe, AZ, USA Toyoaki Nishida, Graduate School of Informatics, Kyoto University, Kyoto, Japan

Federica Pascucci, Dipartimento di Ingegneria, Università degli Studi “Roma Tre”, Rome, Italy

Yong Qin, State Key Laboratory of Rail Traffic Control and Safety, Beijing Jiaotong University, Beijing, China Gan Woon Seng, School of Electrical & Electronic Engineering, Nanyang Technological University, Singapore, Singapore

Joachim Speidel, Institute of Telecommunications, Universität Stuttgart, Stuttgart, Baden-Württemberg, Germany

Germano Veiga, Campus da FEUP, INESC Porto, Porto, Portugal

Haitao Wu, Academy of Opto-electronics, Chinese Academy of Sciences, Beijing, China Junjie James Zhang, Charlotte, NC, USA

(3)

The book series Lecture Notes in Electrical Engineering (LNEE) publishes the latest developments in Electrical Engineering - quickly, informally and in high quality. While original research reported in proceedings and monographs has traditionally formed the core of LNEE, we also encourage authors to submit books devoted to supporting student education and professional training in the variousﬁelds and applications areas of electrical engineering. The series cover classical and emerging topics concerning:

• Communication Engineering, Information Theory and Networks

• Electronics Engineering and Microelectronics

• Signal, Image and Speech Processing

• Wireless and Mobile Communication

• Circuits and Systems

• Energy Systems, Power Electronics and Electrical Machines

• Electro-optical Engineering

• Instrumentation Engineering

• Avionics Engineering

• Control Systems

• Internet-of-Things and Cybersecurity

• Biomedical Devices, MEMS and NEMS

For general information about this book series, comments or suggestions, please contactleontina.

[email protected].

To submit a proposal or request further information, please contact the Publishing Editor in your country:

China

Jasmine Dou, Associate Editor ([email protected]) India

Swati Meherishi, Executive Editor ([email protected]) Aninda Bose, Senior Editor ([email protected])

Japan

Takeyuki Yonezawa, Editorial Director ([email protected]) South Korea

Smith (Ahram) Chae, Editor ([email protected]) Southeast Asia

Ramesh Nath Premnath, Editor ([email protected]) USA, Canada:

Michael Luby, Senior Editor ([email protected]) All other Countries:

Leontina Di Cecco, Senior Editor ([email protected]) Christoph Baumann, Executive Editor ([email protected])

** Indexing: The books of this series are submitted to ISI Proceedings, EI-Compendex, SCOPUS, MetaPress, Web of Science and Springerlink **

More information about this series athttp://www.springer.com/series/7818

(4)

Yingmin Jia

^•

Junping Du

^•

Weicun Zhang

Editors

Proceedings of 2019 Chinese Intelligent Systems Conference

Volume III

123

(5)

Editors Yingmin Jia Beihang University Beijing, China

Junping Du

Beijing University of Posts and Telecommunications Beijing, China

Weicun Zhang University of Science and Technology Beijing Beijing, China

ISSN 1876-1100 ISSN 1876-1119 (electronic) Lecture Notes in Electrical Engineering

ISBN 978-981-32-9697-8 ISBN 978-981-32-9698-5 (eBook) https://doi.org/10.1007/978-981-32-9698-5

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, speciﬁcally the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microﬁlms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed.

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a speciﬁc statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional afﬁliations.

This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd.

The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721, Singapore

(6)

Hierarchical Pooling Based Extreme Learning Machine for Image Classification

Yan Liu¹⁽B⁾_{, Zhi Liu}², and Zhirong Lei³

1 Hengxiang Control Technology Company Limited, Xi’an, China yan [email protected]

2 School of Artiﬁcial Intelligence, Xidian University, Xi’an, China

3 National Key Laboratory of Science and Technology on Aircraft Control, FACRI, Xi’an, China

Abstract. In this paper, a Hierarchical Pooling based Extreme Learn- ing Machine (HPELM) is proposed for image classiﬁcation. Extreme Learning Machine based on Local Receptive Fields (ELM-LRF) has been proved to be powerful for image classiﬁcation. However, ELM-LRF is a shallow network and the features extracted by ELM-LRF is low-level. To obtain better results, one need to enlarge the dimension of the hidden features. This paper extends the concept of deep learning to ELM-LRF.

Random convolutional nodes and hierarchical pooling structures are con- structed for capturing high level semantic features. HPELM has the abil- ity of feature extraction and classiﬁcation. It improves the classiﬁcation performance of ELM-LRF without increasing the number of the neuron in the last hidden layer. Experiments on the MNIST and NORB datasets demonstrate the attractive performance of HPELM even compared with the state-of-the-art algorithms.

Keywords: Local receptive ﬁeld

·

Hierarchical pooling

·

Image classiﬁcation

·

Extreme Learning Machine

1 Introduction

Image classification is an important and fundamental task of computer vision, data mining and machine learning [1]. It has an extremely large number of domains of application ranging from medicine, bioinformatics to industrial automation, robot, etc. Generally, traditional image classification methods con- tain two steps: (1) image feature extraction; (2) feature classification. Feature extraction is a key step in classification task. The extracted features are fed into various classifiers, such as Multi-Layer Perception (MLP) or Support Vec- tor Machines (SVM) for classification. The classification performance is highly depends on the feature extractor [3]. Besides, these traditional Machine Learn- ing (ML) methods mostly use shallow structures. Obviously, the generalization performance of classification will degrade when the objects have complex texture structures.

Springer Nature Singapore Pte Ltd. 2020c

Y. Jia et al. (Eds.): CISC 2019, LNEE 594, pp. 1–9, 2020.

https://doi.org/10.1007/978-981-32-9698-5_1

(14)

2 Y. Liu et al.

In very recent years, Deep Neural Networks (DNN) based Deep Learning (DL) methods have made significant breakthroughs in various areas such as object detection, speech recognition, image classificaiton [19]. Most of the deep models such as AlexNet [16], VGGNet [18], GoogleNet [17] are based on neural networks and have the structures of convolution and pooling. The convolution structure used in deep convolution neural network (CNN) is to imitate the local receptive field of human eye. These models can extract abstract and higher- order features automatically. With the help of big data and powerful computing resources, deep learning has shown excellent performance in various tasks. The training methods of DNN are based on back propagation (BP) algorithm, which suffers from local minima and slow convergence speed etc.

Extreme Learning Machine (ELM) is another model that usually be uti- lized for solving regression and classiﬁcation problems [8]. ELM is a generalization of feedforward neural networks with single-hidden layer. ELM projects the input data to the hidden feature space by random weights. ELMs use the least- square solutions as the solution of the output weight and does not need iterative optimization [7]. ELMs are known for their simplicity, short training time and unusual performance [9].

Recently, more and more studies focus on DNNs with random weights [20–

22]. On the one hand, more and more DL concepts are introduced to ELM.

Hierarchical or Multi-Layer ELMs have been proposed for learning deep representations [11–15]. Huang et al. [2] introduced convolution and pooling structures into ELM and proposed ELM-LRF. The Multi-Scale version of ELM-LRF (ELM-MSLRF) are proposed in [3,6]. Diﬀerent from CNNs, the input weights of ELM-LRF or ELM-MSLRF are generated randomly rather than optimized by iterative training. Since the output weights of ELM-LRF or ELM-MSLRF are calculated analytically without BP learning, the training phase is very fast.

However, both ELM-LRF and ELM-MSLRF are shallow networks. In [4] and [5], the structure of Hierarchical ELM-LRF by stacking convolutional and pooling layers are studied. In this paper, we proposed a Hierarchical Pooling based ELM- LRF (HPELM) by stacking pooling layers. The main works of this paper are two folds. (1) We develop a novel hierarchical structure to learn rich representations.

(2) HPELM can improve the performance of shallow ELM-LRF without increasing the number of the neurons in the last hidden layer. Extensive experiments conducted on the known datasets demonstrate that HPELM is eﬀective.

The rest of this paper is organized as follows. Section2reviews the ELM and ELM-LRF theories brieﬂy. Section3describes the proposed HPELM framework in details. Experiments on known datasets are carried out in Sect.4to evaluate the proposed algorithm. Finally, some conclusions are summarized in Sect.5.

2 Related Works

2.1 ELM

The key idea of ELM is the random feature mapping. Denote x ∈ R^d as the input vector, a_i ∈ R^d as the input weights and b_i ∈ R as the bias, g as the

(15)

Hierarchical Pooling 3

nonlinear piecewise continuous function. The random features can be obtained by u(x) = [u1(x), u2(x),· · · , uL(x)]^T, where u_i(x) = g (a_i, b_i, x), i = 1, 2,· · · , L.

L is the dimension of the features. Consider the problem of m-class classiﬁcation, the outputs of ELM is

f (x) =

L i=1

βiu_i(x) = u(x)β, (1)

where, βi = [β_i1, β_i2,· · · , βim]^T, βL×m = [β1,β2,· · · , βL]^T is the output weights between the hidden layer and the output layer. Equation1 is a Lin- ear regression formulation. Given a training dataset that contains N_s samples S = {(xi, y_i)|}^N_i=1, the goal of ELM is to minimize

minβ :β^σ_p¹+ λUβ − Y^σ_q², (2) where, σ1 > 0, σ2 > 0, p, q > 0 and λ is a weight factor. U is the hidden feature matrix U = [u(x1), u(x2),· · · , u(xN)]^T and Y is the class label matrix Y = [y₁, y2,· · · , yN]^T.

The output weightβ can be computed using many algorithms, such as singu- lar value decomposition, orthogonal projection and so on. When σ1= 2, σ2= 2, p = 2, q = 2, the solution of Eq.2 is

β =

_I

λ + U^TU₋₁

U^TY, if N ≥ L U^T_I

λ+ UU^T−1Y, if N < L. (3) 2.2 ELM-LRF

ELM is essentially a generalization of SLFNs [10]. Therefore, all the variants of SLFNs can be applied to ELM. CNN is an excellent model to deal with visual tasks, in particular image classiﬁcation. The basic idea of CNN is to learn representation and extract feature from small region of the receptive ﬁelds.

CNN mainly contains three types of layers, i.e. convolutional, pooling and nonlinear layer. However, in ELM-LRF, there is no non-linear layers and the weights of the convolutional layer is generated randomly. In order to make the network has the properties of translation invariance and frequency selective, the square-root pooling is adopted in ELM-LRF

h_s,t,k=

^s+e

i=s−e

t+e j=t−e

c²_i,j,k, s, t = 1,· · · , (d − r + 1) (4)

where, c_i,j,k, h_s,t,k represent the neuron (i, j) in the k-th convolutional feature map and the neuron (s, t) in the k-th pooling feature map respectively. d is the size of the previous layer of the convolutional layer, r× r is the size of the convolution kernel and 2e + 1 is the pooling window size. In order to keep the feature dimension unchanged before and after pooling, zeros padding are used before pooling.

(16)

4 Y. Liu et al.

3 HPELM

3.1 HPELM Framework

Fig. 1. The framework of hierarchical pooling based extreme learning machine

The framework of our proposed Hierarchical Pooling based Extreme Learning Machine (HPELM) is shown in Fig.1. It has just one convolutional layer and n hierarchical pooling layers. Denote K as the number of the convolutional kernels.

The output convolutional feature maps of the convolutional layer can be expressed as

c_i,j,k(x) =

r p=1

r q=1

(xi+p−1,j+q−1· ap,q,k) , (5) where, i, j = 1, 2,· · · , d−r+1, K is the number of feature maps of convolutional layer. a_p,q,k denotes the random weight value at (p, q) in the k-th convolutional kernel. c_i,j,k is the output node (i, j) of the k-th feature map. xi+p−1,j+q−1 is the node (i + p− 1, j + q − 1) of the input layer.

The hierarchical square-root pooling layers can be formulated as

h^(l)_s,t,k=

^s+e

i=s−e

t+e j=t−e

h^(l−1)2_i,j,k , h⁽⁰⁾_i,j,k= c_i,j,k, (6)

where, s, t = 1,· · · , (d − r + 1), l = 1, 2, · · · , n, e is the size of pooling, n is the number of pooling layers. The output nodes of the last pooling layer is flattened to a layer (called flatten layer) with size of L = P × Q × K. Where, P, Q is the height and the width of the pooling layer respectively. The flatten layer is the final features nodes and are fully connected with the output layer, the corresponding weight is denoted asβ. Suppose that we have m classes, then the output weightβ has size of L × m and can be solved by

β =

H^T_I

λ+ HH^T₋₁

Y, if N < L

_I

λ+ H^TH₋₁

H^TY, if N ≥ L. (7)

(17)

With these hierarchical pooling operations, the size of L is ﬁxed. That is, increase the number of hierarchical pooling layers does not increase the ﬁnal feature dimension L and the calculation amount ofβ, but increase the time for computing H. In ELM-LRF, in order to obtain better results, we must enlarge the dimension L. However, in HPELM, we can increase the number of hierarchi- cal pooling layers instead of the dimension L.

3.2 Training of HPELM

Similar to ELM-LRF, for training HPELM, generate K convolutional kernels with random weights

A ⁰=

a⁰₁, a⁰₂,· · · , a⁰_K

, a⁰_k ∈ R^r², k = 1,· · · , K (8) where, A⁰∈ R^r²^×K, r is the kernel size.

Utilize the singular value decomposition method to orthogonalize weights A ⁰ and we obtain ˆA. Denote ˆa_k as the column of ˆA, then ˆa_k is the orthogonal basis of A⁰. The weight matrix for the k-th kernel is a_k ∈ R^r×r is reshape from aˆ_k∈ R^r².

After obtain the kernel weights, utilize Eqs.5 and6 to compute the feature maps H, and then use Eq.7to compute the output weight matrixβ.

4 Experimental Results

We test our proposed HPELM method on two datasets: (1) MNIST, (2) NORB.

In experiment, we compare the performances of HPELM, ELM-LRF [2] and ELM-MSLRF [3] and some deep learning based models. Denote conv(r, k) as a convolutional layer that has k number of kernels with size r× r, pool(s) as a pooling layer with pooling window size s respectively.

4.1 MNIST Dataset

The MNIST dataset contains 10 kinds (0 to 9) of digits. It contains 60000 training samples and 10000 testing samples. In this experiment, we set the net structure of ELM-LRF to conv(9, 24)− − > pool(5) and the structure of HPELM to conv(9, 24)− > pool(5) − > pool(3)− > pool(5) − > pool(3)− > pool(5) − >

pool(3)− > pool(7), We evaluate the performance of HPELM and ELM-LRF with diﬀerent number of training samples N_s = [1000, 10000, 20000, 30000, 40000, 50000, 60000]. For each N_s, we test the classiﬁcation errors of HPELM and ELM-LRF with balance factor λ = [0.0001, 0.001, 0.01, 0.1, 0.2, 0.3, 0.4, 0.5].

The minimum error is shown in Fig.2. We can see that our proposed HPELM can improve the classiﬁcation accuracy of ELM-LRF on MNIST datasets without increase the dimension of the hidden feature.

In Table1, the prediction results with diﬀerent methods are shown. From Fig.2and Table1, we can see that our proposed HPELM has competitive results.

(18)

6 Y. Liu et al.

Fig. 2. Classiﬁcation error vs number of training samples (MNIST) Table 1. Prediction error (%) of diﬀerent algorithms on MNIST ELM [15] BinaryConnect [25] ELM-LRF [2] MSLRF-ELM [3] HPELM

1.90 1.01 2.61 1.43 1.07

4.2 NORB Dataset

The NORB dataset contains 24300 stereo images for testing and training respectively [2]. In this experiment, we set the net structure of ELM-LRF to conv(4, 28)−− > pool(7) and the structure of HPELM to conv(4, 28)− > pool(7)

− > pool(9)− > pool(11)− > pool(9) and set Ns = [100, 1000, 10000, 20000, 24300]. The classification performances of HPELM and ELM-LRF with different factor λ = [0.0001, 0.001, 0.01, 0.1, 0.2, 0.3, 0.4, 0.5] are evaluated. The mini- mum error is shown in Fig.3. We can see that our proposed HPELM can also increase the classification accuracy of ELM-LRF on NORB datasets.

Fig. 3. Classiﬁcation error vs number of training samples (NORB)

(19)

Table 2. Classification error rate (%) of different classification methods on NORB DBN [23] RandomWeights [24] ELM-LRF [2] MSLRF-ELM [3] HPELM

6.5 4.8 3.5 2.5 3.0

Under the above experimental parameters, HPELM has 3.0% error on the NORB dataset, slightly higher than MSLRF-ELM, but perform better than DBN, RandomWeights and ELM-LRF based methods (Table2).

5 Conclusions

This paper is devoted to introduce the concept of deep learning into extreme learning machine. We focus on making the shallow ELM-LRF network become more deeper and creating deep models that can be trained fastly. In our proposed HPELM model, hierarchical pooling structures are introduced into ELM-LRF.

Various experiments are conducted and analysed and the results prove that our proposed hierarchical pooling structure can improve the performance of ELM- LRF signiﬁcantly. Future work will focus on studying deep ELM models for very large datasets.

References

1. Bosch A, Zisserman A, Munoz X (2007) Image classiﬁcation using random forests and ferns. In: 11th International conference on computer vision. IEEE Press, Rio de Janeiro, pp 1–8.https://doi.org/10.1109/ICCV.2007.4409066

2. Huang G, Bai Z, Kasun LLC, Vong CM (2015) Local receptive ﬁelds based extreme learning machine. IEEE Comput Intell Mag 10:18–29. https://doi.org/10.1109/

MCI.2015.2405316

3. Huang J, Yu ZL, Cai Z et al (2017) Extreme learning machine with multi-scale local receptive ﬁelds for texture classiﬁcation. Multidim Syst Sign 28:995–1011.

https://doi.org/10.1007/s11045-016-0414-3

4. Liu H, Li F, Xu X, Sun F (2018) Active object recognition using hierarchical local-receptive-ﬁeld-based extreme learning machine. Memetic Comp 10:233–241.

https://doi.org/10.1007/s12293-017-0229-2

5. Lv Q, Niu X, Dou Y, Xu J, Lei Y (2016) Classiﬁcation of hyperspectral remote sensing image using hierarchical local-receptive-ﬁeld-based extreme learning machine. IEEE Geosci Remote Sens Lett 13:1–5. https://doi.org/10.1109/

LGRS.2016.2517178

6. Xu X, Fang J, Li Q, Xie G, Xie J, Ren M (2019) Multi-scale local receptive ﬁeld based online sequential extreme learning machine for material classiﬁcation. In: Sun F, Liu H, Hu D (eds) Cognitive systems and signal processing, vol 1005. Springer, Singapore, pp 37–53.https://doi.org/10.1007/978-981-13-7983-3 4

7. Ding S, Zhao H, Zhang Y, Xu X, Nie R, Ren M (2015) Extreme learning machine:

algorithm, theory and applications. Artif Intell Rev 44:103–115.https://doi.org/

10.1007/s10462-013-9405-z

(20)

8 Y. Liu et al.

8. Huang G, Zhu Q, Siew C (2004) Extreme learning machine: a new learning scheme of feedforward neural networks. In: IEEE international joint conference on neural networks. IEEE Press, New York, pp 985–990. https://doi.org/10.1109/IJCNN.

2004.1380068

9. Huang G, Zhu Q, Siew C (2006) Extreme learning machine: theory and applications. Neurocomputing 70:489–501.https://doi.org/10.1016/j.neucom.2005.12.126 10. Huang G (2014) An insight into extreme learning machines: random neurons, random features and kernels. Cogn Comput 6:376–390. https://doi.org/10.1007/

s12559-014-9255-2

11. Mirza B, Kok S, Dong F (2015) Multi-layer online sequential extreme learning machine for image classiﬁcation. In: Proceedings in adaptation, learning and optimization. Springer, Cham, pp 39–49 (2015). https://doi.org/10.1007/978-3-319- 28397-5 4

12. Cai Y, Liu X, Zhang Y, Cai Z (2018) Hierarchical ensemble of extreme learning machine. Pattern Recogn Lett 116:101–106.https://doi.org/10.1016/j.patrec.2018.

06.015

13. Tang J, Deng C, Huang G (2016) Extreme learning machine for multilayer percep- tron. IEEE Trans Neural Netw Learn Syst 27:809–821. https://doi.org/10.1109/

TNNLS.2015.2424995

14. Yu W, Zhuang F, He Q, Shi Z (2015) Learning deep representations via extreme learning machines. Neurocomputing 149:308–315. https://doi.org/10.

1016/j.neucom.2014.03.077

15. Kasun LLC, Zhou H, Huang G, Vong CM (2013) Representational learning with extreme learning machine for big data. IEEE Intell Syst 4:1–4.https://doi.org/10.

1109/MIS.2013.140

16. Alex K, Sutskever I, Geoﬀrey EH (2012) Imagenet classiﬁcation with deep convolutional neural networks. IEEE Trans Neural Netw Learn Syst 60:84–90.https://

doi.org/10.1145/3065386

17. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: 28th IEEE conference on computer vision and pattern recognition (CVPR). IEEE Press, New York, pp 1–9.https://doi.org/10.1109/CVPR.2015.7298594

18. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition, pp 1–9. arXiv preprintarXiv:1409.1556

19. Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61:85–117.https://doi.org/10.1016/j.neunet.2014.09.003

20. Cao F, Wang D, Zhu H, Wang Y (2016) An iterative learning algorithm for feedforward neural networks with random weights. Inf Sci 328:546–557

21. Raja G, Guillermo S, Alex MB (2016) Deep neural networks with random Gaussian weights: a universal classiﬁcation strategy? IEEE Trans Signal Process 64:3444–

3457

22. Ye H, Cao F, Wang D, Li H (2016) Building feedforward neural networks with random weights for large scale datasets. Expert Syst Appl 106:233–243

23. Vinod N, Geoﬀrey EH (2009) 3D object recognition with deep belief nets. Adv Neural Inf Proces Syst 22:1339–1347

(21)

24. Saxe AM, Koh PW, Chen Z, Bhand M, Suresh B, Andrew Y (2011) On random weights and unsupervised feature learning. In: Proceedings of the 28th international conference on international conference on machine learning. Omnipress, Washing- ton, pp 1089–1096 (2011)

25. Matthieu C, Yoshua B, Jean-Pierre D (2015) BinaryConnect: training deep neural networks with binary weights during propagations. In: Advances in neural information processing systems, vol. 28. Curran Associates, Inc., p 3131 (2015)

(22)

Stability Analysis of Discrete-Time Stochastic Systems with

Borel-Measurable Markov Jumps

Hongji Ma⁽B⁾, Yuechen Cui, and Yongli Wang

Shandong University of Science and Technology, Qingdao 266590, China ma [email protected]

Abstract. This paper is devoted to stability analysis of discrete-time linear systems with Borel-measurable Markov jump parameters and independent multiplicative noises. The relationships are investigated among several stability concepts about the considered dynamics. Speciﬁcally, it is shown that strong exponential stability in the mean square sense can guarantee exponential stability, l₂ input-output stability and stochas- tic stability to hold. Moreover, both exponential stability and l₂ input- output stability give rise to stochastic stability. By a numerical example, it is demonstrated that Borel-measurable Markov jump systems must not be exponentially stable even if it is stochastically stable.

Keywords: Stability

·

Markov jump systems

·

Borel-measurable set

·

Multiplicative noise

1 Introduction

As one of the most important kinds of hybrid systems, stochastic systems with Markov jump parameters have been extensively researched in recent decades.

The impetus for studying Markov jump systems has its root in the potential applications around various fields, such as solar photovoltaic power generation [1], mechanical automation [10] and portfolio optimization [12]. Up to now, a fairly complete sketch of analysis and synthesis has been established for Markov jump systems with finite possible jumps. Among many others, interested read- ers can refer to [2] and [5] for more details about continuous- and discrete-time Markov jump systems with finite Markov jumps. A new trend on the study of Markov jump systems is to generalize the state space of Markov process from finite to infinite. For example, [6] and [8] have addressed the exponential stability of infinite Markov jump systems based on the spectrum of Lyapunov operator.

Optimal control and H2control problems have also been tackled in [7] and [11], respectively. Recently, the state space of Markov jump parameter has been further extended to the case of Borel-measurable set. The more general state space of Markov process has more potential applications in practice, and inevitably makes the related control issues more complex in the same time (c.f. [3] and [4]).

Springer Nature Singapore Pte Ltd. 2020c

Y. Jia et al. (Eds.): CISC 2019, LNEE 594, pp. 10–19, 2020.

https://doi.org/10.1007/978-981-32-9698-5_2

(23)

Borel-Measurable Markov Jump Systems 11

In this paper, our objective is to address the stability of discrete-time linear systems with Borel-measurable Markov jump parameters and independent multiplicative noises. As far as we know, there is few result reported on this type of models. The notions of (strong) exponential stability and l2 input-output stability are proposed for the ﬁrst time. We will focus on the intrinsic relationships among exponential stability, stochastic stability and l2 input-output stability.

The proposed results will lay a solid ground for the further study of optimal and H_∞control problems about the considered models.

The outline of this paper is organized as follows. In Sect.2, some preliminaries are presented for the considered dynamical systems. A useful representation is derived to associate the system state with a Lyapunov operator. Section3 includes the main results of this paper. Necessary and/or suﬃcient conditions are provided for the concerned stability properties. The relationships among them are also clariﬁed. Finally, Sect.4 ends this paper with a brief concluding remark.

Notations. Rⁿ: n-dimensional real Euclidean space; R^m×n: the linear space of all m by n real matrices; · : the Euclidean norm of Rⁿ or the operator norm of R^m×n; A: the transpose of a matrix (or vector) A; S_n: the set of all n× n symmetric matrices; A > 0 (≥ 0): A is positive (semi-)deﬁnite; In: the n× n identity matrix; 1_(·): the indicator function; r(·): the spectral radius of an operator; Z+={0, 1, 2, · · · }.

2 Preliminaries

On a complete probability space (Ω,F , P), we consider the following discrete- time stochastic systems with multiplicative noises:

x_t+1= A0(η_t)x_t+ G0(η_t)v_t+

d k=1

[A_k(η_t)x_t+ G_k(η_t)v_t]w^k_t, (1) where x_t∈ Rⁿ and v_t∈ Rⁿ^v represent the system state and exogenous distur- bance, respectively. For notational simplicity, x0is assumed to be deterministic.

The random vectors w_t={wt = (w_t¹,· · · , w^d_t)} are mutually independent with E(w_t) = 0 and E(w_kw_j) = I_dδ_(k−j). The Markov process{ηt}t∈Z+ takes values in a Borel set S with a σ-ﬁnite measure μ, and has a transition probability functionG (·, ·) associated with the probability density g(·, ·) satisfying

G (, B) = P(ηt+1∈ B|ηt= ) =

B

g(, s)μ(ds), (2)

for any  ∈ S and B ∈ σ(S). In the sequel, the stochastic processes {wt}t∈Z+

are independent of x0 and{ηt}t∈Z+. Denote byFt the σ-algebra generated by {ηk, w_j|0 ≤ k ≤ t, 0 ≤ j ≤ t − 1}. When t = 0, F0= σ{η0}. The initial state of Markov chain η0has a probability distribution π which fulﬁlls

π(B) =

B

ν()μ(d), ∀B ∈ σ(S), (3)

(24)

12 H. Ma et al.

where ν(·) is a nonnegative measurable function deﬁned on S.

Let l²(0,∞; R^m) be the space of R^m-valued stochastic processes {y(t, ω) : Z₊× Ω → R^m}, which are Fk-measurable for all k∈ Z+and_∞

t=0Ey(t)²<

∞. Denote by H^m×n₁ a Banach space {H(·) : S → R^m×n|H() ∈ R^m×n,  ∈ S} with the norm H1 :=

SH()μ(d) < ∞. Similarly, we can introduce another Banach space H^m×n_∞ , where the norm is given by H∞ = ess sup_∈SH(). In the case of m = n, H^m×n₁ will be shorten as Hⁿ1, and so doesH^m×n_∞ . When H()∈ Sn and H()≥ 0 for ∈ S, Hⁿ1 (Hⁿ_∞) will be writ- ten asHⁿ⁺₁ (resp.,Hⁿ⁺_∞). Further, the subset of Hⁿ⁺₁ (resp.,Hⁿ⁺_∞) constituting of all uniformly positive elements that satisfying H()≥ In for some > 0 and any ∈ S will be denoted by ˜Hⁿ⁺1 (resp. ˜Hⁿ⁺_∞).

In (1), we assume that the coeﬃcients A_k ∈ Hⁿ_∞ and G_k ∈ H^n×n_∞ ^v (0≤ k ≤ d). For M, N ∈ Hⁿ⁺₁ , M ≤ N means that M() ≤ N() for all ∈ S. In this case, we have M1 ≤ N1. For a given Banach space X, B(X) indicates the Banach space of all bounded linear operators that map X into X. If Γ ∈ B(X), its uniform induced norm is represented byΓ ξ.

To simplify the expression, we introduce the following linear operators deﬁned onHⁿ_∞: ⎧

⎪⎪

⎪⎨

⎪⎪

⎪⎩

E (U)() =

Sg(, s)U (s)μ(ds), L (U)() = ^d

k=0

S

g(s, )A_k(s)U (s)A_k(s)μ(ds), T (U)() = ^d

k=0

A_k()E (U)()Ak().

(4)

It can be veriﬁed that the operators deﬁned in (4) all belong toB(Hⁿ_∞).

Some fundamental deﬁnitions of stability that will be studied are presented below.

Definition 1. The zero state equilibrium of discrete-time linear infinite Markov jump system

x_t+1= A0(η_t)x_t+

d k=1

A_k(η_t)x_tw^k_t, t∈ Z+, (5) or (A;G ) (A := (A0,· · · , Ad)) for short, is called strongly exponentially mean square stable (SEMSS) if r(L ) < 1, where r(L ) := max{|λ||λ ∈ Λ} denotes the spectral radius of the operatorL and Λ is the spectral set of L .

Definition 2. (A; G ) is called exponentially mean square stable (EMSS) if there exist β≥ 1 and α ∈ (0, 1) such that Ext²≤ βα^tx0²for all t∈ Z+, x0∈ Rⁿ and η₀∈ S. Here, xtis the state of (5) arising from (x₀, η₀) at t = 0.

Definition 3. The perturbed system (1), or (A, G;G ) for short, is said to be l2 input-state stable if for any x0 ∈ Rⁿ and η0 ∈ S, x ∈ l²(0,∞; Rⁿ) whenever v∈ l²(0,∞; Rⁿ^v).

To study the intrinsic relationships among several diﬀerent types of stability, we need the following useful representation associated with (5).

(25)

Borel-Measurable Markov Jump Systems 13

Proposition 1. Let X₀() = x0x₀ν() and X_t() = E[x_tx_tg(η_t−1, )] (t ≥ 1), then X_t+1() =L (Xt)() for t∈ Z+.

Proof. For any x₀∈ Rⁿ, we have X1() = E[x1x₁g(η0, )] =

S

A(s)x0x₀A(s)g(s, )π(ds)

=

S

A(s)x₀x₀A(s)g(s, )ν(s)μ(ds) =L (X0)().

Next let us consider the case of t≥ 1. Substituting the state equation of system (5) into X_t, we will get

X_t+1() = E[xt+1x_t+1g(ηt, )] (6)

= E

E

[A0(ηt)xt+

d k=1

A_k(ηt)xtw^k_t][A0(ηt)xt+

d k=1

A_k(ηt)xtw_t^k]g(ηt, )

Ft−1

.

Taking into account that{w_t^k}^d_k=0are independent ofFt−1, the above equality yields that

Xt+1() = E[xt+1x_t+1g(ηt, )] =

d k=0

E

E[A_k(ηt)xtx_tA_k(ηt)g(ηt, )Ft−1

, (7)

where the assumptions E(w_t) = 0 and E(w_kw_j) = I_dδ_(k−j) have been used.

Then, by the knowledge of stochastic analysis, it can be computed that

X_t+1() = E[x_t+1x_t+1g(η_t, )] =

d k=0

E

S

A_k(s)x_tx_tA_k(s)g(s, )g(η_t−1, s)μ(ds), (8)

which, via Fubini’s theorem, leads to

X_t+1() =

d k=0

S

{Ak(s)E[x_tx_tg(η_t−1, s)]A_k(s)g(s, )}μ(ds)

=

d k=0

S

{Ak(s)X_t(s)A_k(s)g(s, )}μ(ds) = L (Xt)(). (9)

The desired result is obtained.

Remark 1. Since the coeﬃcients of (5) are bounded in the norm · _∞, it can be further shown that X_t∈ Hⁿ⁺₁ .

(26)

14 H. Ma et al.

3 Stability Analysis

In this section, we will clarify the connections among the stability concepts given in the previous section. First of all, we will provide some necessary and suﬃcient conditions for SEMSS of (5) based on Lyapunov equation/inequality.

Theorem 1. (A; G ) is SEMSS if and only if one of the following conditions holds:

(i) for any Y ∈ ˜Hⁿ⁺_∞, there exists a S∈ ˜Hⁿ⁺_∞ such that

S()−

d k=0

A_k()E (S)()Ak() = Y (), ∈ S. (10)

(ii) there exists a Y ∈ ˜Hⁿ⁺_∞ such that the following equation has a solution S ∈ ˜Hⁿ⁺_∞ :

S()−

d k=0

A_k()E (S)()Ak() = I_n, ∈ S. (11)

(iii) there exists a S∈ ˜Hⁿ⁺_∞ such that

S()−^d

k=0

A_k()E (S)()Ak() > αI_n, ∈ S, (12)

where α > 0 is independent of .

Proof. By Deﬁnition1, (A;G ) is SEMSS if and only if L deﬁnes an exponentially stable causal evolution. As shown by Proposition 4.2 [3],T may be regarded as the adjoint operator ofL with respect to the following bilinear operation:

X, Y =

S

T r[X()Y ()]μ(d), X∈ Hⁿ1, Y ∈ Hⁿ_∞, (13)

where T r(·) denotes the trace of a matrix. Then, the statements (1), (2) and (3) can be drawn from Theorem 2.4 (v), (iv) and (vii) of [5], respectively.

The following result reveals that SEMSS can ensure (5) to be EMSS.

Theorem 2. If (A; G ) is SEMSS, then (A; G ) is EMSS.

Lecture Notes in Electrical Engineering 594

Lecture Notes in Electrical Engineering 594

Yingmin Jia Junping Du

Weicun Zhang Editors

Proceedings of 2019 Chinese Intelligent Systems Conference

Volume III

Lecture Notes in Electrical Engineering

Volume 594

Yingmin Jia

Junping Du

Weicun Zhang

Editors

Proceedings of 2019 Chinese Intelligent Systems Conference

Volume III

123

Contents

Hierarchical Pooling Based Extreme Learning Machine for Image Classification

·

·

·

1 Introduction

2 Related Works

3 HPELM

4 Experimental Results

5 Conclusions

References

Stability Analysis of Discrete-Time Stochastic Systems with

Borel-Measurable Markov Jumps

·

·

·

1 Introduction

2 Preliminaries

3 Stability Analysis