Liquid State Machines (LSMs) are a novel computational framework recently in- troduced in . The computing core is a recurrent neural network (RNN). The computing core is referred to as the ‘liquid’; the term liquid is a symbolic repre- sentation of the RNN’s dynamic nature. In the ‘liquid’, each computing ‘unit’ is defined as a mathematical model of spiking integrate-and-fire neuron. These neu- ron models closely resemble functioning of the biological neurons. Thus, the LSM framework is capable of performing real-time computations on time-varying inputs. Its applications are mainly focused in the area of neural microcircuits [13, 23]. The LSM’s architecture can be divided into three distinct sections namely; (i) the pre- processor unit called as the ‘liquid’, (ii) the liquid’s state space (i.e., memory), and (iii) the processing layer called the ‘readout layer’.
In the recent years the movement prediction had became more demanding in wide fields. Probably the most seemingly fields are the robotic applications such as artificial vision systems and militaries applications specifically in air defense (tracking and predicting movement of rockets and aircrafts). Also a very important application the predictor can be used for; is communication systems' bandwidth reduction in video transmission. Since only the prediction error will be transmitted (like the linear predictor used to reduce the bandwidth in voice transmission in mobile phone) which could be classified as data compression method. The Artificial Intelligent Techniques have reached advanced level that’s enable us to build such systems that require a very strong dynamic behavior and large memory capacity. Experts would prefer to choose AI based technique over a specific design (the later will consume much more time, calculation and effort to be achieved). The ReservoirComputing is a developed technique by , under the names of Echo State Network (ESN) and Liquid State Machine (LSM). This neural networks based technique was proven to perform better than any other Dynamic Neural Networks family. But the major difference between the RC tech. and other neural networks is its simple structure and training. The structure consists of two parts; a recurrent neural network with random connection topology usually called the "reservoir". The other part is a feed forward neural network called the "readout layer"; see Figure (1). Its operation based on the following statement: the reservoir computes and process information about the input from
The Markovian characterization of ESNs implies the fact that older input symbols have an exponentially decreasing influence on the reservoir state. Therefore the distance between the states corresponding to input sequences which differ only before a common suffix of length n, is anyhow bounded by a term which exponentially decreases with n. In this way, the reservoir of an ESN is able to differentiate different input sequences in a Markovian suffix-based way even in the absence of learning of the recurrent state transition function. Such state space organization makes the model more suitable for tasks for which more similar targets are associated to input sequences sharing longer suffixes. This is graphically illustrated in Figure 3.1, showing different (symbolic) input sequences and corresponding states computed by a model respecting the suffix-based Markovian organized dynamics. In the simplified example of Figure 3.1, the states corresponding to different input sequences cluster together in a suffix-based fashion. The two sequences ”cbaaba” and ”baaaba”, sharing the suffix ”aaba”, are mapped into close states, whereas the state corresponding to ”cbbaab” is quite distant. Thereby, the readout naturally performs better if the task at hand requires to associate similar outputs to ”cbaaba” and ”baaaba” and a different one to ”cbbaab”. On the contrary, the model is less appropriate whenever the task requires similar outputs for ”cbaaba” and ”cbbaab” and a different one for ”baaaba”. To concretely show the effect of the Markovian factor on possible classes of tasks, in Section 3.5.1 we design two ad-hoc target functions that match and un-match, respectively, the Markovian characterization of state dynamics, thereby characterizing suitable (easy) and un-suitable (hard) tasks for the ESN approach.
Based on the above discussion, the aim of this paper is to extend the distributed ESN protocol presented in  with the inclusion of a decentralized LASSO training algorithm . As we show in the subsequent sections, this allows us to obtain very sparse networks. Apart from achieving a higher accuracy in specific settings (as discussed in Section II), sparseness in the readout strongly reduces the number of parameters required for describing the ESN, allowing to reduce the in-network bandwidth required during the training phase, and the computational cost of performing a prediction. In order to test the validity of the proposal, we investigate two distributed (noisy) prediction problems, involving the Mackey-Glass chaotic time-series and a 10th order nonlinear auto regressive moving average (NARMA) system.
The suggested model has some limitations that could prevent it from achiev- ing the desired performance in certain regimes. The main limitation of ESNS- VMs is the added complexity over ESN, which is represented by the need to optimise the SVMs parameters used in the output layer. This includes the choice of kernel and its associated parameters. The learning in general will take longer, especially when applying ESNSVMs to multiple class problems with medium to large numbers of classes. This is due to the nature of SVMs, which are binary classifiers that require constructing at least equal to the number of problem classes, in the one against all approach, or n ( n 1 ) /2 classifiers where n is the number of classes, in the one against one approach. This limits the ability of ESNSVMs when dealing with the multi-class classification problem with a medium to large number of classes which is required in developing phoneme-based speech recognition systems. In the Arabic domain, using ESNSVMs to develop a phoneme speech recognition system means that 1,260 classifiers needs to constructed in the output layer. This is a computationally expensive and time-consuming procedure that may prove impossible due to hardware limitations. Thus, this developed system has an advantage over the standard ESN approach in regimes where the input dimension is input dimension is very high and there is a limited number of classes.
Up to Chapter 4, we introduced several studies on first-order and second-order memristor devices and networks, focusing on approaches inspired by biology. The crossbar network can efficiently implement bio-inspired computing algorithms due to the co-location of memory and computation with high level of parallelism. However, the maximum size of arrays used in the previous chapters remains relatively small at 16×3, and data processing for more complex algorithm demands larger sizes of memristor arrays to fully take advantage of its parallelism. For example, convolutional neural networks (CNNs) require massive amounts of VMM operations during both training and testing, and can be dramatically speeded up in a hardware system based on larger memristor arrays. In this chapter, we discuss factors that can affect the practical size of memristor arrays from the device fabrication point of view.
We introduce the two types of reservoircomputing and compare them. For most tasks, Echo state network(ESN) has better performance, but Reservoircomputingbased on Kuramoto model(RCK) has a strength in the case of prediction. ESN has a stable result by the Echo state property, but RCK has no such property for stability. Also, RCK has a limitation ac- cording to the number of nodes, but ESN doesn’t. In the case of RCK, there have been many problems with parameter setting because research on parameters has not been conducted much. Performance comparison according to parameters for synchronization of Kuramoto model also seems necessary.
The reservoircomputing paradigm to train only a perceptron-like read-out layer relies on the encoding of inputs in the reservoir by means of a “random, temporal and non-linear kernel” . The mixture of both spatial and temporal components of the input is based upon three key ingredients: (i) the random projection into a high-dimensional state space (ii) with non-linearity introduced by typically sigmoidal activation functions and (iii) the recurrent connections in the reservoir implementing a short-term memory. On the one hand, the non-linear projection into a high-dimensional space is related to spatial kernel expansions which rely on the concept of a non-linear transformation of the original data into a high-dimensional space and the subsequent use of a simple, mostly linear, model. On the other hand, the recurrent connections implement a short-term memory by means of transient network states. Reservoircomputing thereby exploits the “architectural bias“  of recurrent neural networks with small weights to finite memory machines even prior to learning. Due to this short-term memory, reservoir networks are typi- cally utilized for temporal pattern processing such as time-series prediction, classification and generation . In recent work, it is shown that recurrence in the reservoir also enhances the spatial encoding of static inputs in the reservoir . Reservoircomputingbased on attractor states is therefore also suited for static pattern processing [82, 83, 84, 85].
vironments is presented in , in which by adopt- ing a hybrid approach, RC networks have been showed to provide significant benefits to the accuracy of RSS- based localization systems. It is also worth mention- ing the recent results of the European FP7 Project RU- BICON [23,24], whose goal was to design and de- velop a self-learning robotic ecology made up of sen- sors, actuators and robotic devices. Within the aims of the RUBICON project, ESNs distributed on the low- powerful nodes of a Wireless Sensor Network (WSN) have been used to approach supervised computational tasks in the field of AAL, and pertaining to the classi- fication of human activities, using input data streams coming from sensorized environments. In this context, the work in  describes the application of ESNs to a set of supervised tasks pertaining to the recognition of user daily-life activities, within an AAL real-life test bed scenario, including more than 50 sensors of differ- ent types. Moreover, an application of ESNs to adap- tive planning for context-aware robot navigation is pre- sented in . Further applications of ESNs in the area of AAL are reported in , in which ESNs are used to classify physical activities in equestrian sports from a 3-axis accelerometer, a gyroscope and a magnetic sen- sor stripped to a horse rider wrist, and in , in which standard ESNs are used to estimate people count from multiple PIRs in indoor corridors at the Fraunhofer IAIS facilities. Overall, the aforementioned applica- tions of the RC approach to problems in the area of AAL and human activity recognition show the poten- tiality of this methodology, still deserving further in- vestigations and deep experimental assessment, in par- ticular in combination with already established tech- niques in this specific application domain.
The next experiment is accomplished with the first environment S6 from Fig. 4.8 and lasts 180.000 timesteps. The resulting robot oc- cupancy grid can be seen in Fig. 4.12(a): it shows that the reservoir predicts the current location of the robot very well, with a classi- fication rate of 91.6% on test data. Another experiment uses the same environment S6 with 11 additional slow moving obstacles dis- tributed throughout the environment. These dynamic objects alter the behavior of the robot, e.g., by blocking navigation and producing avoidance behaviors, resulting in an extra source of noise to sensory readings. The simulation takes 180.000 timesteps. The respective occupancy grid in Fig. 4.12(b) shows that the RC network correctly predicts the location in 81.1% of the samples. Some of the mispre- dictions are located a bit further from the actual position, due to the new source of dynamics and noise, although they generally tend to be very short. As comparison, the RNN-based room detector in Forster et al. (2007) which has 36 inputs coming from a laser range scanner presents a test classification rate of 81.8% for non-noisy en- vironment and 82.8% with 10 slow moving obstacles for a simulated house environment of 15 rooms.
2008 revived the scientific interest both from academia and industry for this device technology, with several emerging applications including that of logic circuits. Several memristive logic families have been proposed, each with different attributes, in the current quest for energy-efficient computing systems of the future. However, limited endurance of memristor devices and variations (both cycle-to-cycle and device-to-device) are important parameters to be considered in the evaluation of such logic families. In this work we build upon a well-known accurate physics-based model of a bipolar metal-oxide resistive RAM device (supporting parasitics of the device structure and variability of switching voltages and resistance states) and use it to show how performance of memristor-based logic circuits can de degraded owing to both variability and state-drift impact. Based on previous work on CMOS-like memristive logic circuits, we propose a memristive ratioed logic scheme, which is crossbar- compatible, i.e. suitable for in-/near-memory computing, and tolerant to device variability, while also it does not affect the device endurance since computations do not involve switching the memristor states. As a figure of merit, we compare such new logic scheme with MAGIC, focusing on the universal NOR logic gate.
The human brain’s ability to learn and adapt is a hallmark for intelligence. It can process multiple streams of information simultaneously while using very little energy. The average human brain has billions of neurons and trillions of synaptic connections . It is extremely difficult to model the vastness and complexities of the human brain partially because its operation is still not completely understood and partially because our computing technology is not advanced enough. Presently, the fastest supercomputer, OLCF-4, developed by IBM for Oak Ridge National Lab is capable of operating at 200 petaFLOPS (10 15 ) and the human brain is postulated to operate at 1 exaFLOPS (10 18 ) . That said, researchers, today are working to mimic the behavior of complex biological networks using electronic artificial neural networks.
The emerging Cloud Computing paradigm , as exemplified by the Amazon Elastic Com- pute Cloud (EC2), represents a promising conceptual foundation for hosting and deployment of web-based services while theoretically relieving service providers from the responsibility of provisioning the computational resources needed to support these services. Cloud computing offers multiple advantages: it allows individuals or companies with market domain expertise to build and run their Software as a Service (SaaS) company with minimal effort in software devel- opment and without managing any hardware operations. This helps reduce software complexity and costs, expedite time-to-market, and enhance accessibility of consumers.
putational and memory requirements. To understand the details of information processing in a reservoir, we have to understand the effects of the reservoir’s architecture on its fundamental memory and computational capacity. We also have to be able to define the classes of tasks that can be parametrically varied in memory requirement and nonlinearity. Our study reveals that although ESN cannot memorize patterns as well as a memory device or a neural network, it greatly outperforms them in generalizing to novel inputs. Also, increasing reservoir size in ESN improves the performance of generalization, whereas in the DL or the NARX network this will result in increased over- fitting leading to poorer generalization. One solution would be to extend the receiver operation characteristic (ROC) and receiver error characteristic (REC) curve methods to decide on the quality of generalization in ESN –. In the neural network community, methods based on pruning, regularization, cross-validation, and information criterion have been used to alleviate the overfitting problem – . Among these methods, regularization has been suc- cessfully used in ESNs . However, these methods focus on increasing the neural network’s performance and are not suitable to quantify overfitting or to study task hardness. Another area that requires more research is the amount of training that the ESN requires to guarantee a certain performance, as is described in probably approximately correct methods –. To the best of our knowledge these problems have not been addressed in the case of high- dimensional dynamical systems. A well developed theory of computation in reservoircomputing needs to address all of these aspects. In future work, we will study some of these issues experimentally and based on our observations, we will attempt to develop theoretical understanding of computation in the reservoircomputing paradigm.
search; Evolution in Materio and ReservoirComputing. Evo- lution in materio attempts to evolve physical computational machines from often design-less and unconstrained materials through computer controlled evolution . Similar conceptual ideas can be seen in and around the cybernetics movement of the 1940s through pioneering cyberneticians such as Gor- don Pask and Stafford Beer (see ). However, not until Thompson  exploited the low-level physics of silicon-based electronic devices through computer controlled evolution was it so distinctly demonstrated. Thompson’s work showed that blind evolution could harness the unknown physical properties of modern electronic devices to create often unusual solutions. Harding et al.  continued this work, refining the technique and introducing new substrates, such as a liquid crystal display, magnetic quantum dots, a crystal lattice and an optical device. More recently, the EU-funded NASCENCE project  devel- oped new evolvable nanosystems and unconventional hardware interfaces, including carbon nanotube based composites (with static and dynamic structures), disorganised gold-nanoparticle networks, and a bespoke computing platform –. The field of ReservoirComputing was originally conceived from two complementary independent investigations; designing a computational model for real-time continuous cortical micro- circuits (Liquid State Machine) , and an efficient technique for training discrete artificial recurrent neural networks (Echo State Networks) . After its inception, the reservoir model emerged as a potential computational model for many dynam- ical systems and has been applied to several systems, such as a bucket of water , optoelectronic and photonic systems –, and memristive networks , .
Recently, a study of layered deep RC architectures has been proposed in [11, 12], with the introduction of the deepESN model. The study of deepESNs, comprising a hierarchy of multiple untrained reservoir layers, aims on the one hand to better understand the meaning of stacking RNN layers separately from learning aspects, and on the other hand to represent a starting point for the design of efficiently trained deep learning models for temporal data. The preliminary experimental analysis proposed in [11, 12] has evidenced that it is possible to practically exploit the deep architectural design in conjunction with key reservoir hyper-parameters to enhance the time-scale differentiation of layers’ dynamics. The results in [11, 12] show that higher layers in the hierarchy can effectively develop progressively slower dynamics. Moreover, recent literature studies are addressing the impact of ESNs organized in layered architectures in terms of applications on benchmarks as well as on real world tasks. Specifically, in  an ESN-based hierarchical network has been proposed, in which each successive layer is trained to estimate the relevance of the information that is processed at the lower layer, for the final output computation. Being investigated in relation to the problem of temporal feature discovery at different scales, the architecture proposed in  showed promising results on a case study with synthetic data. The advantage of multi-layered RC architectures has also been pointed out on time-series benchmarks in the RC area  and on real world tasks in the area of speech processing using ad-hoc hierarchical RC settings [43, 44]. Overall, such literature works witness the emergence of a growing application interest in this field and further exacerbates the urgency of a timely and theoretically rational support to the set up of hierarchical RC networks.
Distinct from traditional computing, brain-like computing is inspired by biological mechanisms. It is composed of a large number of interconnected neurons working in parallel to solve specific problems. The networks are not programmed since the connections between neurons are “weighted” according to the correlations of data they have already learned. This idea, named “synaptic plasticity”, is also inspired by the studies of biological neural networks. Brain-like computing is a complement to conventional computing rather than a replacement of it because of the distinctions between them. Since conventional computing uses algorithmic approaches, it is competent at fast arithmetic or with unambiguous tasks but lacks abilities such as massive parallelism and fault tolerance. An example is that cognition, perception, and learning on biological spatial and temporal scales can be easily achieved by mammalian brains, however such tasks remain out of reach for modern computers. As a complement, brain-like computing which benefits from similar mechanisms of biological neural networks can do these jobs that conventional computing cannot do well. Artificial neural networks that are foundation stones for brain-like computing have become a thriving research field nowadays. Since the first artificial neural network was proposed, research was split into two distinct approaches which separately focus on biological processes in the brain and the applications of neural networks to artificial intelligence.
Training such RNNs is a bit more complicated than training a feed-forward neural network. However, Werbos (1990) developed a generalization of the Back- Propagation algorithm of Rumelhart et al. (2002) based on the following insight. An RNN can be represented by a single hidden layer where the input also includes the previous neuron states. These previous states can in turn be represented by again a single hidden layer that depends on the previous input and neuron states. An RRN can thus be unfolded over time such that it resembles a multi-layered feed- forward NN. The number of hidden layers within such an unfolded RNN depends on the number of time steps of the training data. The error on the output can again be propagated backwards through the unfolded network and thus time, hence the name for the entire process: Back-Propagation-Trough-Time (BPTT). However, this algorithm is based on the stochastic gradient descent learning approach, caus- ing BPTT to suffer from the same weaknesses: slow convergence times and the possibility to get stuck into a local optimum (not the overall best solution). Addi- tionally, there exists the possibility that gradient information calculated with BPTT fades away due to the number of hidden layers which introduce a large number of non-linearities between the past and present. This fading gradients problem, and the regular bifurcations encountered during training using stochastic gradient de- scent (Bengio et al., 1994; Pearlmutter, 1995; Suykens and Osipov, 2008) make RNNs notoriously difficult to train.
Recently, memristors considered as a new fundamental two terminal circuit devices [2, 3], exhibit the expected advantages in high-density integration, low power consumption, and high-speed read and write . Besides the crossbar architecture of memristor integration can significantly increase memory density while reducing power consumption. It should be highlighted that memristor has been highly considered to be one of the promising candidate for implementation of next generation non-volatile memory, which thus could advance Moore’s Law beyond the present silicon roadmap horizons. Moreover, memristive devices naturally combine the logic, storage and computing function together, which have a wide range of applications, covering almost all existing electronic circuits, and is constantly expanding its application fields [5-7].
Abstract: With the increase in communication bandwidth and frequency, the development level of communication technology is also constantly developing. The scale of the Internet of Things (IoT) has shifted from single point-to-point communication to mesh communication between sensors. However, the large sensors serving the infrastructure place a burden on real-time monitoring, data transmission, and even data analysis. The information processing method is experimentally demonstrated with a non-linear Schmitt trigger oscillator. A neuronally inspired concept called reservoircomputing has been implemented. The synchronization frequency prediction tasks are utilized as benchmarks to reduce the computational load. The oscillator's oscillation frequency is affected by the sensor input, further affecting the storage pattern of the oscillatory neural network. This paper proposes a method of information processing by training and modulating the weights of the intrinsic electronic neural network to achieve the next step prediction. The effects on the frequency of a single oscillator in a coupled oscillatory neural network are studied under asynchronous and synchronization modes. Principle Component Analysis (PCA) is used to reduce the data dimension, and Support Vector Machine (SVM) is used to classify the synchronous and asynchronous data. We define that oscillator with stronger coupling weight (lower coupling resistance) as a leader oscillator. From the spice simulation, when OSC 1 and OSC 2 work as leader oscillator, the ONN almost always achieve synchronization; and the synchronization frequency is