recurrent neural network

This paper described recurrent neural network based quality estimation models of sentence, word and phrase level. We extended the (existing sentence-level) recurrent neural network based quality estimation model to word and phrase level. And we applied these models to sentence, word and phrase-level QE shared task of WMT16. These recurrent neural network based quality esti- mation models are pure neural approaches for QE and achieved excellent performance especially in sentence and phrase-level QE.

Hurricanes are cyclones circulating about a defined center whose closed wind speeds exceed 75 mph originating over tropical and subtropical waters. At landfall, hurricanes can result in severe disasters. The accuracy of predicting their tra- jectory paths is critical to reduce economic loss and save hu- man lives. Given the complexity and nonlinearity of weather data, a recurrent neural network (RNN) could be beneficial in modeling hurricane behavior. We propose the application of a fully connected RNN to predict the trajectory of hur- ricanes. We employed the RNN over a fine grid to reduce typical truncation errors. We utilized their latitude, longitude, wind speed, and pressure publicly provided by the National Hurricane Center (NHC) to predict the trajectory of a hur- ricane at 6-hour intervals. Results show that this proposed technique is competitive to methods currently employed by the NHC and can predict up to approximately 120 hours of hurricane path.

Abstract. In this paper we solve a wide rang of Semidefinite Programming (SDP) Problem by using Recurrent Neural Networks (RNNs). SDP is an important numerical tool for analysis and syn- thesis in systems and control theory. First we reformulate the prob- lem to a linear programming problem, second we reformulate it to a first order system of ordinary differential equations. Then a recurrent neural network model is proposed to compute related pri- mal and dual solutions simultaneously. Illustrative examples are included to demonstrate the validity and applicability of the tech- nique.

A conventional approach for the audio-visual speech recognition task is to infer possible sequences using sequential probability inference models like Hidden Markov Models (HMMs). Since the neural network was introduced to the machine learning community, it has been applied on different kinds of machine learning tasks. In speech recognition research, the recurrent neural network models, which treat speech signals as time-varying inputs, is known to improve the performance of ASR systems [1].

combination of recursive neural network and recurrent neural network, and in turn integrates their respective capabilities: (1) new information can be used to generate the next hidden state, like recurrent neu- ral networks, so that language model and translation model can be integrated natu- rally; (2) a tree structure can be built, as recursive neural networks, so as to gener- ate the translation candidates in a bottom up manner. A semi-supervised training ap- proach is proposed to train the parameter- s, and the phrase pair embedding is ex- plored to model translation confidence di- rectly. Experiments on a Chinese to En- glish translation task show that our pro- posed R 2 NN can outperform the state-

Initial work in discovering new recurrent architectures did not yield promis- ing results ( Klaus et al. (2014)). However, a recent paper from Zoph and Le (2016) showed that policy gradients can be used to train a LSTM network to find better LSTM designs. In Zoph and Le (2016), a recurrent neural network (RNN) was used to generate neural network architectures, and the RNN was trained with re- inforcement learning to maximize the expected accuracy on a learning task. This method uses distributed training and asynchronous parameter updates with 800 graphic processing units (GPUs) to accelerate the reinforcement learning process. Baker et al., (2017) have proposed a meta-modeling approach based on reinforce- ment learning to produce CNN architectures. A Q-learning agent explores and exploits a space of model architectures with an −greedy strategy and experience replay. These approaches adopt the indirect coding scheme for the network repre- sentation, which optimizes generative rules for network architectures such as the RNN. Suganuma et al. (2017) propose a direct coding approach based on Cartesian genetic programming to design the CNN architectures.

This paper describes our submission to the shared task on word/phrase level Qual- ity Estimation (QE) in the First Con- ference on Statistical Machine Trans- lation (WMT16). The objective of the shared task was to predict if the given word/phrase is a correct/incorrect (OK/BAD) translation in the given sen- tence. In this paper, we propose a novel approach for word level Quality Esti- mation using Recurrent Neural Network Language Model (RNN-LM) architecture. RNN-LMs have been found very effective in different Natural Language Processing (NLP) applications. RNN-LM is mainly used for vector space language model- ing for different NLP problems. For this task, we modify the architecture of RNN- LM. The modified system predicts a label (OK/BAD) in the slot rather than predict- ing the word. The input to the system is a word sequence, similar to the standard RNN-LM. The approach is language in- dependent and requires only the translated text for QE. To estimate the phrase level quality, we use the output of the word level QE system.

people across the word. This has led to the advancement in science and technology. Many surveys conclude that network intrusion has registered a consistent increase and lead to personal privacy theft and has become a major platform for attack in the recent years. Network intrusion is unauthorized activity on a computer network. Hence there is a need to develop an effective intrusion detection system. In proposed system acquaint an intrusion detection system that uses improved recurrent neural network (RNN) to detect the type of intrusion. In proposed system also shows a comparison between an intrusion detection system that uses other machine learning algorithm while using smaller subset of kdd- 99 dataset with thousand instances and the KDD-99 dataset.

context recurrent neural network to compress mul- tiple sentences into a single vector. This difficulty in training a recurrent neural network to com- press a long sequence into a single vector has been observed earlier, for instance, in neural machine translation (Cho et al., 2014a). Attention mech- anism, which was found to avoid this problem in machine translation (Bahdanau et al., 2014), is found to solve this problem in our task as well. Perplexity per Part-of-Speech Tag Next, we attempted at discovering why the larger-context recurrent language model outperforms the uncon- ditional one. In order to do so, we computed the perplexity per part-of-speech (POS) tag.

Among various neural network language models (NNLMs), recurrent neural network-based lan- guage models (RNNLMs) are very competitive in many cases. Most current RNNLMs only use one single feature stream, i.e., surface words. However, previous studies proved that language models with additional linguistic information achieve better performance. In this study, we extend RNNLM by explicitly integrating additional linguistic information, including morphological, syn- tactic, or semantic factors. Our proposed RNNLM is called a factored RNNLM that is expected to enhance RNNLMs. A number of experiments are carried out that show the factored RNNLM im- proves the performance for all considered tasks: consistent perplexity and word error rate (WER) reductions. In the Penn Treebank corpus, the relative improvements over n-gram LM and RNNLM are 29.0% and 13.0%, respectively. In the IWSLT-2011 TED ASR test set, absolute WER reduc- tions over RNNLM and n-gram LM reach 0.63 and 0.73 points.

Intrusion detection is a key research area in network security. A common approach for in- trusion detection is detecting anomalies in network tra ffi c, however, network threats are evolv- ing at an unprecedented rate. The di ff erence between the evolution of threats and the current detection response time of a network leave the system vulnerable to attacks [1]. Over the years, a number of machine learning techniques have been developed to detect network intrusions us- ing packet prediction [2]. Recurrent Neural Network (RNN) is the most popular method of performing classification and other analysis on sequences of data. A subset network of RNN is Long Short-term Memory (LSTM), introduced by Hochreiter and Schmidhuber (1997) [3]. LSTM is a key algorithm in regards to the implementation of machine learning tasks that in- volve sequential data. Successful deployment of LSTM has led the industry to heavily invest in implementing the algorithm in a wider range of applications. These applications include voice recognition [4], [5], handwriting recognition [6], machine translation and social media filter- ing, thus making LSTM a natural candidate for Intrusion Detection Systems (IDS). Yet, this

Abstract— This paper proposes a weight initialization strategy for a discrete-time recurrent neural network model. It is based on analyzing the recurrent network as a nonlinear system, and choosing its initial weights to put this system in the boundaries between different dynamics, i.e., its bifurcations. The relationship between the change in dynamics and training error evolution is studied. Two simple examples of the application of this strategy are shown: the detection of a 2-pulse temporal pattern and the detection of a physiological signal, a feature of a visual evoked potential brain signal.

The project demonstrates a web based Solar Power Predictor Application hosted on Flask server. Firstly we trained our model using Long Short Term Memory, Recurrent Neural Network ML Algorithm. The activation functions used in the hidden layer of RNN are Sigmoid and Tanh. The model was trained using the dataset obtained from repository mentioned in Section III of this paper. After training the algorithm we dumped our model in Pickle. To do the prediction of Solar Power user can login into web based app and select the city name for which prediction has to be done. After selecting the city, the API key provided by openweathermap.org and darksky.net will fetch weather data in JSON format. In the background this fetched data will be scaled using the scale factor obtained while training the algorithm. After scaling the data will be passed to the pickle and the output predicted by the model will again be scaled to KWh and displayed as the amount of Solar Polar generated for the queried city. An example of Solar Power predicted for Pune is displayed below which tells us the amount of Solar Power which will be generated by an individual Solar PV System in Pune City.

A feed forward neural network is used to predict duration for Telugu [6]. A Recurrent Neural Network (RNN) is used to predict prosodic information for Persian, Chinese and Mandarin [7]. Recurrent data input also helps to smooth the output parameter tracks [8]. RNNs inherently implement short-term memory by allowing the output of a neuron to influence its input either directly or indirectly via its effect on other neurons [9]. It is obvious that cognitive processes and/or more practical applications will require higher-level architectures. This is a solid reason to investigate recurrent neural networks even if feed forward networks showed good results in many practical applications in different areas, from classification to time-series prediction. In the present work hence it is proposed to predict duration of syllable for Telugu with RNN approach since RNN is better in learning sequence processing tasks than simple feed forward neural network. Linguistic features are used as input nodes of RNN to learn duration rules of the syllable automatically and can be predicted duration of syllable at the output node.

a recursive neural network to predict sentence senti- ment. (Luong et al., 2013) generates better word rep- resentation with recursive neural network. (Cho et al., 2014a) proposed a RNN encoder-decoder model to learn phrase representations in SMT. (Irsoy et al., 2014) introduce a deep recursive neural network, and evaluate this model on the task of fine-grained sentiment classification. (Liu et al., 2014) propose a recursive recurrent neural network to model the end- to-end decoding process for SMT; experiments show that this approach can outperform the state-of-the- art baseline. (Yao et al., 2013) optimized the recur- rent neural network language model to perform lan- guage understanding. (Graves , 2012) apply a RNN based system in probabilistic sequence transduction. 6.2 Loanwords Detection

In this paper, we present the word embedding used for complaint classification which combine with recurrent neural network LSTM and GRU with a single direction and also bidirectional. Our evalua- tion focuses on the comparison of F1 score between various combinations of bidirectional LSTM- GRU. Bidirectional recurrent neural network can surpass the traditional method, TF-IDF (75% F1 score) while using the same amount of training data. The usage time for training is dependent upon the processing unit. It requires about 2-3 hours for the training with a graphic processing unit NVIDIA 660M with 8 GB RAM with 64 word dimensions and 64 hidden units for each architecture. But the execution time requires a few second to predict each sentence.

In the case of recurrent neural network architec- tures such as LSTMs (Hochreiter and Schmidhu- ber, 1997) and GRUs (Cho et al., 2014), there are two types of neural connections involved: many- to-one weighted linear connections, and two-to- one multiplicative interactions. Hence, we restrict our definition of the LRP procedure to these types of connections. Note that, for simplification, we refrain from explicitly introducing a notation for non-linear activation functions; if such an activa- tion is present at a neuron, we always take into account the activated lower-layer neuron’s value in the subsequent formulas.

(2) As a special case of approximation to this, clas- sical n-gram language model keep only sever- al words as history, discarding any information across the sentence boundaries. Recurrent neural network language model (Mikolov et al., 2010) us- es a hidden layer which employs a real-valued vec- tor recurrently as network’s input to keep as many history as possible. This makes RNNLM be able to extend for capturing history beyond a sentence. To prevent the potential exponential decay of the history, the history length in RNN can not be too long. Here we approximate the history information of previous sentences, p(S k |S 1 , S 2 , ..., S k−1 ), by the following:

Recurrent Neural Networks were invented a long time ago, and dozens of different architectures have been published. In this paper we generalize recurrent architectures to a state space model, and we also generalize the numbers the net- work can process to the complex domain. We show how to train the recurrent network in the complex valued case, and we present the theorems and procedures to make the training stable. We also show that the complex valued recurrent neural network is a generalization of the real valued counterpart and that it has specific advantages over the latter. We conclude the paper with a discussion of possible applications and scenarios for using these networks.

and parsing (Lewis and Steedman, 2014). However, their architecture is limited to considering local contexts and does not naturally model sequences of arbitrary length. In this paper, we show how di- rectly capturing sequence information us- ing a recurrent neural network leads to fur- ther accuracy improvements for both su- pertagging (up to 1.9%) and parsing (up to 1% F1), on CCGBank, Wikipedia and biomedical text.