# Questions tagged [recurrent-neural-network]

600 questions

0

votes

0

answer

5

Views

### How to use RNN cell in a network?

I am trying to use a customized RNN cell in my network. I started with the RNN cell example of Keras where the RNN cell is defined as MinimalRNNCell. When I am trying to use the defined cell in my recurrent network, by replacing a simpleRNN that I was using previously with the customized RNN cell, b...

1

votes

0

answer

13

Views

### Inconsistency between GRU and RNN implementation

I'm trying to implement some custom GRU cells using Tensorflow. I need to stack those cells, and I wanted to inherit from tensorflow.keras.layers.GRU. However, when looking at the source code, I noticed that you can only pass a units argument to the __init__ of GRU, while RNN has an argument that is...

0

votes

0

answer

4

Views

### How should the input shape of Keras LSTM layer looks like

I've been reading for a while about training LSTM models using tf.keras, where i did use the same framework for regression problems using simple feedforward NN architectures and i highly understand how should i prepare the input data for such models, however when it comes for training LSTM, i feel...

1

votes

1

answer

693

Views

### Time series classification using LSTM - How to approach?

I am working on an experiment with LSTM for time series classification and I have been going through several HOWTOs, but still, I am struggling with some very basic questions:
Is the main idea for learning the LSTM to take a same sample from every time series?
E.g. if I have time series A (with sam...

1

votes

1

answer

497

Views

### Tensorflow 1.4 Bidirectional RNN not working as expected

I am trying to use Bidirectional RNN and pass the output through a CNN for text classification. However, I am getting all sorts of shape errors with bidirectional RNN. Although, If I use two dynamic rnn with reverse op in the second layer, it appears to work fine:
Here is bidirectional RNN code that...

1

votes

0

answer

83

Views

### Tensorflow error with input to dynamic_rnn ValueError: Cannot feed value of shape (2,) for Tensor 'Placeholder:0', which has shape '(1, ?)'

I'm a newer to tensorflow and python. I would like to use dynamic RNN to construct sentence embeddings. I want to use variable-length sequences. I would have for each sentence its embedding. I really need your help.
Here is a part of my code.
graph = tf.Graph()
with graph.as_default():
x_data = tf.p...

1

votes

0

answer

75

Views

### Neural Network error is constant

I am building a Recurrent Neural Network based on the one at this link: intro to RNN in Tensorflow
But when I try to run it, the accuracy is stuck at 19.1% for the test set and similar values for the train and validation set.
From my understanding, it looks like the network is not updating the weigh...

1

votes

2

answer

346

Views

### How to modify the initial value in BasicLSTMcell in tensorflow

I want to initial the value for weight and bias in BasicLSTMcell in Tensorflow with my pre-trained value (I get them by .npy). But as I use get_tensor_by_name to get the tensor, it seems that it just returns a copy for me and the raw value is never changed. I need your help!

1

votes

0

answer

68

Views

### Indices and Slicing of TensorFlow's global variables of kernels for hidden weights and recurrent states of LSTM

I have a question about detailed indices of global variables which are generated by LSTM cells.
placeholders = {'inputs':tf.placeholder(tf.float32, shape=[None, None, 1000])}
cell = tf.nn.rnn_cell.BasicLSTMCell(80)
outs, states = tf.nn.dynamic_rnn(cell=cell, inputs=placeholders['inputs'], dtype=tf.f...

1

votes

0

answer

273

Views

### Confusion about many-to-one, many-to-many LSTM architectures in Keras

My task is the following: I have a (black box) method which computes a sequence starting from an initial element. At each step, my method reads an input from an external source of memory and outputs an action which potentially changes this memory. You can think of this method as a function f: (exter...

1

votes

1

answer

477

Views

### Inputting both Word-level and Char-level embedding in LSTM for PoS tagging

I am referring to this research paper 'Learning Character-level Representations for Part-of-Speech Tagging', where the author says: 'The proposed neural network uses a convolutional
layer that allows effective feature extraction from
words of any size. At tagging time, the convolutional layer
gener...

1

votes

0

answer

51

Views

### How to fit cubic equation on pybrain

I am trying to fit a Neural Network on a cubic Equation, but after many tries changing the number of the neurons on the hidden layer and increasing the number of epochs I could only get this:
Could you guys help me on this?
from pybrain.datasets import SupervisedDataSet
from pybrain.tools.shortcuts...

1

votes

0

answer

249

Views

### Implementing Hierarchical Attention for Classification

I am trying to implement the Hierarchical Attention paper for text classification. One of the challenges that I am finding is how to manage batching and updates to the weights of the network by the optimizer. The architecture of the network is made of two encoders stacked one after the other: a sen...

1

votes

0

answer

70

Views

### How to get results from all Tensorflow's while_loop() iterations as a list of not specified length?

I want to get results from all while_loop() iterations, so that tf.shape(result) == [batch_size, input_length].
batch_size and input_length may differ from mini-batch to mini-batch, so I evaluate their shapes dynamically.
I created this code:
batch_size = tf.shape(x)[0]
input_length = tf.shape(x)[1...

1

votes

1

answer

48

Views

### Artificial life simulator not producing any results

I have been experimenting with evolving artificial creatures, but so far all creatures just die. To initialize the creatures that do not result from asexual reproduction; I create around 8 random neurons which both have a connection in and a connection out. I'm using mutation to get a set of weights...

1

votes

1

answer

132

Views

### Time series prediction with LSTM

I'm currently learning on LSTMs and time series prediction with LSTMs. And I try to predict speeds for a road segment.
from sklearn.preprocessing import MinMaxScaler
sc = MinMaxScaler()
training_set = sc.fit_transform(training_set)
X_train = training_set[0:1257] //speed at (t)
y_train = training_set...

1

votes

0

answer

42

Views

### Multiple Sequence to Sequence RNN with Various Time Scales

I'm trying to forecast a time series based on several influencing factors.
For example, forecasting how many shoppers per hour there will be in my shop tomorrow will depend on three things:
Yesterday's weather and shoppers/hour
Tomorrow's weather
How many tourists there are in town
The first two are...

1

votes

1

answer

34

Views

### What is the difference of using the same cell or two different cells for backward and forward cells in bidirectional RNN in tensorflow?

I have this question that I couldn't figured out.
What is the difference between a bidirectional RNN which have two different cells (one for forward and one for backward) and a bidirectional RNN that share the same cell (same cell for backward and forward) ?
here the MWE of the two codes :
1) Using...

1

votes

1

answer

251

Views

### Tensorflow: gradients are zero for LSTM and GradientDescentOptimizer

Gradients which are computed by GradientDescentOptimizer for LSTM network are always zero. They are zero even on the first step, so, I think it is not vanishing gradient problem. The same issue happens for AdamOptimizer.
I have reduced input to one point of time series and label (expected output) to...

1

votes

0

answer

255

Views

### How do I combine features like word embeddings and sentiment polarity for text classification using LSTM neural networks?

Embeddings layer of LSTM is fed with the weights=embedding_matrix from the vocab, and model.fit has X_train which is the tokenized text data.
My X_train has shape (12,000 , 100) and embeddings_matrix has shape (34613, 300) where 34613 are the number of tokens(vocab from complete data ~15000 sentence...

1

votes

0

answer

53

Views

### Many to one recurrent network for episodic patterns in Keras

I'm trying to build a recurrent neural network using Keras. I'm using as base the discussion presented here.
However, in the solution proposed on the original discussion, as far as I understand, there is no concept of an 'episode'. Let me explain what I mean by that.
Imagine you have 6 instances x1...

1

votes

0

answer

212

Views

### Using pytorch cuda for RNNs on google colaboratory

I have a code (a code we saw in a class) of a recurrent neural network that reads a given text and tries to produce its own text similar to the example. The code is written in python and uses the pytorch library. I wanted to modify to see whether I could increase its speed by using GPU instead of CP...

1

votes

0

answer

39

Views

### Understanding distributing deep learning with tensorflow

I have written an RNN with tensorflow that trains on some time series data. I am training the model on moving window taking in 2 time slices and predicting for the third.
I batch my inputs and labels and run the entire set of batches over 30 epochs. The model runs fine as a single TF application....

1

votes

1

answer

358

Views

### Feed stacked RNN output into fully connected layer

I'm trying to solve a regression problem with a stacked RNN in tensorflow. The RNN output should be fed into a fully connected layer for the final prediction. Currently I'm struggeling on how to feed the RNN output into the final fully_connected layer.
My input is of shape [batch_size, max_sequence...

1

votes

0

answer

426

Views

### combining DropoutWrapper and ResidualWrapper with variational_recurrent=True

I'm trying to create a MultiRNNCell of LSTM cells wrapped with both DropoutWrapper and ResidualWrapper. For using variational_recurrent=True, we must provide input_size parameter to DropoutWrapper. I'm not able figure out what input_size should be passed to each LSTM layer, since ResidualWrapper als...

1

votes

1

answer

502

Views

### confusion about sequence length in dynamic rnn version of tensor flow

i am training a RNN to classify a sequence of input (13 values at each time step) as a particular class.
Here is relevant part of my code (line numbers prefixed)
50 data = tf.placeholder(tf.float32, [None, 40,13]) #Number of examples, number of input, dimension of each input
51 target = tf.placehold...

1

votes

1

answer

52

Views

### What is the error term in backpropagation through time if I only have one output?

In this question, RNN: Back-propagation through time when output is taken only at final timestep I've seen that if I only have one output at final time step T, which is y(T), then the error at earlier time step is unneeded.
Then, is the loss function term E = sum(E(t)) instead the value of E = E(T)...

1

votes

1

answer

437

Views

### How to interpret Tensorflow Histograms and Distribution plots

I am implementing a two-layer RNN. When I run the code, I have the following plots for weights and biases for the two layers of RNN.
(Sorry not enough reputation to include images on SO, please click on links.)
Plot for RNN cell 0
Plot for RNN cell 1
By looking at these plots, I don't understand if...

1

votes

0

answer

108

Views

### validation set get higher accuracy than training set for LSTM RNN as learning rate gets smaller

I'm currently trying to test the differences of behavior between a LSTM RNN and a GRU RNN on the prediction of time series ( classification 1/0 if time series goes up or down). I use the fit_generator method (as in Keras François Chollet book's)
I feed 30 point to the network and the next point ha...

1

votes

0

answer

35

Views

### How to get the input word as output word in neural network

I am trying to get the input word as output word with LSTM(copy a word). For that, I have seen some snippets. In those examples, I observed that we need to convert characters of a word into integers. But Those examples were written to predict the next characters.
an example of what I am trying
input...

1

votes

0

answer

220

Views

### How to implement a Many-to-Many RNN in tensorflow?

The below code gives me all the hidden state values of the unrolled RNN.
hidden_states,final_hidden_state = tf.nn.dyanamic_rnn(...)
How do I multiply each of the hidden_states with the weight 'Why' shown in the figure.
I find this confusing because the shape of hidden_states is [mini_batch_size,max_...

1

votes

1

answer

67

Views

### Running RNN in Tensorflow

If I have an array of 20 elements of type float.
Based on the values of the first ten elements I want a RNN to predict what the value of the last ten elements are. Using various online resources and books I have gotten a RNN built that reads the first 10 elements and processes them. However I don't...

1

votes

0

answer

272

Views

### Concatenate encoder hidden states/cells/outputs from different sources for attention calculation - issues?

I am using Pytorch for an LSTM encoder-decoder sequence-to-sequence prediction problem. As a first step, I would like to forecast 2D trajectories (trajectory x, trajectory y) from multivariate input - 2-D or more (trajectory x, trajectory y, speed, rotation, etc.)
I am following the below notebook (...

1

votes

0

answer

156

Views

### Keras embedding layer with LSTM

My original model is:
input = Input(shape=(1, 6)) # 1 time step, 6 features
LSTM_layer = LSTM(self.lstm_units, return_sequences=False, return_state=True)
lstm_output, out_h, out_c = LSTM_layer(embedded, initial_state=[h, c])
The input is a one hot vector, eg [0, 1, 0, 0, 0, 0]
Now I've read that it'...

1

votes

0

answer

44

Views

### Using incremental integer in Neural Network

I am using Brain.js to develop a neural network for a project of mine. The idea is to have something like this:
{ input: [day, month, hour, minute], output: some categoryID }
Basically, I will train the network to learn user's habits over time, so that later the network can predict a user's favourit...

1

votes

0

answer

127

Views

### Prediction of Words from the corpus

I am trying to build a model where, as the user types, my model should predict the word (Name of a medicine in this case). I have a corpus of all the words that are going to be typed. When I tried to use rnn with lstm to predict these words, I'm getting words that are similar to the words present in...

1

votes

0

answer

47

Views

### How to print (or see) i, j, f and o gates values and dimensions while using BasicLSTMCell tensorflow for sequence classification?

I am using this tensorflow example written by Aymeric Damien to classify some sequences using BasicLSTMCell. How can I change this example to print i, j, f and o (gates) values and shapes at each time step during the training?
Thanks!

1

votes

0

answer

50

Views

### Learning a non-linear mapping using LSTM units: encountering overfitting

I am trying to use a recurrent neural network to perform sensor fusion for an inertial measurement unit. IMUs are commonly used in conjunction with a Kalman filter (KF), which performs both fusion of accelerometer and gyroscope data along with 'smoothing', which results in a filtered, final output o...

1

votes

0

answer

115

Views

### Linear Regression using RNN tensorflow python

Can anyone please site an example on how linear regression can be implemented using RNN in pure tensorflow other than using keras API..
Eg :
x_train = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
I ll divide them into 5 batches of 2 element each so x_train will be of shape [5, 2]
[1, 2]
[3, 4]
[5, 6]..
y_train...

1

votes

0

answer

26

Views

### Passing the context of one RNN to another

I have 'n' number of RNNs and I wish to pass the context of first RNN along with second input to second RNN, then the context of this(second)RNN and third input to third RNN and so on. How can I achieve this in Keras. I wish to use keras.layers.SimpleRNN. Is there a way to fetch the context of previ...