Questions tagged [pytorch]

1

votes
2

answer
1.9k

Views

Create a nnModule that's just the identity

I'm trying to debug a pretty complex interaction between different nnModules. It would be very helpful for me to be able to replace one of them with just an identity network for debugging purposes. For example: net_a = NetworkA() net_b = NetworkB() net_c = NetworkC() input = Autograd.Variable(torch....
Sam Bobel
3

votes
0

answer
20

Views

Computational graph vs (computer algebra) symbolic expression

I was reading Baydin et al, Automatic Differentiation in Machine Learning: a Survey, 2018 (Arxiv), which differentiates between symbolic differentiation and automatic differentiation (AD). It then says: AD Is Not Symbolic Differentiation. Symbolic differentiation is the automatic manipulation of [sy...
Albert
1

votes
1

answer
47

Views

not able to predict using pytorch [MNIST]

pytorch noob here,trying to learn. link to my notebook: https://gist.github.com/jagadeesh-kotra/412f371632278a4d9f6cb31a33dfcfeb I get validation accuracy of 95%. i use the following to predict: m.eval() testset_predictions = [] for batch_id,image in enumerate(test_dataloader): image = torch.autogra...
Jagadeesh Kotra
1

votes
1

answer
65

Views

PyTorch - GPU is not used by tensors despite CUDA support is detected

As the title of the question clearly describes, even though torch.cuda.is_available() returns True, CPU is used instead of GPU by tensors. I have set the device of the tensor to GPU through the images.to(device) function call after defining the device. When I debug my code, I am able to see that the...
talha06
1

votes
2

answer
40

Views

PyTorch doesn't seem to be optimizing correctly

I have posted this question on Data Science StackExchange site since StackOverflow does not support LaTeX. Linking it here because this site is probably more appropriate. The question with correctly rendered LaTeX is here: https://datascience.stackexchange.com/questions/48062/pytorch-does-not-seem-t...
wny
1

votes
2

answer
54

Views

Why is torch.nn.Sigmoid a class instead of a method?

I'm trying to understand how pytorch works a little bit better. Usually, when defining a neural network class, in the init() constructor, people write self.sigmoid = nn.Sigmoid(), so that in the forward() method they can call the sigmoid function multiple times with having to reinstantiate nn.Sigmoi...
user49404
1

votes
1

answer
41

Views

Model learns with SGD but not Adam

I was going through a basic PyTorch MNIST example here and noticed that when I changed the optimizer from SGD to Adam the model did not converge. Specifically, I changed line 106 from optimizer = optim.SGD(model.parameters(), lr=args.lr, momentum=args.momentum) to optimizer = optim.Adam(model.parame...
thefxperson
1

votes
0

answer
7

Views

Detailed examples of what Keras cannot do but Pytorch can?

I read many articles saying Keras is too high level and hard to be used for research. I found Keras has Lambda layer, and custom layer, so what are some detailed examples of what Pytorch can achieve while Keras cannot or is very tricky to? Thanks.
Herbert
1

votes
0

answer
8

Views

How can I run pytorch with multiple graphic cards?

I have 4 graphic cards which I want to utilize to pytorch. I have this net: class Net(nn.Module): def __init__(self): super(Net, self).__init__() self.conv1 = nn.Conv2d(1, 20, 5, 1) self.conv2 = nn.Conv2d(20, 50, 5, 1) self.fc1 = nn.Linear(4*4*50, 500) self.fc2 = nn.Linear(500, 10) def forward(self,...
RNN
0

votes
0

answer
5

Views

Check if the expected object is of backend CUDA or CPU?

I am trying to run a code on both CPU and CUDA. The problem arise when I create objects, as I need to know what's expected. I need to determine if the computer is expecting a CUDA or CPU tensor, before it is created. Code: def initilize(self, input): self.x = torch.nn.Parameter(torch.zeros((1,M)) de...
brolija
1

votes
2

answer
1.8k

Views

save predictions from pytorch model

Im following the pytorch transfer learning tutorial and applying it to the kaggle seed classification task,Im just not sure how to save the predictions in a csv file so that i can make the submission, Any suggestion would be helpful,This is what i have , use_gpu = torch.cuda.is_available() model = m...
Ryan
1

votes
2

answer
725

Views

How to iterate over layers in Pytorch

Let's say I have a network model object called m. Now I have no prior information about the number of layers this network has. How can create a for loop to iterate over its layer? I am looking for something like: Weight=[] for layer in m._modules: Weight.append(layer.weight)
Infintyyy
0

votes
0

answer
4

Views

Plotting torch.tensor in Pytorch: Invalid dimension error

I have a torch.Tensor object and this has the shape of torch.Size([9, 1, 28, 28])). I tried something like for digit in range(10): similar_img = create_interpolates(/*something...*/) plt.figure(figsize=(10,10)) plt.imshow(similar_img.detach().numpy()) But this gives me: TypeError: Invalid dimensions...
Dawn17
1

votes
0

answer
16

Views

How to free gpu memory by deleting tensors?

Suppose that I create a tensor and put it on gpu, then I don't need it and want to free gpu memory allocated by it. How to do that? import torch a=torch.randn(3,4).cuda() # nvidia-smi shows that some mem has been allocated. # do something # a does not exist and nvidia-smi shows that mem has been fre...
Linghao.Chen
1

votes
0

answer
506

Views

Pytorch: Randomly subsample loss tensors using `torch.randperm`

I'm trying to randomly subsample the prediction and target array for my loss calculation. idx = torch.randperm(target.shape[0]) target = target.index_select(0, idx[0, sample_size] However I'm getting this error message. index_select(): argument 'index' (position 2) must be Variable, not torch.Long...
mcExchange
1

votes
0

answer
296

Views

Input size mismatch error when using pre-trained inceptionV3 model for image classification

I'm facing trouble when training a model using pre-trained inceptionV3 for my own image data set. I'm loading images using data.Dataset loader and 'transforms' for image transformation. Here's my inceptionV3 model inceptionV3 = torchvision.models.inception_v3(pretrained=True) pretrained_model = nn.S...
Sam
1

votes
1

answer
187

Views

How to ignore specific columns for calculating softmax attention?

I am implementing a model which is based on MemoryNetworks. I have triplets data of (context, query, answer). And I want to calculate attention. The attention indicates which sentences in a context should be focused. To formulate mini-batch, I use zero-paddings to create context data. So the followi...
jef
1

votes
1

answer
1.7k

Views

Getting CUDA out of memory

Im trying to train a network but i get, I set my batch-size as 300 and i get this error,but even if i reduce this to 100 i still get this error,and more frustratingly for running 10 epoch on ~1200 images it takes about 40 minutes,any suggestions what is going wrong and how may i speed the process! A...
Ryan
1

votes
2

answer
1.6k

Views

in pytorch, cuda.is_availbale(), but every operation fails with out of memory

I've set up a new virtual machine (on GCP) with a K80 GPU on Ubuntu 16. Followed installation instructions for the CUDA toolkit 9.1 nvidia-smi returns correctly: NVIDIA-SMI 390.12 Driver Version: 390.12 ... in pytorch, cuda.is_available() returns True, but any operation fails: torch....
Paul
1

votes
0

answer
210

Views

Is this a correct reimplementation of Pytorch Seq2Seq model?

I made a code that sort of change the tutorial script of seq2seq provided by Pytorch. Here’s the model: class Seq2Seq(nn.Module): def __init__(self, encoder, batch_size, vocab_size, input_size, output_size, hidden_dim, embedding_dim, n_layers=2, dropout_p=0.5): super(Seq2Seq, self).__init__() self...
Aryo Pradipta Gema
1

votes
1

answer
2.1k

Views

Pytorch tutorial LSTM

I was trying to implement the exercise about Sequence Models and Long-Short Term Memory Networks with Pytorch. The idea is to add an LSTM part-of-speech tagger character-level features but I can't seem to work it out. They gave as a hint that there should be two LSTMs involved, one that will output...
David
1

votes
1

answer
553

Views

setuptools: installing pytorch from download link: 403 Forbidden

I am trying to include pytorch in the requirements list for setuptools: install_requires=[ 'torch' ], dependency_links=[ 'http://download.pytorch.org/whl/cpu/torch-0.3.0.post4-cp27-cp27mu-linux_x86_64.whl' '@develop#egg=torch' ], But after running python setup.py develop I receive: error: Can't down...
Gerry
1

votes
1

answer
231

Views

Reading multiple images as custom dataset for PyTorch?

I want to read in multiple images for the main_image set and blur_image set. For example, 5 main images and 5 blurred images. The goal is determine what values for the kernel in the convolutional layer convert the main images to the blurred images. The assumption is that the same kernel is used to b...
user5739619
1

votes
1

answer
224

Views

Adding matrix rows to columns in numpy

Say I have two 3D matrices/tensors with dimensions: [10, 3, 1000] [10, 4, 1000] How do I add each combination of the third dimensions of each vector together such that to get a dimension of: [10, 3, 4, 1000] So each row if you will, in the second x third dimension for each of the vectors adds to the...
Matt
1

votes
1

answer
453

Views

Pytorch: TypeError 'torch.LongTensor' object is not reversible

I'm trying to do a NLP task by pytorch and I used following code to pack my batch of sentences. for iter in range(0, n_iters, batch_size): # batch size * max length Variable input_batch = input_data[iter:iter + batch_size] target_batch = target_data[iter:iter + batch_size] # batch size * 1 LongTenso...
Shiloh_C
1

votes
1

answer
255

Views

Loss is not decreasing at all for RNN

I have already tried to change the weights initialization parameters, learning rate and the batch size and the activation functions to ReLu Still no decrease in the loss This is the code: import torch import torchvision.datasets as dsets import torchvision.transforms as transforms from torch.autogr...
sai venkatesh
1

votes
1

answer
880

Views

PyTorch: RuntimeError: element 0 of variables tuple is volatile

I'm training a LSTM based model in PyTorch 0.3.1. My problem is that after increasing the learning rate I always get a RuntimeError saying: element 0 of variables tuple is volatile. This does not happen at the beginning, but after some training, like in epoch 3, 4, 5 .. etc. When looking after this...
blue-phoenox
1

votes
0

answer
249

Views

Implementing Hierarchical Attention for Classification

I am trying to implement the Hierarchical Attention paper for text classification. One of the challenges that I am finding is how to manage batching and updates to the weights of the network by the optimizer. The architecture of the network is made of two encoders stacked one after the other: a sen...
Jadiel de Armas
1

votes
0

answer
45

Views

Only backpropagate up to a given variable

I’m working on a GAN where the discriminator operates on the latent space vector produced by the encoder. The details aren’t important, but the paper describing the model is https://arxiv.org/abs/1706.00409 if you want to take a look. Essentially, my problem is that my training code requires an...
Eric Elmoznino
1

votes
0

answer
386

Views

Pytorch no gradient calculated one for variable

I defined 2 variables (D, ht) to received gradients. However after the loss function and backward() calculation, only D has gradients, ht does not have any gradients calculated. Can you please help me to understand why it is the case? thank you. import torch.nn as nn import torch from torch.autograd...
W.S.
1

votes
1

answer
767

Views

Pytorch: Error in DataParallel for RNN model

I'm trying to use torch.nn.DataParallel for a RNN model. My model looks like this: class EncoderRNN(nn.Module): def __init__(self, vocal_size, hidden_size): super(EncoderRNN, self).__init__() self.hidden_size = hidden_size self.embedding = nn.Embedding(vocal_size, hidden_size) self.gru = nn.GRU(hidd...
Shiloh_C
1

votes
0

answer
149

Views

Assertion error in DataParallel when applying to RNN

I'm trying to apply DataParallel to a RNN model. this is part of my code: if use_cuda: encoder = encoder.cuda() decoder = decoder.cuda() encoder = nn.DataParallel(encoder, dim=0) decoder = nn.DataParallel(decoder, dim=0) class EncoderRNN(nn.Module): def __init__(self, vocal_size, hidden_size): supe...
Shiloh_C
1

votes
0

answer
389

Views

PyTorch RNN gradients for variable length input sequences are very small

I'm aiming to do multiclass classification on sentences. The input to the my RNN(LSTM or GRU) is a batched input of variable length sequences(which are indexed using Glove embeddings). This input is right padded with zeros. The redefined forward for my GRU RNN is: def last_timestep(self, unpacked, l...
Venkat
1

votes
0

answer
133

Views

Understanding Actor-Advantage Critic

I am attempting to implement an A2C neural network in C++ from this tutorial. I could only find Python tutorials and this Python tutorial made the most sense to me. I have already implemented a regular back propagation neural network, however, I want to upgrade this to use reinforcement learning. I...
anon
1

votes
0

answer
346

Views

Uneven usage of GPU memory by DataParallel causes out-of-memory error

I am running my NN model using DataParallel on 3 GPUs. In one GPU, the memory usage rises to 12gb and as a result, the program stops after giving an out-of-memory error. While one GPU uses a large amount of memory (~12gb), the other two GPU memory usage is quite low (~2-3gb). Is there any way I can...
Wasi Ahmad
1

votes
0

answer
212

Views

Using pytorch cuda for RNNs on google colaboratory

I have a code (a code we saw in a class) of a recurrent neural network that reads a given text and tries to produce its own text similar to the example. The code is written in python and uses the pytorch library. I wanted to modify to see whether I could increase its speed by using GPU instead of CP...
Sina
1

votes
0

answer
507

Views

pytorch AttributeError: module 'torch.optim.lr_scheduler' has no attribute 'CosineAnnealingLR'

When trying to run a python notebook on google colab, mentioned in https://medium.com/@markn_67491/run-allennlp-models-on-free-gpus-using-googles-colab-notebooks-4db9359970c1 available from: https://drive.google.com/file/d/1JH6dz8GJbwh9GhPoZQKwR-EipeR5JBrV/view after installing pytorch and AllenNlp...
Oren Bochman
1

votes
2

answer
597

Views

Loss doesn't decrease in training the pytorch RNN

Here is the RNN network I designed for a sentiment. class rnn(nn.Module): def __init__(self, input_size, hidden_size, output_size): super().__init__() self.hidden_size = hidden_size self.i2h = nn.Linear(input_size, hidden_size) self.h2o = nn.Linear(hidden_size, output_size) self.h2h = nn.Linear(hidd...
Ayush
1

votes
0

answer
653

Views

Dropout Decreases Test and Train Accuracy in one layer LSTM in Pytorch

I have a one layer lstm with pytorch on Mnist data. I know that for one layer lstm dropout option for lstm in pytorch does not operate. So, I have added a drop out at the beginning of second layer which is a fully connected layer. However, I observed that without dropout I get 97.75% accuracy on the...
Hadi Gharibi
1

votes
1

answer
142

Views

PyTorch - WSD using LSTM

I'm trying to replicate Google's research paper on WSD with neural models using PyTorch. I'm having some issues traying to overfit the model before training on large datasets. Using this training set: The film was also intended to be the first in a trilogy. this model definition: class WordGuesser(n...
Emanuele Giona

View additional questions