Questions tagged [pickle]

1

votes
0

answer
249

Views

Big CSV to pickled dict

I have a 1 GB CSV file like this (16-char-max-string;int) hello;1798732 you;1762 ... that is slow to be parsed into a dict, that's why I would like to do it once, save it as pickled file (or any other serialization method), and then reopen this already-processed file for further use. I tried:...
Basj
1

votes
2

answer
282

Views

Loading pickled (dill) file containing dictionary of functions

This is a follow-up to a question I previously asked: Loading functions in pickle file that uses class wrapper, where I wanted to pickle a dictionary of function under class wrappers, but the class wrapper module couldn't load. I have since solved this using dill instead of pickle: func_dict = lowes...
MoMiJi
1

votes
1

answer
70

Views

Scipy: pickle an ode solver instance

I am running long simulations which involve integration of an ode. Due to the long running times, I decided to generate fallbacks which allow to continue the simulations from a checkpoint. For the ode integration, I decided to pickle the scipy ode solver instance. This approach works for the reentra...
p.py
1

votes
0

answer
130

Views

Is there a way to query a pickle dataset using something like where

Consider I have an object with 3 properties (viz. Name, Size, Type). I have 1000 such objects pickled and store in a file. Is it possible to load the dataset and then query it for the particular object with name 'Main'. Currently, I am doing the following, but I hope so there is much effective way:...
Rishik Mani
1

votes
0

answer
58

Views

Limit Python pickle to simple types

I need to serialize and transfer Python objects between interpreters, and pickle is an ideal choice. However, pickle allows the user of mu lib to serialize globa functions, classes (and their instances) and modules. However, these may not be present on the receiving end, e.g. this runs without any e...
Tomáš Gavenčiak
1

votes
0

answer
24

Views

Truncated bytearray using pyodbc in Linux

I'm using a kalman filter in pykalman, then I pickle the filter and save it into Sybase as a binary array that is sufficiently long. I'm using pyodbc as the connection. I run the script on a Linux server, then get the same filter from my Windows desktop and unpickle it, it works fine. However, if I...
swmfg
1

votes
0

answer
638

Views

How to load a .pickle file on python from google cloud storage

My problem is the following: I have two .p (pickle files) in my bucket in the google cloud storage and I would like to load them on my jupyter notebook (where I run my code on the VM instance). I tried the following url = 'http:\\\tree\storage.googleapis.com\' data_x = pickle.load(open(url + '\.p'...
Tom
1

votes
1

answer
198

Views

EOFError using numpy.load()

I have a command to load data from a npy file: utable = numpy.load('utable.npy') But this results in EOFError: File '/home/divyat/anaconda2/lib/python2.7/site-packages/numpy/lib/npyio.py', line 419, in load pickle_kwargs=pickle_kwargs) File '/home/divyat/anaconda2/lib/python2.7/site-packages/numpy/l...
Divyat
1

votes
0

answer
682

Views

Python pickle custom class object leads to Segmentation Fault

I am using dlib's HOG face detector. Trying to set and load the model to/from redis (localhost) by pickling/unpickling leads to a segmentation fault. Below is the code for the same. from redis import Redis from pickle import dumps, loads redis = Redis(host='localhost', port=6379, db=0, password='')...
Vibhor Kalra
1

votes
0

answer
841

Views

Can't Pickle Thread Lock Object

I am trying to run the following code in python: from pyspark.sql.types import StringType from pyspark.sql.functions import udf from pyspark import SparkContext from pyspark import SparkConf import pyspark.sql.functions as pf import logging import sys from pyspark.sql import SQLContext log = loggi...
Gring
1

votes
0

answer
72

Views

Pickling and Unpickling a dynamically generated class

I have a function that returns a customized class type which I then want to pickle and unpickle. I am trying to follow this answer and the __reduce__ documentation. I am using Python 2.7. The difference from that answer is that my function accepts an arbitrary class and returns a class derived from...
buzjwa
1

votes
0

answer
300

Views

Getting a pandas DataFrame from a subprocess

I have a python 3 script that calls a sub process. Below is line of code I use in my script: Note: Path is my python exe file (the sub process is ran in 2.7 where as the main script is in python 3. The rest are some parameters passed to the sub process. Proc = subprocess.check_output([path,'ListProc...
Campo21
1

votes
0

answer
35

Views

Loading from pickle slow/fails

I'm using Gauss-Legendre integration to calculate an integral. To get the necessary roots of the Legendre polynomials, I have followed steps outlined here and it works. To save me the time of generating the roots for high order polynomials, I have added a try:, except: routine to store and load root...
Huang_d
1

votes
0

answer
111

Views

self-store an object in python

I have a class in Python. class myclass: def bar(): # do stuff I need to implement some mechanism to store/cache part of the results of a function using some store()/load() functions. I am constrained to use an external object, somewriter which offers an API for making objects persistant using a rea...
00__00__00
1

votes
0

answer
276

Views

Can't pickle <function <lambda> when using multiprocessing Pool.map()

I'm trying to parallelize my python script with the multiprocessing library. My function is part of a class and I used Pool.map. import numpy as np import pandas as pd import netCDF4 import itertools import multiprocessing as mpp from tqdm import tqdm Class catch2grid(object): def __init__(self): ''...
Robo
1

votes
3

answer
113

Views

Load all pickled objects [duplicate]

This question already has an answer here: Saving and loading multiple objects in pickle file? 5 answers import pickle ListNames = [['Name1','City1','Email1'],['Name2','City2','Number2']] ListNumbers = [1,2,3,4,5,6,7,8] with open ('TestPickle.pickle','wb') as fileSaver: pickle.dump(ListNames,fileSav...
jsmith613
1

votes
1

answer
36

Views

Issues with pickling in python - items not saving to file once program exits

I am trying to create a python program to save my friends' birthdays and access them easily and check for birthdays each day(I am not great at remembering dates and I never use facebook), but when I add a new birthday it is only accessible until I end the program - it then disappears again. I have b...
Hannah Mishow
1

votes
0

answer
176

Views

cannot pickle pyspark dataframe

I want to create a decision tree model using spark submit. from pyspark.mllib.regression import LabeledPoint from pyspark.mllib.tree import DecisionTree from pyspark import SparkConf, SparkContext from numpy import array from pyspark.sql import SparkSession spark = SparkSession \ .builder \ .appName...
betty bth
1

votes
1

answer
316

Views

Loading pickled model with Pyspark

I'm trying to load a pickled model from S3 using PySpark, then make a prediction using the model. I can load the model all right, but when I try to feed the model to the method that makes the prediction, I am running into PicklingError: Cannot pickle files that are not opened for reading I've read t...
Meghan
1

votes
0

answer
25

Views

C implementation _pickle doesn't respect dispatch_table for native types in Python 3.6

I am trying to pickle lambdas with the native pickle module by providing custom serializing function as: from six.moves import copyreg import pickle, types copyreg.pickle(types.FunctionType, _custom_pickle_func) However, this doesn't work for following types which pickle supports natively: import pi...
Saim Raza
1

votes
0

answer
178

Views

What is member_descriptor and how to access/modify its contents in Python?

Can somebody please explain what member_descriptor is and how to access/modify its content from Python interpretor? Python version = 3.6 In [1]: import _pickle In [2]: _pickle.Pickler.dispatch_table Out[2]: In [3]: type(_pickle.Pickler.dispatch_table) Out[3]: member_descriptor I could find only thi...
Saim Raza
1

votes
0

answer
151

Views

Pickle error multiprocessing between Python 3.6.3 and 3.5.2

My code is running completely fine on Python 3.5.2, but when I use Python 3.6.3 (without changing a single line of code) I get a pickle error from multiprocessing and I am not sure why. TypeError: can't pickle 'MyClass instance' objects Any idea why ? Thanks !
sulo
1

votes
0

answer
343

Views

Pickling a ctypes.Structure with ct.Pointer

I have the following classes of a ctypes Structure. import ctypes as ct class my_array(ct.Structure): _fields_ = [('_data', ct.POINTER(ct.c_uint32)), ('_size', ct.c_size_t)] def __init__(self, data): self._data = (ct.c_uint32 * len(data))() for i in range(0, len(data)): self._data[i] = data[i] self....
Tohiko
1

votes
0

answer
51

Views

Python3: How can I inspect lambda source from pickle load?

Is there a way I can see the python source of loaded cloudpickle object ? For example, below I'm loading object In [17]: new_squared = pickle.loads (b'\x80\x02ccloudpickle.cloudpickle\n_fill_function\nq\x00(ccloudpickle.cloudpickle\n_make_skel_func\nq\x01ccloudpickle.cloudpickle\n_builtin_type\nq\x0...
user3458988
1

votes
0

answer
379

Views

How do we import a .pkl file in Matlab?

I trained a neural model in python and saved it in my directory in pkl format. I want to import this pkl file in Matlab. Thanks in advance.
Raj Shrivastava
1

votes
1

answer
49

Views

Unifying pickled numpy arrays into a single file efficiently

I have got a folder of thousands of pickled one-dimensional numpy arrays, of which each array has a length of 921603 integer values (up to 3 digits each). Like So: folder/ |0.pkl |1.pkl |2.pkl ... |5000.pkl The goal is to convert them into a final merged.csv file, so that each datapoint in form of...
Cryptic Pug
1

votes
1

answer
343

Views

Pass QuerySet to Celery Task in Django 1.11

I want to move some long query to the async task in Celery and next retrieve that extra-info with AJAX. Now I get QuerySet like this: brands = Brand.objects.filter(shops__shop_name__in=[shop]) Task: @task() def brand_count(querystr): querystr.annotate(amount_of_products=Count('products')) I want to...
Chiefir
1

votes
0

answer
111

Views

dill.load unicodeDecodeError

I try to dill.load() some large pickle file that will cause momery error if I use _pickle.load(). However, I got this error UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 1 What should I do?
Philip Fu
1

votes
0

answer
144

Views

ImportError: No module named multiarray

When I run this code: from numpy.core import multiarray import cPickle as pickle exp_number = 13 model.summary() losses = pickle.load(open('/home/ubuntu/nilmtk/nilmtk/neuralnilm/tmpdata/history_exp' + str(exp_number) + '.pickle' , 'rb')) test_set = pickle.load(open('/home/ubuntu/nilmtk/nilmtk/neural...
ikram ikram
1

votes
0

answer
110

Views

KeyError: 1744843431 when loading a pickle file

so I was trying to load the pickled file like I used to do using set_x = pickle.load(open('retaindata/h50_cl2_all.visits.train', 'rb'), encoding='bytes') But somehow it gives me an error that I couldn't find a solution to on google: error msg Can someone help and let me know what went wrong?
Ginny Zhu
1

votes
0

answer
55

Views

What causes the difference in execution time between pickled transfomer and normal one?

I have trained a dimensionality reduction model in scikit-learn. It applies PCA on term frequencies from a text. After it has been trained, running the model takes around 1.7s. When I now pickle, using joblib or dill, and then unpickle the model in the same python shell, the execution time goes up...
1

votes
1

answer
104

Views

Python typing, pickle and serialisation

I've started learning the typing system in python and came across an issue in defining function arguments that are picklable. Not everything in python can be pickled, can I define a type annotation that says 'only accept objects that can are picklable'? At first it sounds like something that should...
Matti Lyra
1

votes
0

answer
1.5k

Views

Pickle AttributeError: Can't get attribute 'Wishart' on <module '__main__' from 'app.py'>

I already run my code to load my variable saved by pickle. This my code import pickle last_priors_file = open('simpanan/priors', 'rb') priors = pickle.load(last_priors_file) and i get error like this : AttributeError: Can't get attribute 'Wishart' on
1

votes
0

answer
100

Views

Inserting binary data with sqlite3

I would like to insert binary data in a database with sqlite3. compressed = zlib.compress(pickle.dumps(foo)) database.insert(cursor, 'bar', config=compressed) Here, database is from another module from another class, that performs execute. In that class, the command looks like this: cursor.execute('...
Brynhildr Xie
1

votes
2

answer
61

Views

Class to textfile to list

I am currently working on a project trying to make a simple bank system. The basics of the system is done as far as im concerned but there is one problem. I am trying to make a file saving system for my 'bank accounts'. When a new account is created (aka, a new class is made) it's supposed to be sav...
Filthy student
1

votes
0

answer
36

Views

Can I build a file by mmap incrementally?

I'm new to python. I'm trying to build a local DB for a huge data by mmap. Data are written as: import os import mmap import cPickle as pickle import operator from functools32 import lru_cache with open(path + '.data', 'wb') as f: index = [] for key, value in iter_kvs: index.append((key, f.tell()))...
1

votes
0

answer
127

Views

Save on disk h264 packages received using PyAV

I'm using PyAv to read packets from H.264 stream during 5 minutes. I need to save the packets encoded on disk and load and decode them later. I tried to use pickle to save the packets, but it cannot serialize them. What can I do? Here is part of my code (an exception is thrown when I try to save the...
Vítor Cézar
1

votes
0

answer
168

Views

How to use Klepto package Python

Hellow World! I have a pickle file (containing images in byte form) to open but when I open it with pickle I get this error: Traceback (most recent call last): File 'test.py', line 20, in if __name__ == '__main__': main() File 'test.py', line 10, in main dump = pickle.load(file) UnicodeDecodeError:...
Stefano Pozzi
1

votes
0

answer
307

Views

Pickle suddenly won't pickle cv2.VideoCapture object

as part of my work I need to pickle a VideoCapture object (from opencv), and it was no problem until suddenly it started giving me this error: TypeError: can't pickle cv2.VideoCapture objects That is with the following code: video = cv2.VideoCapture('some_video.mp4') pickle_out = gzip.open('file_nam...
Nadav Schweiger
1

votes
0

answer
266

Views

can't pickle weakref objects

I'm trying to use spark to extract spatial data from a csv and represent it in a folium map. While using the foreach() function i get a PicklingError: Could not serialize object: TypeError: can't pickle weakref objects i understand as described here that not all types can be pickled, however i need...
A.HADDAD

View additional questions