Questions tagged [pickle]

1

votes
2

answer
742

Views

writing python objects to disk without loading into memory?

I'm running a large number of computations whose results I want to save to disk one item at a time, since whole data is too big to hold in memory. I tried using shelve to save it but I get the error: HASH: Out of overflow pages. Increase page size my code is below. What is the right way to do this...
mvd
1

votes
2

answer
28

Views

How to save and reload pickled data to/from a tkinter Entry box?

I am looking for a way to save the data that was typed into an Entry box on my Tkinter GUI and then immediately load that data back into the Entry box when the program is opened after it has been closed. I have done some research and found that the Pickle module is the best way to do this kind of st...
Mihkel
1

votes
1

answer
80

Views

my function save to pickle not working

I have a script that calls the function below to create a dict from pickle load as in: def load_obj(name ): try: with open('obj/' + name + '.pkl', 'rb') as f: return pickle.load(f) except: return dict(name) I update the pickle loaded dict to add a new key:value record then and save the pickle using...
1

votes
1

answer
18

Views

Create new object if object doesn't exist in pickle file and load it if it does (best way with multiple objects?)

I know how to do this with a single object in a pickle file or with a single object missing, but I don't know how to do something like this if more than 1 object is missing from the pickle file. What I got now: try: user1 = pickle.load(open("users.pkl", "rb")) except (OSError, IOError) as e: user1 =...
Mr Squidr
0

votes
0

answer
25

Views

How to dump objects of a running Python program?

I have a Python program which basically keeps list of Counter objects and then writes them to disk. After four days counting has finished but the system is almost out of 63GB memory and it has already swapped out 50GB and no progress. Here is a simplified version of my code. import os import time fr...
user1480788
1

votes
2

answer
2.4k

Views

TypeError: can't pickle function objects (can't pickle sklearn estimator)

It only happened when using jieba my code: from sklearn.feature_extraction.text import TfidfVectorizer import jieba data = ["十二届全国政协副秘书长黄小祥被免职撤委员资格-人事任免-时政频道-中工网", "银联持卡人境外可获紧急现金支援-财经网", "国...
Mithril
1

votes
2

answer
157

Views

How to serialize/store the ciphertext encrypted by hybrid CPabe_BSW07 in Charm

I want to store the ciphertext encrypted by hybrid cpabe_BSW07 in files, but I found errors when pickling the ciphertext: raise TypeError, "can't pickle %s objects" % base.__name__ TypeError: can't pickle Element objects from charm.toolbox.pairinggroup import PairingGroup from charm.schemes.abenc.ab...
Lex
1

votes
1

answer
1.4k

Views

module 'pickle' has no attribute 'dump'

import pickle imelda = ('More Mayhem', 'IMelda May', '2011', ((1, 'Pulling the Rug'), (2, 'Psycho'), (3, 'Mayhem'), (4, 'Kentish Town Waltz'))) with open("imelda.pickle", "wb") as pickle_file: pickle.dump(imelda, pickle_file) I am trying to execute this code, but the console keeps telling me: module...
Champa
1

votes
2

answer
77

Views

Unpickle binary file to text [duplicate]

This question already has an answer here: Is there a way to view cPickle or Pickle file contents without loading Python in Windows? 2 answers I need to do some maintenance on a system that basically looks like: (Complicated legacy Python program) -> binary pickle file -> (Another complicated legacy...
rwallace
1

votes
1

answer
887

Views

Can a pickled object be reliably encoded as Postgres string (to be sent with NOTIFY)?

Postgres NOTIFY supports only string literals (no binary data). I've managed to use it with a pickled object in a following way: message = cPickle.dumps(objectFoo) cursor = connection.cursor() # this is psycopg2 connection cursor.execute("NOTIFY channelFoo, %s", [message]) However, if I some tricky...
Jan Wrobel
1

votes
1

answer
1.5k

Views

dill dump_session with ipython

I'm trying to use the dill module to save my ipython session using dump_session() but I'm getting an error message. I'm using Ipython 1.0.0 and dill 0.2-a-dev 20120503. Does anyone out there have any insight? Thanks in advance. Niall Here's the enormous traceback: -----------------------------------...
nrob
1

votes
3

answer
472

Views

create working data structure in file

I am creating a very large array. Rather than having this array stored in memory, I want to be able to write it to a file. This needs to be in a format I can later import. I would use pickle but it appears pickle is used for completed file structures. In the following example, I need a way for the o...
Michael
1

votes
4

answer
4.3k

Views

Using Python's pickle to open and save a dictionary

In the final days of my intro comp sci class, we got to creating dictionaries. A homework program in our book asks us to create something that can look up, add, change, and delete a set of names and email addresses. It asks us to pickle the dictionary, but the kicker for me is that it stipulates tha...
Zachary Matthew Perry
1

votes
2

answer
1.4k

Views

Can I pickle a string without any dump?

Say I have a string "[A,B,C]" and I want to convert it into a list [A,B,C], I have googled around and I know I should use pickle, but it seems that before I pickle.loads I seems to have to pickle.dumps the object, is that true? If yes, how can I walk around this?
dorafmon
1

votes
1

answer
137

Views

How to avoid penalty of pickling a dict vs pickling a list

Introduction I have a dictionary with the following format: dict_list = {'S0':[[list of int],[list of int]], 'S1':[[list of int],[list of int]], ...} with S0's list of ints accessed via dict_list['S0'][0] and dict_list['S0'][1] To improve code readability, I changed the "list of list" to "dict of li...
1

votes
1

answer
139

Views

Problematic function pickling

In one module, I have a class which requires a function as one of its parameters: class Foo(): def __init__(self, fun): self.fun = fun def call_fun(self, arg): self.fun(arg) In another module, I have a couple of functions that I eventually pass as parameters when creating this class' objects. I'm bi...
boobookum
1

votes
2

answer
2.2k

Views

Python for the absolute beginner, Chapter 7 challenge 2

I've been going through the exercises in this book and I've hit a bit of a road block. The challenge is to: "Improve the Trivia Challenge game so that it maintains a high-scores list in a file. The program should record the player's name and score. Store the high scores using a pickled object." I've...
Toby
1

votes
2

answer
161

Views

Proper way to save python dictionaries and retrieve them at a later stage

following an earlier question I asked here (Most appropriate way to combine features of a class to another?) I got an answer that I finally grown to understand. In short what I intend to now is have a bunch of dictionaries, each dictionary will look somewhat like this: { "url": "http://....", "parse...
fred
1

votes
3

answer
440

Views

In a pickle with pickling in python

I have gone through this website and many others but no one seems to give me the simplest possible answer. In the scrip bellow there are 2 different variables that need to be placed into a single pickle (aka 'test1' and 'test2'); but I am wholly unable to get even the simpler one of the two to load....
Arad Mann
1

votes
1

answer
1.9k

Views

Python 3: Pickling and UnPickling class instances returning “no persistent load” error

I am trying to make a program that collects together lots of data about when certain Players in a band are available for busking this Christmas, and I'm struggling to get the pickle function to do what I want... The data is stored in class instances of the class below, Player: import pickle class Pl...
rjmcf
1

votes
3

answer
580

Views

How do I remove the memory limit on openmpi processes?

I'm running a process with mpirun and 2 cores and it gets killed at the point when I'm mixing values between the two processes. Both processes use about 15% of the machines memory and even though the memory will increase when mixing, there should still be plenty of memory left. So I'm assuming that...
user1364735
1

votes
3

answer
189

Views

I am trying to save high scores with pickle, but how do I add to an already pickled document and then get the max?

I am trying to save high scores in a game that I am creating, but each time I do a pickle.dump, it overwrites my previous data. Any help?
1

votes
1

answer
2.2k

Views

how to pickle customized vectorizer?

I'm having trouble pickling a vectorizer after I customize it. from sklearn.feature_extraction.text import TfidfVectorizer import pickle tfidf_vectorizer = TfidfVectorizer(analyzer=str.split) pickle.dump(tfidf_vectorizer, open('test.pkl', "wb")) this results in "TypeError: can't pickle method_des...
Ziggy Eunicien
1

votes
1

answer
152

Views

Python fast dictionary creation with Pickle?

I'm trying to write an anagram service. The first stage of the program is to go through a dictionary of words and create a Python dictionary with keys for the word lengths and values of the words of those lengths, ie: def processedDictionary(): d = defaultdict(list) f = open(dictionaryFile, "r") f.c...
Laurence
1

votes
1

answer
288

Views

error when unpickling a class from its constructor

I would like to implement a class (ideally a singleton) for which the state should be restored using cPickle mechanism at initialization stage. To do so I wrote the following snippet: import cPickle import collections class Test(collections.OrderedDict): path = "test.cp" def __init__(self): self.pat...
Eurydice
1

votes
1

answer
285

Views

Saving and loading classes in python

I am trying to save and load some classes using pickle. In my project I have some classes with variables containing lists and/or dictionaries and/or tuples of other classes. when I load the saved classes the classes within are different. I found a great tutorial about the static method __new__ here....
Yannis
1

votes
1

answer
91

Views

re-reading a dictionary structure saved as text file , again as dictionary

I've parsed a big corpus and I've saved the data I needed in a dictionary structure. But at the end of my code I've saved it as a .txt file 'cause I needed to manually check something. now in another part of my work I need that dictionary as my input. I wanted to know if there are other ways than ju...
Pari
1

votes
2

answer
281

Views

use pickle to save dictionary in python

I have a file hash_db.pickle that when I created it I saved a dictionary on it: v = {hash_value:{"file name":file_name,"file size":file_size,"last scanned time":scanned_time}} {123dfre345:{"file name":calc.pdf,"file size":234,"last scanned time":12:23 24/12/2013}} {3gcdshj754:{"file name":star.pdf,"...
user3832061
1

votes
2

answer
851

Views

Pickling Matplotlib plot raising PicklingError: Can't pickle 'RendererAgg' object

I have a program that creates plots - sometimes line plots, sometimes NonUniformImages - using matplotlib. I'd like to be able to pickle the plots to reopen them at a later time without going through the whole creation process again. For whatever reason, it keeps throwing a PicklingError: Can't pick...
wes3449
1

votes
1

answer
866

Views

Python - pickle.load() takes one positional argument (2 given)

This produces an error: pickle.load() takes one positional argument (2 given) Here is my code: import pickle, os.path created = False phoneBook = {} name = input("Please enter a name(or press enter to end input): ") while name != '': number = input("Please enter number: ") phoneBook[name] = number...
1

votes
2

answer
800

Views

Can't pickle StringO objects (while trying to serialize the signature returned from librsync-python module)

https://pypi.python.org/pypi/python-librsync/0.1-5 import librsync # The destination file. dst = file('Resume-v1.0.pdf', 'rb') # Step 1: prepare signature of the destination file signature = librsync.signature(dst) I want to store the signature in a file (preferably as a dictionary entry using pickl...
aste123
1

votes
4

answer
2.5k

Views

pickling pandas dataframe does multiply by 5 the file size

I am reading a 800 Mo csv file with panda.csv_reader, and then use the original pythoin pickle.dump(datfarame) to save it. The result is a 4 Gb pkl file, so the csv size is multipled by 5. I expected pickle to compress data rather than extend it. Also because I can do a gzip on the csv file which c...
Romain Jouin
1

votes
2

answer
433

Views

Pickle and exec in python

¡Hi everybody! What I have to do is this: import pickle exec "def f(): print 'Hi!'" code = pickle.dumps(f) print code As you can see, it works fine. But, in order to avoid defining global variables, (as for example f in previous code), I'd like to do something like this: import pickle d = {} ex...
1

votes
2

answer
583

Views

Is there any difference between Pickling and Serialization?

I have came across these two terms so often while reading about python objects. However, there is a confusion between pickling and serialization since at one place I read The pickle module implements an algorithm for turning an arbitrary Python object into a series of bytes. This process is also cal...
Ajay Gupta
1

votes
1

answer
1.1k

Views

how to deepcopy when pickling is not possible

I'm trying to deepcopy a dictionary in Python 3.4 but some of the values are not "pickable", i.e. when I try to deepcopy the dictionary, it raises a RuntimeError exception *** RuntimeError: Pickling of "DataObject" instances is not enabled (http://www.boost.org/libs/python/doc/v2/pickle.html) What w...
zom-pro
1

votes
1

answer
430

Views

How to use class decorators with pickle?

I would like to use class decorators (not function decorators!), e.g. def class_decorator(cls): class new_cls(cls): def run(self, *args, **kwargs): print 'In decorator' super(new_cls,self).run(*args, **kwargs) return new_cls @class_decorator class cls(object): ''' a class ''' def run(self): print 'c...
alex
1

votes
1

answer
5.2k

Views

can't pickle instancemethod objects

I met a problem of pickle, Code is that: import cPickle class A(object): def __init__(self): self.a = 1 def methoda(self): print(self.a) class B(object): def __init__(self): self.b = 2 a = A() self.b_a = a.methoda def methodb(self): print(self.b) if __name__ == '__main__': b = B() with open('best_mo...
DemoRunner
1

votes
2

answer
304

Views

Convert lambda function into pickleable object

Is there a function or class that is able to take lambda function as an argument and return object or function that can be pickled using Pickle? I am trying to parallelize some code using joblib.Parallel that has a problem with lambda function and turning all lambda functions into named functions ma...
BoZenKhaa
1

votes
2

answer
2.8k

Views

Is there a way to save multiple variables in python pickle?

I am trying to save multiple variables to a file. E.G saving an item in a shop so I'm trying to save the items price,name and code to one file, and multiple items to the same file def enter_item_info(): count = 0 count1 = 0 print("how many items are you entering?") amount = int(input("Items = ")) da...
Joseph
0

votes
0

answer
2

Views

Error in loading mnist dataset in Python 3

When trying to load the MNIST dataset with a mnist_loader script, I keep on getting the following error: 'ascii' codec can't decode byte 0x90 in position 614: ordinal not in range(128)' The code used to load the data is shown below. Any suggestions? Many thanks! def load_data(): f = gzip.open('m...
Sibren De Preter

View additional questions