Cpickle Memory Error
Contents |
here for a quick overview of the site Help Center Detailed answers to any questions you might have Meta Discuss the workings and policies of this site About Us Learn more about Stack Overflow the company Business Learn more about pickle load memory error hiring developers or posting ads with us Stack Overflow Questions Jobs Documentation Tags Users Badges cpickle eof error Ask Question x Dismiss Join the Stack Overflow Community Stack Overflow is a community of 4.7 million programmers, just like you, helping each python pickle load memory error other. Join them; it only takes a minute: Sign up Pickle dump huge file without memory error up vote 5 down vote favorite 3 I have a program where I basically adjust the probability of certain things happening based
Streaming-pickle
on what is already known. My file of data is already saved as a pickle Dictionary object at Dictionary.txt. The problem is, is that everytime that I run the program it pulls in the Dictionary.txt, turns it into a dictionary object, makes it's edits and overwrites Dictionary.txt. This is pretty memory intensive as the Dictionary.txt is 123 MB. When I dump I am getting the MemoryError, everything seems fine when I pull it in.. Is there a better python pickle memory usage (more efficient) way of doing the edits? (Perhaps w/o having to overwrite the entire file everytime) Is there a way that I can invoke garbage collection (through gc module)? (I already have it auto-enabled via gc.enable()) I know that besides readlines() you can read line-by-line. Is there a way to edit the dictionary incrementally line-by-line when I already have a fully completed Dictionary object File in the program. Any other solutions? Thank you for your time. python memory file-io pickle share|improve this question asked Jul 7 '13 at 14:31 user2543682 2813 There are a few compressive and other libraries. Personally, I like dill and H5Py for large objects. If you are using scikit learn and have to use a model base on dictionary, perhaps you could use joblib as well (only really for these models). –Andrew Scott Evans Nov 5 '15 at 22:44 add a comment| 6 Answers 6 active oldest votes up vote 2 down vote Have you tried using streaming pickle: https://code.google.com/p/streaming-pickle/ I have just solved a similar memory error by switching to streaming pickle. share|improve this answer answered Jul 11 '13 at 15:16 Chris Wheadon 485315 streaming-pickle does not appear well-suited to the present case of just one single dictionary object. When it is better suited (like for a huge list), normal pickle can do the trick as well, because you can dump seve
here for a quick overview of the site Help Center Detailed answers to any questions you might have Meta klepto python Discuss the workings and policies of this site About Us Learn
Cpickle Vs Pickle
more about Stack Overflow the company Business Learn more about hiring developers or posting ads with us Stack
Pickle Protocol
Overflow Questions Jobs Documentation Tags Users Badges Ask Question x Dismiss Join the Stack Overflow Community Stack Overflow is a community of 4.7 million programmers, just like you, http://stackoverflow.com/questions/17513036/pickle-dump-huge-file-without-memory-error helping each other. Join them; it only takes a minute: Sign up MemoryError while pickling data in python up vote 5 down vote favorite I am trying to dump a dictionary into pickle format, using 'dump' command provided in python. The file size of the dictionary is around 150 mb, but an exception occurs when only 115 http://stackoverflow.com/questions/16403633/memoryerror-while-pickling-data-in-python mb of the file is dumped. The exception is: Traceback (most recent call last): File "C:\Python27\generate_traffic_pattern.py", line 32, in
Support Search GitHub This repository Watch https://github.com/waf-project/waf/issues/1330 35 Star 402 Fork 122 waf-project/waf Code Issues 7 http://www.gossamer-threads.com/lists/python/dev/714806 Pull requests 0 Projects 0 Pulse Graphs New issue Out of memory during cPickle.dumps() #1330 Closed GoogleCodeExporter opened this Issue Apr 3, 2015 · 4 comments Projects None yet Labels auto-migrated Type-Other Milestone No milestone memory error Assignees No one assigned 1 participant GoogleCodeExporter commented Apr 3, 2015 >> operating system: Windows 7 (64-bit) >> python version: 2.7.1 (32-bit) >> waf version (or svn revision): 1.7.10 >> observed output and expected output: Traceback (most recent call last): File "submodules\external\tools\waf\waflib\Scripting.py", line 138, pickle load memory in waf_entry_point run_commands() File "submodules\external\tools\waf\waflib\Scripting.py", line 232, in run_commands ctx = run_command(cmd_name) File "submodules\external\tools\waf\waflib\Scripting.py", line 219, in run_command ctx.execute() File "submodules\external\tools\waf\waflib\Scripting.py", line 576, in execute return execute_method(self) File "submodules\external\tools\waf\waflib\Build.py", line 251, in execute self.execute_build() File "submodules\external\tools\waf\waflib\Build.py", line 273, in execute_build self.compile() File "submodules\external\tools\waf\waflib\Build.py", line 369, in compile self.store() File "submodules\external\tools\waf\waflib\extras\relocation.py", line 24, in store old1(self) File "submodules\external\tools\waf\waflib\Utils.py", line 669, in f ret = fun(*k, **kw) File "submodules\external\tools\waf\waflib\Build.py", line 334, in store x = cPickle.dumps(data, -1) MemoryError: out of memory >> how to reproduce the problem? This happens after building our large project many times. We have thousands of compiled files and a few variants, so the pickle file can grow to 100MB. This error can crop up with the file anywhere between 50-100MB, it isn't consistent. Once in this state
Post #1 of 13 (2051 views) Permalink writing large dictionaries to file using cPickle hello all, i have a large dictionary which contains about 10 keys, each key has a value which is a list containing about 1 to 5 million (small) dictionaries. for example, mydict = {key1: [.{'a': 1, 'b': 2, 'c': 'hello'}, {'d', 3, 'e': 4, 'f': 'world'}, ...], key2: [...]} in total there are about 10 to 15 million lists if we concatenate together all the values of every key in 'mydict'. mydict is a structure that represents data in a very large file (about 800 megabytes). what is the fastest way to pickle 'mydict' into a file? right now i am experiencing a lot of difficulties with cPickle when using it like this: from cPickle import pickle pfile = open(my_file, 'w') pickle.dump(mydict, pfile) pfile.close() this creates extremely large files (~ 300 MB) though it does so *extremely* slowly. it writes about 1 megabyte per 5 or 10 seconds and it gets slower and slower. it takes almost an hour if not more to write this pickle object to file. is there any way to speed this up? i dont mind the large file... after all the text file with the data used to make the dictionary was larger (~ 800 MB) than the file it eventually creates, which is 300 MB. but i do care about speed... i have tried optimizing this by using this: s = pickle.dumps(mydict, 2) pfile.write(s) but this takes just as long... any ideas ? is there a different module i could use that's more suitable for large dictionaries ? thank you very much. -- http://mail.python.org/mailman/listinfo/python-list python at bdurham Jan28,2009,8:32AM Post #2 of 13 (2016