Numpy Concatenate Memory Error
Contents |
here for a quick overview of the site Help Center Detailed answers to any questions you might have Meta Discuss the workings and policies of this site About Us Learn more about Stack Overflow the company Business Learn more pandas memory error about hiring developers or posting ads with us Stack Overflow Questions Jobs Documentation Tags Users numpy concatenate vs append Badges Ask Question x Dismiss Join the Stack Overflow Community Stack Overflow is a community of 6.2 million programmers, just like you, helping
Memory Error Python
each other. Join them; it only takes a minute: Sign up Concatenate large numpy arrays in RAM up vote 1 down vote favorite I have some 3D image data and want to build a stack of RGB images
Numpy Merge Arrays
out of single channel stacks, i.e. I try to concatenate three arrays of shape (358, 1379, 1042) into one array of shape (358, 1379, 1042, 3). Inspired by skimage.color.gray2rgb I tried np.concatenate(( stack1[..., np.newaxis], stack2[..., np.newaxis], stack3[..., np.newaxis]), axis=-1) However, even though each of these stacks is only about 1GiB this fills my empty ~12GiB RAM immediately ... So I tried to pre-allocate an array of the final shape and then fill it with the stacks, like hstack rgb_stack = np.zeros(stack1.shape + (3,)) rgb_stack[:,:,:,0] = stack1 which also exhausted my RAM once I execute the second line. Finally I tried to explicitly copy the data from stack1 into rgb_stack by rgb_stack = np.zeros(stack1.shape + (3,)) rgb_stack[:,:,:,0] = stack1.copy() with the same result. What am I doing wrong? python arrays numpy ram scikit-image share|improve this question asked Jul 7 '15 at 14:09 qiv 659 1 If you don't specify a dtype, zeros will assume you want float64, and a (358, 1379, 1042, 3) float64 array will take up ~11.5 GiB. What dtype are your input arrays? –ali_m Jul 7 '15 at 14:24 1 Some operating systems (notably Linux) will happily overcommit memory. When you create your array of zeros, the kernel does not immediately set aside a correspondingly sized chunk of RAM - this only occurs when you actually try to write to those memory addresses, hence why you only see the MemoryError when you try to assign to rgb_stack. –ali_m Jul 7 '15 at 15:05 1 Do you have the data saved elsewhere? If you do I would try to help python's garbage collector by saying del stack1 after you have put it into the rgb_stack, same thing after stack2 has been inserted. –jfish003 Jul 7 '15 at 15:17 1 I was also using np.zeros as default stack3 when playing around with np.con
here for a quick overview of the site Help Center Detailed answers to any questions you might have Meta Discuss the workings and policies of this site About Us Learn more about Stack Overflow the company Business
Pytables
Learn more about hiring developers or posting ads with us Stack Overflow Questions Jobs Documentation numpy append Tags Users Badges Ask Question x Dismiss Join the Stack Overflow Community Stack Overflow is a community of 6.2 million programmers, just numpy vstack like you, helping each other. Join them; it only takes a minute: Sign up Numpy memory error creating huge matrix up vote 2 down vote favorite 3 I am using numpy and trying to create a huge matrix. http://stackoverflow.com/questions/31270968/concatenate-large-numpy-arrays-in-ram While doing this, I receive a memory error Because the matrix is not important, I will just show the way how to easily reproduce the error. a = 10000000000 data = np.array([float('nan')] * a) not surprisingly, this throws me MemoryError There are two things I would like to tell: I really need to create and to use a big matrix I think I have enough RAM to handle this matrix (I have 24 Gb or http://stackoverflow.com/questions/19085012/numpy-memory-error-creating-huge-matrix RAM) Is there an easy way to handle big matrices in numpy? Just to be on the safe side, I previously read these posts (which sounds similar): Python Numpy Very Large Matrices Python/Numpy MemoryError Processing a very very big data set in python - memory error P.S. apparently I have some problems with multiplication and division of numbers, which made me think that I have enough memory. So I think it is time for me to go to sleep, review math and may be to buy some memory. May be during this time some genius might come up with idea how to actually create this matrix using only 24 Gb of Ram. Why I need this big matrix I am not going to do any manipulations with this matrix. All I need to do with it is to save it into pytables. python memory numpy share|improve this question edited Sep 30 '13 at 1:18 asked Sep 30 '13 at 0:53 Salvador Dali 50.3k40232310 2 How do you expect to fit 10 billion floats in 24 GB? If a float were 2.4 bytes, and 100% of your RAM were devoted to holding this array - sure ;-) –Tim Peters Sep 30 '13 at 0:56 What do you need to do with this matrix? That might give an insight to a workaro
here for a quick overview of the site Help Center Detailed answers to any questions you might have Meta Discuss the workings and policies of http://stackoverflow.com/questions/23819119/numpy-hstack-explodes-memory this site About Us Learn more about Stack Overflow the company Business Learn more about hiring developers or posting ads with us Stack Overflow Questions Jobs Documentation Tags Users Badges Ask Question x Dismiss Join the Stack Overflow Community Stack Overflow is a community of 6.2 million programmers, just like you, helping each other. Join them; it only takes a minute: Sign up memory error Numpy hstack explodes memory [duplicate] up vote 2 down vote favorite This question already has an answer here: How to extend an array in-place in Numpy? 3 answers I am new to Python and am uncertain why I am seeing memory usage spike so dramatically when I use Numpy hstack to join together two pandas data frames. The performance with pandas.concat was even numpy concatenate memory worse - if it would finish at all - so I am using NumPy. The two data frames are relatively large, but I have 20 gb free RAM (using 11GB, including the two data frames I want to copy). The data frames a and b have shapes: a.shape (66377, 30) b.shape (66377, 11100) when I use np.hstack((a,b)) the free 20GB is had is completely used up. python numpy pandas share|improve this question edited May 23 '14 at 0:25 Simon Kuang 1,72911140 asked May 23 '14 at 0:16 B_Miner 309419 marked as duplicate by Saullo Castro, Code Lღver, Aniket Kulkarni, Soner Gönül, EdChum May 23 '14 at 7:09 This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question. what is the dtype of your data? For float64, b should be about 5.5 GB, so the results of np.hstack should only add about 5.5 GB as well. –JoshAdel May 23 '14 at 0:39 They are float64 –B_Miner May 23 '14 at 2:52 add a comment| 2 Answers 2 active oldest