Out Of Memory Error In Perl Script
Contents |
here for a quick overview of the site Help Center Detailed answers to any questions you might have Meta Discuss the workings and policies of this site About Us Learn more about Stack Overflow the
Out Of Memory Error While Running Perl Script
company Business Learn more about hiring developers or posting ads with us Stack Overflow Questions perl out of memory windows Jobs Documentation Tags Users Badges Ask Question x Dismiss Join the Stack Overflow Community Stack Overflow is a community of 6.2 million
Perl Out Of Memory Reading Large File
programmers, just like you, helping each other. Join them; it only takes a minute: Sign up Why does my Perl script die with an “out of memory” exception? up vote 1 down vote favorite I need to how to solve out of memory error in perl read a 200mb "space"-separated file line-by-line and collect its contents into an array. Every time I run the script, Perl throws an "out of memory" exception, but I don't understand why! Some advice please? #!/usr/bin/perl -w use strict; use warnings; open my $fh, "<", "../cnai_all.csd"; my @parse = (); while (<$fh>) { my @words = split(/\s/,$_); push (@parse, \@words); } print scalar @parse; the cnai file looks like this: it contains 11000 rows perl ulimit and 4200 values, seperated by "space", per line. VALUE_GROUP_A VALUE_GROUP_B VALUE_GROUP_C VALUE_GROUP_A VALUE_GROUP_B VALUE_GROUP_C VALUE_GROUP_A VALUE_GROUP_B VALUE_GROUP_C VALUE_GROUP_A VALUE_GROUP_B VALUE_GROUP_C The code above is just a stripped down sample.The final script will store all values in a hash and write it to a database later . But first, I have to solve that memory problem! database perl memory share|improve this question edited Aug 17 '10 at 2:21 Greg Bacon 75.8k18148197 asked Feb 4 '10 at 16:45 Floopy-Doo 63117 Code said while (<$fh>) but it was not displayed in the markup. –mob Feb 4 '10 at 16:50 @floppy-doo Please edit your question to give us an idea what the contents of cnai_all.csd look like? –Greg Bacon Feb 4 '10 at 16:55 See also: stackoverflow.com/questions/1663498/finding-a-perl-memory-leak –Ether Feb 4 '10 at 21:53 If all you want is the number of lines (the output of print scalar @parse), just do `my $i=0; $i++ while <$f>; print $i;". –daotoad Feb 5 '10 at 3:28 Well, how much memory do you have and what else is using it up? :) –brian d foy Feb 5 '10 at 4:22 | show 2 more comments 6 Answers 6 active oldest votes up vote 6 down vote That would be because... you're running out of memory! You're
RecentThreads NewestNodes Donate What'sNew on Feb 07, 2007 at 04:23UTC ( #598685=perlquestion: print w/replies, xml ) Need Help?? jesuashok has asked for the wisdom of the Perl Monks concerning the following question: Dear monks, I have a perl
Perl Catch Out Of Memory Error
script which is trying to read some files and storing those lines perl memory usage into a hash. But each file size comes around 3MB or more. when I run the script I got the following error [belief] /apps/inst2/metrica/analysis_ericsson/schema_analysis> perl s +chema_analysis.pl -r rename_columns -u /apps/inst2/metrica/anthony/Er +icson_R10_Onsite/VFOZ_BACKUP/summaryspr/ -y /apps/inst2/metrica/anth +ony/Ericson_R10_Onsite/VFOZ_BACKUP/metalayer/ \ > -o only_in_old_schema -p /apps/inst2/metrica/anthony/Ericson_R10_Ons +ite/VFOZ_BACKUP/reportspr/ Out of memory! [download] Is there any way, can http://stackoverflow.com/questions/2201432/why-does-my-perl-script-die-with-an-out-of-memory-exception I resolve this Issue. Is it possible to control the memory usage. Comment on Out of memory!Download Code Replies are listed 'Best First'. Re: Out of memory! by GrandFather (Sage) on Feb 07, 2007 at 04:29UTC Is there any way you can post sample code that demonstrates the issue and indicate how many files are being manipulated? We need a little more information http://www.perlmonks.org/?node_id=598685 than "I have a problem with large hashes. How do I solve it?" if you want a better answer than "Use a tied hash" or "Install more memory". DWIM is Perl's answer to Gödel [reply] Re: Out of memory! by Tanktalus (Canon) on Feb 07, 2007 at 04:55UTC Generally speaking, when I've had that problem, it was that there was a memory limit via ulimit - removing that limit 'solved' the problem. At least, insofaras letting me use way more memory. Perhaps you have an underlying memory problem - wasting memory or leaking it, we can't be sure from your description. Assuming that not to be the case, though, it's probably a ulimit on memory.[reply] Re: Out of memory! by chargrill (Parson) on Feb 07, 2007 at 04:55UTC "Doctor, it hurts when I do this!" "Then don't do that." So the answer is simple - don't write perl programs that consume more memory than your system can allocate. --chargrill s**lil*; $*=join'',sort split q**; s;.*;grr; &&s+(.(.)).+$2$1+; $; = qq-$_-;s,.*,ahc,;$,.=chop for split q,,,reverse;print for($,,$;,$*,$/) [download] [reply][d/l] Re: Out of memory! by quester (Vicar) on Feb 07, 2007 at 08:34UTC
PERL Beginners I wrote a small script that uses message ID's as unique values and extracts recipient address info. The goal is to count 1019 events per message ID. It also http://www.justskins.com/forums/out-of-memory-error-115248.html gets the sum of recipients per message ID. The script works fine but when it runs against a very large file (2GB+) I receive an out of memory error. Is there a more efficient way of handling the hash portion that is less memory intense and preferably faster? --Paul # Tracking log pr use strict; my $recips; my %event_id; my $counter; my $total_recips; my $count; # out of Get log file die "You ... Thread Tools Show Printable Version Email this Page… Subscribe to this Thread… Display Linear Mode Switch to Hybrid Mode Switch to Threaded Mode December 16th,07:15 PM #1 Out of memory error problem I wrote a small script that uses message ID's as unique values and extracts recipient address info. The goal is to count 1019 events per message ID. It also out of memory gets the sum of recipients per message ID. The script works fine but when it runs against a very large file (2GB+) I receive an out of memory error. Is there a more efficient way of handling the hash portion that is less memory intense and preferably faster? --Paul # Tracking log pr use strict; my $recips; my %event_id; my $counter; my $total_recips; my $count; # Get log file die "You must enter a tracking log. \n" if $#ARGV <0; my $logfile = shift; open (LOGFILE, $logfile) || die "Unable to open $logfile because\n $!\n"; foreach (