[Bioperl-l] perl memory problem

[Jason Stajich]

On Oct 26, 2005, at 11:55 AM, Zaigang Liu wrote:

>
>
>
>
> Hello, everyone,
>
> I am new to this list and I am looking for help for one problem I met.
>
> I am trying to run a perl program doing calculation on large data  
> set. It
> uses an array to store the input data and the array size will reach 8G
> after all the data is loaded by my estimation. The process always  
> stops and
> gives a "out of memory" error when the memory usage reached about  
> 2G. I
> used a solaris machine with 40G memory so it is not really run out of
> memory when the program stopped.
>
there is probably a limit on how much memory perl can address in the  
version you have compiled.

Have you considered using a data structure which does not require the  
entire array to be in memory at once.  the DB_File module is very  
useful for this as you can tie your array to a flatfile Berkeley DB.   
You can also use something like SQLite or a full fledged relational  
database instead if you are crunching such a large number of items.


> Does anyone know where could this program come from and how to fix it.
>
> Thanks in advance,
>
> Zaigang
>
> This communication is for use by the intended recipient and contains
> information that may be Privileged, confidential or copyrighted under
> applicable law. If you are not the intended recipient, you are hereby
> formally notified that any use, copying or distribution of this e- 
> mail,
> in whole or in part, is strictly prohibited. Please notify the  
> sender by
> return e-mail and delete this e-mail from your system. Unless  
> explicitly
> and conspicuously designated as "E-Contract Intended", this e-mail  
> does
> not constitute a contract offer, a contract amendment, or an  
> acceptance
> of a contract offer. This e-mail does not constitute a consent to the
> use of sender's contact information for direct marketing purposes  
> or for
> transfers of data to third parties.
>
> Francais Deutsch Italiano  Espanol  Portugues  Japanese  Chinese   
> Korean
>
>            http://www.DuPont.com/corp/email_disclaimer.html
> _______________________________________________
> Bioperl-l mailing list
> Bioperl-l at portal.open-bio.org
> http://portal.open-bio.org/mailman/listinfo/bioperl-l
>

--
Jason Stajich
Duke University
http://www.duke.edu/~jes12