[Bioperl-l] load_seqdatabase.pl memory requirements unusually large

[Chris Fields]

There was a known memory issue with Bio::Species (), but I think that  
may be resolved (http://bugzilla.open-bio.org/show_bug.cgi?id=2594).   
The odd thing about the failed in 01dbadaptor.t is it appears to be  
related to rolling back changes.

Could that have something to do with using a loaded database for  
tests?  I thought tests were supposed to be run with a clean database  
(with the biosql schema but nothing else).

chris

On Feb 6, 2009, at 2:10 PM, Hilmar Lapp wrote:

> It could be, but that doesn't explain the test failures. They need  
> not have the same cause, but they could, and the tests failing the  
> way they do is certainly not right.
>
> 	-hilmar
>
> On Feb 6, 2009, at 3:04 PM, Florent Angly wrote:
>
>> Out of the blue, I'm going to ask: could it be some memory leak due  
>> to a circular reference?
>> Florent
>>
>> Hilmar Lapp wrote:
>>> Something seems to cause Perl to be crapping out. It it were a  
>>> programmatic exception you would see the message and the trace.
>>>
>>> Could you run these tests by themselves:
>>>
>>>   ./Build test --test-files t/11locuslink.t
>>>
>>> If that doesn't reveal the error, add a verbose=1 argument.
>>>
>>> Let us know what you find.
>>>
>>>   -hilmar
>>>
>>> On Feb 6, 2009, at 11:03 AM, Michael Muratet wrote:
>>>
>>>> Greetings
>>>>
>>>> I have use bioperl-db and load_seqdatabase.pl many times in the  
>>>> past and it's worked pretty much out of the box.
>>>>
>>>> I have been trying to load fasta files from hg18. For chr1, the  
>>>> virtual and resident memory quickly builds to 14 GB or so, then  
>>>> starts using up the 2GB swap until it's full and then the system  
>>>> hangs. The system is an 8-core Dell with 16GB of physical memory.  
>>>> chr1.fa is ~242MB. All of the disk storage is network mounted on  
>>>> an EMC system which (I am told) has a proprietary version of  
>>>> something that's NFS-like.
>>>>
>>>> I loaded chrM (~17kb) and load_seqdatabase grew to over 4 GB  
>>>> before it completed.
>>>>
>>>> I am using MySQL 5.0.51a-community, DBI 1.607, perl 5.85, bioperl  
>>>> 1.6.0, bioperl-db 1.006900.
>>>>
>>>> I have the innodb engine enabled in MySQL and the buffers and  
>>>> caches set for a 'large' system.
>>>>
>>>> I had some errors during the bioperl-db install:
>>>>
>>>> Test Summary Report
>>>> -------------------
>>>> t/01dbadaptor.t (Wstat: 256 Tests: 23 Failed: 1)
>>>> Failed test:  23
>>>> Non-zero exit status: 1
>>>> t/10ensembl.t   (Wstat: 65280 Tests: 5 Failed: 0)
>>>> Non-zero exit status: 255
>>>> Parse errors: Bad plan.  You planned 18 tests but ran 5.
>>>> t/11locuslink.t (Wstat: 65280 Tests: 7 Failed: 0)
>>>> Non-zero exit status: 255
>>>> Parse errors: Bad plan.  You planned 113 tests but ran 7.
>>>> t/15cluster.t   (Wstat: 65280 Tests: 7 Failed: 0)
>>>> Non-zero exit status: 255
>>>> Parse errors: Bad plan.  You planned 162 tests but ran 7.
>>>> Files=16, Tests=1205, 21 wallclock secs ( 0.51 usr  0.12 sys +  
>>>> 14.90 cusr  2.69 csys = 18.22 CPU)
>>>> Result: FAIL
>>>> Failed 4/16 test programs. 1/1205 subtests failed.
>>>>
>>>> The most recent bioperl-db documentation says that a workable  
>>>> version may be possible after some errors and so I went ahead  
>>>> with the install.
>>>>
>>>> The mailing list archive has some discussion about throughput but  
>>>> nothing really about filling up memory.
>>>>
>>>> Can anyone offer any clues about what's going on or where to  
>>>> start looking?
>>>>
>>>> Thanks
>>>>
>>>> Mike
>>>> _______________________________________________
>>>> Bioperl-l mailing list
>>>> Bioperl-l at lists.open-bio.org
>>>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>>>
>
> -- 
> ===========================================================
> : Hilmar Lapp  -:-  Durham, NC  -:-  hlapp at gmx dot net :
> ===========================================================
>
>
>
> _______________________________________________
> Bioperl-l mailing list
> Bioperl-l at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/bioperl-l