[Bioperl-l] load_seqdatabase.pl memory requirements unusually large

Mon Feb 9 21:01:58 UTC 2009

Thanks Mike. Apparently in your version of Perl you don't see the  
stack traces unless you run them individually. Based on the output  
below I think that Chris was on the right track when he suspected lack  
of transaction support.

To be sure, could you run

	./Build test --test-files t/01dbadaptor.t

The last test is for transaction support. If the above prints  
something like "your RDBMS does not have transactions enabled" and a  
failure of test #23, then that is the problem. In this case, which  
RDBMS are you using, and if it is MySQL, did you enable Innodb fully  
in the configuration file (you need to enable it *and* give the  
location of the data files)?

Once you fixed that, you'll need to empty your database.

	-hilma

On Feb 9, 2009, at 3:36 PM, Michael Muratet wrote:

>
> On Feb 6, 2009, at 1:34 PM, Hilmar Lapp wrote:
>
>> Something seems to cause Perl to be crapping out. It it were a  
>> programmatic exception you would see the message and the trace.
>>
>> Could you run these tests by themselves:
>>
>> 	./Build test --test-files t/11locuslink.t
>>
>> If that doesn't reveal the error, add a verbose=1 argument.
>>
>> Let us know what you find.
>
> Here's the result:
>
> [root at srv-cf1 bioperl-db]# ../Build test --test-files t/11locuslink.t
> bash: ../Build: No such file or directory
> [root at srv-cf1 bioperl-db]# ./Build test --test-files t/11locuslink.t
> Copying scripts/biosql/terms/importrelation.pl -> blib/script/ 
> importrelation.pl
> blib/script/importrelation.pl -> blib/script/bp_importrelation.pl
> Copying scripts/biosql/merge-unique-ann.pl -> blib/script/merge- 
> unique-ann.pl
> blib/script/merge-unique-ann.pl -> blib/script/bp_merge-unique-ann.pl
> Copying scripts/biosql/update-on-new-date.pl -> blib/script/update- 
> on-new-date.pl
> blib/script/update-on-new-date.pl -> blib/script/bp_update-on-new- 
> date.pl
> Copying scripts/biosql/terms/add-term-annot.pl -> blib/script/add- 
> term-annot.pl
> Deleting blib/script/add-term-annot.pl.bak
> blib/script/add-term-annot.pl -> blib/script/bp_add-term-annot.pl
> Copying scripts/corba/caching_corba_server.pl -> blib/script/ 
> caching_corba_server.pl
> Deleting blib/script/caching_corba_server.pl.bak
> blib/script/caching_corba_server.pl -> blib/script/ 
> bp_caching_corba_server.pl
> Copying scripts/biosql/load_ontology.pl -> blib/script/ 
> load_ontology.pl
> Deleting blib/script/load_ontology.pl.bak
> blib/script/load_ontology.pl -> blib/script/bp_load_ontology.pl
> Copying scripts/biosql/load_seqdatabase.pl -> blib/script/ 
> load_seqdatabase.pl
> Deleting blib/script/load_seqdatabase.pl.bak
> blib/script/load_seqdatabase.pl -> blib/script/bp_load_seqdatabase.pl
> Copying scripts/biosql/terms/interpro2go.pl -> blib/script/ 
> interpro2go.pl
> blib/script/interpro2go.pl -> blib/script/bp_interpro2go.pl
> Copying scripts/biosql/clean_ontology.pl -> blib/script/ 
> clean_ontology.pl
> blib/script/clean_ontology.pl -> blib/script/bp_clean_ontology.pl
> Copying scripts/corba/test_bioenv.pl -> blib/script/test_bioenv.pl
> Deleting blib/script/test_bioenv.pl.bak
> blib/script/test_bioenv.pl -> blib/script/bp_test_bioenv.pl
> Copying scripts/biosql/update-on-new-version.pl -> blib/script/ 
> update-on-new-version.pl
> blib/script/update-on-new-version.pl -> blib/script/bp_update-on-new- 
> version.pl
> Copying scripts/corba/bioenv_server.pl -> blib/script/bioenv_server.pl
> Deleting blib/script/bioenv_server.pl.bak
> blib/script/bioenv_server.pl -> blib/script/bp_bioenv_server.pl
> Copying scripts/biosql/bioentry2flat.pl -> blib/script/ 
> bioentry2flat.pl
> Deleting blib/script/bioentry2flat.pl.bak
> blib/script/bioentry2flat.pl -> blib/script/bp_bioentry2flat.pl
> Copying scripts/biosql/load_interpro.pl -> blib/script/ 
> load_interpro.pl
> blib/script/load_interpro.pl -> blib/script/bp_load_interpro.pl
> Copying scripts/biosql/cgi-bin/getentry.pl -> blib/script/getentry.pl
> Deleting blib/script/getentry.pl.bak
> blib/script/getentry.pl -> blib/script/bp_getentry.pl
> Copying scripts/biosql/del-assocs-sql.pl -> blib/script/del-assocs- 
> sql.pl
> blib/script/del-assocs-sql.pl -> blib/script/bp_del-assocs-sql.pl
> Copying scripts/biosql/freshen-annot.pl -> blib/script/freshen- 
> annot.pl
> blib/script/freshen-annot.pl -> blib/script/bp_freshen-annot.pl
> t/11locuslink.t....1/113
> ------------- EXCEPTION: Bio::Root::Exception -------------
> MSG: Unique key query in Bio::DB::BioSQL::SpeciesAdaptor returned 2  
> rows instead of 1. Query was [name_class="scientific  
> name",binomial="Homo sapiens"]
> STACK: Error::throw
> STACK: Bio::Root::Root::throw /usr/lib/perl5/site_perl/5.8.5/Bio/ 
> Root/Root.pm:357
> STACK: Bio::DB::BioSQL::BasePersistenceAdaptor::_find_by_unique_key / 
> root/mmroot/bioperl-db/blib/lib/Bio/DB/BioSQL/ 
> BasePersistenceAdaptor.pm:965
> STACK: Bio::DB::BioSQL::BasePersistenceAdaptor::find_by_unique_key / 
> root/mmroot/bioperl-db/blib/lib/Bio/DB/BioSQL/ 
> BasePersistenceAdaptor.pm:860
> STACK: Bio::DB::BioSQL::BasePersistenceAdaptor::create /root/mmroot/ 
> bioperl-db/blib/lib/Bio/DB/BioSQL/BasePersistenceAdaptor.pm:182
> STACK: Bio::DB::Persistent::PersistentObject::create /root/mmroot/ 
> bioperl-db/blib/lib/Bio/DB/Persistent/PersistentObject.pm:244
> STACK: Bio::DB::BioSQL::BasePersistenceAdaptor::create /root/mmroot/ 
> bioperl-db/blib/lib/Bio/DB/BioSQL/BasePersistenceAdaptor.pm:169
> STACK: Bio::DB::Persistent::PersistentObject::create /root/mmroot/ 
> bioperl-db/blib/lib/Bio/DB/Persistent/PersistentObject.pm:244
> STACK: t/11locuslink.t:30
> -----------------------------------------------------------
> # Looks like you planned 113 tests but ran 7.
> # Looks like your test exited with 255 just after 7.
> t/11locuslink.t.... Dubious, test returned 255 (wstat 65280, 0xff00)
> Failed 106/113 subtests
>
> Test Summary Report
> -------------------
> t/11locuslink.t (Wstat: 65280 Tests: 7 Failed: 0)
>  Non-zero exit status: 255
>  Parse errors: Bad plan.  You planned 113 tests but ran 7.
> Files=1, Tests=7,  2 wallclock secs ( 0.01 usr  0.00 sys +  0.40  
> cusr  0.06 csys =  0.47 CPU)
> Result: FAIL
> Failed 1/1 test programs. 0/7 subtests failed.
>
> Thanks
>
> Mike
>
>>
>>
>> 	-hilmar
>>
>> On Feb 6, 2009, at 11:03 AM, Michael Muratet wrote:
>>
>>> Greetings
>>>
>>> I have use bioperl-db and load_seqdatabase.pl many times in the  
>>> past and it's worked pretty much out of the box.
>>>
>>> I have been trying to load fasta files from hg18. For chr1, the  
>>> virtual and resident memory quickly builds to 14 GB or so, then  
>>> starts using up the 2GB swap until it's full and then the system  
>>> hangs. The system is an 8-core Dell with 16GB of physical memory.  
>>> chr1.fa is ~242MB. All of the disk storage is network mounted on  
>>> an EMC system which (I am told) has a proprietary version of  
>>> something that's NFS-like.
>>>
>>> I loaded chrM (~17kb) and load_seqdatabase grew to over 4 GB  
>>> before it completed.
>>>
>>> I am using MySQL 5.0.51a-community, DBI 1.607, perl 5.85, bioperl  
>>> 1.6.0, bioperl-db 1.006900.
>>>
>>> I have the innodb engine enabled in MySQL and the buffers and  
>>> caches set for a 'large' system.
>>>
>>> I had some errors during the bioperl-db install:
>>>
>>> Test Summary Report
>>> -------------------
>>> t/01dbadaptor.t (Wstat: 256 Tests: 23 Failed: 1)
>>> Failed test:  23
>>> Non-zero exit status: 1
>>> t/10ensembl.t   (Wstat: 65280 Tests: 5 Failed: 0)
>>> Non-zero exit status: 255
>>> Parse errors: Bad plan.  You planned 18 tests but ran 5.
>>> t/11locuslink.t (Wstat: 65280 Tests: 7 Failed: 0)
>>> Non-zero exit status: 255
>>> Parse errors: Bad plan.  You planned 113 tests but ran 7.
>>> t/15cluster.t   (Wstat: 65280 Tests: 7 Failed: 0)
>>> Non-zero exit status: 255
>>> Parse errors: Bad plan.  You planned 162 tests but ran 7.
>>> Files=16, Tests=1205, 21 wallclock secs ( 0.51 usr  0.12 sys +  
>>> 14.90 cusr  2.69 csys = 18.22 CPU)
>>> Result: FAIL
>>> Failed 4/16 test programs. 1/1205 subtests failed.
>>>
>>> The most recent bioperl-db documentation says that a workable  
>>> version may be possible after some errors and so I went ahead with  
>>> the install.
>>>
>>> The mailing list archive has some discussion about throughput but  
>>> nothing really about filling up memory.
>>>
>>> Can anyone offer any clues about what's going on or where to start  
>>> looking?
>>>
>>> Thanks
>>>
>>> Mike
>>> _______________________________________________
>>> Bioperl-l mailing list
>>> Bioperl-l at lists.open-bio.org
>>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>>
>> -- 
>> ===========================================================
>> : Hilmar Lapp  -:-  Durham, NC  -:-  hlapp at gmx dot net :
>> ===========================================================
>>
>>
>>
>
> Michael Muratet
> mmuratet at hudsonalpha.org
>
>

-- 
===========================================================
: Hilmar Lapp  -:-  Durham, NC  -:-  hlapp at gmx dot net :
===========================================================