[Bioperl-l] Blast temporary open files not closed
Hilmar Lapp
hlapp at gmx.net
Sun Sep 10 13:40:46 UTC 2006
On Sep 10, 2006, at 8:19 AM, ende wrote:
> El 10/09/2006, a las 5:59, Hilmar Lapp escribió:
>
>> Your code snippet leaves plenty of opportunities for temp files
>> accumulating, depending on what you do with the report object. So
>> long as you use the report object the associated result file will
>> remain present and open.
>>
>
> It sounds possible.. I have thought it previously... but I returns
> a local copy of the result of $blastMachine->blastall(..); not the
> $blastMachine at all. How it could be possible the original object
> could get caught with that copy?
Never assume inner workings to be as you think they ought to be. In
this case it is not the original object but the file because it is
parsed on-demand. I.e., by the time the function returns the file has
not been fully parsed yet and therefore remains open.
>
>> You will have to provide the calling code as well if you want
>> someone to look at the problem.
>>
>
>
> From the main object I call this DoBlast
>
> ########################
> # #
> # ANALIZE EACH SEQ #
> # #
> ########################
> #
> for (my $i=0; $i < $self->{inSeqArrTotal}; ++$i) {
> my $s = ${$self->{inSeqArr}}[$i];
> my $ID = $s->id();
> local *x = \$self->{SQ}{$ID}; # short for ref curr seq
>
>
> $x->{seq} = $s;
>
>
> # search for cloning vector seqs. in all the query seqs.
> #
> my $blastTemp = $self->DoBlastSeq($self->
> {cloningVectorsFilename}, $s);
>
>
> and then use this $blastTemp and save it in another place of the
> object:
>
> $x->{blastcv} = $blastTemp;
Well this is still not the complete story but sounds like you're
storing all the report objects which will have the files remain open.
-hilmar
>
> ...
>
>
>
>> On Sep 9, 2006, at 7:50 PM, ende wrote:
>>
>>>
>>> Thank all of you for your quick and precise answer!!!
>>>
>>> My bioperl version must be 1.20 :(
>>>
>>> I can read it in the file Bio::Perl.pm since
>>>
>>> perl -MCPAN -e shell
>>> and then
>>> i Bio::Perl
>>>
>>> does not inform me about any local version but says instead:
>>>
>>> trange distribution name [Bio:Perl]
>>> Module id = Bio::Perl
>>> CPAN_USERID BIRNEY (Ewan Birney <birney at sanger.ac.uk>)
>>> CPAN_VERSION undef
>>> CPAN_FILE B/BI/BIRNEY/bioperl-1.4.tar.gz
>>> UPLOAD_DATE 2003-12-23
>>> DSLIP_STATUS (,,,,)
>>> MANPAGE Bio::Perl - Functional access to BioPerl for
>>> people
>>> who don't know objects
>>> INST_FILE /Library/Perl/5.8.6/Bio/Perl.pm
>>> INST_VERSION undef
>>>
>>> I read 1.20 at top of /Library/5.8.6/Bio/Perl.pm file!
>>>
>>> After many (many) attempts of installing Bio::Perl via -MCPAN always
>>> it ends with the same message (of course using force!!):
>>>
>>>
>>> Failed 3/25 tests, 88.00% okay
>>> t/WABA.......................ok
>>> t/XEMBL_DB...................SOAP::Lite and/or XML::DOM not
>>> installed. This means that Bio::DB::XEMBL module is not usable.
>>> Skipping tests.
>>> t/XEMBL_DB...................ok
>>> Failed Test Stat Wstat Total Fail Failed List of Failed
>>> --------------------------------------------------------------------
>>> ----
>>> -------
>>> t/BioFetch_DB.t 27 1 3.70% 8
>>> t/EMBL_DB.t 15 3 20.00% 6 13-14
>>> t/Ontology.t 255 65280 50 100 200.00% 1-50
>>> t/TreeIO.t 41 1 2.44% 42
>>> t/Variation_IO.t 25 3 12.00% 15 20 25
>>> t/simpleGOparser.t 255 65280 98 196 200.00% 1-98
>>> 121 subtests skipped.
>>> Failed 6/179 test scripts, 96.65% okay. 154/8273 subtests failed,
>>> 98.14% okay.
>>> make: *** [test_dynamic] Error 2
>>> /usr/bin/make test -- NOT OK
>>> Running make install
>>> make test had returned bad status, won't install without force
>>> Failed during this command:
>>> BIRNEY/bioperl-1.4.tar.gz : make_test NO
>>>
>>>
>>> ..and I am using "force" (exactly:)
>>>
>>> cpan> force install B/BI/BIRNEY/bioperl-1.4.tar.gz
>>>
>>>
>>> I have installed many other modules without no problems but this
>>> seems to reject me. Finally I have dowloaded the bioperl
>>> 1.5.1 .tar.gz and installed ignoring the many errors make test gave
>>> me. sudo make install and wow!!! the bioperl version changed and
>>>
>>> lsof +p pidofperl | grep stat | wc -l
>>>
>>> again grows during the execution until reach 250 (!) and crashes.
>>>
>>> It was also imposible from CPAN shell to obtain the local bioperl
>>> version (as was also impossible in a remote Linux installation, that
>>> also said) INST_VERSION undef.
>>>
>>> But now the Bio/Perl.pm file heads:
>>>
>>> # Perl.pm,v 1.23.2.1 2005/10/09 15:16:18 jason Exp
>>> #
>>> # BioPerl module for Bio::Perl
>>>
>>>
>>>
>>>
>>>
>>> (I am using 10.4.7 on the MacOSX)
>>>
>>> the code launch blast all for each seq inside a perl object (use of
>>> $blastMachine->io->_io_cleanup(); did not resolve the problem)
>>>
>>>> sub DoBlastSeq ($$$$) {
>>>> my ($self, $db, $seq, $outFileName) = @_;
>>>> my %params = (
>>>> program => "blastn",
>>>> outfile => $outFileName,
>>>> database => "$self->{path_db}/$db",
>>>> # q => "-5",
>>>> G => 3, # si
>>>> E => 3, # si
>>>> F => "\"m D\"", # "mD", # si
>>>> e => 700, # si
>>>> Y => 1.75e12, # si
>>>> best => 1,
>>>> );
>>>> if (!$params{outfile}) {
>>>> delete $params{outfile};
>>>> }
>>>> my $outErrs = ($outFileName || "blastErrs.err"),
>>>>
>>>>
>>>> my $blastMachine = Bio::Tools::Run::StandAloneBlast->new(%
>>>> params);
>>>>
>>>> open(OLDSTDERR, ">&", \*STDERR) or die "Can't dup STDERR: $!";
>>>> open(STDERR, ">", $outErrs ) or die "ERROR reopening STDERR:
>>>> $!";
>>>>
>>>> print "Running Blast with id: ", $seq->id(), "\n" if $DEBUG;
>>>>
>>>> my $blastResult = $blastMachine->blastall($seq);
>>>>
>>>>
>>>> close(STDERR);
>>>> open(STDERR, ">&", \*OLDSTDERR) or die "Can't dup OLDSTDERR:
>>>> $!";
>>>> unlink ($outErrs) if (-z $outErrs);
>>>>
>>>> return $blastResult;
>>>> }
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> El 09/09/2006, a las 15:45, Brian Osborne escribió:
>>>
>>>> Juan,
>>>>
>>>> I recall a bug like this was fixed a while back - what version
>>>> Bioperl are
>>>> you using? By the way, always give version numbers when reporting a
>>>> bug, the
>>>> answer "already fixed" is very common.
>>>>
>>>> Brian O.
>>>>
>>>>
>>>> On 9/9/06 4:31 AM, "ende" <mmacho at gmail.com> wrote:
>>>>
>>>>>
>>>>> Processing a fasta file with about 500 dna seqs.. my MacOSX (that
>>>>> has the max number of opened files up to 512) crashes... You
>>>>> need to
>>>>> divide the problem in pieces or (in bash shell, with ulimit -n
>>>>> 1024)
>>>>> augment that max number of opened files.
>>>>>
>>>>> This has no sense for me since my perl program nor leave any open
>>>>> file without its corresponding closing. On the other side, the
>>>>> problem arises when the number of dnas grows _in one file_.
>>>>>
>>>>> In the code I run blast (StandAloneBlast... $blastMachine-
>>>>> >blastall)
>>>>> for each seq.
>>>>>
>>>>>
>>>>> Then sniffing int the perl program stopped perl program I
>>>>> confirmed
>>>>> my suspects. BioPerl (StandAloneBlast) does not closes temporary
>>>>> opened files. Those files seems to be created to save seqs for
>>>>> to be
>>>>> then processed by blastall program... The output of lsof
>>>>> indicates
>>>>> (as MacOSX System Monitor) that those files are left opened but
>>>>> not
>>>>> there (!?)
>>>>>
>>>>> The output of lsof +p pidofperlprogram
>>>>>
>>>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>>>>> perl 21977 juanfc cwd VDIR 14,7 238 6835993 /Users/
>>>>> juanfc/Documents/programperl 21977 juanfc txt VREG 14,7
>>>>> 19280 1589055 /usr/bin/perl
>>>>> perl 21977 juanfc txt VREG 14,7 23476 1580272 /System/
>>>>> Library/Perl/5.8.6/darwin-thread-multi-2level/auto/IO/IO.bundle
>>>>> perl 21977 juanfc txt VREG 14,7 17772 1580263 /System/
>>>>> Library/Perl/5.8.6/darwin-thread-multi-2level/auto/Fcntl/
>>>>> Fcntl.bundle
>>>>> perl 21977 juanfc txt VREG 14,7 114116 1580381 /System/
>>>>> Library/Perl/5.8.6/darwin-thread-multi-2level/auto/POSIX/
>>>>> POSIX.bundle
>>>>> perl 21977 juanfc txt VREG 14,7 23684 1580265 /System/
>>>>> Library/Perl/5.8.6/darwin-thread-multi-2level/auto/File/Glob/
>>>>> Glob.bundle
>>>>> perl 21977 juanfc txt VREG 14,7 1797788 6275687 /usr/
>>>>> lib/
>>>>> dyld
>>>>> perl 21977 juanfc txt VREG 14,7 4379472 6276030 /usr/
>>>>> lib/
>>>>> libSystem.B.dylib
>>>>> perl 21977 juanfc txt VREG 14,7 1086420 6276221 /System/
>>>>> Library/Perl/5.8.6/darwin-thread-multi-2level/CORE/libperl.dylib
>>>>> perl 21977 juanfc 0u VCHR 4,2 0t3748 63113092 /dev/
>>>>> ttyp2
>>>>> perl 21977 juanfc 1u VCHR 4,2 0t3748 63113092 /dev/
>>>>> ttyp2
>>>>> perl 21977 juanfc 2u VCHR 4,2 0t3748 63113092 /dev/
>>>>> ttyp2
>>>>> perl 21977 juanfc 3u VCHR 4,2 0t3748 63113092 /dev/
>>>>> ttyp2
>>>>> perl 21977 juanfc 4r VREG stat(/
>>>>> ...
>>>>> Please, help.
>>>>>
>>>>>
>>>>> --
>>>>> Juan Falgueras
>>>>> Profesor del Depto. de Lenguajes y Ciencias de la Computación
>>>>> Universidad de Málaga
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Bioperl-l mailing list
>>>>> Bioperl-l at lists.open-bio.org
>>>>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>>>>
>>>>
>>>
>>>
>>>
>>> ---- ende
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Bioperl-l mailing list
>>> Bioperl-l at lists.open-bio.org
>>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>>>
>>
>> --
>> ===========================================================
>> : Hilmar Lapp -:- Durham, NC -:- hlapp at gmx dot net :
>> ===========================================================
>>
>>
>>
>>
>>
>
>
>
> ---- ende
>
>
>
--
===========================================================
: Hilmar Lapp -:- Durham, NC -:- hlapp at gmx dot net :
===========================================================
More information about the Bioperl-l
mailing list