[Bioperl-l] Asking for advice on full EMBL extraction
Jason Stajich
jason at bioperl.org
Thu May 7 17:54:39 EDT 2009
Russell -
I am not sure how that will help as only 1 sequence is parsed at a
time by SeqIO parsers and they use the "//" delimiter.
If the equivalent data exists in genbank format at NCBI I think
_that_ module (Bio::SeqIO::genbank) has the ability to ignore
annotations/features. Really we have to re-work the whole thing to be
more lightweight and lazy-parse.
-jason
On May 7, 2009, at 2:24 PM, Smithies, Russell wrote:
> I'm not sure if this will help with your problem or how it deals
> with memory management but using "ordinary" Perl to split the large
> EMBL file might work.
> Give this a go:
>
> ============================
> #!perl -w
>
> use Bio::SeqIO;
> use IO::String;
>
> use constant SEP => "//\n";
>
> open($fh, "gunzip -c rel_ann_mus_01_r99.dat.gz |") or die;
>
> my $index = 1;
>
> while(my $stringfh = new IO::String(get_next_record($fh))){
>
> my $seqio = Bio::SeqIO->new( -fh => $stringfh,-format =>
> "EMBL" ) or die $!;
>
> while ( my $seq_object = $seqio->next_seq ) {
> print "Dealing with entry: ".$index++."\t".$seq_object->id."\n";
>
> # show the features
> for my $feat_object ($seq_object->get_SeqFeatures) {
> print "primary tag: ", $feat_object->primary_tag, "\n";
> for my $tag ($feat_object->get_all_tags) {
> print " tag: ", $tag, "\n";
> for my $value ($feat_object->get_tag_values($tag)) {
> print " value: ", $value, "\n";
> }
> }
> }
> }
>
> }
>
>
> sub get_next_record{
> my($fh) = @_;
> (my $old_sep,$/) = ($/,SEP);
> my $record = <$fh>;
> $/ = $old_sep;
> return $record;
> }
> ========================================
>
>
> --Russell
>
>
>> -----Original Message-----
>> From: bioperl-l-bounces at lists.open-bio.org [mailto:bioperl-l-
>> bounces at lists.open-bio.org] On Behalf Of brian li
>> Sent: Friday, 8 May 2009 1:00 a.m.
>> To: Chris Fields
>> Cc: bioperl-l at lists.open-bio.org
>> Subject: Re: [Bioperl-l] Asking for advice on full EMBL extraction
>>
>> My server has 32 GB RAM.
>>
>> The os of my server is 64-bit version of Ubuntu Server Edition 8.04
>> LTS. And I have run my example code on another server with 32-bit
>> version of Ubuntu Server Edition 8.04 and 4 GB RAM. Segfault again.
>>
>> -Brian
>>
>> On Thu, May 7, 2009 at 8:07 PM, Chris Fields
>> <cjfields at illinois.edu> wrote:
>>> I noticed that Russell has 16GB RAM on his setup. Was yours
>>> equivalent?
>>>
>>> chris
>>>
>>> On May 7, 2009, at 12:32 AM, brian li wrote:
>>>
>>>> Thank you very much for your offer.
>>>>
>>>> The director of our lab wants me to do the extraction every time
>>>> a new
>>>> release of EMBL is published. I can't push the task to you every
>>>> time.
>>>>
>>>> I can offer more information of the server I run my script on if
>>>> needed.
>>>>
>>>> -Brian
>>>>
>>>> On Thu, May 7, 2009 at 1:01 PM, Smithies, Russell
>>>> <Russell.Smithies at agresearch.co.nz> wrote:
>>>>>
>>>>> Sadly, that's the same code as I ran but I had a Data::Dump in the
>>>>> middle.
>>>>> Versions of Perl and BioPerl are the same.
>>>>> We're running RHEL 5 (kernel 2.6.18-92.1.18.el5) with 16GB RAM
>>>>>
>>>>> If you get a full script running on a smaller dataset, I could
>>>>> probably
>>>>> run it on the bigger stuff and give you back tab-separated (or
>>>>> is that
>>>>> tab\tseparated ?) data for loading into your db.
>>>>>
>>>>> --Russell
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: brian li [mailto:brianli.cas at gmail.com]
>>>>>> Sent: Thursday, 7 May 2009 4:50 p.m.
>>>>>> To: Smithies, Russell
>>>>>> Cc: bioperl-l at lists.open-bio.org
>>>>>> Subject: Re: [Bioperl-l] Asking for advice on full EMBL
>>>>>> extraction
>>>>>>
>>>>>> Dear Russell,
>>>>>>
>>>>>> My example code is as following. I omit the parse process and
>>>>>> these
>>>>>> lines give me "Segmentation Fault" too.
>>>>>>
>>>>>> # Start of code
>>>>>> my $seqio = Bio::SeqIO->new(-file => 'rel_ann_mus_01_r99.dat',
>>>>>> -format => 'EMBL');
>>>>>> my $index = 1;
>>>>>> while (my $seq = $seqio->next_seq)
>>>>>> {
>>>>>> print "Dealing with entry: $index\n";
>>>>>> $index++;
>>>>>> }
>>>>>> # End
>>>>>>
>>>>>> The platform I run this code on:
>>>>>> BioPerl 1.6.0
>>>>>> Perl 5.8.8
>>>>>> Ubuntu 8.04 LTS Server 64-bit version (Linux 2.6.24-23-server)
>>>>>>
>>>>>> I have monitored the memory usage when I run the code above.
>>>>>> There is
>>>>>> always around 20GB free memory (buffer size counted in) left.
>>>>>> So I
>>>>>> suppose the segfault can't be explained just by memory shortage.
>>>>>>
>>>>>> Brian
>>>>>>
>>>>>>
>>>>>> On Thu, May 7, 2009 at 11:32 AM, Smithies, Russell
>>>>>> <Russell.Smithies at agresearch.co.nz> wrote:
>>>>>>>
>>>>>>> Hi Brian,
>>>>>>> I hate to say it but it worked OK for me using
>>>>>>> rel_ann_mus_01_r99.dat.gz and
>>>>>>
>>>>>> simple example Bio::SeqIO code from bugzilla
>>>>>>>
>>>>>>> It's not using more than 1GB memory on our server and doesn't
>>>>>>> segfault.
>>>>>>>
>>>>>>> Send me your example code and I'll give it a go if you like.
>>>>>>>
>>>>>>>
>>>>>>> Russell Smithies
>>>>>>>
>>>>>>> Bioinformatics Applications Developer
>>>>>>> T +64 3 489 9085
>>>>>>> E russell.smithies at agresearch.co.nz
>>>>>>>
>>>>>>> Invermay Research Centre
>>>>>>> Puddle Alley,
>>>>>>> Mosgiel,
>>>>>>> New Zealand
>>>>>>> T +64 3 489 3809
>>>>>>> F +64 3 489 9174
>>>>>>> www.agresearch.co.nz
>>>>>>>
>>>>>>>
>>>>> =
>>>>> =
>>>>> =
>>>>> =
>>>>> =
>>>>> ==================================================================
>>>>> Attention: The information contained in this message and/or
>>>>> attachments
>>>>> from AgResearch Limited is intended only for the persons or
>>>>> entities
>>>>> to which it is addressed and may contain confidential and/or
>>>>> privileged
>>>>> material. Any review, retransmission, dissemination or other use
>>>>> of, or
>>>>> taking of any action in reliance upon, this information by
>>>>> persons or
>>>>> entities other than the intended recipients is prohibited by
>>>>> AgResearch
>>>>> Limited. If you have received this message in error, please
>>>>> notify the
>>>>> sender immediately.
>>>>> =
>>>>> =
>>>>> =
>>>>> =
>>>>> =
>>>>> ==================================================================
>>>>>
>>>>
>>>> _______________________________________________
>>>> Bioperl-l mailing list
>>>> Bioperl-l at lists.open-bio.org
>>>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>>>
>>>
>>
>> _______________________________________________
>> Bioperl-l mailing list
>> Bioperl-l at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>
> _______________________________________________
> Bioperl-l mailing list
> Bioperl-l at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/bioperl-l
Jason Stajich
jason at bioperl.org
More information about the Bioperl-l
mailing list