[Bioperl-l] Next-gen modules

[Sendu Bala]

Chris Fields wrote:
> On Jun 17, 2009, at 5:10 PM, Sendu Bala wrote:
 >
>>> I'm personally wondering if this could be done as a sequence 
>>> database, something similar in theme to Lincoln's SeqFeature::Store, 
>>> but sequence only, and returns quality objects in a similar manner 
>>> (ala Storable)?  Not sure whether that's feasible, but it's appears 
>>> at least scalable.
>>
>> I think not. Well, at least SeqFeature::Store doesn't scale. Try 
>> storing millions of features in a database and watch it crawl to 
>> complete unusability. I can't imagine a db scaling to holding hundreds 
>> of TB of data either. I'm also not sure what the benefit is. There are 
>> already high-speed ways of indexing your fastq or bam files.
> 
> Interesting that you ran into issues with SF::Store; wonder if object 
> storage is the limiting factor there, or if it is something else.

Object storage certainly was an issue, which is why I patched it to 
(optionally) not store objects. That helped a great deal, but ultimately 
only increased the number of features you could store before it slowed 
down; it didn't solve the problem completely.