[Bioperl-l] Next-gen modules
Elia Stupka
e.stupka at ucl.ac.uk
Wed Jun 17 18:01:28 UTC 2009
If we reach a consensus on how/who/what, I will be happy to contribute
some coding time in the coming days.
Would it be a good starting point to start adding the different
formats as named in BioPython, and test support for reading/wrting
them? I could start playing with that.
regards,
Elia
On 17 Jun 2009, at 18:52, Chris Fields wrote:
> I think this is a top priority for a fall BioPerl release, maybe
> 1.6.2 (I am planning on a summer 1.6.1 release still). Made it into
> a bug report for tracking:
>
> http://bugzilla.open-bio.org/show_bug.cgi?id=2857
>
> If no one works on this I may take it up after the 1.6.1 release.
>
> chris
>
> On Jun 17, 2009, at 12:13 PM, Mark A. Jensen wrote:
>
>> I'm on the case! (but maybe not in realtime, today!)
>>
>> ----- Original Message ----- From: "Chris Fields" <cjfields at illinois.edu
>> >
>> To: "Peter" <biopython at maubp.freeserve.co.uk>
>> Cc: "BioPerl List" <bioperl-l at lists.open-bio.org>; "Elia Stupka" <e.stupka at ucl.ac.uk
>> >; "Heikki Lehvaslaiho" <heikki at sanbi.ac.za>
>> Sent: Wednesday, June 17, 2009 1:06 PM
>> Subject: Re: [Bioperl-l] Next-gen modules
>>
>>
>>>
>>> On Jun 17, 2009, at 8:25 AM, Peter wrote:
>>>
>>>> On Wed, Jun 17, 2009 at 1:57 PM, Chris
>>>> Fields<cjfields at illinois.edu> wrote:
>>>>>
>>>>> Elia,
>>>>>
>>>>> As Mark indicated, we recently discussed the lack of support
>>>>> for next-gen on
>>>>> list, at least re: fastq. I may be hit with the same thing in
>>>>> a few months
>>>>> time myself, and I recall Jason and a few others also
>>>>> mentioning the same.
>>>>> Heikki wrote some code for Illumina FASTQ for SeqIO and related
>>>>> modules but
>>>>> I don't believe it has been committed to trunk yet, so maybe he
>>>>> can answer.
>>>>>
>>>>> From prior discussions IIRC the issues were:
>>>>>
>>>>> 1) distinguishing the various FASTQ versions (Sanger, Illumina
>>>>> 1.0, Illumina
>>>>> 1.3) from one another (so maybe some optional validation), and
>>>>
>>>> Following the python rule of thumb for being explicit, Biopython
>>>> makes
>>>> the user specify which FASTQ variant is being used. I don't think
>>>> you
>>>> can do anything else. Any attempted validation would have to be
>>>> heuristic based on the ASCII characters found, and would risk false
>>>> positive warnings.
>>>
>>> Right; I'm thinking along the same lines. If anything the most
>>> we would allow is some level of validation, so if there were a
>>> degree of uncertainty about the format one could set a validation
>>> flag to check bounds during the parse and warn if they are
>>> exceeded.
>>>
>>>>> 2) having a way for the Seq object to either 'know' what format is
>>>>> contained, or we use phred score and convert back and forth
>>>>> from that (I
>>>>> think the latter makes more sense).
>>>>
>>>> I think it could make sense for BioPerl to convert Solexa scores
>>>> to/ from
>>>> PHRED scores on the fly (especially now that Illumina is abandoning
>>>> the Solexa score system). Python style tries to avoid implicit
>>>> conversions,
>>>> so Biopython doesn't automatically do a conversion from Solexa to
>>>> PHRED scores on parsing (but will on writing if the requested
>>>> output
>>>> format requires this).
>>>>
>>>>> Peter's suggestions also are reasonable, though does biopython
>>>>> have a
>>>>> separate module for each of these variations? Our version (I
>>>>> believe)
>>>>> mainly varied the conversion within Bio::SeqIO::fastq itself
>>>>> based on the
>>>>> fastq variant passed in as a separate named argument.
>>>>
>>>> Biopython's SeqIO gives the three FASTQ variants their own unique
>>>> names. This format name is a required argument for parsing/writing
>>>> (we don't try and guess the file format from the data contents).
>>>> Internally
>>>> we have three separate FASTQ parsers/writers although they do share
>>>> code.
>>>
>>> We could easily do the same if others agree. Actually, if we
>>> specified that shorthand for a variant on a format would be
>>> designated as -format => 'format-variant', I think we could
>>> easily hack SeqIO to deal with that by splitting on '-' and
>>> passing everything to the constructor as (-format => 'format', -
>>> variant => 'variant'). Very little repeated code in this case,
>>> just an additional named parameter indicating the format variant
>>> (and the SeqIO class can do the type checking on that within the
>>> constructor).
>>>
>>>> Other issues to keep in mind:
>>>>
>>>> (3) There should be no warning parsing files where the optional
>>>> repeated
>>>> title is missing on the "+" lines (as discussed earlier on the
>>>> BioPerl list).
>>>
>>> Agreed, though we'll have to check the current fastq parser to see
>>> if that's currently the case. I thought that was fixed but maybe
>>> not?
>>>
>>>> (4) When writing FASTQ files should BioPerl omit the optional
>>>> repeated
>>>> title on the "+" line? Biopython omits this as I understand this
>>>> to be
>>>> common practice, and can make a big different to file sizes -
>>>> especially
>>>> on short read data from Solexa/Illumina.
>>>
>>> Agreed, particularly if it's commonly encountered.
>>>
>>>> (5) Also test reading and writing files with an optional
>>>> description (as well
>>>> as an identifier) on the "@" (and "+") lines. See the NCBI SRA
>>>> for examples,
>>>> e.g.
>>>>
>>>> @SRR001666.1 071112_SLXA-EAS1_s_7:5:1:817:345 length=36
>>>> GGGTGATGGCCGCTGCCGATGGCGTCAAATCCCACC
>>>> +SRR001666.1 071112_SLXA-EAS1_s_7:5:1:817:345 length=36
>>>> IIIIIIIIIIIIIIIIIIIIIIIIIIIIII9IG9IC
>>>
>>> Should be easy enough to implement with a simple regex.
>>>
>>>> (6) Test reading and writing files where the encoded quality
>>>> string starts
>>>> with a "@" or a "+" character, e.g.
>>>> http://lists.open-bio.org/pipermail/bioperl-l/2009-May/029911.html
>>>>
>>>> Peter
>>>
>>> Mark, getting all that? ;>
>>>
>>> chris
>>>
>>>
>>>
>>> _______________________________________________
>>> Bioperl-l mailing list
>>> Bioperl-l at lists.open-bio.org
>>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>>>
>>
>> _______________________________________________
>> Bioperl-l mailing list
>> Bioperl-l at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>
---
Senior Lecturer, Bioinformatics
UCL Cancer Institute
Paul O' Gorman Building
University College London
Gower Street
WC1E 6BT
London
UK
Office (UCL): +44 207 679 6493
Office (ICMS): +44 0207 8822374
Mobile: +44 7597 566 194
Mobile (Italy): +39 338 8448801
More information about the Bioperl-l
mailing list