[Biopython] File format autodetection.

[Jan Kim]

On Tue, Jun 24, 2014 at 02:00:53PM -0400, Ivan Gregoretti wrote:
> Indeed, the STDIN stream is the challenge. That is why I though that
> the question was worth documenting in the Biopython list.
> 
> Would anybody mind showing how peekline() is used? I tried using it on
> a SeqIO.parse generator but I get an error:
> 
> AttributeError: 'generator' object has no attribute 'peekline'
> 
> I am using Biopython 1.61 and Python 2.7.3 on linux 64bit.

I haven't tried but my general understanding is that UndoHandle
replaces the file handle in the typical Python "duck typing" way.
So I'd try something like

# p = Bio.SeqIO.parse(f, 'fastq')
u = Bio.File.UndoHandle(f)
firstLine = u.peekline()
if firstLine[0] == '@' :
  p = Bio.SeqIO.parse(u, 'fastq')
elif firstLine[0] == '>' :
  p = Bio.SeqIO.parse(u, 'fasta')
else
  raise StandardError, 'cannot auto-identify sequence format'
for s in p :
  # process the record...

Another hack may be to pipe the input through EMBOSS's seqret application,
that's pretty good at auto-detection.

Personally, I'd prefer that solution now thinking of it, as using a tool
made for a task (automagic sequence format detection and conversion in
this case) is better than writing yet another ad-hoc heuristic.

And finally, I'd also encourage you to educate those responsible for
outputting sequences in a format that's somewhat out of control (if at
all possible).

Best regards & good luck, Jan

> Thank you,
> 
> Ivan
> 
> 
> 
> 
> Ivan Gregoretti, PhD
> Bioinformatics
> 
> 
> On Tue, Jun 24, 2014 at 1:41 PM, Fields, Christopher J
> <cjfields at illinois.edu> wrote:
> > On Jun 24, 2014, at 11:54 AM, Peter Cock <p.j.a.cock at googlemail.com> wrote:
> >
> >> Hi Ivan,
> >>
> >> Biopython's SeqIO does not (and will not) do automatic file
> >> format detection, it is just too hard to get right so instead
> >> that's the user's task:
> >>
> >> Zen of Python: Explicit is better than implicit.
> >> http://legacy.python.org/dev/peps/pep-0020/
> >>
> >> (BioPerl's SeqIO can do format guessing)
> >
> > (somewhat)
> >
> > You are welcome to try it, but Bio::Tools::GuessSeqFormat is IMHO one of the misbegotten step-children of Bioperl; if you delve into it, you???ll find it also tries to guess whether something is a sequence or an alignment file.  My general feeling is that if you don???t know the source of your data (and from that the format) then there is only so much we can do to help.  Doing so from STDIN is even trickier.
> >
> > So, it???s there, it works in most cases so we keep it around, but caveat emptor.  We really don???t really maintain that module any more than very routine bugs fixes.
> >
> >> Your use case is one which highlights a technical reason
> >> why this is hard - you are using stdin, a read-once handle.
> >> You cannot peek at the file, guess the format, seek back to
> >> the beginning, and then give the handle to a specific parser.
> >>
> >> You could use Biopython's UndoHandle here, but it will
> >> impose a (modest) performance overhead.
> >>
> >> from Bio.File import UndoHandle
> >> help(UndoHandle)
> >>
> >> e.g. Use the .peekline() method to spot FASTA vs FASTQ?
> >>
> >> Peter
> >
> > That seems like a pretty reasonable option.
> >
> > chris
> >
> >> On Tue, Jun 24, 2014 at 5:16 PM, Ivan Gregoretti <ivangreg at gmail.com> wrote:
> >>> Hello Biopythoneers,
> >>>
> >>> The question:
> >>>
> >>> What is the strategy currently used for file format autodetection?
> >>>
> >>>
> >>> The context:
> >>>
> >>> I have written a command line program that gets a stream of FASTQ data
> >>> and reports how many records are contained. You can visualise it like
> >>> this
> >>>
> >>> zcat myfile.fq.gz | fxcounttags.py -i /dev/stdin -o /dev/stdout > myfile.counts
> >>>
> >>> That works fine for FASTQ but I need to extend the functionality to
> >>> FASTA streams. How would you write fxcounttags.py to detect
> >>> FASTQ/FASTA?
> >>>
> >>> Thank you,
> >>>
> >>> Ivan
> >>>
> >>>
> >>>
> >>> Ivan Gregoretti, PhD
> >>> Bioinformatics
> >>> _______________________________________________
> >>> Biopython mailing list  -  Biopython at mailman.open-bio.org
> >>> http://mailman.open-bio.org/mailman/listinfo/biopython
> >> _______________________________________________
> >> Biopython mailing list  -  Biopython at mailman.open-bio.org
> >> http://mailman.open-bio.org/mailman/listinfo/biopython
> >
> 
> _______________________________________________
> Biopython mailing list  -  Biopython at mailman.open-bio.org
> http://mailman.open-bio.org/mailman/listinfo/biopython

-- 
 +- Jan T. Kim -------------------------------------------------------+
 |             email: jttkim at gmail.com                                |
 |             WWW:   http://www.jtkim.dreamhosters.com/              |
 *-----=<  hierarchical systems are for files, not for humans  >=-----*