[Biopython] Bio.Entrez.epost error with Python 3.2

[Nicolas Joannin]

Hi Michiel,

Thank you for your help!
I'm still a bit confused about how to work around this problem...
I tried adding a post=False, and different variants of this, in the epost
call, but I still get the same error. So, I'm guessing that it is actually
in the __init__.py file that I should modify it, but I don't know if that
is what you are suggesting or not (considering your second email).

I am actually trying to retrieve a very large dataset from Genbank (all
records for a specific species, e.g. ~90000 records). I initially tried
with a simple esearch+usehistory and then using efetch to retrieve the
results in batches of 500 records, but I encountered a strange error in xml
format (that actually didn't break the process) with this information:
"<ERROR>Unable to obtain query #1</ERROR>".

Googling the problem, I found this thread:
http://comments.gmane.org/gmane.comp.python.bio.general/6962

Their solution involves using epost, which brings us to my initial question
:)

I'd really appreciate if you could let me know:

   1. How best to work around the epost problem...
   2. If you know of any alternate solution to the "Unable to obtain query
   #1" error

Thanks in advance!
Best regards,
Nicolas





Nicolas Joannin, Ph.D.
Bioinformatics Center
Kyoto University, Uji campus, Japan



On Sun, Jan 27, 2013 at 1:41 PM, Michiel de Hoon <mjldehoon at yahoo.com>wrote:

> Looking at this some more, I found this on the mailing list explaining why
> we are using post=True:
>
> http://lists.open-bio.org/pipermail/biopython/2009-May/005152.html
>
> This page provides some explanation on urllib.parse.urlencode in Python3:
>
> http://docs.python.org/3/library/urllib.request.html#urllib-examples
>
> Best,
> -Michiel.
>
> --- On Sat, 1/26/13, Michiel de Hoon <mjldehoon at yahoo.com> wrote:
>
> > From: Michiel de Hoon <mjldehoon at yahoo.com>
> > Subject: Re: [Biopython] Bio.Entrez.epost error with Python 3.2
> > To: "Biopython Mailing List" <biopython at lists.open-bio.org>, "Nicolas
> Joannin" <nicolas.joannin at gmail.com>
> > Date: Saturday, January 26, 2013, 11:06 PM
> > For some reason Entrez.epost switched
> > to post=False to post=True in the call to _open in
> > Bio.Entrez. I don't know why; the other Entrez functions all
> > use post=False, and if I use post=False in Entrez.epost it
> > seems to work fine. The switch from post=False to post=True
> > was made in this commit; Peter, do you remember why we
> > switched to post=True?
> >
> >
> https://github.com/biopython/biopython/commit/c928057b6c811c8959daf806ee6159eb09e0928f#Bio/Entrez/__init__.py
> >
> > Best,
> > -Michiel.
> >
> > --- On Sat, 1/26/13, Nicolas Joannin <nicolas.joannin at gmail.com>
> > wrote:
> >
> > > From: Nicolas Joannin <nicolas.joannin at gmail.com>
> > > Subject: [Biopython] Bio.Entrez.epost error with Python
> > 3.2
> > > To: "Biopython Mailing List" <biopython at lists.open-bio.org>
> > > Date: Saturday, January 26, 2013, 7:42 AM
> > > Hello everyone,
> > >
> > > I am having trouble with using Bio.Entrez.epost: see
> > details
> > > below.
> > > I have tried converting the variables to bytes, but
> > > everything I've tried
> > > gives the same error message.
> > >
> > > I'm guessing that this might be a bug in biopython when
> > used
> > > with Python 3.
> > > If not, could you please tell me where I got this
> > wrong, and
> > > how I can fix
> > > it?
> > >
> > > Best regards,
> > > Nicolas
> > >
> > >
> > > Here are the details:
> > > I have tried the following:
> > >
> > >    -
> > > post_h=Entrez.epost("nuccore",id="160418,160351")
> > >    -
> > > post_h=Entrez.epost(b"nuccore",id=b"160418,160351")
> > >    -
> > >
> >
>    post_h=Entrez.epost("nuccore".encode("utf-8"),id="160418,160351".encode("utf-8")
> > >
> > >
> > > >>>
> > > post_h=Entrez.epost("nuccore",id="160418,160351")
> > > Traceback (most recent call last):
> > >   File "<stdin>", line 1, in <module>
> > >   File
> > >
> >
> "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/Bio/Entrez/__init__.py",
> > > line 97, in epost
> > >     return _open(cgi, variables, post=True)
> > >   File
> > >
> >
> "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/Bio/Entrez/__init__.py",
> > > line 436, in _open
> > >     handle = urllib.request.urlopen(cgi,
> > > data=options)
> > >   File
> > >
> >
> "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/urllib/request.py",
> > > line 138, in urlopen
> > >     return opener.open(url, data, timeout)
> > >   File
> > >
> >
> "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/urllib/request.py",
> > > line 367, in open
> > >     req = meth(req)
> > >   File
> > >
> >
> "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/urllib/request.py",
> > > line 1066, in do_request_
> > >     raise TypeError("POST data should be bytes"
> > > TypeError: POST data should be bytes or an iterable of
> > > bytes. It cannot be
> > > str.
> > >
> > >
> > > Nicolas Joannin, Ph.D.
> > > Bioinformatics Center
> > > Kyoto University, Uji campus, Japan
> > > _______________________________________________
> > > Biopython mailing list  -  Biopython at lists.open-bio.org
> > > http://lists.open-bio.org/mailman/listinfo/biopython
> > >
> >
> > _______________________________________________
> > Biopython mailing list  -  Biopython at lists.open-bio.org
> > http://lists.open-bio.org/mailman/listinfo/biopython
> >
>