[Biopython] help with ncbiWWW

[Peter Cock]

On Wed, Jul 26, 2017 at 4:58 PM, Nabeel Ahmed
<chaudhrynabeelahmed at gmail.com> wrote:
>>
>> > Suggestion 3: Make direct API calls using 'requests' package.
>> > In case the API calls are simple (you can easily do so) use request to
>> > make
>> > a call, with timeout flag, once the HTTP request will timeout it'll
>> > raise
>> > Timeout exception, which you can catch and in that block make the second
>> > call (which as per you, works perfectly fine)
>>
>>
>> This is essentially the idea I was initially suggesting, but the problem
>> isn't actually in the online request (currently done by urlopen).
>> With the NCBI BLAST you typically submit a query, wait, check for
>> progress, wait (repeat), and then download the results. This loop in
>> Biopython has no timeout - it relies on the NCBI returning results
>> eventually - or giving an error.
>
>
>  Yh, what you're saying is, the response in qblast call depends on lot of
> factors, and varies from a job to job i.e.
> data size, NCBI servers' response time, etc.
> Given this, there isn't any point in having a timeout param for this call.
>
> But, in case in case, it's required, can patch qblast with a new param
> 'timeout', and I have looked into the source code (for Python 2.x) it's
> using urlopen from urllib2 package (line 172)
> This urlopen allows timeout as an argument.
> Can have an optional param - timeout=0 for qblast, and pass it to urlopen:
>
> handle = _urlopen(request, timeout=timeout)  - Line 132 and 176

Good point - I'd forgotten we were using urllib2.urlopen on Python 2.
That might be worth doing, but doubt it would solve Pej's problem.

Rather we need probably need a time out on this loop:

https://github.com/biopython/biopython/blob/biopython-170/Bio/Blast/NCBIWWW.py#L160

i.e. Record the time of the initial query and check the total elapsed
wait time does not exceed a total wait limit (like 5 or 10 minutes).

Peter