[BioPython] Bio.Blast.NCBIWWW.blast returns before search is complete

[Noah Hoffman]

Hello biopython list -

This is my first attempt to use the biopython tools, so I thought I'd
check with the mailing list first before submitting it as a bug to see
if I'm missing something obvious (I wouldn't be surprised!). I'm
running Biopython version 1.30 on mac os 10.3.8, python version 2.3.

Just doing a simple blast search, following the example in the
cookbook. The job is dispatched properly, and often (but not always)
returns the "search pending" message rather than the search results
itself. I believe that I've found the code in Bio/Blast/NCBIWWW.py
responsible for the failure to wait for the results to be ready on the
NCBI end - it actually looks like the python code is correct, and that
the server is supplying misleading data (held in the "comment"
variable below).

Code snippets from the module with my debugging code (flagged by ####
NH) follows:

% cat -n /System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/Bio/Blast/NCBIWWW.py
| grep -A3 -B3 "#### NH"

   769      while 1:
   770          # pause before trying to get the results
   771
   772          print '(line %s) pausing: refresh_delay = %s' %
(line_no(), refresh_delay)  #### NH
   773
   774          time.sleep(refresh_delay)
   775
--
   783          ready, results_cgi, results_params =
_parse_blast_results_page(handle)
   784
   785          print '(line %s) checking search status; ready = %s' \
   786          % (line_no(), ready) #### NH
   787
   788          results_params['FORMAT_TYPE']=format_type
   789          results_params['NCBI_GI']='yes'
--
   869              # //--></SCRIPT>
   870              comment = string.lower(comment)
   871
   872              print '(line %s) comment:\n--->%s<---' %
(line_no(),comment.strip()) #### NH
   873
   874              if string.find(comment, 'status=ready') >= 0:
   875                  self.ready = 1
--
   982          outhandle.write(data)
   983      sock.close()
   984
   985  def line_no(): #### NH
   986          import inspect
   987          try: raise AttributeError
   988          except AttributeError: return inspect.stack()[1][2]

... and the output resulting from the dispatch of two sequence files,
445.fasta and 457.fasta:

searching 445.fasta against the nr database using tblastn
(line 772) pausing: refresh_delay = 56
(line 872) comment:
--->qblastinfobegin
        status=waiting
qblastinfoend<---
(line 872) comment:
--->settimeout('document.forms[0].submit();',11000);
//<---
(line 785) checking search status; ready = 0
(line 772) pausing: refresh_delay = 56
(line 872) comment:
--->qblastinfobegin
        status=waiting
qblastinfoend<---
(line 872) comment:
--->settimeout('document.forms[0].submit();',12000);
//<---
(line 785) checking search status; ready = 0
(line 772) pausing: refresh_delay = 56
(line 872) comment:
--->qblastinfobegin
        status=ready
qblastinfoend<---
(line 785) checking search status; ready = 1
writing 445_tblastn.html

searching 457.fasta against the nr database using tblastn
(line 772) pausing: refresh_delay = 39
(line 872) comment:
--->qblastinfobegin
        status=waiting
qblastinfoend<---
(line 872) comment:
--->settimeout('document.forms[0].submit();',17000);
//<---
(line 785) checking search status; ready = 0
(line 772) pausing: refresh_delay = 39
(line 872) comment:
--->qblastinfobegin
        status=waiting
qblastinfoend<---
(line 872) comment:
--->settimeout('document.forms[0].submit();',17000);
//<---
(line 785) checking search status; ready = 0
(line 772) pausing: refresh_delay = 39
(line 872) comment:
--->qblastinfobegin
        status=ready
qblastinfoend<---
(line 785) checking search status; ready = 1
writing 457_tblastn.html

% du -h *tblastn.html
4.0k    445_tblastn.html
28k     457_tblastn.html

As you can see from the file sizes, only the search using 457.fasta
returned actual results, despite the fact that the while loop seemed
to end appropriately once status=ready appeared in the comment
variable.

Anyone have any ideas? Is this simply erratic behavior by the NCBI
server that we can not hope to have any control over? Is this a known
problem that has been addressed in the CVS repository?

Thanks a lot,
Noah