[BioPython] Fwd: NCBI Abuse Activity with BioPython

[Renato Alves]

you mean 3 seconds no?

Quoting Chris Fields on 06/25/2008 04:34 PM:
> Just as a note from the BioPerl side, BioPerl modules which access 
> eutils use the 3 min sleep rule, and we specify in the documentation 
> the NCBI rules.  The modules also identify the tool/agent used as 
> 'bioperl', I believe.
>
> chris
>
> On Jun 25, 2008, at 10:08 AM, Chris Dagdigian wrote:
>
>>
>> Can someone from the biopython dev team respond officially to Scott 
>> please?
>>
>> Regards,
>> Chris
>>
>>
>> Begin forwarded message:
>>
>>> From: "Mcginnis, Scott (NIH/NLM/NCBI) [E]" <mcginnis at ncbi.nlm.nih.gov>
>>> Date: June 25, 2008 10:54:28 AM EDT
>>> To: <biopython-owner at lists.open-bio.org>
>>> Subject: NCBI Abuse Activity with BioPython
>>>
>>> Dear Colleague:
>>>
>>>
>>>
>>> My name is Scott McGinnis and I am responsible for monitoring the web
>>> page at NCBI and blocking users with excessive access.
>>>
>>>
>>>
>>> I am seeing more and more activity with BioPython and it is us concern.
>>> Mainly the BioPython suite does not appear to be written to the
>>> recommendations made on the main NCBI E-utilities web page
>>> (http://eutils.ncbi.nlm.nih.gov/entrez/query/static/eutils_help.html).Pr 
>>>
>>> inciply the following are not being done by BioPython tools.
>>>
>>>
>>>
>>> *  Send E-utilities requests to http://eutils.ncbi.nlm.nih.gov
>>> <http://eutils.ncbi.nlm.nih.gov/> , not the standard NCBI Web address.
>>>
>>> *  Make no more than one request every 3 seconds.
>>>
>>>
>>>
>>> In fact I recently cc'd you on an event when a user was coming in at
>>> over 18 requests per second. We really wish that you would alter you
>>> scripts to run with a some sort of sleep in it in order to not send
>>> requests more than once per 3 seconds and to not send these to the main
>>> www web servers but use the  http://eutils.ncbi.nlm.nih.gov
>>> <http://eutils.ncbi.nlm.nih.gov/> .
>>>
>>>
>>>
>>> Also, there is the problem of huge searches in order to build local
>>> databases. With you package it seems that if one were so inclined you
>>> would send a search for all human sequences (over 10,000,000 sequences)
>>> and you program would then retrieve these one ID at a time. Regardless
>>> of the fact that this is an extreme example, we would much prefer if
>>> your program could webenv from the Esearch  and  use the search history
>>> and webenv to retrieve sets of sequences at 200 - 200 at a time.
>>>
>>>
>>>
>>> History: Requests utility to maintain results in user's environment.
>>> Used in conjunction with WebEnv.
>>>
>>> usehistory=y
>>>
>>> Web Environment: Value previously returned in XML results from ESearch
>>> or EPost. This value may change with each utility call. If WebEnv is
>>> used, History search numbers can be included in an ESummary URL, e.g.,
>>> term=cancer+AND+%23X (where %23 replaces # and X is the History search
>>> number).
>>>
>>> Note: WebEnv is similar to the cookie that is set on a user's computers
>>> when accessing PubMed on the web.  If the parameter usehistory=y is
>>> included in an ESearch URL both a WebEnv (cookie string) and query_key
>>> (history number) values will be returned in the results. Rather than
>>> using the retrieved PMIDs in an ESummary or EFetch URL you may simply
>>> use the WebEnv and query_key values to retrieve the records. WebEnv 
>>> will
>>> change for each ESearch query, but a sample URL would be as follows:
>>>
>>> http://www.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pubmed
>>> &WebEnv=%3D%5DzU%5D%3FIJIj%3CC%5E%5DA%3CT%5DEACgdn%3DF%5E%3Eh
>>> GFA%5D%3CIFKGCbQkA%5E_hDFiFd%5C%3D
>>> &query_key=6&retmode=html&rettype=medline&retmax=15
>>>
>>> WebEnv=WgHmIcDG]B etc.
>>>
>>> Display Numbers:
>>>
>>> retstart=x  (x= sequential number of the first record retrieved -
>>> default=0 which will retrieve the first record)
>>> retmax=y  (y= number of items retrieved)
>>>
>>>
>>>
>>> Otherwise we will end up blocking more of your users which we are
>>> unfortunately already doing in some cases.
>>>
>>>
>>>
>>> Sincerely,
>>> Scott D. McGinnis, M.S.
>>> DHHS/NIH/NLM/NCBI
>>> www.ncbi.nlm.nih.gov
>>>
>>>
>>>
>>
>> _______________________________________________
>> BioPython mailing list  -  BioPython at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/biopython
>
> Christopher Fields
> Postdoctoral Researcher
> Lab of Dr. Marie-Claude Hofmann
> College of Veterinary Medicine
> University of Illinois Urbana-Champaign
>
>
>
>
> _______________________________________________
> BioPython mailing list  -  BioPython at lists.open-bio.org
> http://lists.open-bio.org/mailman/listinfo/biopython