From krishnaroskin at gmail.com Tue Apr 1 02:41:04 2014 From: krishnaroskin at gmail.com (Krishna Roskin) Date: Mon, 31 Mar 2014 23:41:04 -0700 Subject: [Biopython-dev] interest module for sequence based clustering tools? Message-ID: Hey all, Long time fan, taking my first crack at contributing. I've built a basic module to run and parse the result of sequence based clustering tools such as DNACLUST and CD-HIT. I've written subclasses of AbstractCommandline to run dnaclust and cd-hit. I've also written classes to store the clusters and their members and loaders for the output formats used by those programs. I posting here to gauge interest and get some feedback and maybe some beta testers. My code is available at: https://github.com/krishnaroskin/biopython.git under the seqcluster branch. I've started writing some test code at: Tests/seqcluster/test_seqcluster.py that also severs as example code. I've pasted that at the end of this message so people can get an idea of how it works without having to checkout code. If there is interest, I'm planning on adding a seqclust.cluster function that takes a list of SeqRecords and returns a clustering using one of the supported tools. I envision that function being the main interface to this module. I also want to write a something that will map the cluster membership (given by ids) back to collections of SeqRecords. Other to-dos: Test that all the flavors of CD-HIT work (there are many) Add support for other sequence based clustering tools (suggestions?) Documentation Tutorial Test code -krish #!/usr/bin/env python from __future__ import print_function import StringIO from Bio.seqcluster.applications import DNAClustCommandline from Bio.seqcluster import DNAClustIterator from Bio.seqcluster.applications import CDHITCommandline from Bio.seqcluster import CDHITClustIterator cmd = DNAClustCommandline(similarity=0.8, header=True, threads=2, inputfile= "test_sequences.fasta") stdout, stderr = cmd() clusters = DNAClustIterator(StringIO.StringIO(stdout)) for cluster in clusters: print(cluster.name) for member in cluster: if member == cluster.representative: print("\t" + member.name + "*") else: print("\t" + member.name) print() # blank line cmd = CDHITCommandline(cutoff=0.8, threads=2, inputfile= "test_sequences.fasta", outputfile="tmp") stdout, stderr = cmd() clusters = CDHITClustIterator(open("tmp.clstr", "r")) for cluster in clusters: print(cluster.name) for member in cluster: if member == cluster.representative: print("\t" + member.name + "*") else: print("\t" + member.name) From tra at popgen.net Tue Apr 1 06:46:20 2014 From: tra at popgen.net (Tiago Antao) Date: Tue, 1 Apr 2014 11:46:20 +0100 Subject: [Biopython-dev] Fwd: [Numpy-discussion] EuroSciPy 2014 Call for Abstracts In-Reply-To: References: <20140320114815.70210fd5@grandao> <87txathtux.fsf@chupacabra.windows.ebi.ac.uk> Message-ID: <20140401114620.67490302@lnx> Hi, On Mon, 31 Mar 2014 17:30:21 +0100 Peter Cock wrote: > I'm glad to see there are several people interested in EuroSciPy. > One one of you like to submit a Biopython talk? The deadline > is 14 April: https://www.euroscipy.org/2014/calls/abstracts/ I know I have fuelled all this, but I will only know next week if I am in the US or not. I could offer to do the presentation, if I am around. In any case, I still believe it might be a good geographical place for some Biopython developers to meet. Brandon: We can talk face-to-face on Thursday, as we will be in the same place ;) Tiago From p.j.a.cock at googlemail.com Tue Apr 1 09:08:45 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 1 Apr 2014 14:08:45 +0100 Subject: [Biopython-dev] Fwd: [Numpy-discussion] EuroSciPy 2014 Call for Abstracts In-Reply-To: <20140401114620.67490302@lnx> References: <20140320114815.70210fd5@grandao> <87txathtux.fsf@chupacabra.windows.ebi.ac.uk> <20140401114620.67490302@lnx> Message-ID: On Tue, Apr 1, 2014 at 11:46 AM, Tiago Antao wrote: > Hi, > > On Mon, 31 Mar 2014 17:30:21 +0100 > Peter Cock wrote: > >> I'm glad to see there are several people interested in EuroSciPy. >> One one of you like to submit a Biopython talk? The deadline >> is 14 April: https://www.euroscipy.org/2014/calls/abstracts/ > > > I know I have fuelled all this, but I will only know next week if I am > in the US or not. I could offer to do the presentation, if I am around. > In any case, I still believe it might be a good geographical place for > some Biopython developers to meet. > > Brandon: We can talk face-to-face on Thursday, as we will be in the > same place ;) > > Tiago OK, let's sort this out next week then - this week is the BOSC deadline (4 April), and Bow has volunteered to present the Biopython talk. Thanks, Peter From p.j.a.cock at googlemail.com Tue Apr 1 17:37:45 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 1 Apr 2014 22:37:45 +0100 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: References: Message-ID: On Mon, Mar 31, 2014 at 10:08 PM, Wibowo Arindrarto wrote: > Hi Peter, everyone, > > A LaTeX template would be great :). I'm still preparing the abstract, > should be ready for everyone to check soon. > > Cheers, > Bow LaTeX template sent. I will be travelling the next few days and may be largely out of email contact - so if you don't hear from me, don't worry and submit the abstract anyway. I'm confident the others will have good advice and suggestions (and I'm not the only one who does email list moderation - but it might be simpler to send round plain text and a link to the PDF if you want to show it and not worry about the attachment triggering moderation). As usual this will get reviewed by the BOSC panel, and you'll have a chance to revise the abstract at that point - for the project updates is it OK if the initial text has a couple of details "to be confirmed", like a summary of the forthcoming release Biopython 1.64 (time for another thread, maybe we can aim for the end of this month?). Thanks Bow, Peter From clements at galaxyproject.org Wed Apr 2 16:06:56 2014 From: clements at galaxyproject.org (Dave Clements) Date: Wed, 2 Apr 2014 13:06:56 -0700 Subject: [Biopython-dev] Talk Abstracts Due April 4: 2014 Galaxy Community Conference (GCC2014) Message-ID: Hello all, This is a reminder that talk abstract submissions for the 2014 Galaxy Community Conference *are due this Friday, April 4*. See http://bit.ly/gcc2014abs for details. This is a great opportunity to share your work in high-throughput biology. Accepted abstracts will be considered for the *GigaScience * "Galaxy: Data Intensive and Reproducible Research " series (announced for GCC2013)*. * BGI is also continuing to cover the article processing charges until the end of the year Poster abstracts are being accepted until April 25. Early registration (save a bundle, really) is also open. Thanks, and we do hope you'll join us in Baltimore! Dave C, on behalf of the GCC2014 Organizing Committee On Sat, Feb 22, 2014 at 3:21 PM, Dave Clements wrote: > *2014 Galaxy Community Conference (GCC2014) > * > http://galaxyproject.org/GCC2014 > June 30 - July 2, 2014 > Homewood Campus > > Johns Hopkins University > Baltimore, Maryland , United > States > > ------ > The *2014 Galaxy Community Conference *(*GCC2014*, > http://galaxyproject.org/GCC2014) features two full days of > presentations, discussions, poster sessions, lightning talks and > birds-of-a-feather, *all about data-intensive biology and the tools that > support it*. GCC2014 also includes a Training Day with > five concurrent tracks and in-depth coverage of thirteen different topics. > > GCC2014 will be held at the Homewood > Campus > of Johns Hopkins University , in Baltimore, Maryland, > United States, from June 30 through July 2, 2014. > > Galaxy is an easily extensible data > integration and analysis platform for life sciences research that supports > hundreds of bioinformatics analysis tools. Galaxy is open-source and can be > locally installed or run on the cloud. There are hundreds of local > installs, and over 50 publicly accessible serversaround the world. > > *Early registration > * is now open. > Early combined registration (Training Day + > main meeting) starts at $140 for post-docs and students. Registration is > capped this year at 250 participants, *and we expect to hit that limit*. > Registering early assures you a place at the conference and also a spot in > the Training Day workshops you want to attend. > > You can also book affordable conference housing at the same time you > register. See the conference Logistics page for > details on this and other housing options. > > *Abstract submission > * for both oral > presentations and posters is also open. Abstract submission for oral > presentations closes April 4, and poster submission closes April 25. The *GigaScience > * "Galaxy: Data Intensive and > Reproducible Research " > series (announced for GCC2013) *is continuing to take submissions for > this year's meeting and beyond. * BGI is also continuing to cover the > article processing charges until the end of the year, and for more > information see their latest update > . > > Thanks, and hope to see you in Baltimore! > > The GCC2014 Organizing Committee > -- http://galaxyproject.org/GCC2014 http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ https://wiki.galaxyproject.org/ From w.arindrarto at gmail.com Wed Apr 2 17:54:21 2014 From: w.arindrarto at gmail.com (Wibowo Arindrarto) Date: Wed, 2 Apr 2014 23:54:21 +0200 Subject: [Biopython-dev] Fwd: [Open-bio-l] Proposed BLAST XML Changes In-Reply-To: References: Message-ID: Dear all, Regarding the update on querying SearchIO hits with multiple keys, I've just added some (not so) small changes to the submodule here: https://github.com/biopython/biopython/pull/307. The proposed changes affect our current SearchIO core object model, so I suppose it's better if more than one person take a look at this first :). Cheers, Bow On Tue, Mar 18, 2014 at 12:48 PM, Wibowo Arindrarto wrote: > On Tue, Mar 18, 2014 at 11:58 AM, Peter Cock wrote: >> On Tue, Mar 18, 2014 at 10:33 AM, Wibowo Arindrarto >> wrote: >>> On Tue, Mar 18, 2014 at 11:17 AM, Peter Cock wrote: >>>> On Tue, Mar 18, 2014 at 9:52 AM, Wibowo Arindrarto >>>> wrote: >>>>> Hi Peter, everyone, >>>>> >>>>> Thanks for the heads up. If implemented as it is, the updates will >>>>> change our underlying SearchIO model (aside from the blast-xml parser >>>>> itself), by allowing a Hit retrieval using multiple different keys. >>>> >>>> Could you clarify what you mean by multiple keys here? >>> >>> Currently, we can retrieve hits from a query using its ID, aside from >>> its numeric index. With their proposed changes to the Hit element >>> here: ftp://ftp.ncbi.nlm.nih.gov/blast/documents/NEWXML/ProposedBLASTXMLChanges.pdf, >>> it means that a given Hit can now be annotated with more than one ID. >> >> But this happens already in the current output from merged entries >> in databases like NR - we effectively use the first alternative ID as >> the hit ID. See for example the nasty > separated entries in >> the legacy BLAST XML's tag where only the first ID >> appears in the tag: >> >> http://blastedbio.blogspot.co.uk/2012/05/blast-tabular-missing-descriptions.html >> >> See also the new optional fields in the tabular output which >> explicitly list all the aliases for the merge record (e.g. sallseqid). > > In the BLAST outputs, yes. However, there's no explicit support yet in > SearchIO for this. Currently we only parse whatever is in as > the ID and as the description. If the tag has is > separated by semicolons / has more than one IDs, the current parser > does not try to split it into multiple IDs. Instead it takes the whole > string as the ID. > > Also, in the blast tabular format, even though sallseqid is parsed, > it's merely stored as an attribute of the hit object, not something > that can be used to retrieve Hits from the QueryResult object. > >>> Ideally, this should also be reflected in the QueryResult object: a >>> hit item should be retrievable using any of the IDs it has. >>> >>> This will also affect membership checking on the QueryResult object. >> >> This looks like something we should review anyway, regardless >> of the new BLAST XML format. > > Of course :). > >>>>> I have a feeling it will be difficult to jam all the new changes into >>>>> a backwards-compatible parser. One way to make it transparent to users >>>>> is to use the underlying DTD to do validation before parsing (for the >>>>> two BLAST DTDs, use the one which the file can be validated against). >>>>> However, this comes at a price. Since the standard library-bundled >>>>> elementtree doesn't seem to support validation, we have to use another >>>>> library (lxml is my choice). This means adding 3rd party dependency >>>>> which require compiling (lxml is also partly written in C). >>>> >>>> We can probably tell by sniffing the first few lines... but how >>>> to do that without using a handle seek to rewind may be >>>> tricky (desirable to support parsing streams, e.g. stdin). >>> >>> Ah yes. We have a rewindable file seek object in Bio.File, don't we >>> :)? I'll have to play around with some real datasets first, I think. >> >> Yes, the UndoHandle in Bio.File might be the best solution >> here for auto-detection. But two explicit formats is probably better. >> >>> The other thing we should take into account is the Xinclude tag. Would >>> we want to make it possible to query *either* the single query XML >>> results or the master Xinclude document (point 2 of the proposed >>> change)? Or should we restrict our parser only to the single query >>> files? >> >> I think single files is a reasonable restriction... assuming BLAST >> will still have the option of producing a big multi-query XML? >> Probably we should ask the NCBI about that... > > In a way, the Xinclude file is the file containing multi-query XML. I > have a feeling that if Xinclude is proposed, producing multi-output > BLAST XML files will not be an option anymore (otherwise it seems > redundant). But yes, NCBI should has more info about this. > >> I would hope the Bio.SearchIO.index_db(...) approach could >> be used on a colloection of little XML files, one for each query. >> >>>>> The other option is to introduce a new format name (e.g. >>>>> 'blast-xml2'), which makes the user responsible for knowing which >>>>> BLAST XML he/she is parsing. It feels more explicit this way, so I am >>>>> leaning towards this option, despite 'blast-xml2' not sounding very >>>>> nice to me ;). >>>>> >>>>> Any other thoughts? >>>>> >>>>> Best, >>>>> Bow >>>> >>>> I agree for the SearchIO interface, two format names makes >>>> sense - unless there is a neat way to auto-detect this on input. >>>> >>>> Using "blast-xml2" would work, or maybe something like >>>> "blast-xml-2014" (too long?). >>>> >>>> We could even go for "blast-xml-old" and "blast-xml" perhaps? >>> >>> Hmm..'blast-xml-old', may make it difficult to adapt for future XML >>> schema changes. How about renaming the current parser to >>> 'blast-xml-legacy', and the new one to just 'blast-xml'? >> >> A possible downside of 'blast-xml-legacy' over 'blast-xml-old' >> is this may be confused with the "legacy" BLAST in C to the >> current BLAST+ in C++ move (which happened well before >> this XML format change). > > Hmm. In this case then I am leaning to 'blast-xml2', I think. It's the > shortest and most future-proof (subsequent changes to the XML format > could be denoted as 'blast-xml3'). But it does make it slightly > inconsistent with the names we have for HMMER (i.e. 'hmmer2-text' is > for HMMER version 2 text output, 'hmmer3-text' is for HMMER version 3 > text output). > > Cheers, > Bow From w.arindrarto at gmail.com Thu Apr 3 03:06:29 2014 From: w.arindrarto at gmail.com (Wibowo Arindrarto) Date: Thu, 3 Apr 2014 09:06:29 +0200 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: References: Message-ID: Hi everyone, I'm forwarding the previous message to the mailing list now. I just realized that my message did not get through yesterday due to the attachment :(. Apologies for the quite late check, should've checked this earlier. I have now reuploaded our abstract here: http://bit.ly/biopython_bosc_2014. As noted below, please let me know if there are additions / deletions you would like me to make before the abstract is submitted. Best, Bow On Wed, Apr 2, 2014 at 11:52 AM, Wibowo Arindrarto wrote: > Hello everyone, > > I've finished the first draft of our update (attached). Please let me > know if there are additions / deletions you would like me to make > before I submit the abstract. > > Also, some things I would like to note: > > * I am missing the formal affiliations of everybody in the authors > list except for me and Peter. Could you please send me your most > recent affiliation? > > * I'm not sure about whether to include Tiago's recent Biopython > docker container (I think it would be nice). Tiago, do we have the > container up and running for our tests? > > Cheers, > Bow > > On Tue, Apr 1, 2014 at 11:37 PM, Peter Cock wrote: >> On Mon, Mar 31, 2014 at 10:08 PM, Wibowo Arindrarto >> wrote: >>> Hi Peter, everyone, >>> >>> A LaTeX template would be great :). I'm still preparing the abstract, >>> should be ready for everyone to check soon. >>> >>> Cheers, >>> Bow >> >> LaTeX template sent. >> >> I will be travelling the next few days and may be largely out of >> email contact - so if you don't hear from me, don't worry and >> submit the abstract anyway. I'm confident the others will have >> good advice and suggestions (and I'm not the only one who >> does email list moderation - but it might be simpler to send >> round plain text and a link to the PDF if you want to show it >> and not worry about the attachment triggering moderation). >> >> As usual this will get reviewed by the BOSC panel, and you'll >> have a chance to revise the abstract at that point - for the >> project updates is it OK if the initial text has a couple of details >> "to be confirmed", like a summary of the forthcoming release >> Biopython 1.64 (time for another thread, maybe we can aim >> for the end of this month?). >> >> Thanks Bow, >> >> Peter From eric.talevich at gmail.com Thu Apr 3 18:18:01 2014 From: eric.talevich at gmail.com (Eric Talevich) Date: Thu, 3 Apr 2014 15:18:01 -0700 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: References: Message-ID: Hi Bow, The abstract looks good. It's great to see you're carrying the torch this year. In the second paragraph, the sentence beginning "New features since version 1.62" confused me. Should this be "New features *in* version 1.62"? My recollection is that the listed features were actually introduced after 1.63, in which case that sentence would move to the end of the paragraph as "new features since version *1.63*". I'm looking at: http://news.open-bio.org/news/2013/08/biopython-1-62-released/ http://news.open-bio.org/news/2013/12/biopython-1-63-released/ Maybe you could streamline that paragraph to say that since BOSC 2013 Biopython has successfully supported Python 2 and Python 3 (and Pypy and Jython 2.7) with a single codebase, and leave the details of the transition for the talk. Then describe the other changes/enhancements. Also, in the same paragraph: "various enhancements" could be swapped for "other significant enhancements". Cheers, Eric On Thu, Apr 3, 2014 at 12:06 AM, Wibowo Arindrarto wrote: > Hi everyone, > > I'm forwarding the previous message to the mailing list now. I just > realized that my message did not get through yesterday due to the > attachment :(. Apologies for the quite late check, should've checked > this earlier. > > I have now reuploaded our abstract here: http://bit.ly/biopython_bosc_2014 > . > > As noted below, please let me know if there are additions / deletions > you would like me to make before the abstract is submitted. > > Best, > Bow > > On Wed, Apr 2, 2014 at 11:52 AM, Wibowo Arindrarto > wrote: > > Hello everyone, > > > > I've finished the first draft of our update (attached). Please let me > > know if there are additions / deletions you would like me to make > > before I submit the abstract. > > > > Also, some things I would like to note: > > > > * I am missing the formal affiliations of everybody in the authors > > list except for me and Peter. Could you please send me your most > > recent affiliation? > > > > * I'm not sure about whether to include Tiago's recent Biopython > > docker container (I think it would be nice). Tiago, do we have the > > container up and running for our tests? > > > > Cheers, > > Bow > > > > On Tue, Apr 1, 2014 at 11:37 PM, Peter Cock > wrote: > >> On Mon, Mar 31, 2014 at 10:08 PM, Wibowo Arindrarto > >> wrote: > >>> Hi Peter, everyone, > >>> > >>> A LaTeX template would be great :). I'm still preparing the abstract, > >>> should be ready for everyone to check soon. > >>> > >>> Cheers, > >>> Bow > >> > >> LaTeX template sent. > >> > >> I will be travelling the next few days and may be largely out of > >> email contact - so if you don't hear from me, don't worry and > >> submit the abstract anyway. I'm confident the others will have > >> good advice and suggestions (and I'm not the only one who > >> does email list moderation - but it might be simpler to send > >> round plain text and a link to the PDF if you want to show it > >> and not worry about the attachment triggering moderation). > >> > >> As usual this will get reviewed by the BOSC panel, and you'll > >> have a chance to revise the abstract at that point - for the > >> project updates is it OK if the initial text has a couple of details > >> "to be confirmed", like a summary of the forthcoming release > >> Biopython 1.64 (time for another thread, maybe we can aim > >> for the end of this month?). > >> > >> Thanks Bow, > >> > >> Peter > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev > From w.arindrarto at gmail.com Thu Apr 3 20:57:07 2014 From: w.arindrarto at gmail.com (Wibowo Arindrarto) Date: Fri, 4 Apr 2014 02:57:07 +0200 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: References: Message-ID: Hi Eric, Thanks for the feedback. I have updated the abstract and you can access it here: http://bit.ly/biopython_bosc_2014_02. In addition to the changes you suggested, I added Michiel in the authors list as well. May I also ask what I should put as your affiliation :)? I think I only have mine and Peter's at the moment, and I'm not sure if I could get the most recent affiliations by scouring the web ;). Best, Bow On Fri, Apr 4, 2014 at 12:18 AM, Eric Talevich wrote: > Hi Bow, > > The abstract looks good. It's great to see you're carrying the torch this > year. > > In the second paragraph, the sentence beginning "New features since version > 1.62" confused me. Should this be "New features *in* version 1.62"? My > recollection is that the listed features were actually introduced after > 1.63, in which case that sentence would move to the end of the paragraph as > "new features since version *1.63*". I'm looking at: > http://news.open-bio.org/news/2013/08/biopython-1-62-released/ > http://news.open-bio.org/news/2013/12/biopython-1-63-released/ > > Maybe you could streamline that paragraph to say that since BOSC 2013 > Biopython has successfully supported Python 2 and Python 3 (and Pypy and > Jython 2.7) with a single codebase, and leave the details of the transition > for the talk. Then describe the other changes/enhancements. Also, in the > same paragraph: "various enhancements" could be swapped for "other > significant enhancements". > > Cheers, > Eric > > > On Thu, Apr 3, 2014 at 12:06 AM, Wibowo Arindrarto > wrote: >> >> Hi everyone, >> >> I'm forwarding the previous message to the mailing list now. I just >> realized that my message did not get through yesterday due to the >> attachment :(. Apologies for the quite late check, should've checked >> this earlier. >> >> I have now reuploaded our abstract here: >> http://bit.ly/biopython_bosc_2014. >> >> As noted below, please let me know if there are additions / deletions >> you would like me to make before the abstract is submitted. >> >> Best, >> Bow >> >> On Wed, Apr 2, 2014 at 11:52 AM, Wibowo Arindrarto >> wrote: >> > Hello everyone, >> > >> > I've finished the first draft of our update (attached). Please let me >> > know if there are additions / deletions you would like me to make >> > before I submit the abstract. >> > >> > Also, some things I would like to note: >> > >> > * I am missing the formal affiliations of everybody in the authors >> > list except for me and Peter. Could you please send me your most >> > recent affiliation? >> > >> > * I'm not sure about whether to include Tiago's recent Biopython >> > docker container (I think it would be nice). Tiago, do we have the >> > container up and running for our tests? >> > >> > Cheers, >> > Bow >> > >> > On Tue, Apr 1, 2014 at 11:37 PM, Peter Cock >> > wrote: >> >> On Mon, Mar 31, 2014 at 10:08 PM, Wibowo Arindrarto >> >> wrote: >> >>> Hi Peter, everyone, >> >>> >> >>> A LaTeX template would be great :). I'm still preparing the abstract, >> >>> should be ready for everyone to check soon. >> >>> >> >>> Cheers, >> >>> Bow >> >> >> >> LaTeX template sent. >> >> >> >> I will be travelling the next few days and may be largely out of >> >> email contact - so if you don't hear from me, don't worry and >> >> submit the abstract anyway. I'm confident the others will have >> >> good advice and suggestions (and I'm not the only one who >> >> does email list moderation - but it might be simpler to send >> >> round plain text and a link to the PDF if you want to show it >> >> and not worry about the attachment triggering moderation). >> >> >> >> As usual this will get reviewed by the BOSC panel, and you'll >> >> have a chance to revise the abstract at that point - for the >> >> project updates is it OK if the initial text has a couple of details >> >> "to be confirmed", like a summary of the forthcoming release >> >> Biopython 1.64 (time for another thread, maybe we can aim >> >> for the end of this month?). >> >> >> >> Thanks Bow, >> >> >> >> Peter >> _______________________________________________ >> Biopython-dev mailing list >> Biopython-dev at lists.open-bio.org >> http://lists.open-bio.org/mailman/listinfo/biopython-dev > > From w.arindrarto at gmail.com Fri Apr 4 12:05:36 2014 From: w.arindrarto at gmail.com (Wibowo Arindrarto) Date: Fri, 4 Apr 2014 18:05:36 +0200 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: References: Message-ID: Hello everyone, Our abstract have been submitted. You can access the submitted version here: http://bit.ly/biopython_bosc_2014_submitted. There are still some parts we need to update during the abstract review (hopefully including a newly released 1.64 :) ), but they are minimum. Have a nice weekend, Bow On Fri, Apr 4, 2014 at 2:57 AM, Wibowo Arindrarto wrote: > Hi Eric, > > Thanks for the feedback. I have updated the abstract and you can > access it here: http://bit.ly/biopython_bosc_2014_02. In addition to > the changes you suggested, I added Michiel in the authors list as > well. > > May I also ask what I should put as your affiliation :)? I think I > only have mine and Peter's at the moment, and I'm not sure if I could > get the most recent affiliations by scouring the web ;). > > Best, > Bow > > On Fri, Apr 4, 2014 at 12:18 AM, Eric Talevich wrote: >> Hi Bow, >> >> The abstract looks good. It's great to see you're carrying the torch this >> year. >> >> In the second paragraph, the sentence beginning "New features since version >> 1.62" confused me. Should this be "New features *in* version 1.62"? My >> recollection is that the listed features were actually introduced after >> 1.63, in which case that sentence would move to the end of the paragraph as >> "new features since version *1.63*". I'm looking at: >> http://news.open-bio.org/news/2013/08/biopython-1-62-released/ >> http://news.open-bio.org/news/2013/12/biopython-1-63-released/ >> >> Maybe you could streamline that paragraph to say that since BOSC 2013 >> Biopython has successfully supported Python 2 and Python 3 (and Pypy and >> Jython 2.7) with a single codebase, and leave the details of the transition >> for the talk. Then describe the other changes/enhancements. Also, in the >> same paragraph: "various enhancements" could be swapped for "other >> significant enhancements". >> >> Cheers, >> Eric >> >> >> On Thu, Apr 3, 2014 at 12:06 AM, Wibowo Arindrarto >> wrote: >>> >>> Hi everyone, >>> >>> I'm forwarding the previous message to the mailing list now. I just >>> realized that my message did not get through yesterday due to the >>> attachment :(. Apologies for the quite late check, should've checked >>> this earlier. >>> >>> I have now reuploaded our abstract here: >>> http://bit.ly/biopython_bosc_2014. >>> >>> As noted below, please let me know if there are additions / deletions >>> you would like me to make before the abstract is submitted. >>> >>> Best, >>> Bow >>> >>> On Wed, Apr 2, 2014 at 11:52 AM, Wibowo Arindrarto >>> wrote: >>> > Hello everyone, >>> > >>> > I've finished the first draft of our update (attached). Please let me >>> > know if there are additions / deletions you would like me to make >>> > before I submit the abstract. >>> > >>> > Also, some things I would like to note: >>> > >>> > * I am missing the formal affiliations of everybody in the authors >>> > list except for me and Peter. Could you please send me your most >>> > recent affiliation? >>> > >>> > * I'm not sure about whether to include Tiago's recent Biopython >>> > docker container (I think it would be nice). Tiago, do we have the >>> > container up and running for our tests? >>> > >>> > Cheers, >>> > Bow >>> > >>> > On Tue, Apr 1, 2014 at 11:37 PM, Peter Cock >>> > wrote: >>> >> On Mon, Mar 31, 2014 at 10:08 PM, Wibowo Arindrarto >>> >> wrote: >>> >>> Hi Peter, everyone, >>> >>> >>> >>> A LaTeX template would be great :). I'm still preparing the abstract, >>> >>> should be ready for everyone to check soon. >>> >>> >>> >>> Cheers, >>> >>> Bow >>> >> >>> >> LaTeX template sent. >>> >> >>> >> I will be travelling the next few days and may be largely out of >>> >> email contact - so if you don't hear from me, don't worry and >>> >> submit the abstract anyway. I'm confident the others will have >>> >> good advice and suggestions (and I'm not the only one who >>> >> does email list moderation - but it might be simpler to send >>> >> round plain text and a link to the PDF if you want to show it >>> >> and not worry about the attachment triggering moderation). >>> >> >>> >> As usual this will get reviewed by the BOSC panel, and you'll >>> >> have a chance to revise the abstract at that point - for the >>> >> project updates is it OK if the initial text has a couple of details >>> >> "to be confirmed", like a summary of the forthcoming release >>> >> Biopython 1.64 (time for another thread, maybe we can aim >>> >> for the end of this month?). >>> >> >>> >> Thanks Bow, >>> >> >>> >> Peter >>> _______________________________________________ >>> Biopython-dev mailing list >>> Biopython-dev at lists.open-bio.org >>> http://lists.open-bio.org/mailman/listinfo/biopython-dev >> >> From w.arindrarto at gmail.com Wed Apr 2 05:52:07 2014 From: w.arindrarto at gmail.com (Wibowo Arindrarto) Date: Wed, 2 Apr 2014 11:52:07 +0200 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: References: Message-ID: Hello everyone, I've finished the first draft of our update (attached). Please let me know if there are additions / deletions you would like me to make before I submit the abstract. Also, some things I would like to note: * I am missing the formal affiliations of everybody in the authors list except for me and Peter. Could you please send me your most recent affiliation? * I'm not sure about whether to include Tiago's recent Biopython docker container (I think it would be nice). Tiago, do we have the container up and running for our tests? Cheers, Bow On Tue, Apr 1, 2014 at 11:37 PM, Peter Cock wrote: > On Mon, Mar 31, 2014 at 10:08 PM, Wibowo Arindrarto > wrote: >> Hi Peter, everyone, >> >> A LaTeX template would be great :). I'm still preparing the abstract, >> should be ready for everyone to check soon. >> >> Cheers, >> Bow > > LaTeX template sent. > > I will be travelling the next few days and may be largely out of > email contact - so if you don't hear from me, don't worry and > submit the abstract anyway. I'm confident the others will have > good advice and suggestions (and I'm not the only one who > does email list moderation - but it might be simpler to send > round plain text and a link to the PDF if you want to show it > and not worry about the attachment triggering moderation). > > As usual this will get reviewed by the BOSC panel, and you'll > have a chance to revise the abstract at that point - for the > project updates is it OK if the initial text has a couple of details > "to be confirmed", like a summary of the forthcoming release > Biopython 1.64 (time for another thread, maybe we can aim > for the end of this month?). > > Thanks Bow, > > Peter -------------- next part -------------- A non-text attachment was scrubbed... Name: Biopython_BOSC_abstract_2014.pdf Type: application/pdf Size: 258281 bytes Desc: not available URL: From tra at popgen.net Mon Apr 7 06:41:51 2014 From: tra at popgen.net (Tiago Antao) Date: Mon, 7 Apr 2014 11:41:51 +0100 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: References: Message-ID: <20140407114151.23947d6b@lnx> Hi, 1. Shouldn't Peter be the first author on the list? 2. We have 2 containers: 1. a container with all (almost) the dependencies, which can be used to deploy biopython on the fly (for normal usage): https://github.com/tiagoantao/my-containers/blob/master/biopython/Biopython 2. A buildbot test container (that is in used for builbot testing): https://github.com/tiagoantao/my-containers/blob/master/biopython/Biopython-Test 3. My address: Vector Biology Department, Liverpool School of Tropical Medicine, Pembroke Place, Liverpool, United Kingdom On Wed, 2 Apr 2014 11:52:07 +0200 Wibowo Arindrarto wrote: > Hello everyone, > > I've finished the first draft of our update (attached). Please let me > know if there are additions / deletions you would like me to make > before I submit the abstract. > > Also, some things I would like to note: > > * I am missing the formal affiliations of everybody in the authors > list except for me and Peter. Could you please send me your most > recent affiliation? > > * I'm not sure about whether to include Tiago's recent Biopython > docker container (I think it would be nice). Tiago, do we have the > container up and running for our tests? > > Cheers, > Bow > > On Tue, Apr 1, 2014 at 11:37 PM, Peter Cock > wrote: > > On Mon, Mar 31, 2014 at 10:08 PM, Wibowo Arindrarto > > wrote: > >> Hi Peter, everyone, > >> > >> A LaTeX template would be great :). I'm still preparing the > >> abstract, should be ready for everyone to check soon. > >> > >> Cheers, > >> Bow > > > > LaTeX template sent. > > > > I will be travelling the next few days and may be largely out of > > email contact - so if you don't hear from me, don't worry and > > submit the abstract anyway. I'm confident the others will have > > good advice and suggestions (and I'm not the only one who > > does email list moderation - but it might be simpler to send > > round plain text and a link to the PDF if you want to show it > > and not worry about the attachment triggering moderation). > > > > As usual this will get reviewed by the BOSC panel, and you'll > > have a chance to revise the abstract at that point - for the > > project updates is it OK if the initial text has a couple of details > > "to be confirmed", like a summary of the forthcoming release > > Biopython 1.64 (time for another thread, maybe we can aim > > for the end of this month?). > > > > Thanks Bow, > > > > Peter From w.arindrarto at gmail.com Mon Apr 7 07:00:42 2014 From: w.arindrarto at gmail.com (Wibowo Arindrarto) Date: Mon, 7 Apr 2014 13:00:42 +0200 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: <20140407114151.23947d6b@lnx> References: <20140407114151.23947d6b@lnx> Message-ID: Hi Tiago, 1. I have no objections to that. Could we change it post submission? If you noticed, I listed the authors initially alphabetically based on last name. After talking with Eric, I updated the order with the last one submitted. But in general, I'm ok with any ordering so long as we all agree. 2. I didn't have chance to add this prior to submission, unfortunately :/. But adding this would be nice. Is it possible to do this during abstract review? 3. Noted, thanks :). Best, Bow On Mon, Apr 7, 2014 at 12:41 PM, Tiago Antao wrote: > Hi, > > 1. Shouldn't Peter be the first author on the list? > > 2. We have 2 containers: > 1. a container with all (almost) the dependencies, which can be used > to deploy biopython on the fly (for normal usage): > https://github.com/tiagoantao/my-containers/blob/master/biopython/Biopython > 2. A buildbot test container (that is in used for builbot testing): > https://github.com/tiagoantao/my-containers/blob/master/biopython/Biopython-Test > > 3. My address: Vector Biology Department, Liverpool School of Tropical > Medicine, Pembroke Place, Liverpool, United Kingdom > > On Wed, 2 Apr 2014 11:52:07 +0200 > Wibowo Arindrarto wrote: > >> Hello everyone, >> >> I've finished the first draft of our update (attached). Please let me >> know if there are additions / deletions you would like me to make >> before I submit the abstract. >> >> Also, some things I would like to note: >> >> * I am missing the formal affiliations of everybody in the authors >> list except for me and Peter. Could you please send me your most >> recent affiliation? >> >> * I'm not sure about whether to include Tiago's recent Biopython >> docker container (I think it would be nice). Tiago, do we have the >> container up and running for our tests? >> >> Cheers, >> Bow >> >> On Tue, Apr 1, 2014 at 11:37 PM, Peter Cock >> wrote: >> > On Mon, Mar 31, 2014 at 10:08 PM, Wibowo Arindrarto >> > wrote: >> >> Hi Peter, everyone, >> >> >> >> A LaTeX template would be great :). I'm still preparing the >> >> abstract, should be ready for everyone to check soon. >> >> >> >> Cheers, >> >> Bow >> > >> > LaTeX template sent. >> > >> > I will be travelling the next few days and may be largely out of >> > email contact - so if you don't hear from me, don't worry and >> > submit the abstract anyway. I'm confident the others will have >> > good advice and suggestions (and I'm not the only one who >> > does email list moderation - but it might be simpler to send >> > round plain text and a link to the PDF if you want to show it >> > and not worry about the attachment triggering moderation). >> > >> > As usual this will get reviewed by the BOSC panel, and you'll >> > have a chance to revise the abstract at that point - for the >> > project updates is it OK if the initial text has a couple of details >> > "to be confirmed", like a summary of the forthcoming release >> > Biopython 1.64 (time for another thread, maybe we can aim >> > for the end of this month?). >> > >> > Thanks Bow, >> > >> > Peter > From p.j.a.cock at googlemail.com Mon Apr 7 08:48:03 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 7 Apr 2014 13:48:03 +0100 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: References: <20140407114151.23947d6b@lnx> Message-ID: Hi all, Thanks for handling this Bow, and I'm sorry I didn't see the emails while I was away - thank you to Eric, Tiago and anyone else who had commented directly. Regarding (1), authorship: Bow is presenting, so I think he should be first author and corresponding author. But Tiago has raised an important point: Past BOSC talk authorship has been largely unplanned - often just named presenter and anyone directly contributing to the talk slides. Last year I'd just put Peter Cock et al. which was not very explicit. We could and perhaps should use something like "W. Arindrarato, ... and the Biopython Consortium" or "Team" or "Contributors"? (The phrase "Biopython Consortium" is what it says on our PyPy page, but it sounds very formal to me.) Regarding (2), yes - assuming the reviewers don't reject the abstract completely (which would be a shock!), we will have a chance to revise the abstract. This is usually to address reviewers' comments, but for the project update talks it helps ensure that the abstract is as up to date as possible. Regards, Peter On Mon, Apr 7, 2014 at 12:00 PM, Wibowo Arindrarto wrote: > Hi Tiago, > > 1. I have no objections to that. Could we change it post submission? > If you noticed, I listed the authors initially alphabetically based on > last name. After talking with Eric, I updated the order with the last > one submitted. But in general, I'm ok with any ordering so long as we > all agree. > > 2. I didn't have chance to add this prior to submission, unfortunately > :/. But adding this would be nice. Is it possible to do this during > abstract review? > > 3. Noted, thanks :). > > Best, > Bow > > On Mon, Apr 7, 2014 at 12:41 PM, Tiago Antao wrote: >> Hi, >> >> 1. Shouldn't Peter be the first author on the list? >> >> 2. We have 2 containers: >> 1. a container with all (almost) the dependencies, which can be used >> to deploy biopython on the fly (for normal usage): >> https://github.com/tiagoantao/my-containers/blob/master/biopython/Biopython >> 2. A buildbot test container (that is in used for builbot testing): >> https://github.com/tiagoantao/my-containers/blob/master/biopython/Biopython-Test >> >> 3. My address: Vector Biology Department, Liverpool School of Tropical >> Medicine, Pembroke Place, Liverpool, United Kingdom >> >> On Wed, 2 Apr 2014 11:52:07 +0200 >> Wibowo Arindrarto wrote: >> >>> Hello everyone, >>> >>> I've finished the first draft of our update (attached). Please let me >>> know if there are additions / deletions you would like me to make >>> before I submit the abstract. >>> >>> Also, some things I would like to note: >>> >>> * I am missing the formal affiliations of everybody in the authors >>> list except for me and Peter. Could you please send me your most >>> recent affiliation? >>> >>> * I'm not sure about whether to include Tiago's recent Biopython >>> docker container (I think it would be nice). Tiago, do we have the >>> container up and running for our tests? >>> >>> Cheers, >>> Bow >>> >>> On Tue, Apr 1, 2014 at 11:37 PM, Peter Cock >>> wrote: >>> > On Mon, Mar 31, 2014 at 10:08 PM, Wibowo Arindrarto >>> > wrote: >>> >> Hi Peter, everyone, >>> >> >>> >> A LaTeX template would be great :). I'm still preparing the >>> >> abstract, should be ready for everyone to check soon. >>> >> >>> >> Cheers, >>> >> Bow >>> > >>> > LaTeX template sent. >>> > >>> > I will be travelling the next few days and may be largely out of >>> > email contact - so if you don't hear from me, don't worry and >>> > submit the abstract anyway. I'm confident the others will have >>> > good advice and suggestions (and I'm not the only one who >>> > does email list moderation - but it might be simpler to send >>> > round plain text and a link to the PDF if you want to show it >>> > and not worry about the attachment triggering moderation). >>> > >>> > As usual this will get reviewed by the BOSC panel, and you'll >>> > have a chance to revise the abstract at that point - for the >>> > project updates is it OK if the initial text has a couple of details >>> > "to be confirmed", like a summary of the forthcoming release >>> > Biopython 1.64 (time for another thread, maybe we can aim >>> > for the end of this month?). >>> > >>> > Thanks Bow, >>> > >>> > Peter >> From mjldehoon at yahoo.com Tue Apr 8 22:35:54 2014 From: mjldehoon at yahoo.com (Michiel de Hoon) Date: Tue, 8 Apr 2014 19:35:54 -0700 (PDT) Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: Message-ID: <1397010954.15323.YahooMailBasic@web164006.mail.gq1.yahoo.com> -------------------------------------------- On Mon, 4/7/14, Peter Cock wrote: > We could and perhaps should use something > like "W. Arindrarato, ... and the Biopython Consortium" This sounds good to me. Perhaps it is a bit formal, but it also implies that we are serious about Biopython. I would suggest to write consortium without a capital letter though, since it is not an official name. Best, -Michiel. From w.arindrarto at gmail.com Wed Apr 9 04:46:36 2014 From: w.arindrarto at gmail.com (Wibowo Arindrarto) Date: Wed, 9 Apr 2014 10:46:36 +0200 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: References: <20140407114151.23947d6b@lnx> Message-ID: Hi Peter, everyone, > Thanks for handling this Bow, and I'm sorry I didn't see the emails > while I was away - thank you to Eric, Tiago and anyone else who > had commented directly. Happy to help :). > Regarding (1), authorship: Bow is presenting, so I think he should > be first author and corresponding author. But Tiago has raised > an important point: > > Past BOSC talk authorship has been largely unplanned - often > just named presenter and anyone directly contributing to the > talk slides. Last year I'd just put Peter Cock et al. which was not > very explicit. We could and perhaps should use something > like "W. Arindrarato, ... and the Biopython Consortium" or > "Team" or "Contributors"? > > (The phrase "Biopython Consortium" is what it says on our > PyPy page, but it sounds very formal to me.) I have no strong preference for this, but "Biopython developers" seems more appealing and appropriate than "Biopython consortium" I think. If not developers, perhaps "Biopython contributors" is ok. > Regarding (2), yes - assuming the reviewers don't reject the > abstract completely (which would be a shock!), we will have > a chance to revise the abstract. This is usually to address > reviewers' comments, but for the project update talks it helps > ensure that the abstract is as up to date as possible. Ok, I'll post the latest updates on our abstract along the way then. Having it up to date makes sense ~ which reminds me of something Peter mentioned some emails ago: do we have date target on releasing 1.64?. Cheers, Bow From tra at popgen.net Wed Apr 9 04:56:52 2014 From: tra at popgen.net (Tiago Antao) Date: Wed, 9 Apr 2014 09:56:52 +0100 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: References: <20140407114151.23947d6b@lnx> Message-ID: <20140409095652.13ee5258@lnx> On Wed, 9 Apr 2014 10:46:36 +0200 Wibowo Arindrarto wrote: > I have no strong preference for this, but "Biopython developers" > seems more appealing and appropriate than "Biopython consortium" I > think. If not developers, > perhaps "Biopython contributors" is ok. Another opinion that consortium is over-formal. Developers or contributors sounds more appealing for me... From p.j.a.cock at googlemail.com Wed Apr 9 05:02:41 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 9 Apr 2014 10:02:41 +0100 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: <20140409095652.13ee5258@lnx> References: <20140407114151.23947d6b@lnx> <20140409095652.13ee5258@lnx> Message-ID: On Wed, Apr 9, 2014 at 9:56 AM, Tiago Antao wrote: > On Wed, 9 Apr 2014 10:46:36 +0200 > Wibowo Arindrarto wrote: > >> I have no strong preference for this, but "Biopython developers" >> seems more appealing and appropriate than "Biopython consortium" I >> think. If not developers, perhaps "Biopython contributors" is ok. > > Another opinion that consortium is over-formal. Developers > or contributors sounds more appealing for me... > I agree Biopython Consortium is too formal - should we change the name on PyPI as well? Time for a little vote? (a) The Biopython Consortium (as on PyPI now) (b) (The) Biopython consortium (lower case c) (c) (The) Biopython developers (suggestive of a core team) (d) (The) Biopython contributors (even more inclusive) Or any other ideas? Biopython Team? Or getting to informal, Biopythoneers or Biopythonistas? ;) Regards, Peter From tra at popgen.net Wed Apr 9 05:09:31 2014 From: tra at popgen.net (Tiago Antao) Date: Wed, 9 Apr 2014 10:09:31 +0100 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: References: <20140407114151.23947d6b@lnx> <20140409095652.13ee5258@lnx> Message-ID: <20140409100931.1fd685b8@lnx> On Wed, 9 Apr 2014 10:02:41 +0100 Peter Cock wrote: > (a) The Biopython Consortium (as on PyPI now) > (b) (The) Biopython consortium (lower case c) > (c) (The) Biopython developers (suggestive of a core team) > (d) (The) Biopython contributors (even more inclusive) My vote: d>c>>b=a The reason I sligthly prefer contributors to developers is to suggest that there are many ways to help that are not development (e.g. documentation writing). From christian at brueffer.de Wed Apr 9 05:52:30 2014 From: christian at brueffer.de (Christian Brueffer) Date: Wed, 09 Apr 2014 11:52:30 +0200 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: <20140409100931.1fd685b8@lnx> References: <20140407114151.23947d6b@lnx> <20140409095652.13ee5258@lnx> <20140409100931.1fd685b8@lnx> Message-ID: <5345185E.6060102@brueffer.de> On 4/9/14 11:09 AM, Tiago Antao wrote: > On Wed, 9 Apr 2014 10:02:41 +0100 > Peter Cock wrote: > >> (a) The Biopython Consortium (as on PyPI now) >> (b) (The) Biopython consortium (lower case c) >> (c) (The) Biopython developers (suggestive of a core team) >> (d) (The) Biopython contributors (even more inclusive) > > My vote: > d>c>>b=a > > The reason I sligthly prefer contributors to developers is to suggest > that there are many ways to help that are not development (e.g. > documentation writing). +1 Chris From anaryin at gmail.com Thu Apr 10 03:36:27 2014 From: anaryin at gmail.com (=?UTF-8?Q?Jo=C3=A3o_Rodrigues?=) Date: Thu, 10 Apr 2014 09:36:27 +0200 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: <5345185E.6060102@brueffer.de> References: <20140407114151.23947d6b@lnx> <20140409095652.13ee5258@lnx> <20140409100931.1fd685b8@lnx> <5345185E.6060102@brueffer.de> Message-ID: +1 too for contributors. Sorry for the time it took to answer, but I was on a trip. Bow, thanks for the abstract and volunteering, looks great. Jo?o 2014-04-09 11:52 GMT+02:00 Christian Brueffer : > On 4/9/14 11:09 AM, Tiago Antao wrote: > > On Wed, 9 Apr 2014 10:02:41 +0100 > > Peter Cock wrote: > > > >> (a) The Biopython Consortium (as on PyPI now) > >> (b) (The) Biopython consortium (lower case c) > >> (c) (The) Biopython developers (suggestive of a core team) > >> (d) (The) Biopython contributors (even more inclusive) > > > > My vote: > > d>c>>b=a > > > > The reason I sligthly prefer contributors to developers is to suggest > > that there are many ways to help that are not development (e.g. > > documentation writing). > > +1 > > Chris > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev > From arklenna at gmail.com Sun Apr 13 16:16:13 2014 From: arklenna at gmail.com (Lenna Peterson) Date: Sun, 13 Apr 2014 16:16:13 -0400 Subject: [Biopython-dev] Coordinate Mapper pull request Message-ID: Hi all, I've finally submitted a pull request for my coordinate mapper (many thanks to Reece Hart for the original code). An example of its usage is on the cookbook: http://biopython.org/wiki/Coordinate_mapping The example code can also be found in this gist: https://gist.github.com/lennax/10600113 Comments are welcome on the list or the pull request: https://github.com/biopython/biopython/pull/309 Cheers, Lenna From p.j.a.cock at googlemail.com Mon Apr 14 05:20:25 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 14 Apr 2014 10:20:25 +0100 Subject: [Biopython-dev] interest module for sequence based clustering tools? In-Reply-To: References: Message-ID: Hi Krishna, This looks interesting, although it is not something I can remember doing myself with Python before. Do you think the framework would be general enough to cover other sequence clustering tools like UCLUST http://drive5.com/usearch/manual/uclust_algo.html or OrthoMCL? Peter On Tue, Apr 1, 2014 at 7:41 AM, Krishna Roskin wrote: > Hey all, > > Long time fan, taking my first crack at contributing. > > I've built a basic module to run and parse the result of sequence based > clustering tools such as DNACLUST and CD-HIT. I've written subclasses of > AbstractCommandline to run dnaclust and cd-hit. I've also written classes > to store the clusters and their members and loaders for the output formats > used by those programs. > > I posting here to gauge interest and get some feedback and maybe some beta > testers. > > My code is available at: > > https://github.com/krishnaroskin/biopython.git > > under the seqcluster branch. I've started writing some test code at: > > Tests/seqcluster/test_seqcluster.py > > that also severs as example code. I've pasted that at the end of this > message so people can get an idea of how it works without having to > checkout code. > > If there is interest, I'm planning on adding a seqclust.cluster function > that takes a list of SeqRecords and returns a clustering using one of the > supported tools. I envision that function being the main interface to this > module. I also want to write a something that will map the cluster > membership (given by ids) back to collections of SeqRecords. > > Other to-dos: > > Test that all the flavors of CD-HIT work (there are many) > Add support for other sequence based clustering tools (suggestions?) > Documentation > Tutorial > Test code > > -krish > > #!/usr/bin/env python > > > from __future__ import print_function > > > import StringIO > > > from Bio.seqcluster.applications import DNAClustCommandline > > from Bio.seqcluster import DNAClustIterator > > > from Bio.seqcluster.applications import CDHITCommandline > > from Bio.seqcluster import CDHITClustIterator > > > cmd = DNAClustCommandline(similarity=0.8, header=True, threads=2, inputfile= > "test_sequences.fasta") > > stdout, stderr = cmd() > > clusters = DNAClustIterator(StringIO.StringIO(stdout)) > > for cluster in clusters: > > print(cluster.name) > > for member in cluster: > > if member == cluster.representative: > > print("\t" + member.name + "*") > > else: > > print("\t" + member.name) > > > print() # blank line > > > cmd = CDHITCommandline(cutoff=0.8, threads=2, inputfile= > "test_sequences.fasta", outputfile="tmp") > > stdout, stderr = cmd() > > clusters = CDHITClustIterator(open("tmp.clstr", "r")) > > for cluster in clusters: > > print(cluster.name) > > for member in cluster: > > if member == cluster.representative: > > print("\t" + member.name + "*") > > else: > > print("\t" + member.name) > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev From p.j.a.cock at googlemail.com Mon Apr 14 07:36:23 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 14 Apr 2014 12:36:23 +0100 Subject: [Biopython-dev] Problem on Mac OS X since update Xcode 5.1 In-Reply-To: References: Message-ID: On Sun, Mar 16, 2014 at 9:15 PM, Peter Cock wrote: > On Sun, Mar 16, 2014 at 8:53 PM, Lenna Peterson wrote: >> On Wed, Mar 12, 2014 at 5:59 AM, Peter Cock >> wrote: >>> >>> ... >>> >>> This looks relevant, especially this reply from Paul Kehrer which >>> suggests this is entirely Apple's fault for shipping a Python and >>> clang compiler which don't get along with the default settings: >>> >>> >>> http://stackoverflow.com/questions/22313407/clang-error-unknown-argument-mno-fused-madd-psycopg2-installation-failure >>> >> >>> The suggested workaround seems to do the trick, >>> >>> $ export CFLAGS=-Qunused-arguments >>> $ export CPPFLAGS=-Qunused-arguments >>> >> >> I encountered the same problem (clean install of Mavericks, vanilla Python, >> latest XCode from App Store). >> >> One answer [1] suggests this is not a guaranteed solution but offers a >> different flag (which I did not test). >> >> I chose to edit system python files [2] which is definitely not the best >> option for most users. >> >> [1]: http://stackoverflow.com/a/22315129 >> [2]: http://stackoverflow.com/a/22322068 >> >>> Perhaps we can add this hack to our setup.py on Mac OS X... >>> it seems harmless under gcc (e.g. my locally compiled version >>> of Python 3.3 used gcc rather than clang)? >> >> Do you mean editing environment variables with `os.environ`? I don't know >> enough about the details of how packages are built to know what will work >> with both compiling from source, easy_install, pip, etc. > > Yes, I was thinking about editing the environment variables in > setup.py via the os module. I agree there are potential risks > with 3rd party installers, but adding -Qunused-arguments to > any existing CFLAGS (within the scope of the Biopython install) > is hopefully low risk... > >>> Or it could be done via the buildbot setup, or on this buildslave >>> directly (e.g. the ~/.bash_profile). >> >> It's a dilemma, because asking users to edit their .bashrc or .bash_profile >> before installation is annoying and easy to overlook, but modifying them in >> setup.py feels hacky (i.e. how long will this solution work?). Crossing my >> fingers and hoping Apple fixes this in an update... >> > > Fingers crossed Apple pushes another update in the next few > weeks to resolve this... > > Peter I just retested with XCode 5.1.1 (released last week and updated via the App Store), and the problem persists. $ cc -v Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn) Target: x86_64-apple-darwin13.1.0 Thread model: posix $ which python-config /Library/Frameworks/Python.framework/Versions/Current/bin/python-config $ python-config --cflags -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 i.e. Apple's current Python still says to use C compliler options which Apple's current clang compiler rejects :( The good news is the environment variable trick still works. It appears Apple are not going to fix this, so perhaps we do need to add a hack to our setup.py file? Peter From krishnaroskin at gmail.com Mon Apr 14 13:00:32 2014 From: krishnaroskin at gmail.com (Krishna Roskin) Date: Mon, 14 Apr 2014 10:00:32 -0700 Subject: [Biopython-dev] interest module for sequence based clustering tools? In-Reply-To: References: Message-ID: Peter, It should be general enough. UCLUST was on my list of programs to support but I back-burnered it because it's not free software/open source. But I'll take a look and checkout OrthoMCL as well. I've since added tools that take an iterable of SeqRecords and returns a clustering. Writing to a temporary FASTA file, running the program, and parsing the results are all taken care of. I'm also working on some documentation. I'm planning on having examples of doing sequence deduplication, building motifs for each cluster, and recalculating cluster medoids using multiple sequence alignments. Any other suggestions are more than welcome. -krish On Mon, Apr 14, 2014 at 2:20 AM, Peter Cock wrote: > Hi Krishna, > > This looks interesting, although it is not something I can remember > doing myself with Python before. Do you think the framework would > be general enough to cover other sequence clustering tools like > UCLUST http://drive5.com/usearch/manual/uclust_algo.html > or OrthoMCL? > > Peter > > On Tue, Apr 1, 2014 at 7:41 AM, Krishna Roskin > wrote: > > Hey all, > > > > Long time fan, taking my first crack at contributing. > > > > I've built a basic module to run and parse the result of sequence based > > clustering tools such as DNACLUST and CD-HIT. I've written subclasses of > > AbstractCommandline to run dnaclust and cd-hit. I've also written classes > > to store the clusters and their members and loaders for the output > formats > > used by those programs. > > > > I posting here to gauge interest and get some feedback and maybe some > beta > > testers. > > > > My code is available at: > > > > https://github.com/krishnaroskin/biopython.git > > > > under the seqcluster branch. I've started writing some test code at: > > > > Tests/seqcluster/test_seqcluster.py > > > > that also severs as example code. I've pasted that at the end of this > > message so people can get an idea of how it works without having to > > checkout code. > > > > If there is interest, I'm planning on adding a seqclust.cluster function > > that takes a list of SeqRecords and returns a clustering using one of the > > supported tools. I envision that function being the main interface to > this > > module. I also want to write a something that will map the cluster > > membership (given by ids) back to collections of SeqRecords. > > > > Other to-dos: > > > > Test that all the flavors of CD-HIT work (there are many) > > Add support for other sequence based clustering tools (suggestions?) > > Documentation > > Tutorial > > Test code > > > > -krish > > > > #!/usr/bin/env python > > > > > > from __future__ import print_function > > > > > > import StringIO > > > > > > from Bio.seqcluster.applications import DNAClustCommandline > > > > from Bio.seqcluster import DNAClustIterator > > > > > > from Bio.seqcluster.applications import CDHITCommandline > > > > from Bio.seqcluster import CDHITClustIterator > > > > > > cmd = DNAClustCommandline(similarity=0.8, header=True, threads=2, > inputfile= > > "test_sequences.fasta") > > > > stdout, stderr = cmd() > > > > clusters = DNAClustIterator(StringIO.StringIO(stdout)) > > > > for cluster in clusters: > > > > print(cluster.name) > > > > for member in cluster: > > > > if member == cluster.representative: > > > > print("\t" + member.name + "*") > > > > else: > > > > print("\t" + member.name) > > > > > > print() # blank line > > > > > > cmd = CDHITCommandline(cutoff=0.8, threads=2, inputfile= > > "test_sequences.fasta", outputfile="tmp") > > > > stdout, stderr = cmd() > > > > clusters = CDHITClustIterator(open("tmp.clstr", "r")) > > > > for cluster in clusters: > > > > print(cluster.name) > > > > for member in cluster: > > > > if member == cluster.representative: > > > > print("\t" + member.name + "*") > > > > else: > > > > print("\t" + member.name) > > _______________________________________________ > > Biopython-dev mailing list > > Biopython-dev at lists.open-bio.org > > http://lists.open-bio.org/mailman/listinfo/biopython-dev > From arklenna at gmail.com Mon Apr 14 13:32:00 2014 From: arklenna at gmail.com (Lenna Peterson) Date: Mon, 14 Apr 2014 13:32:00 -0400 Subject: [Biopython-dev] Problem on Mac OS X since update Xcode 5.1 In-Reply-To: References: Message-ID: On Apr 14, 2014 7:36 AM, "Peter Cock" wrote: > > On Sun, Mar 16, 2014 at 9:15 PM, Peter Cock wrote: > > On Sun, Mar 16, 2014 at 8:53 PM, Lenna Peterson wrote: > >> On Wed, Mar 12, 2014 at 5:59 AM, Peter Cock > >> wrote: > >>> > >>> ... > >>> > >>> This looks relevant, especially this reply from Paul Kehrer which > >>> suggests this is entirely Apple's fault for shipping a Python and > >>> clang compiler which don't get along with the default settings: > >>> > >>> > >>> http://stackoverflow.com/questions/22313407/clang-error-unknown-argument-mno-fused-madd-psycopg2-installation-failure > >>> > >> > >>> The suggested workaround seems to do the trick, > >>> > >>> $ export CFLAGS=-Qunused-arguments > >>> $ export CPPFLAGS=-Qunused-arguments > >>> > >> > >> I encountered the same problem (clean install of Mavericks, vanilla Python, > >> latest XCode from App Store). > >> > >> One answer [1] suggests this is not a guaranteed solution but offers a > >> different flag (which I did not test). > >> > >> I chose to edit system python files [2] which is definitely not the best > >> option for most users. > >> > >> [1]: http://stackoverflow.com/a/22315129 > >> [2]: http://stackoverflow.com/a/22322068 > >> > >>> Perhaps we can add this hack to our setup.py on Mac OS X... > >>> it seems harmless under gcc (e.g. my locally compiled version > >>> of Python 3.3 used gcc rather than clang)? > >> > >> Do you mean editing environment variables with `os.environ`? I don't know > >> enough about the details of how packages are built to know what will work > >> with both compiling from source, easy_install, pip, etc. > > > > Yes, I was thinking about editing the environment variables in > > setup.py via the os module. I agree there are potential risks > > with 3rd party installers, but adding -Qunused-arguments to > > any existing CFLAGS (within the scope of the Biopython install) > > is hopefully low risk... > > > >>> Or it could be done via the buildbot setup, or on this buildslave > >>> directly (e.g. the ~/.bash_profile). > >> > >> It's a dilemma, because asking users to edit their .bashrc or .bash_profile > >> before installation is annoying and easy to overlook, but modifying them in > >> setup.py feels hacky (i.e. how long will this solution work?). Crossing my > >> fingers and hoping Apple fixes this in an update... > >> > > > > Fingers crossed Apple pushes another update in the next few > > weeks to resolve this... > > > > Peter > > I just retested with XCode 5.1.1 (released last week and updated > via the App Store), and the problem persists. > > $ cc -v > Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn) > Target: x86_64-apple-darwin13.1.0 > Thread model: posix > > $ which python-config > /Library/Frameworks/Python.framework/Versions/Current/bin/python-config > > $ python-config --cflags > -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 > -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 > -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk > -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd > -fno-common -dynamic -DNDEBUG -g -O3 > > i.e. Apple's current Python still says to use C compliler options which > Apple's current clang compiler rejects :( But Python is distributed with the OS, not XCode. I'm still hoping the next update to Mavericks will fix it. > > The good news is the environment variable trick still works. > > It appears Apple are not going to fix this, so perhaps we do need > to add a hack to our setup.py file? I suppose modifying env variables is easier when it doesn't need to be cross platform! > > Peter From manlio.calvi at gmail.com Tue Apr 15 04:59:16 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Tue, 15 Apr 2014 10:59:16 +0200 Subject: [Biopython-dev] I could lend a little help Message-ID: Hello devs, I'm Manlio and I'm a MSc in medicinal chemistry with some stints in python coding and a budding interest in bioinformatics. At the moment don't think I could really help you coding, I'm picking up my skills in python really, but I want to help in some manner here, maybe some testing, documentation or some other related task and in meantime learn more of either python and its application in this field(s) I'm interested in. So, I would help but this is my first project I collaborate in and I don't really know how it works and what I could really do here, a pointer or two on where to start working on could be useful. Thank you in advance to all. Cheers, Manlio From tra at popgen.net Wed Apr 16 05:03:49 2014 From: tra at popgen.net (Tiago Antao) Date: Wed, 16 Apr 2014 10:03:49 +0100 Subject: [Biopython-dev] Fwd: [Numpy-discussion] EuroSciPy 2014 Call for Abstracts In-Reply-To: References: <20140320114815.70210fd5@grandao> <87txathtux.fsf@chupacabra.windows.ebi.ac.uk> <20140401114620.67490302@lnx> Message-ID: <20140416100349.5bd8e4a9@lnx> Dear all, On Tue, 1 Apr 2014 14:08:45 +0100 Peter Cock wrote: > OK, let's sort this out next week then - this week is the BOSC > deadline (4 April), and Bow has volunteered to present the > Biopython talk. The abstract deadline for euroscipy has just been extended. I am not in the UK at that time, unfortunately. Is there still anyone interested on going? From p.j.a.cock at googlemail.com Wed Apr 16 08:58:04 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 16 Apr 2014 13:58:04 +0100 Subject: [Biopython-dev] Volunteer buildslave machines? e.g. Windows & 32 bit Linux In-Reply-To: References: Message-ID: On Mon, Mar 24, 2014 at 11:57 AM, Peter Cock wrote: > Hello all, > > Tiago and I have been looking after a range of machines > covering different operating systems and Python versions, > running as volunteer buildslaves for Biopython using buildbot: > http://testing.open-bio.org/biopython/tgrid > > Does anyone else have a lab/home server which could be > setup to run nightly Biopython tests for us via buildbot? > Ideally the machine needs to be online overnight (European > time) when the server is currently setup to schedule tests: > > http://www.biopython.org/wiki/Continuous_integration > > Our elderly 32 bit Linux desktop which has been running > as a Biopython buildslave for the last few years is finally > failing (hard drive problem). > > I would particularly like to see new buildslaves for: > > * 32 bit Linux > * 64 bit Windows > * Windows 7 or 8 (we have a 32 bit XP machine) > > If you think you might be able to help, the first hurdle is > verifying you can checkout Biopython from github, and > then compile the source (this is non-trivial on Windows, > especially for 64 bit Windows). > > Note that this is separate from the continuous integration > testing done for use via TravisCI whenever the GitHub > repository is updated - this is very useful but currently > only covers Linux: > https://travis-ci.org/biopython/biopython/builds > > The key benefit of the buildbot server is cross platform > testing - but this requires a range of volunteer machines. > > Thanks, > > Peter > > Anyone? Right now I'm particularly interesting in Windows 32bit, since my own machine has started showing test failures with SQLite and I would like to know if this is unique or a common issue (e.g. due to a Windows update): http://lists.open-bio.org/pipermail/biopython-dev/2014-March/011169.html Also a couple of people have suggested using VMs, which would be a neat solution but some physically redundancy seems helpful. Thanks, Peter From manlio.calvi at gmail.com Wed Apr 16 09:30:26 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Wed, 16 Apr 2014 15:30:26 +0200 Subject: [Biopython-dev] Volunteer buildslave machines? e.g. Windows & 32 bit Linux In-Reply-To: References: Message-ID: I have a question, exactly what it takes? I mean what kind of machine you need, must be a server? Long story short, could be a common home box?My PC in this case (a mid/mid-high range gaming machine) and what entail exactly as software installed(apart biopython and dependencies, of course) and internet access? I'm keeping general because I believe that knowing what is exactly needed in a detailed way could be useful here. In all cases my machine is a Win7 x64 box, I'm currently running python 2.7 x64. Cheers, Manlio On Wed, Apr 16, 2014 at 2:58 PM, Peter Cock wrote: > On Mon, Mar 24, 2014 at 11:57 AM, Peter Cock wrote: > >> Hello all, >> >> Tiago and I have been looking after a range of machines >> covering different operating systems and Python versions, >> running as volunteer buildslaves for Biopython using buildbot: >> http://testing.open-bio.org/biopython/tgrid >> >> Does anyone else have a lab/home server which could be >> setup to run nightly Biopython tests for us via buildbot? >> Ideally the machine needs to be online overnight (European >> time) when the server is currently setup to schedule tests: >> >> http://www.biopython.org/wiki/Continuous_integration >> >> Our elderly 32 bit Linux desktop which has been running >> as a Biopython buildslave for the last few years is finally >> failing (hard drive problem). >> >> I would particularly like to see new buildslaves for: >> >> * 32 bit Linux >> * 64 bit Windows >> * Windows 7 or 8 (we have a 32 bit XP machine) >> >> If you think you might be able to help, the first hurdle is >> verifying you can checkout Biopython from github, and >> then compile the source (this is non-trivial on Windows, >> especially for 64 bit Windows). >> >> Note that this is separate from the continuous integration >> testing done for use via TravisCI whenever the GitHub >> repository is updated - this is very useful but currently >> only covers Linux: >> https://travis-ci.org/biopython/biopython/builds >> >> The key benefit of the buildbot server is cross platform >> testing - but this requires a range of volunteer machines. >> >> Thanks, >> >> Peter >> >> > Anyone? > > Right now I'm particularly interesting in Windows 32bit, > since my own machine has started showing test failures > with SQLite and I would like to know if this is unique or > a common issue (e.g. due to a Windows update): > http://lists.open-bio.org/pipermail/biopython-dev/2014-March/011169.html > > Also a couple of people have suggested using VMs, > which would be a neat solution but some physically > redundancy seems helpful. > > Thanks, > > Peter > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev From p.j.a.cock at googlemail.com Wed Apr 16 09:51:13 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 16 Apr 2014 14:51:13 +0100 Subject: [Biopython-dev] Volunteer buildslave machines? e.g. Windows & 32 bit Linux In-Reply-To: References: Message-ID: Hi Manlio, >From a hardware point of view, that sounds fine. The Biopython tests are deliberately not computationally heavy - each full test suite should take less than 10 minutes. One of the BuildBots is my old desktop machine at home, Ideally your machine would be turned on, with internet access, all the time - but in particular overnight European time when the nightly tests are currently triggered automatically. See: http://www.biopython.org/wiki/Continuous_integration Have you ever compiled Biopython from source on Windows? That would be the first step (without worrying about the buildbot side of things for now), although ... if that is too much ... we could maybe just run the PyPy tests on your machine (no need for a C compiler which makes it much simpler)? Regards, Peter On Wed, Apr 16, 2014 at 2:30 PM, Manlio Calvi wrote: > > I have a question, > exactly what it takes? I mean what kind of machine you need, must be a > server? Long story short, could be a common home box?My PC in this > case (a mid/mid-high range gaming machine) and what entail exactly as > software installed(apart biopython and dependencies, of course) and > internet access? > I'm keeping general because I believe that knowing what is exactly > needed in a detailed way could be useful here. > > In all cases my machine is a Win7 x64 box, I'm currently running python 2.7 x64. > > Cheers, > Manlio > > > On Wed, Apr 16, 2014 at 2:58 PM, Peter Cock wrote: > > On Mon, Mar 24, 2014 at 11:57 AM, Peter Cock wrote: > > > >> Hello all, > >> > >> Tiago and I have been looking after a range of machines > >> covering different operating systems and Python versions, > >> running as volunteer buildslaves for Biopython using buildbot: > >> http://testing.open-bio.org/biopython/tgrid > >> > >> Does anyone else have a lab/home server which could be > >> setup to run nightly Biopython tests for us via buildbot? > >> Ideally the machine needs to be online overnight (European > >> time) when the server is currently setup to schedule tests: > >> > >> http://www.biopython.org/wiki/Continuous_integration > >> > >> Our elderly 32 bit Linux desktop which has been running > >> as a Biopython buildslave for the last few years is finally > >> failing (hard drive problem). > >> > >> I would particularly like to see new buildslaves for: > >> > >> * 32 bit Linux > >> * 64 bit Windows > >> * Windows 7 or 8 (we have a 32 bit XP machine) > >> > >> If you think you might be able to help, the first hurdle is > >> verifying you can checkout Biopython from github, and > >> then compile the source (this is non-trivial on Windows, > >> especially for 64 bit Windows). > >> > >> Note that this is separate from the continuous integration > >> testing done for use via TravisCI whenever the GitHub > >> repository is updated - this is very useful but currently > >> only covers Linux: > >> https://travis-ci.org/biopython/biopython/builds > >> > >> The key benefit of the buildbot server is cross platform > >> testing - but this requires a range of volunteer machines. > >> > >> Thanks, > >> > >> Peter > >> > >> > > Anyone? > > > > Right now I'm particularly interesting in Windows 32bit, > > since my own machine has started showing test failures > > with SQLite and I would like to know if this is unique or > > a common issue (e.g. due to a Windows update): > > http://lists.open-bio.org/pipermail/biopython-dev/2014-March/011169.html > > > > Also a couple of people have suggested using VMs, > > which would be a neat solution but some physically > > redundancy seems helpful. > > > > Thanks, > > > > Peter > > _______________________________________________ > > Biopython-dev mailing list > > Biopython-dev at lists.open-bio.org > > http://lists.open-bio.org/mailman/listinfo/biopython-dev From manlio.calvi at gmail.com Wed Apr 16 10:01:55 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Wed, 16 Apr 2014 16:01:55 +0200 Subject: [Biopython-dev] Volunteer buildslave machines? e.g. Windows & 32 bit Linux In-Reply-To: References: Message-ID: Hi Peter, No as I said my (bio)python distibution is from binary, if you could say me what tools is needed to compile it I think I could manage (or try to) build up a python enviroment from source(I have some past stints in building linuxes from source). Let me know Best, Manlio On Wed, Apr 16, 2014 at 3:51 PM, Peter Cock wrote: > Hi Manlio, > > From a hardware point of view, that sounds fine. The Biopython > tests are deliberately not computationally heavy - each full test > suite should take less than 10 minutes. One of the BuildBots > is my old desktop machine at home, > > Ideally your machine would be turned on, with internet access, > all the time - but in particular overnight European time when the > nightly tests are currently triggered automatically. > > See: http://www.biopython.org/wiki/Continuous_integration > > Have you ever compiled Biopython from source on Windows? > That would be the first step (without worrying about the buildbot > side of things for now), although ... if that is too much ... we > could maybe just run the PyPy tests on your machine (no > need for a C compiler which makes it much simpler)? > > Regards, > > Peter > > > On Wed, Apr 16, 2014 at 2:30 PM, Manlio Calvi wrote: >> >> I have a question, >> exactly what it takes? I mean what kind of machine you need, must be a >> server? Long story short, could be a common home box?My PC in this >> case (a mid/mid-high range gaming machine) and what entail exactly as >> software installed(apart biopython and dependencies, of course) and >> internet access? >> I'm keeping general because I believe that knowing what is exactly >> needed in a detailed way could be useful here. >> >> In all cases my machine is a Win7 x64 box, I'm currently running python 2.7 x64. >> >> Cheers, >> Manlio >> >> >> On Wed, Apr 16, 2014 at 2:58 PM, Peter Cock wrote: >> > On Mon, Mar 24, 2014 at 11:57 AM, Peter Cock wrote: >> > >> >> Hello all, >> >> >> >> Tiago and I have been looking after a range of machines >> >> covering different operating systems and Python versions, >> >> running as volunteer buildslaves for Biopython using buildbot: >> >> http://testing.open-bio.org/biopython/tgrid >> >> >> >> Does anyone else have a lab/home server which could be >> >> setup to run nightly Biopython tests for us via buildbot? >> >> Ideally the machine needs to be online overnight (European >> >> time) when the server is currently setup to schedule tests: >> >> >> >> http://www.biopython.org/wiki/Continuous_integration >> >> >> >> Our elderly 32 bit Linux desktop which has been running >> >> as a Biopython buildslave for the last few years is finally >> >> failing (hard drive problem). >> >> >> >> I would particularly like to see new buildslaves for: >> >> >> >> * 32 bit Linux >> >> * 64 bit Windows >> >> * Windows 7 or 8 (we have a 32 bit XP machine) >> >> >> >> If you think you might be able to help, the first hurdle is >> >> verifying you can checkout Biopython from github, and >> >> then compile the source (this is non-trivial on Windows, >> >> especially for 64 bit Windows). >> >> >> >> Note that this is separate from the continuous integration >> >> testing done for use via TravisCI whenever the GitHub >> >> repository is updated - this is very useful but currently >> >> only covers Linux: >> >> https://travis-ci.org/biopython/biopython/builds >> >> >> >> The key benefit of the buildbot server is cross platform >> >> testing - but this requires a range of volunteer machines. >> >> >> >> Thanks, >> >> >> >> Peter >> >> >> >> >> > Anyone? >> > >> > Right now I'm particularly interesting in Windows 32bit, >> > since my own machine has started showing test failures >> > with SQLite and I would like to know if this is unique or >> > a common issue (e.g. due to a Windows update): >> > http://lists.open-bio.org/pipermail/biopython-dev/2014-March/011169.html >> > >> > Also a couple of people have suggested using VMs, >> > which would be a neat solution but some physically >> > redundancy seems helpful. >> > >> > Thanks, >> > >> > Peter >> > _______________________________________________ >> > Biopython-dev mailing list >> > Biopython-dev at lists.open-bio.org >> > http://lists.open-bio.org/mailman/listinfo/biopython-dev From p.j.a.cock at googlemail.com Wed Apr 16 10:13:41 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 16 Apr 2014 15:13:41 +0100 Subject: [Biopython-dev] Volunteer buildslave machines? e.g. Windows & 32 bit Linux In-Reply-To: References: Message-ID: Hi Manlio, If you are willing to try getting Biopython to compile under Windows 7 with 64 bit Python that would be great. I hope Tiago will be able to share some pointers to get started. The slightly less ambitious plan is compiling with 32 bit Python. My experience with 32bit Windows XP should help here. The least ambitious plan is avoiding compiling as required for C Python, and only testing under PyPy and Jython - or maybe try IronPython if that interests you? Peter On Wed, Apr 16, 2014 at 3:01 PM, Manlio Calvi wrote: > Hi Peter, > > No as I said my (bio)python distibution is from binary, if you could > say me what tools is needed to compile it I think I could manage (or > try to) build up a python enviroment from source(I have some past > stints in building linuxes from source). > Let me know > > Best, > Manlio > > > On Wed, Apr 16, 2014 at 3:51 PM, Peter Cock wrote: >> Hi Manlio, >> >> From a hardware point of view, that sounds fine. The Biopython >> tests are deliberately not computationally heavy - each full test >> suite should take less than 10 minutes. One of the BuildBots >> is my old desktop machine at home, >> >> Ideally your machine would be turned on, with internet access, >> all the time - but in particular overnight European time when the >> nightly tests are currently triggered automatically. >> >> See: http://www.biopython.org/wiki/Continuous_integration >> >> Have you ever compiled Biopython from source on Windows? >> That would be the first step (without worrying about the buildbot >> side of things for now), although ... if that is too much ... we >> could maybe just run the PyPy tests on your machine (no >> need for a C compiler which makes it much simpler)? >> >> Regards, >> >> Peter >> >> >> On Wed, Apr 16, 2014 at 2:30 PM, Manlio Calvi wrote: >>> >>> I have a question, >>> exactly what it takes? I mean what kind of machine you need, must be a >>> server? Long story short, could be a common home box?My PC in this >>> case (a mid/mid-high range gaming machine) and what entail exactly as >>> software installed(apart biopython and dependencies, of course) and >>> internet access? >>> I'm keeping general because I believe that knowing what is exactly >>> needed in a detailed way could be useful here. >>> >>> In all cases my machine is a Win7 x64 box, I'm currently running python 2.7 x64. >>> >>> Cheers, >>> Manlio >>> >>> >>> On Wed, Apr 16, 2014 at 2:58 PM, Peter Cock wrote: >>> > On Mon, Mar 24, 2014 at 11:57 AM, Peter Cock wrote: >>> > >>> >> Hello all, >>> >> >>> >> Tiago and I have been looking after a range of machines >>> >> covering different operating systems and Python versions, >>> >> running as volunteer buildslaves for Biopython using buildbot: >>> >> http://testing.open-bio.org/biopython/tgrid >>> >> >>> >> Does anyone else have a lab/home server which could be >>> >> setup to run nightly Biopython tests for us via buildbot? >>> >> Ideally the machine needs to be online overnight (European >>> >> time) when the server is currently setup to schedule tests: >>> >> >>> >> http://www.biopython.org/wiki/Continuous_integration >>> >> >>> >> Our elderly 32 bit Linux desktop which has been running >>> >> as a Biopython buildslave for the last few years is finally >>> >> failing (hard drive problem). >>> >> >>> >> I would particularly like to see new buildslaves for: >>> >> >>> >> * 32 bit Linux >>> >> * 64 bit Windows >>> >> * Windows 7 or 8 (we have a 32 bit XP machine) >>> >> >>> >> If you think you might be able to help, the first hurdle is >>> >> verifying you can checkout Biopython from github, and >>> >> then compile the source (this is non-trivial on Windows, >>> >> especially for 64 bit Windows). >>> >> >>> >> Note that this is separate from the continuous integration >>> >> testing done for use via TravisCI whenever the GitHub >>> >> repository is updated - this is very useful but currently >>> >> only covers Linux: >>> >> https://travis-ci.org/biopython/biopython/builds >>> >> >>> >> The key benefit of the buildbot server is cross platform >>> >> testing - but this requires a range of volunteer machines. >>> >> >>> >> Thanks, >>> >> >>> >> Peter >>> >> >>> >> >>> > Anyone? >>> > >>> > Right now I'm particularly interesting in Windows 32bit, >>> > since my own machine has started showing test failures >>> > with SQLite and I would like to know if this is unique or >>> > a common issue (e.g. due to a Windows update): >>> > http://lists.open-bio.org/pipermail/biopython-dev/2014-March/011169.html >>> > >>> > Also a couple of people have suggested using VMs, >>> > which would be a neat solution but some physically >>> > redundancy seems helpful. >>> > >>> > Thanks, >>> > >>> > Peter >>> > _______________________________________________ >>> > Biopython-dev mailing list >>> > Biopython-dev at lists.open-bio.org >>> > http://lists.open-bio.org/mailman/listinfo/biopython-dev From tra at popgen.net Wed Apr 16 10:17:40 2014 From: tra at popgen.net (Tiago Antao) Date: Wed, 16 Apr 2014 15:17:40 +0100 Subject: [Biopython-dev] Volunteer buildslave machines? e.g. Windows & 32 bit Linux In-Reply-To: References: Message-ID: <20140416151740.3a7d6c10@lnx> On Wed, 16 Apr 2014 15:13:41 +0100 Peter Cock wrote: > If you are willing to try getting Biopython to compile under > Windows 7 with 64 bit Python that would be great. I hope Tiago > will be able to share some pointers to get started. I agree with you: 32 bits is a better place to start. Regarding 64 bits, the only thing that I have to share is pain and gried. When I tried (twice in the long past) I always failed. It is obviously possible, as there are unofficial 64-bit distributions - but above my pay grade. The process now might be easier as things are maturing a bit... Tiago From manlio.calvi at gmail.com Wed Apr 16 10:33:20 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Wed, 16 Apr 2014 16:33:20 +0200 Subject: [Biopython-dev] Volunteer buildslave machines? e.g. Windows & 32 bit Linux In-Reply-To: <20140416151740.3a7d6c10@lnx> References: <20140416151740.3a7d6c10@lnx> Message-ID: Well I'm currently running on the WinPython 64bit binary distribution with all the py packages from Gohlke. I could try to build from scratch but what I don't understand is if I need to build every single piece of python and dependencies of biopython from scratch or we could start with a binary distro and some toolchain for building from C. For the other thing PyPy, Jithon and IronPython, I never used it but that means nothing as I could get interested in it in future. I guess for PyPy is only a battery of tests against it and as far I understood things either Jython and IronPython needs their toolchain to be tested.At the moment I could build only one thing though. On Wed, Apr 16, 2014 at 4:17 PM, Tiago Antao wrote: > On Wed, 16 Apr 2014 15:13:41 +0100 > Peter Cock wrote: > >> If you are willing to try getting Biopython to compile under >> Windows 7 with 64 bit Python that would be great. I hope Tiago >> will be able to share some pointers to get started. > > > I agree with you: 32 bits is a better place to start. > Regarding 64 bits, the only thing that I have to share is pain and > gried. When I tried (twice in the long past) I always failed. It is > obviously possible, as there are unofficial 64-bit distributions - but > above my pay grade. The process now might be easier as things are > maturing a bit... > > Tiago From p.j.a.cock at googlemail.com Wed Apr 16 10:41:12 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 16 Apr 2014 15:41:12 +0100 Subject: [Biopython-dev] Volunteer buildslave machines? e.g. Windows & 32 bit Linux In-Reply-To: References: <20140416151740.3a7d6c10@lnx> Message-ID: On Wed, Apr 16, 2014 at 3:33 PM, Manlio Calvi wrote: > Well I'm currently running on the WinPython 64bit binary distribution > with all the py packages from Gohlke. > I could try to build from scratch but what I don't understand is if I > need to build every single piece of python and dependencies of > biopython from scratch or we could start with a binary distro and some > toolchain for building from C. You should be OK using the Python binary installer, plus the matching NumPy binary installer from Christoph Gohlke, and a C compiler tool chain matching what he uses: http://www.lfd.uci.edu/~gohlke/pythonlibs/ > For the other thing PyPy, Jithon and IronPython, I never used it but > that means nothing as I could get interested in it in future. I guess > for PyPy is only a battery of tests against it and as far I understood > things either Jython and IronPython needs their toolchain to be > tested.At the moment I could build only one thing though. I've only tried IronPython briefly, getting Biopython to work nicely there is a project in itself. If anyone is interested, great, otherwise this is not a personal priority. Testing under PyPy should be trivial - download and install one of the provided binaries for PyPy, put them somewhere like C:\pypy-2.2, and then from the Biopython source do something like: C:\pypy2.2\pypy setup.py build C:\pypy2.2\pypy setup.py test C:\pypy2.2\pypy setup.py install Jython 2.7 ought to be similar, however the current release is only a beta and if I recall correctly I had to manually fix at least one bug in Jython for the Biopython tests to pass. Peter From manlio.calvi at gmail.com Wed Apr 16 11:03:16 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Wed, 16 Apr 2014 17:03:16 +0200 Subject: [Biopython-dev] Volunteer buildslave machines? e.g. Windows & 32 bit Linux In-Reply-To: References: <20140416151740.3a7d6c10@lnx> Message-ID: OK one step at time, giving that WinPython is a self contained distro(or so is claimed) I'm downloading that and all the binaries Christoph Gohlke use of compiling his tools (this mean all the dependencies of biopython are built against the Gohlke's toolchain) I'm downloading two of them one contain python 2.7 and the other 3.3, I'll try to build biopython against those two when all the downloading and installing went well. Then I'll report here. Only I don't know if could to all today Manlio On Wed, Apr 16, 2014 at 4:41 PM, Peter Cock wrote: > On Wed, Apr 16, 2014 at 3:33 PM, Manlio Calvi wrote: >> Well I'm currently running on the WinPython 64bit binary distribution >> with all the py packages from Gohlke. >> I could try to build from scratch but what I don't understand is if I >> need to build every single piece of python and dependencies of >> biopython from scratch or we could start with a binary distro and some >> toolchain for building from C. > > You should be OK using the Python binary installer, plus the > matching NumPy binary installer from Christoph Gohlke, and > a C compiler tool chain matching what he uses: > http://www.lfd.uci.edu/~gohlke/pythonlibs/ > >> For the other thing PyPy, Jithon and IronPython, I never used it but >> that means nothing as I could get interested in it in future. I guess >> for PyPy is only a battery of tests against it and as far I understood >> things either Jython and IronPython needs their toolchain to be >> tested.At the moment I could build only one thing though. > > I've only tried IronPython briefly, getting Biopython to work > nicely there is a project in itself. If anyone is interested, great, > otherwise this is not a personal priority. > > Testing under PyPy should be trivial - download and install > one of the provided binaries for PyPy, put them somewhere > like C:\pypy-2.2, and then from the Biopython source do > something like: > > C:\pypy2.2\pypy setup.py build > C:\pypy2.2\pypy setup.py test > C:\pypy2.2\pypy setup.py install > > Jython 2.7 ought to be similar, however the current release > is only a beta and if I recall correctly I had to manually fix at > least one bug in Jython for the Biopython tests to pass. > > Peter From p.j.a.cock at googlemail.com Wed Apr 16 12:03:37 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 16 Apr 2014 17:03:37 +0100 Subject: [Biopython-dev] Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? Message-ID: On Wed, Apr 16, 2014 at 4:03 PM, Manlio Calvi wrote: > OK one step at time, giving that WinPython is a self contained > distro(or so is claimed) I'm downloading that and all the binaries > Christoph Gohlke use of compiling his tools (this mean all the > dependencies of biopython are built against the Gohlke's toolchain) > I'm downloading two of them one contain python 2.7 and the other 3.3, > I'll try to build biopython against those two when all the downloading > and installing went well. > Then I'll report here. > Only I don't know if could to all today > > Manlio Good luck - even if we don't reach the buildbot stage, this should help establish proper Windows 64 bit support for Biopython :) Thank you, Peter From marco.galardini at unifi.it Wed Apr 16 13:11:25 2014 From: marco.galardini at unifi.it (Marco Galardini) Date: Wed, 16 Apr 2014 19:11:25 +0200 Subject: [Biopython-dev] Interested in a Phenotype Microarray parser? In-Reply-To: <533A0164.9010700@unifi.it> References: <1389160399.57779.YahooMailBasic@web164004.mail.gq1.yahoo.com> <52CD2948.7050102@unifi.it> <533213FC.8010304@unifi.it> <20140326142642.9o3hmuipcocsw8g8@webmail.unifi.it> <533A0164.9010700@unifi.it> Message-ID: <20140416191125.ysfq235mtc44ksg4@webmail.unifi.it> Hi, regarding further additions to the Bio.Phenotype module I was considering the following solution to add support for sigmoid curve fitting and parameters extraction (which is of interest when analysing this kind of data). Since the easiest way to do the curve fitting is by using the scipy package, a solution may be to implement it as a "optional" feature, like the ability to draw trees with the Phylo module using matplotlib. An exception would be raised if the "function" is called with no scipy installed. Would it that be ok? Alternatively some other way to perform curve fitting may be found, but the maintenance may become very difficult. Marco ----- Messaggio da marco.galardini at unifi.it --------- Data: Tue, 01 Apr 2014 00:59:32 +0100 Da: Marco Galardini Rispondi-A:Marco Galardini Oggetto: Re: [Biopython-dev] Interested in a Phenotype Microarray parser? A: Peter Cock Cc: Biopython-Dev Mailing List > Hi, > > as suggested, I've made a few changes to the proposed Bio.Phenotype > module (apart from the less-omics name). > > The PlateRecord object now can be indexed in a similar fashion as > AlignIO multiple alignments: it is still possible to use the WellRecord > identifier as an index, but when integers or slices are used, new > sub-plates or single wells are returned. The system uses the well > identifier as a mean to divide the plate into rows/column. Thanks for > pointing out the AlignIO system, it has been very useful. > I've left the two getColumns and getRows functions, since for some > people it may still be useful to use the wells identifiers. If you feel > like they are too confusing I can remove them. > > The updated branch is here: > https://github.com/mgalardini/biopython/tree/phenomics > > Kind regards, > Marco > > > On 26/03/2014 13:26, Marco Galardini wrote: >> Hi, >> >> many thanks for your comments, below some replies: >> >> ----- Messaggio da p.j.a.cock at googlemail.com --------- >> Data: Wed, 26 Mar 2014 10:14:53 +0000 >> Da: Peter Cock >> Rispondi-A:Peter Cock >> Oggetto: Re: [Biopython-dev] Interested in a Phenotype Microarray parser? >> A: Marco Galardini >> Cc: Biopython-Dev Mailing List >> >> >>> On Tue, Mar 25, 2014 at 11:40 PM, Marco Galardini >>> wrote: >>>> Hi all, >>>> >>>> following your suggestions (as well as the other modules implementations) >>>> I've just committed a couple of commits to my biopython fork, >>>> featuring the >>>> Bio.Phenomics module. >>>> The module capabilities are limited to reading/writing Phenotype >>>> Microarray >>>> files and basic operations on the PlateRecord/WellRecord objects. >>>> The module >>>> requires numpy to interpolate the signal when the user request a >>>> time point >>>> that wasn't in the input file (this way the WellRecord object can >>>> be queried >>>> with slices). >>>> I'm thinking on how to implement the parameters extraction from WellRecord >>>> objects without the use of scipy. >>>> >>>> Here's the link to my branch: >>>> https://github.com/mgalardini/biopython/tree/phenomics >>>> The module and functions have been documented taking inspiration from the >>>> other modules: hope they are clear enough for you to try it out. >>>> Some example files can be found in Tests/Phenomics. >>>> >>>> Marco >>> >>> Hi Marco, >>> >>> I've not worked with kind of data so my comments are not on >>> the application specifics. But I'm pleased to see unit tests :) >>> >>> One thought was while you define (Java like?) getRow and getColumn >>> methods, your __getitem__ does not support (NumPy like) access, >>> which is something we do for multiple sequence alignments. I guess >>> while most plates are laid out in a grid, the row/column for each >>> sample is not the most important thing - the sample identifier is? >>> >>> Thinking out loud, would properties `rows` and `columns` etc be >>> nicer than `getRow` and `getColumn`, supporting iteration over >>> the rows/columns/etc and indexing? >> >> Yeah, absolutely: I'll work on some changes to have a more >> straightforward way to select multiple WellRecords on row/column >> basis. >> >>> >>> Minor: Your longer function docstrings do not follow PEP257, >>> specifically starting with a one line summary, then a blank line, >>> then the details. Also you are using triple single-quotes, rather >>> than triple double-quotes (like the rest of Biopthon). >>> http://legacy.python.org/dev/peps/pep-0257/ >> >> Whoops, I'll change it, thanks >> >>> >>> Peter >>> >>> P.S. Also, I'm not very keen on the module name, phenomics - >>> I wonder if it would earn Biopython a badomics award? ;) >>> http://dx.doi.org/10.1186/2047-217X-1-6 >> >> That's meta-omics right? :p >> What about 'Phenotype' then? Maybe it's too general, but future >> extensions may include other phenotypic readouts. >> >> Marco >>> _______________________________________________ >>> Biopython-dev mailing list >>> Biopython-dev at lists.open-bio.org >>> http://lists.open-bio.org/mailman/listinfo/biopython-dev >>> >> >> >> ----- Fine del messaggio da p.j.a.cock at googlemail.com ----- >> >> >> >> Marco Galardini >> Postdoctoral Fellow >> EMBL-EBI - European Bioinformatics Institute >> Wellcome Trust Genome Campus >> Hinxton, Cambridge CB10 1SD, UK >> Phone: +44 (0)1223 49 2547 >> >> >> _______________________________________________ >> Biopython-dev mailing list >> Biopython-dev at lists.open-bio.org >> http://lists.open-bio.org/mailman/listinfo/biopython-dev > > -- > ------------------------------------------------- > Marco Galardini, PhD > Dipartimento di Biologia > Via Madonna del Piano, 6 - 50019 Sesto Fiorentino (FI) > > e-mail: marco.galardini at unifi.it > www: http://www.unifi.it/dblage/CMpro-v-p-51.html > phone: +39 055 4574737 > mobile: +39 340 2808041 > ------------------------------------------------- > > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev ----- Fine del messaggio da marco.galardini at unifi.it ----- Marco Galardini Postdoctoral Fellow EMBL-EBI - European Bioinformatics Institute Wellcome Trust Genome Campus Hinxton, Cambridge CB10 1SD, UK Phone: +44 (0)1223 49 2547 From arklenna at gmail.com Wed Apr 16 13:28:46 2014 From: arklenna at gmail.com (Lenna Peterson) Date: Wed, 16 Apr 2014 13:28:46 -0400 Subject: [Biopython-dev] Coordinate Mapper pull request In-Reply-To: References: Message-ID: On Sun, Apr 13, 2014 at 4:16 PM, Lenna Peterson wrote: > Hi all, > > I've finally submitted a pull request for my coordinate mapper (many > thanks to Reece Hart for the original code). > > An example of its usage is on the cookbook: > http://biopython.org/wiki/Coordinate_mapping > The example code can also be found in this gist: > https://gist.github.com/lennax/10600113 > > Comments are welcome on the list or the pull request: > > https://github.com/biopython/biopython/pull/309 > I fixed it for Python 3 and caught a logic error, but it's now passing all tests on Travis CI. > > > Cheers, > > Lenna > From manlio.calvi at gmail.com Thu Apr 17 02:57:58 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Thu, 17 Apr 2014 08:57:58 +0200 Subject: [Biopython-dev] Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? In-Reply-To: References: Message-ID: Update from the first round of fights :) Apparently even using the ginormous windows toolchains available online is the compilers hit a snags on architectures in the build.py script (isn't a build.py problem, I think), that's because every main version of python needs a different toolchain in windows and must be the FULL windows toolchain because they put in limitation to the compilers on the x64 architectures, the only (legal) way to have the full win tools is buy it and I can't afford them. So tl;dr I'm going to try mingw-64 toolchains, hoping those goes well. Bye to all, Manlio On Wed, Apr 16, 2014 at 6:03 PM, Peter Cock wrote: > On Wed, Apr 16, 2014 at 4:03 PM, Manlio Calvi wrote: >> OK one step at time, giving that WinPython is a self contained >> distro(or so is claimed) I'm downloading that and all the binaries >> Christoph Gohlke use of compiling his tools (this mean all the >> dependencies of biopython are built against the Gohlke's toolchain) >> I'm downloading two of them one contain python 2.7 and the other 3.3, >> I'll try to build biopython against those two when all the downloading >> and installing went well. >> Then I'll report here. >> Only I don't know if could to all today >> >> Manlio > > Good luck - even if we don't reach the buildbot stage, this > should help establish proper Windows 64 bit support for > Biopython :) > > Thank you, > > Peter From reece at harts.net Fri Apr 18 01:57:23 2014 From: reece at harts.net (Reece Hart) Date: Thu, 17 Apr 2014 22:57:23 -0700 Subject: [Biopython-dev] Coordinate Mapper pull request In-Reply-To: References: Message-ID: Lenna- I hope that grad school is treating you well. As the diagnostic sequencing space has heated up, accurate transcript-genome mapping has become ever more relevant to genome interpretations. The mapping code from that original post, now 4 years old I think, was quite primitive. My recollection is that the original post didn't even handle minus strand transcripts. Perhaps you've improved it since then. A team of us at Invitae have recently released a much more robust HGVS parser, formatter, and mapper (Apache 2.0 licensed). It's available at https://bitbucket.org/invitae/hgvs. This package currently relies on transcripts from the Universal Transcript Archive, which has current and recent historical transcripts from multiple sources and alignments to reference genomes and patches using splign and blat. UTA is available via postgresql/libpq at uta.invitae.com:5432; code is at https://bitbucket.org/invitae/uta, but only required for loading the database). This pair is reasonably accurate, but does have shortcomings. Users are advised to read through issues to understand limitations and the development roadmap. We are actively working on improving these tools. Patches are certainly welcome. (For those going to HVP in May, I'll be speaking about it there and would love to connect with users.) -Reece From p.j.a.cock at googlemail.com Fri Apr 18 14:26:00 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Fri, 18 Apr 2014 19:26:00 +0100 Subject: [Biopython-dev] TogoWS test failure, was: Compiling Biopython on 64 bit Windows Message-ID: On Thu, Apr 17, 2014 at 7:57 AM, Manlio Calvi wrote: > Update from the first round of fights :) > Following a successful build (the emails didn't get CC'd to the list - can we forward them?), Manlio reported some test failures including with TogoWS (both Python 2.7 and 3.4), pasted at the end of this email. Something is amiss in the call to Bio.TogoWS.search_count(...) where we expect TogoWS to return a plain text file with an integer count in it. Somehow some XHTML is being returned instead. Has anyone else seen this before? I wonder if it is a proxy problem or something like that? Thanks, Peter -- ====================================================================== ERROR: test_pubmed_search_bioruby (test_TogoWS.TogoSearch) Bio.TogoWS.search_iter("pubmed", "BioRuby") etc ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", line 425, in test_pubmed_search_bioruby "20739307", "20015970", "14693808"]) File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", line 463, in check search_count = TogoWS.search_count(database, search_term) File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\TogoWS\__init__.py", line 167, in search_count count = int(handle.read().strip()) ValueError: invalid literal for int() with base 10: '\n\n\n \n\n\n \n\n\n References: <20140417124408.5af66518@lnx> Message-ID: ---------- Forwarded message ---------- From: Manlio Calvi Date: Thu, Apr 17, 2014 at 10:11 PM Subject: Re: Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? To: Tiago Antao Cc: Peter Cock People, announcement! :) I'm successful! (well, mostly) I compiled on Win7 x64 against both native Python 2.7.6 x64 and Python 3.4.0 x64 How I did: 1. Installed MS Visual Studio 2008 express 2. Installed Windows SDK for Windows Server 2008 and .NET Framework 3.5 because the x64 toolchains was placed there 3. Run windows update to fix and patch things in those packages 4. Installed both Python native binary for last 2.7 and 3.4 (I discovered that is better put in this order) 5. Install from C.Gohlke all the x64 deps biopython could have + SciPy just to be sure. A pair of caveat here, I tried to produce the same configuration for both the python versions and in that site some optional deps for 3.4 are missing. I did an exception later for networkX, I downloaded from git and compiled (I'll tell you when in this process) Now coming the tricky part: 6. I found the only way to make the compilation start is to call a shell with this command - *cmd /k ""C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\vcvars64.bat""* - this set the enviroment variable for that specific shell to the x64 toolchain, otherwise don't compile (I opened two of those, one for major python version) 7. For this experiments I created two separate copies of biopython in two different dir where I cloned the reps. 8. I set the python env variable, for the Python 2.7 dedicated shell - SET PATH=C:\python27\;%PATH% - and for Python 3.4 dedicated shell - SET PATH=C:\python34\;%PATH; - of course. If you don't do that is possible you compile against the wrong python version, and in my specific case it compile (that strange but that's how happened to me) 9. Now I downloaded and installed networkx from git for test the "assembly", it worked 10. Cloned, build both the package "versions" flawlessy or so seems 11. Tested the two, a lot of skipping for missing soft dependencies. A Note here, I see a lot packages skipped because missing optional packages (in and out python) a lot MORE than the one listed on the biopython site. The results of the test are: Biopython x64 - Python 3.4 Ran 224 test and failed 3 on SearchIO_blast_xml.py, on SubsMat and on TogoWS(more than one kind of error generated here) Biopython x64 - Python 2.7.6 Ran 225 test and failed 2 on TogoWS(The same errors seems at first look) and a Docstring test on test_Tutorial.py OK that's all folks for now. What's next? :) Manlio From manlio.calvi at gmail.com Fri Apr 18 14:35:42 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Fri, 18 Apr 2014 20:35:42 +0200 Subject: [Biopython-dev] Fwd: Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? In-Reply-To: References: <20140417124408.5af66518@lnx> Message-ID: ---------- Forwarded message ---------- From: Peter Cock Date: Fri, Apr 18, 2014 at 12:01 PM Subject: Re: Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? To: Manlio Calvi Cc: Tiago Antao Manlio Excellent work - well done :) We should add some of this to the "Installing from source on Windows" section of the installation notes (which we also need to update regarding newer versions of Python and which compiler works): https://github.com/biopython/biopython/blob/master/Doc/install/Installation.tex Step 6 sounds familiar, I think there is a similar batch file to set the environment variables for the 43bit MSVC compiler. Regarding step 7, Python should use separate subdirectoroes under build/ for each version of Python, so that shouldn't be a needed. However, if you do use the same folder some of our tests use hard coded temporary filenames so it is best to only test one version of Python at a time. Regarding step 8, normally on Windows I don't bother with setting the $PATH environment variable and simply call the explicit version of Python I am testing, e.g. C:\Python27\python.exe setup.py build C:\Python27\python.exe setup.py test C:\Python27\python.exe setup.py install (This is what the current buildbot instructions do where we have automated the tests). As to what's text - let's try to fix these test failures :) Can you post the test output please? TogoWS is an online service, so from time to time it does have transient failures (so does the NCBI Entrez API). The fact this broke under both Python 2.7 and 3.4 suggests this is not a Python 3 issue. I am a little surprised you didn't see this problem with Python 3.4, but Windows does unicode a little differently so that may be why? https://github.com/biopython/biopython/issues/305 How did you install git & clone Biopython? Specifically, what is it doing about text files (i.e. Unix versus DOS/Windows new lines). This was something which has caused test failures in the past - perhaps some new cases have arisen. Thanks! Peter From arklenna at gmail.com Fri Apr 18 14:53:56 2014 From: arklenna at gmail.com (Lenna Peterson) Date: Fri, 18 Apr 2014 14:53:56 -0400 Subject: [Biopython-dev] TogoWS test failure, was: Compiling Biopython on 64 bit Windows In-Reply-To: References: Message-ID: I see similar errors (Ubuntu 12.04, Python 2.7.3, university network). I can test whether the same thing happens while I'm off-campus. Lenna On Fri, Apr 18, 2014 at 2:26 PM, Peter Cock wrote: > On Thu, Apr 17, 2014 at 7:57 AM, Manlio Calvi > wrote: > > Update from the first round of fights :) > > > > Following a successful build (the emails didn't get CC'd to > the list - can we forward them?), Manlio reported some test > failures including with TogoWS (both Python 2.7 and 3.4), > pasted at the end of this email. > > Something is amiss in the call to Bio.TogoWS.search_count(...) > where we expect TogoWS to return a plain text file with an > integer count in it. > > Somehow some XHTML is being returned instead. > > Has anyone else seen this before? I wonder if it is a proxy > problem or something like that? > > Thanks, > > Peter > > -- > > > ====================================================================== > ERROR: test_pubmed_search_bioruby (test_TogoWS.TogoSearch) > Bio.TogoWS.search_iter("pubmed", "BioRuby") etc > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > line 425, in test_pubmed_search_bioruby > "20739307", "20015970", "14693808"]) > File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > line 463, in check > search_count = TogoWS.search_count(database, search_term) > File > "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\TogoWS\__init__.py", > line 167, in search_count > count = int(handle.read().strip()) > ValueError: invalid literal for int() with base 10: ' PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" > "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n xmlns="http://www.w3.org/1999/xhtml">\n\n http-equiv="Content-Type" c > > ====================================================================== > ERROR: test_pubmed_search_porin (test_TogoWS.TogoSearch) > Bio.TogoWS.search_iter("pubmed", "human porin") etc > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > line 434, in test_pubmed_search_porin > self.check("pubmed", "human porin", ["21189321", "21835183"]) > File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > line 463, in check > search_count = TogoWS.search_count(database, search_term) > File > "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\TogoWS\__init__.py", > line 167, in search_count > count = int(handle.read().strip()) > ValueError: invalid literal for int() with base 10: ' PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" > "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n xmlns="http://www.w3.org/1999/xhtml">\n\n http-equiv="Content-Type" c > > ====================================================================== > ERROR: test_pubmed_search_togows (test_TogoWS.TogoSearch) > Bio.TogoWS.search_iter("pubmed", "TogoWS") etc > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > line 420, in test_pubmed_search_togows > self.check("pubmed", "TogoWS", ["20472643"]) > File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > line 463, in check > search_count = TogoWS.search_count(database, search_term) > File > "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\TogoWS\__init__.py", > line 167, in search_count > count = int(handle.read().strip()) > ValueError: invalid literal for int() with base 10: ' PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" > "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n xmlns="http://www.w3.org/1999/xhtml">\n\n http-equiv="Content-Type" c > > ---------------------------------------------------------------------- > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev > From p.j.a.cock at googlemail.com Fri Apr 18 16:24:53 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Fri, 18 Apr 2014 21:24:53 +0100 Subject: [Biopython-dev] TogoWS test failure, was: Compiling Biopython on 64 bit Windows In-Reply-To: References: Message-ID: Thanks Lenna, Strange - although I was sure it was working perfectly on Linux at work this afternoon, I can reproduce this as well now. This commit adds a more informative error message: https://github.com/biopython/biopython/commit/e5c656e7ce7496db0291f0055da61c492c5054aa e.g. ValueError: Expected an integer from URL http://togows.dbcls.jp/search/pubmed/TogoWS/count, got: '\n\n\n \n\n\n
1
\n\n\n\n' i.e. Rather than plain text, it is HTML with the value inside a
tag. This may be a content negotiation problem, but I will raise this with the TogoWS team. Manlio - we can ignore this for now for 64 bit Windows testing, Thanks, Peter On Fri, Apr 18, 2014 at 7:53 PM, Lenna Peterson wrote: > I see similar errors (Ubuntu 12.04, Python 2.7.3, university network). > > I can test whether the same thing happens while I'm off-campus. > > Lenna > > > On Fri, Apr 18, 2014 at 2:26 PM, Peter Cock > wrote: >> >> On Thu, Apr 17, 2014 at 7:57 AM, Manlio Calvi >> wrote: >> > Update from the first round of fights :) >> > >> >> Following a successful build (the emails didn't get CC'd to >> the list - can we forward them?), Manlio reported some test >> failures including with TogoWS (both Python 2.7 and 3.4), >> pasted at the end of this email. >> >> Something is amiss in the call to Bio.TogoWS.search_count(...) >> where we expect TogoWS to return a plain text file with an >> integer count in it. >> >> Somehow some XHTML is being returned instead. >> >> Has anyone else seen this before? I wonder if it is a proxy >> problem or something like that? >> >> Thanks, >> >> Peter >> >> -- >> >> >> ====================================================================== >> ERROR: test_pubmed_search_bioruby (test_TogoWS.TogoSearch) >> Bio.TogoWS.search_iter("pubmed", "BioRuby") etc >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", >> line 425, in test_pubmed_search_bioruby >> "20739307", "20015970", "14693808"]) >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", >> line 463, in check >> search_count = TogoWS.search_count(database, search_term) >> File >> "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\TogoWS\__init__.py", >> line 167, in search_count >> count = int(handle.read().strip()) >> ValueError: invalid literal for int() with base 10: '> PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" >> "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n> xmlns="http://www.w3.org/1999/xhtml">\n\n > http-equiv="Content-Type" c >> >> ====================================================================== >> ERROR: test_pubmed_search_porin (test_TogoWS.TogoSearch) >> Bio.TogoWS.search_iter("pubmed", "human porin") etc >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", >> line 434, in test_pubmed_search_porin >> self.check("pubmed", "human porin", ["21189321", "21835183"]) >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", >> line 463, in check >> search_count = TogoWS.search_count(database, search_term) >> File >> "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\TogoWS\__init__.py", >> line 167, in search_count >> count = int(handle.read().strip()) >> ValueError: invalid literal for int() with base 10: '> PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" >> "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n> xmlns="http://www.w3.org/1999/xhtml">\n\n > http-equiv="Content-Type" c >> >> ====================================================================== >> ERROR: test_pubmed_search_togows (test_TogoWS.TogoSearch) >> Bio.TogoWS.search_iter("pubmed", "TogoWS") etc >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", >> line 420, in test_pubmed_search_togows >> self.check("pubmed", "TogoWS", ["20472643"]) >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", >> line 463, in check >> search_count = TogoWS.search_count(database, search_term) >> File >> "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\TogoWS\__init__.py", >> line 167, in search_count >> count = int(handle.read().strip()) >> ValueError: invalid literal for int() with base 10: '> PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" >> "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n> xmlns="http://www.w3.org/1999/xhtml">\n\n > http-equiv="Content-Type" c >> >> ---------------------------------------------------------------------- >> _______________________________________________ >> Biopython-dev mailing list >> Biopython-dev at lists.open-bio.org >> http://lists.open-bio.org/mailman/listinfo/biopython-dev > > From manlio.calvi at gmail.com Fri Apr 18 16:57:41 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Fri, 18 Apr 2014 22:57:41 +0200 Subject: [Biopython-dev] TogoWS test failure, was: Compiling Biopython on 64 bit Windows In-Reply-To: References: Message-ID: Maybe they did a slight change to the output and thought was so minor that no one get disturbed by it. However if remain in this form I think it's not too difficult extract it with a regex. I haven't time to look to the other issues in the Win 64 build raised, they are different from python 2.7 to 3.4. I'll build manually more or less once a day until we get the buildbot in order. If something else get raised up, I'll post here. Manlio On Fri, Apr 18, 2014 at 10:24 PM, Peter Cock wrote: > Thanks Lenna, > > Strange - although I was sure it was working perfectly on Linux > at work this afternoon, I can reproduce this as well now. > > This commit adds a more informative error message: > > https://github.com/biopython/biopython/commit/e5c656e7ce7496db0291f0055da61c492c5054aa > > e.g. > > ValueError: Expected an integer from URL > http://togows.dbcls.jp/search/pubmed/TogoWS/count, got: ' html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" > "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n xmlns="http://www.w3.org/1999/xhtml">\n\n http-equiv="Content-Type" content="text/html; charset=UTF-8" > />\n\n\n
1
\n\n\n\n' > > i.e. Rather than plain text, it is HTML with the value > inside a
tag. This may be a content negotiation > problem, but I will raise this with the TogoWS team. > > Manlio - we can ignore this for now for 64 bit Windows > testing, > > Thanks, > > Peter > > On Fri, Apr 18, 2014 at 7:53 PM, Lenna Peterson > wrote: > > I see similar errors (Ubuntu 12.04, Python 2.7.3, university network). > > > > I can test whether the same thing happens while I'm off-campus. > > > > Lenna > > > > > > On Fri, Apr 18, 2014 at 2:26 PM, Peter Cock > > wrote: > >> > >> On Thu, Apr 17, 2014 at 7:57 AM, Manlio Calvi > >> wrote: > >> > Update from the first round of fights :) > >> > > >> > >> Following a successful build (the emails didn't get CC'd to > >> the list - can we forward them?), Manlio reported some test > >> failures including with TogoWS (both Python 2.7 and 3.4), > >> pasted at the end of this email. > >> > >> Something is amiss in the call to Bio.TogoWS.search_count(...) > >> where we expect TogoWS to return a plain text file with an > >> integer count in it. > >> > >> Somehow some XHTML is being returned instead. > >> > >> Has anyone else seen this before? I wonder if it is a proxy > >> problem or something like that? > >> > >> Thanks, > >> > >> Peter > >> > >> -- > >> > >> > >> ====================================================================== > >> ERROR: test_pubmed_search_bioruby (test_TogoWS.TogoSearch) > >> Bio.TogoWS.search_iter("pubmed", "BioRuby") etc > >> ---------------------------------------------------------------------- > >> Traceback (most recent call last): > >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > >> line 425, in test_pubmed_search_bioruby > >> "20739307", "20015970", "14693808"]) > >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > >> line 463, in check > >> search_count = TogoWS.search_count(database, search_term) > >> File > >> > "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\TogoWS\__init__.py", > >> line 167, in search_count > >> count = int(handle.read().strip()) > >> ValueError: invalid literal for int() with base 10: ' >> PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" > >> "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n >> xmlns="http://www.w3.org/1999/xhtml">\n\n >> http-equiv="Content-Type" c > >> > >> ====================================================================== > >> ERROR: test_pubmed_search_porin (test_TogoWS.TogoSearch) > >> Bio.TogoWS.search_iter("pubmed", "human porin") etc > >> ---------------------------------------------------------------------- > >> Traceback (most recent call last): > >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > >> line 434, in test_pubmed_search_porin > >> self.check("pubmed", "human porin", ["21189321", "21835183"]) > >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > >> line 463, in check > >> search_count = TogoWS.search_count(database, search_term) > >> File > >> > "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\TogoWS\__init__.py", > >> line 167, in search_count > >> count = int(handle.read().strip()) > >> ValueError: invalid literal for int() with base 10: ' >> PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" > >> "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n >> xmlns="http://www.w3.org/1999/xhtml">\n\n >> http-equiv="Content-Type" c > >> > >> ====================================================================== > >> ERROR: test_pubmed_search_togows (test_TogoWS.TogoSearch) > >> Bio.TogoWS.search_iter("pubmed", "TogoWS") etc > >> ---------------------------------------------------------------------- > >> Traceback (most recent call last): > >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > >> line 420, in test_pubmed_search_togows > >> self.check("pubmed", "TogoWS", ["20472643"]) > >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > >> line 463, in check > >> search_count = TogoWS.search_count(database, search_term) > >> File > >> > "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\TogoWS\__init__.py", > >> line 167, in search_count > >> count = int(handle.read().strip()) > >> ValueError: invalid literal for int() with base 10: ' >> PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" > >> "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n >> xmlns="http://www.w3.org/1999/xhtml">\n\n >> http-equiv="Content-Type" c > >> > >> ---------------------------------------------------------------------- > >> _______________________________________________ > >> Biopython-dev mailing list > >> Biopython-dev at lists.open-bio.org > >> http://lists.open-bio.org/mailman/listinfo/biopython-dev > > > > > From manlio.calvi at gmail.com Fri Apr 18 14:40:43 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Fri, 18 Apr 2014 20:40:43 +0200 Subject: [Biopython-dev] Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? In-Reply-To: References: <20140417124408.5af66518@lnx> Message-ID: And at last of this forward I'm posting the build and tests logs (more a dump than a log) for those who want peek into it. As you'll see the only consistent error was the TogoWS' XHTML error. Cheers, Manlio On Fri, Apr 18, 2014 at 8:35 PM, Manlio Calvi wrote: > > > ---------- Forwarded message ---------- > From: Peter Cock > Date: Fri, Apr 18, 2014 at 12:01 PM > Subject: Re: Compiling Biopython on 64 bit Windows, was: Volunteer > buildslave machines? > To: Manlio Calvi > Cc: Tiago Antao > > Manlio > > Excellent work - well done :) > > We should add some of this to the "Installing from source on Windows" > section of the installation notes (which we also need to update regarding > newer versions of Python and which compiler works): > > > https://github.com/biopython/biopython/blob/master/Doc/install/Installation.tex > > Step 6 sounds familiar, I think there is a similar batch file to set the > environment variables for the 43bit MSVC compiler. > > Regarding step 7, Python should use separate subdirectoroes under > build/ for each version of Python, so that shouldn't be a needed. > However, if you do use the same folder some of our tests use > hard coded temporary filenames so it is best to only test one version > of Python at a time. > > Regarding step 8, normally on Windows I don't bother with setting the > $PATH environment variable and simply call the explicit version of > Python I am testing, e.g. > > C:\Python27\python.exe setup.py build > C:\Python27\python.exe setup.py test > C:\Python27\python.exe setup.py install > > (This is what the current buildbot instructions do where we > have automated the tests). > > As to what's text - let's try to fix these test failures :) > Can you post the test output please? > > TogoWS is an online service, so from time to time it does have > transient failures (so does the NCBI Entrez API). The fact this > broke under both Python 2.7 and 3.4 suggests this is not a > Python 3 issue. > > I am a little surprised you didn't see this problem with Python 3.4, > but Windows does unicode a little differently so that may be why? > https://github.com/biopython/biopython/issues/305 > > How did you install git & clone Biopython? Specifically, what > is it doing about text files (i.e. Unix versus DOS/Windows > new lines). This was something which has caused test > failures in the past - perhaps some new cases have arisen. > > Thanks! > > Peter > > -------------- next part -------------- A non-text attachment was scrubbed... Name: test27.log Type: application/octet-stream Size: 13944 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: build27.log Type: application/octet-stream Size: 63254 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test34.log Type: application/octet-stream Size: 29282 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: build34.log Type: application/octet-stream Size: 63350 bytes Desc: not available URL: From p.j.a.cock at googlemail.com Sat Apr 19 08:23:35 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Sat, 19 Apr 2014 13:23:35 +0100 Subject: [Biopython-dev] TogoWS test failure, was: Compiling Biopython on 64 bit Windows In-Reply-To: References: Message-ID: The TogoWS team confirmed this is a problem with the server and expect it to be fixed shortly. Peter On Fri, Apr 18, 2014 at 9:57 PM, Manlio Calvi wrote: > Maybe they did a slight change to the output and thought was so minor that > no one get disturbed by it. However if remain in this form I think it's not > too difficult extract it with a regex. > > I haven't time to look to the other issues in the Win 64 build raised, they > are different from python 2.7 to 3.4. > > I'll build manually more or less once a day until we get the buildbot in > order. If something else get raised up, I'll post here. > > Manlio > > > On Fri, Apr 18, 2014 at 10:24 PM, Peter Cock > wrote: >> >> Thanks Lenna, >> >> Strange - although I was sure it was working perfectly on Linux >> at work this afternoon, I can reproduce this as well now. >> >> This commit adds a more informative error message: >> >> https://github.com/biopython/biopython/commit/e5c656e7ce7496db0291f0055da61c492c5054aa >> >> e.g. >> >> ValueError: Expected an integer from URL >> http://togows.dbcls.jp/search/pubmed/TogoWS/count, got: '> html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" >> "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n> xmlns="http://www.w3.org/1999/xhtml">\n\n > http-equiv="Content-Type" content="text/html; charset=UTF-8" >> />\n\n\n
1
\n\n\n\n' >> >> i.e. Rather than plain text, it is HTML with the value >> inside a
tag. This may be a content negotiation >> problem, but I will raise this with the TogoWS team. >> >> Manlio - we can ignore this for now for 64 bit Windows >> testing, >> >> Thanks, >> >> Peter >> From p.j.a.cock at googlemail.com Mon Apr 21 09:37:54 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 21 Apr 2014 14:37:54 +0100 Subject: [Biopython-dev] C code review - Bio.trie fix for Python 3.4 Message-ID: Hi all, Do we have any Python 3 bytes/unicode C API experts lurking on the list who could review my patch here please? https://github.com/biopython/biopython/issues/305 Thanks, Peter From manlio.calvi at gmail.com Mon Apr 21 10:41:39 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 21 Apr 2014 16:41:39 +0200 Subject: [Biopython-dev] C code review - Bio.trie fix for Python 3.4 In-Reply-To: References: Message-ID: I don't know if it is related (but probably is) but I understood why I don't get a trie error in my Test config... simply the module don't compile itself! Looking at the build dump seems to be compiled (with a lot of warning and deprecations) My test machine is Win7 x64, Python 3.4 x64 binary from Python.org That's is the command I enter and the copy of the test dump I obtained ############################################################## C:\Users\Public\BioPython3\biopython>python Tests/run_tests.py test_trie.py > TestTrie.log 2>&1 test_trie ... skipping. Could not import Bio.trie, check C code was compiled. ---------------------------------------------------------------------- Ran 1 test in 0.006 seconds Python version: 3.4.0 (v3.4.0:04f714765c13, Mar 16 2014, 19:25:23) [MSC v.1600 64 bit (AMD64)] Operating system: nt win32 ################################## What follows is part of the build dump related to the trie c module,instead ################################################ building 'Bio.trie' extension C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IBio -IC:\Python34\include -IC:\Python34\include /TcBio/triemodule.c /Fobuild\temp.win-amd64-3.4\Release\Bio/triemodule.obj triemodule.c Bio/triemodule.c(578) : warning C4133: 'initializing' : incompatible types - from 'long (__cdecl *)(PyObject *)' to 'hashfunc' Bio/triemodule.c(750) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'const int', possible loss of data C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IBio -IC:\Python34\include -IC:\Python34\include /TcBio/trie.c /Fobuild\temp.win-amd64-3.4\Release\Bio/trie.obj trie.c Bio/trie.c(11) : warning C4996: 'strcpy': This function or variable may be unsafe. Consider using strcpy_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details. Bio/trie.c(161) : warning C4996: 'strncpy': This function or variable may be unsafe. Consider using strncpy_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details. C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE\string.h(157) : see declaration of 'strncpy' Bio/trie.c(294) : warning C4267: 'initializing' : conversion from 'size_t' to 'int', possible loss of data Bio/trie.c(310) : warning C4996: 'strncat': This function or variable may be unsafe. Consider using strncat_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details. C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE\string.h(144) : see declaration of 'strncat' Bio/trie.c(379) : warning C4267: 'initializing' : conversion from 'size_t' to 'int', possible loss of data Bio/trie.c(382) : warning C4996: 'strcat': This function or variable may be unsafe. Consider using strcat_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details. Bio/trie.c(395) : warning C4267: 'function' : conversion from 'size_t' to 'const int', possible loss of data Bio/trie.c(466) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data Bio/trie.c(467) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data Bio/trie.c(494) : warning C4267: 'initializing' : conversion from 'size_t' to 'int', possible loss of data Bio/trie.c(501) : warning C4996: 'strcat': This function or variable may be unsafe. Consider using strcat_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details. Bio/trie.c(547) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data Bio/trie.c(548) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data Bio/trie.c(575) : warning C4267: 'initializing' : conversion from 'size_t' to 'int', possible loss of data Bio/trie.c(581) : warning C4996: 'strncat': This function or variable may be unsafe. Consider using strncat_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details. C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE\string.h(144) : see declaration of 'strncat' Bio/trie.c(669) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Python34\libs /LIBPATH:C:\Python34\PCbuild\amd64 /EXPORT:PyInit_trie build\temp.win-amd64-3.4\Release\Bio/triemodule.obj build\temp.win-amd64-3.4\Release\Bio/trie.obj /OUT:build\lib.win-amd64-3.4\Bio\trie.pyd /IMPLIB:build\temp.win-amd64-3.4\Release\Bio\trie.lib /MANIFESTFILE:build\temp.win-amd64-3.4\Release\Bio\trie.pyd.manifest Creating library build\temp.win-amd64-3.4\Release\Bio\trie.lib and object build\temp.win-amd64-3.4\Release\Bio\trie.exp On Mon, Apr 21, 2014 at 3:37 PM, Peter Cock wrote: > Hi all, > > Do we have any Python 3 bytes/unicode C API experts > lurking on the list who could review my patch here please? > > https://github.com/biopython/biopython/issues/305 > > Thanks, > > Peter > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev > From p.j.a.cock at googlemail.com Mon Apr 21 10:53:50 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 21 Apr 2014 15:53:50 +0100 Subject: [Biopython-dev] C code review - Bio.trie fix for Python 3.4 In-Reply-To: References: Message-ID: On Mon, Apr 21, 2014 at 3:41 PM, Manlio Calvi wrote: > I don't know if it is related (but probably is) ... but I understood why I don't > get a trie error in my Test config... simply the module don't compile > itself! > Looking at the build dump seems to be compiled (with a lot of warning and > deprecations) That is strange - all I can see is warnings, no errors. > My test machine is Win7 x64, Python 3.4 x64 binary from Python.org > Referring to your test log for Python 3.4 [*], it looks like other C code extensions are also not working (but they are for Python 2.7): running test test_Ace ... ok ... test_Cluster ... skipping. If you want to use Bio.Cluster, install NumPy first and then reinstall Biopython ... test_KDTree ... skipping. C module in Bio.KDTree not compiled ... test_PDB_KDTree ... skipping. C module in Bio.KDTree not compiled ... test_pairwise2 ... ok ... test_trie ... skipping. Could not import Bio.trie, check C code was compiled. Let's continue this on the other thread - its a bigger issue than the Bio.trie unicode problem under Python 3.4 :( Thanks, Peter [*] available on the archive here, http://lists.open-bio.org/pipermail/biopython-dev/2014-April/011265.html From p.j.a.cock at googlemail.com Mon Apr 21 10:59:46 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 21 Apr 2014 15:59:46 +0100 Subject: [Biopython-dev] Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? In-Reply-To: References: <20140417124408.5af66518@lnx> Message-ID: On Fri, Apr 18, 2014 at 7:40 PM, Manlio Calvi wrote: > And at last of this forward I'm posting the build and tests logs (more a > dump than a log) for those who want peek into it. > As you'll see the only consistent error was the TogoWS' XHTML error. > > Cheers, > Manlio Thanks Manlio, Following the discussion here, I've looked at your Python 3.4 test log in more detail - and sadly it looks like none of the C code extensions are also not working (but they are for Python 2.7): running test test_Ace ... ok ... test_Cluster ... skipping. If you want to use Bio.Cluster, install NumPy first and then reinstall Biopython ... test_KDTree ... skipping. C module in Bio.KDTree not compiled ... test_PDB_KDTree ... skipping. C module in Bio.KDTree not compiled ... test_pairwise2 ... ok ... test_trie ... skipping. Could not import Bio.trie, check C code was compiled. Note that test_pairwise2 is probably testing the pure Python fallback here (assuming its C code also didn't get used). My hunch in the absence of compilation errors, is something is amiss with Python 3.4 finding the compiled modules when we run the tests. Can you try installing Biopython under Python 3.4, and trying to import any of the C code modules? e.g. [Mac] $ python3.4 -c "from Bio import trie; print(trie.__file__)" /Users/pjcock/lib/python3.4/site-packages/Bio/trie.so [Linux] $ python3.4 -c "from Bio import trie; print(trie.__file__)" /home_local/buildslave/lib/python3.4/site-packages/Bio/trie.cpython-34m.so [32 bit Windows] > c:\python34\python -c "from Bio import trie; print(trie.__file__)" c:\python34\lib\site-packages\Bio\trie.pyd Thanks, Peter [*] available on the archive here, http://lists.open-bio.org/pipermail/biopython-dev/2014-April/011265.html From manlio.calvi at gmail.com Mon Apr 21 11:45:40 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 21 Apr 2014 17:45:40 +0200 Subject: [Biopython-dev] Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? In-Reply-To: References: <20140417124408.5af66518@lnx> Message-ID: > test_Cluster ... skipping. If you want to use Bio.Cluster, install NumPy first and then reinstall Biopython And that's strange, Numpy IS installed in my case and usable by other scripts... I tried to reinstall and rebuild, all seems the same as before. Apart from Numpy, seems the .pyd file is there but isn't importable by python as module as it raise an "ImportError" kind of error. This is the dump: #################### Traceback (most recent call last): File "", line 1, in ImportError: cannot import name 'trie' #################### Now, seems strange to me that File thing Manlio On Mon, Apr 21, 2014 at 4:59 PM, Peter Cock wrote: > On Fri, Apr 18, 2014 at 7:40 PM, Manlio Calvi wrote: >> And at last of this forward I'm posting the build and tests logs (more a >> dump than a log) for those who want peek into it. >> As you'll see the only consistent error was the TogoWS' XHTML error. >> >> Cheers, >> Manlio > > Thanks Manlio, > > Following the discussion here, I've looked at your Python 3.4 test > log in more detail - and sadly it looks like none of the C code > extensions are also not working (but they are for Python 2.7): > > running test > test_Ace ... ok > ... > test_Cluster ... skipping. If you want to use Bio.Cluster, install > NumPy first and then reinstall Biopython > ... > test_KDTree ... skipping. C module in Bio.KDTree not compiled > ... > test_PDB_KDTree ... skipping. C module in Bio.KDTree not compiled > ... > test_pairwise2 ... ok > ... > test_trie ... skipping. Could not import Bio.trie, check C code was compiled. > > > Note that test_pairwise2 is probably testing the pure Python > fallback here (assuming its C code also didn't get used). > > My hunch in the absence of compilation errors, is something is > amiss with Python 3.4 finding the compiled modules when we > run the tests. > > Can you try installing Biopython under Python 3.4, and trying > to import any of the C code modules? e.g. > > [Mac] $ python3.4 -c "from Bio import trie; print(trie.__file__)" > /Users/pjcock/lib/python3.4/site-packages/Bio/trie.so > > [Linux] $ python3.4 -c "from Bio import trie; print(trie.__file__)" > /home_local/buildslave/lib/python3.4/site-packages/Bio/trie.cpython-34m.so > > [32 bit Windows] > c:\python34\python -c "from Bio import trie; > print(trie.__file__)" > c:\python34\lib\site-packages\Bio\trie.pyd > > Thanks, > > Peter > > [*] available on the archive here, > http://lists.open-bio.org/pipermail/biopython-dev/2014-April/011265.html From p.j.a.cock at googlemail.com Mon Apr 21 12:12:20 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 21 Apr 2014 17:12:20 +0100 Subject: [Biopython-dev] Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? In-Reply-To: References: <20140417124408.5af66518@lnx> Message-ID: On Mon, Apr 21, 2014 at 4:45 PM, Manlio Calvi wrote: >> test_Cluster ... skipping. If you want to use Bio.Cluster, install NumPy first and then reinstall Biopython > > And that's strange, Numpy IS installed in my case and usable by other > scripts... I tried to reinstall and rebuild, all seems the same as > before. The error message is just triggered if "from Bio import Cluster" fails with an ImportError - much like all the other C code modules' imports are failing on your Python 3.4 setup. > Apart from Numpy, seems the .pyd file is there but isn't importable by > python as module as it raise an "ImportError" kind of error. > > This is the dump: > #################### > Traceback (most recent call last): > File "", line 1, in > ImportError: cannot import name 'trie' > #################### > > Now, seems strange to me that File thing > > Manlio What does this do? python3.4 -c "import numpy; print(numpy.__file__)" If it works, can you see any compiled modules in that folder, and how they are named? Peter From manlio.calvi at gmail.com Mon Apr 21 12:25:40 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 21 Apr 2014 18:25:40 +0200 Subject: [Biopython-dev] Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? In-Reply-To: References: <20140417124408.5af66518@lnx> Message-ID: Works as expected because I have Numpy modules running in other scripts (dumps the path to __init__.py) In C:\Python34\Lib\site-packages\Bio the non-py files are two: cpairwise2.pyd trie.pyd The same are in the folder I used to build them, and by the extension should be callable/importable. I tried to import cpairwise2 but (don't) work the same way, same error raised as before. On Mon, Apr 21, 2014 at 6:12 PM, Peter Cock wrote: > On Mon, Apr 21, 2014 at 4:45 PM, Manlio Calvi wrote: >>> test_Cluster ... skipping. If you want to use Bio.Cluster, install NumPy first and then reinstall Biopython >> >> And that's strange, Numpy IS installed in my case and usable by other >> scripts... I tried to reinstall and rebuild, all seems the same as >> before. > > The error message is just triggered if "from Bio import Cluster" > fails with an ImportError - much like all the other C code modules' > imports are failing on your Python 3.4 setup. > >> Apart from Numpy, seems the .pyd file is there but isn't importable by >> python as module as it raise an "ImportError" kind of error. >> >> This is the dump: >> #################### >> Traceback (most recent call last): >> File "", line 1, in >> ImportError: cannot import name 'trie' >> #################### >> >> Now, seems strange to me that File thing >> >> Manlio > > What does this do? > > python3.4 -c "import numpy; print(numpy.__file__)" > > If it works, can you see any compiled modules in that > folder, and how they are named? > > Peter From p.j.a.cock at googlemail.com Mon Apr 21 12:27:57 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 21 Apr 2014 17:27:57 +0100 Subject: [Biopython-dev] Pickle problem on 64 bit Windows with Python 3.4 Message-ID: Hi Manlio, One of the new errors you reported under Python 3.4 on Windows 7 was: ====================================================================== ERROR: test_SubsMat ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Users\Public\BioPython3\biopython\Tests\run_tests.py", line 408, in runTest suite = unittest.TestLoader().loadTestsFromName(name) File "C:\Python34\lib\unittest\loader.py", line 105, in loadTestsFromName module = __import__('.'.join(parts_copy)) File "C:\Users\Public\BioPython3\biopython\Tests\test_SubsMat.py", line 38, in acc_rep_mat = pickle.load(handle) _pickle.UnpicklingError: the STRING opcode argument must be quoted Except copied from test log here: http://lists.open-bio.org/pipermail/biopython-dev/2014-April/011265.html I suspect this could be a problem in Python itself, since pickle was changed in Python 3.4: https://docs.python.org/3.4/whatsnew/2.3.html#pep-307-pickle-enhancements http://www.python.org/dev/peps/pep-0307 Can you try this example on your machine - from my Mac: $ cd Tests/SubsMat $ python3.4 -c "import pickle; h = open('acc_rep_mat.pik', 'rb'); m = pickle.load(h); h.close(); print(m)" {('D', 'W'): 22, ..., ('A', 'E'): 207} >From 32 bit Windows XP, it also works for me: C:\repositories\biopython\Tests\SubsMat>c:\python34\python -c "import pickle; h = open('acc_rep_mat.pik', 'rb'); m = pickle.load(h); h.close(); print(m)" {('H', 'S'): 59, ..., ('K', 'P'): 94} Note the dictionary order is expected to change. Also I am not using a with statement in order to make a single command line friendly test you can copy & paste into the Windows terminal. It would be more work, but if you could try Python 3.3 that would be useful (and helpful if we get to run your desktop as a buildslave). Thanks, Peter From p.j.a.cock at googlemail.com Mon Apr 21 12:33:03 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 21 Apr 2014 17:33:03 +0100 Subject: [Biopython-dev] Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? In-Reply-To: References: <20140417124408.5af66518@lnx> Message-ID: On Mon, Apr 21, 2014 at 5:25 PM, Manlio Calvi wrote: > Works as expected because I have Numpy modules running in other > scripts (dumps the path to __init__.py) > > In C:\Python34\Lib\site-packages\Bio the non-py files are two: > cpairwise2.pyd > trie.pyd > > The same are in the folder I used to build them, and by the extension > should be callable/importable. > I tried to import cpairwise2 but (don't) work the same way, same error > raised as before. > So, if you've done an install & can't import the C modules, that tells me this isn't a simply problem in our test harness (run_tests.py does some magic with the Python path to try to test using the just built files rather than any installed files). This is a deeper problem with building/installing under Python 3.4 on your setup. I have for the moment run out of ideas :( (Other than trying Python 3.3 which may behave differently - and will be useful anyway for checking the other test failures and if they are in part due to the new Python version). Thanks, Peter From p.j.a.cock at googlemail.com Mon Apr 21 12:47:02 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 21 Apr 2014 17:47:02 +0100 Subject: [Biopython-dev] Python 2.7 failure from test_Tutorial.py, was: Compiling Biopython on 64 bit Windows Message-ID: Hi Manlio, Moving on to Python 2.7, apart from the TogoWS issue, http://lists.open-bio.org/pipermail/biopython-dev/2014-April/011259.html your system showed one other failure: ====================================================================== ERROR: test_doctests (test_Tutorial.TutorialTestCase) Run tutorial doctests. ---------------------------------------------------------------------- Traceback (most recent call last): File "test_Tutorial.py", line 167, in test_doctests ValueError: 1 Tutorial doctests failed: test_from_line_04424 ---------------------------------------------------------------------- You should be able to rerun the tutorial tests alone (from memory only if Biopython is installed) with just: c:\python27\python test_Tutorial.py This is from the following doctest in the Tutorial.tex file, %This example fails under PyPy 2.0, https://bugs.pypy.org/issue1546 %doctest examples lib:numpy \begin{verbatim} >>> import numpy as np >>> from Bio import AlignIO >>> alignment = AlignIO.read("PF05371_seed.sth", "stockholm") >>> align_array = np.array([list(rec) for rec in alignment], np.character) >>> align_array.shape (7, 52) \end{verbatim} Could you try running that example "by hand" and seeing what happens? Note the doctest framework is very very fragile and the slightest output difference will break a test. Thanks, Peter On Fri, Apr 18, 2014 at 7:40 PM, Manlio Calvi wrote: > And at last of this forward I'm posting the build and tests logs (more a > dump than a log) for those who want peek into it. > As you'll see the only consistent error was the TogoWS' XHTML error. > > Cheers, > Manlio > > > > On Fri, Apr 18, 2014 at 8:35 PM, Manlio Calvi wrote: > >> >> >> ---------- Forwarded message ---------- >> From: Peter Cock >> Date: Fri, Apr 18, 2014 at 12:01 PM >> Subject: Re: Compiling Biopython on 64 bit Windows, was: Volunteer >> buildslave machines? >> To: Manlio Calvi >> Cc: Tiago Antao >> >> Manlio >> >> Excellent work - well done :) >> >> We should add some of this to the "Installing from source on Windows" >> section of the installation notes (which we also need to update regarding >> newer versions of Python and which compiler works): >> >> >> https://github.com/biopython/biopython/blob/master/Doc/install/Installation.tex >> >> Step 6 sounds familiar, I think there is a similar batch file to set the >> environment variables for the 43bit MSVC compiler. >> >> Regarding step 7, Python should use separate subdirectoroes under >> build/ for each version of Python, so that shouldn't be a needed. >> However, if you do use the same folder some of our tests use >> hard coded temporary filenames so it is best to only test one version >> of Python at a time. >> >> Regarding step 8, normally on Windows I don't bother with setting the >> $PATH environment variable and simply call the explicit version of >> Python I am testing, e.g. >> >> C:\Python27\python.exe setup.py build >> C:\Python27\python.exe setup.py test >> C:\Python27\python.exe setup.py install >> >> (This is what the current buildbot instructions do where we >> have automated the tests). >> >> As to what's text - let's try to fix these test failures :) >> Can you post the test output please? >> >> TogoWS is an online service, so from time to time it does have >> transient failures (so does the NCBI Entrez API). The fact this >> broke under both Python 2.7 and 3.4 suggests this is not a >> Python 3 issue. >> >> I am a little surprised you didn't see this problem with Python 3.4, >> but Windows does unicode a little differently so that may be why? >> https://github.com/biopython/biopython/issues/305 >> >> How did you install git & clone Biopython? Specifically, what >> is it doing about text files (i.e. Unix versus DOS/Windows >> new lines). This was something which has caused test >> failures in the past - perhaps some new cases have arisen. >> >> Thanks! >> >> Peter >> >> > > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev > From manlio.calvi at gmail.com Mon Apr 21 13:00:34 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 21 Apr 2014 19:00:34 +0200 Subject: [Biopython-dev] Pickle problem on 64 bit Windows with Python 3.4 In-Reply-To: References: Message-ID: I think we hit a stumbling block here, and maybe a serious one (but we'll be sure with more tests), because this following error suggest me the way we used the pickling in the code changed or something strange happened in my config. Apart from installing a 3.3, I'll try to go experimental with 3.4, I want to understand if it is a problem with my build of 3.4 or if it is a general issue of 3.4 with some piece of BioPython. C:\Users\Public\BioPython3\biopython\Tests\SubsMat>python -c "import pickle; h = open('acc_rep_mat.pik', 'rb'); m = pickle.load(h);h.close();print(m); Traceback (most recent call last): File "", line 1, in _pickle.UnpicklingError: the STRING opcode argument must be quoted On Mon, Apr 21, 2014 at 6:27 PM, Peter Cock wrote: > Hi Manlio, > > One of the new errors you reported under Python 3.4 on Windows 7 was: > > ====================================================================== > ERROR: test_SubsMat > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Users\Public\BioPython3\biopython\Tests\run_tests.py", line > 408, in runTest > suite = unittest.TestLoader().loadTestsFromName(name) > File "C:\Python34\lib\unittest\loader.py", line 105, in loadTestsFromName > module = __import__('.'.join(parts_copy)) > File "C:\Users\Public\BioPython3\biopython\Tests\test_SubsMat.py", > line 38, in > acc_rep_mat = pickle.load(handle) > _pickle.UnpicklingError: the STRING opcode argument must be quoted > > Except copied from test log here: > http://lists.open-bio.org/pipermail/biopython-dev/2014-April/011265.html > > I suspect this could be a problem in Python itself, since pickle was > changed in Python 3.4: > > https://docs.python.org/3.4/whatsnew/2.3.html#pep-307-pickle-enhancements > http://www.python.org/dev/peps/pep-0307 > > Can you try this example on your machine - from my Mac: > > $ cd Tests/SubsMat > $ python3.4 -c "import pickle; h = open('acc_rep_mat.pik', 'rb'); m = > pickle.load(h); h.close(); print(m)" > {('D', 'W'): 22, ..., ('A', 'E'): 207} > > From 32 bit Windows XP, it also works for me: > > C:\repositories\biopython\Tests\SubsMat>c:\python34\python -c "import > pickle; h = open('acc_rep_mat.pik', 'rb'); m = pickle.load(h); > h.close(); print(m)" > {('H', 'S'): 59, ..., ('K', 'P'): 94} > > Note the dictionary order is expected to change. Also I am not > using a with statement in order to make a single command line > friendly test you can copy & paste into the Windows terminal. > > It would be more work, but if you could try Python 3.3 that would be > useful (and helpful if we get to run your desktop as a buildslave). > > Thanks, > > Peter From zruan1991 at gmail.com Mon Apr 21 13:02:36 2014 From: zruan1991 at gmail.com (Zheng Ruan) Date: Mon, 21 Apr 2014 13:02:36 -0400 Subject: [Biopython-dev] [biopython] Code for Codon Alignment GSoC (#259) In-Reply-To: References: Message-ID: Hi Peter and Eric, I just made some changes to my repository. The old codonalignment branch has been renamed to gsoc2013 for backup. I will write new code to the codonalignment branch. Eric: I think the whole module should be marked as experimental like the Bio.SearchIO module. I anticipate to enhance the codon alignment construction method as well as add more methods for dn/ds ratio estimation and Bayes Empirical Bayes this summer. I place a warning in the `__init__.py`. ( https://github.com/zruan/biopython/commit/1c9e20c8a15febb7fd9219c632afb96127a006f1 ) Peter: The NEWS file is updated ( https://github.com/zruan/biopython/commit/0dcb39bcfd48cc009edc290ceaaa3b2c2e8f806f ). I will update the wiki shortly and make the announcement. Do I need to make a separate pull request to make the above change to biopython master? Or you may help me do that? I am also planning to include my latex document to Tutorial.tex. I typically use the Latex-Suite plugin to write latex in vim. However, it took a long time to open the Tutorial.tex, which made it difficult to manage. Why not put the latex code of each chapter into different tex file and use \include{} to import them into the main tex file? Thank you! Zheng Ruan On Mon, Apr 21, 2014 at 6:41 AM, Peter Cock wrote: > Congratulations all - could you also add something to the NEWS file, email > the main list to announce this, and update the wiki as needed? > > ? > Reply to this email directly or view it on GitHub > . > From manlio.calvi at gmail.com Mon Apr 21 13:02:47 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 21 Apr 2014 19:02:47 +0200 Subject: [Biopython-dev] Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? In-Reply-To: References: <20140417124408.5af66518@lnx> Message-ID: I'm running the tests against a WinPython x64 3.3.3.3 at the moment (so a non-Python.org distro) and will see if that happens there too, I'm planning to install a native one(if the time permits). Manlio On Mon, Apr 21, 2014 at 6:33 PM, Peter Cock wrote: > On Mon, Apr 21, 2014 at 5:25 PM, Manlio Calvi wrote: >> Works as expected because I have Numpy modules running in other >> scripts (dumps the path to __init__.py) >> >> In C:\Python34\Lib\site-packages\Bio the non-py files are two: >> cpairwise2.pyd >> trie.pyd >> >> The same are in the folder I used to build them, and by the extension >> should be callable/importable. >> I tried to import cpairwise2 but (don't) work the same way, same error >> raised as before. >> > > So, if you've done an install & can't import the C modules, that > tells me this isn't a simply problem in our test harness (run_tests.py > does some magic with the Python path to try to test using the just > built files rather than any installed files). > > This is a deeper problem with building/installing under Python 3.4 > on your setup. I have for the moment run out of ideas :( > > (Other than trying Python 3.3 which may behave differently - and > will be useful anyway for checking the other test failures and if they > are in part due to the new Python version). > > Thanks, > > Peter From p.j.a.cock at googlemail.com Mon Apr 21 13:06:25 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 21 Apr 2014 18:06:25 +0100 Subject: [Biopython-dev] [biopython] Code for Codon Alignment GSoC (#259) In-Reply-To: References: Message-ID: On Mon, Apr 21, 2014 at 6:02 PM, Zheng Ruan wrote: > > Hi Peter and Eric, > > I just made some changes to my repository. The old codonalignment branch has been > renamed to gsoc2013 for backup. I will write new code to the codonalignment branch. OK. You could write a final GSoC blog post about this too ;) > Eric: > I think the whole module should be marked as experimental like the Bio.SearchIO > module. I anticipate to enhance the codon alignment construction method as well > as add more methods for dn/ds ratio estimation and Bayes Empirical Bayes this > summer. > > I place a warning in the `__init__.py`. (https://github.com/zruan/biopython/commit/1c9e20c8a15febb7fd9219c632afb96127a006f1) That sounds wise. > Peter: > The NEWS file is updated (https://github.com/zruan/biopython/commit/0dcb39bcfd48cc009edc290ceaaa3b2c2e8f806f). > I will update the wiki shortly and make the announcement. > > Do I need to make a separate pull request to make the above change to biopython > master? Or you may help me do that? I think Eric or I can do that from those links, Eric - do you want to do this? > I am also planning to include my latex document to Tutorial.tex. I typically use > the Latex-Suite plugin to write latex in vim. However, it took a long time to open the > Tutorial.tex, which made it difficult to manage. Why not put the latex code of each > chapter into different tex file and use \include{} to import them into the main tex file? > > Thank you! > Zheng Ruan We could split the Tutorial.tex file like that - although it would mean a bit of extra work on test_Tutorial.py (but with the benefit of getting per-chapter line numbers). Thank you, Peter From p.j.a.cock at googlemail.com Mon Apr 21 13:12:34 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 21 Apr 2014 18:12:34 +0100 Subject: [Biopython-dev] Pickle problem on 64 bit Windows with Python 3.4 In-Reply-To: References: Message-ID: On Mon, Apr 21, 2014 at 6:00 PM, Manlio Calvi wrote: > I think we hit a stumbling block here, and maybe a serious one (but > we'll be sure with more tests), because this following error suggest > me the way we used the pickling in the code changed or something > strange happened in my config. > Apart from installing a 3.3, I'll try to go experimental with 3.4, I > want to understand if it is a problem with my build of 3.4 or if it is > a general issue of 3.4 with some piece of BioPython. > > C:\Users\Public\BioPython3\biopython\Tests\SubsMat>python -c "import pickle; h = > open('acc_rep_mat.pik', 'rb'); m = pickle.load(h);h.close();print(m); > > Traceback (most recent call last): > File "", line 1, in > _pickle.UnpicklingError: the STRING opcode argument must be quoted > Good - I was hoping that would happen - we have a clear test case separate from the Biopython code now. The pickle file acc_rep_mat.pik was created long ago (15 Nov 2000) with Python 2, and I would have hoped it would be forward compatible. This could be a regression in Python 3.4 itself (under Windows 64 bit only)? If anyone else has a 64 bit Windows machine and Python 3.4, please try this and let us know what happens. Thanks, Peter From manlio.calvi at gmail.com Mon Apr 21 13:45:43 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 21 Apr 2014 19:45:43 +0200 Subject: [Biopython-dev] Pickle problem on 64 bit Windows with Python 3.4 In-Reply-To: References: Message-ID: >From what I read here http://hg.python.org/cpython/rev/4a6b8f86b081 could be a problem related to that file. Seems to me they stripped the check for a quote that must be in, and looking at the pickle apparently isn't On Mon, Apr 21, 2014 at 7:12 PM, Peter Cock wrote: > On Mon, Apr 21, 2014 at 6:00 PM, Manlio Calvi wrote: >> I think we hit a stumbling block here, and maybe a serious one (but >> we'll be sure with more tests), because this following error suggest >> me the way we used the pickling in the code changed or something >> strange happened in my config. >> Apart from installing a 3.3, I'll try to go experimental with 3.4, I >> want to understand if it is a problem with my build of 3.4 or if it is >> a general issue of 3.4 with some piece of BioPython. >> >> C:\Users\Public\BioPython3\biopython\Tests\SubsMat>python -c "import pickle; h = >> open('acc_rep_mat.pik', 'rb'); m = pickle.load(h);h.close();print(m); >> >> Traceback (most recent call last): >> File "", line 1, in >> _pickle.UnpicklingError: the STRING opcode argument must be quoted >> > > Good - I was hoping that would happen - we have a clear test > case separate from the Biopython code now. > > The pickle file acc_rep_mat.pik was created long ago (15 Nov 2000) > with Python 2, and I would have hoped it would be forward compatible. > This could be a regression in Python 3.4 itself (under Windows 64 bit > only)? > > If anyone else has a 64 bit Windows machine and Python 3.4, > please try this and let us know what happens. > > Thanks, > > Peter From manlio.calvi at gmail.com Mon Apr 21 17:51:28 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 21 Apr 2014 23:51:28 +0200 Subject: [Biopython-dev] Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? In-Reply-To: References: <20140417124408.5af66518@lnx> Message-ID: OK test on WinPython concluded. Analyzing it we have, the expected TogoWS problem, a skipped test for a non-compiled C module, two tests really but dependant from the same uncompiled module ( ################################### test_KDTree ... skipping. C module in Bio.KDTree not compiled ################################### and this errors: ====================================================================== ERROR: test_unicode_exception (test_Phylo.IOTests) ---------------------------------------------------------------------- Traceback (most recent call last): File ".\test_Phylo.py", line 57, in test_unicode_exception tree = Phylo.read(EX_NEWICK_BOM, 'newick') File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\_io.py", line 65, in read tree = next(tree_gen) File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\_io.py", line 53, in parse for tree in getattr(supported_formats[format], 'parse')(fp, **kwargs): File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\NewickIO.py", line 107, in parse for line in self.handle: File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 15: character maps to ====================================================================== ERROR: test_write_0 (test_Phylo_CDAO.WriterTests) Write and re-parse the phylogenies in test.cdao. ---------------------------------------------------------------------- Traceback (most recent call last): File ".\test_Phylo_CDAO.py", line 62, in test_write CDAOIO.write([t1], outfile) File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\CDAOIO.py", line 74, in write return Writer(trees).write(handle, plain=plain, **kwargs) File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\CDAOIO.py", line 268, in write handle.write('@prefix %s: <%s> .\n' % (k, v)) TypeError: 'str' does not support the buffer interface ################################################# The test output for CDAO reports a series of slightly messages (more than an hundred) that looks like this: file://C:\WinPython3.3\3.3 - Biopython test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Tests\CDAO\test.cdao/tree0000001#node0000001 does not look like a valid URI, trying to serialize this will break. Phylo.py related files was a recent commit, right? Manlio On Mon, Apr 21, 2014 at 7:02 PM, Manlio Calvi wrote: > I'm running the tests against a WinPython x64 3.3.3.3 at the moment > (so a non-Python.org distro) and will see if that happens there too, > I'm planning to install a native one(if the time permits). > > Manlio > > On Mon, Apr 21, 2014 at 6:33 PM, Peter Cock wrote: >> On Mon, Apr 21, 2014 at 5:25 PM, Manlio Calvi wrote: >>> Works as expected because I have Numpy modules running in other >>> scripts (dumps the path to __init__.py) >>> >>> In C:\Python34\Lib\site-packages\Bio the non-py files are two: >>> cpairwise2.pyd >>> trie.pyd >>> >>> The same are in the folder I used to build them, and by the extension >>> should be callable/importable. >>> I tried to import cpairwise2 but (don't) work the same way, same error >>> raised as before. >>> >> >> So, if you've done an install & can't import the C modules, that >> tells me this isn't a simply problem in our test harness (run_tests.py >> does some magic with the Python path to try to test using the just >> built files rather than any installed files). >> >> This is a deeper problem with building/installing under Python 3.4 >> on your setup. I have for the moment run out of ideas :( >> >> (Other than trying Python 3.3 which may behave differently - and >> will be useful anyway for checking the other test failures and if they >> are in part due to the new Python version). >> >> Thanks, >> >> Peter From manlio.calvi at gmail.com Tue Apr 22 01:51:45 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Tue, 22 Apr 2014 07:51:45 +0200 Subject: [Biopython-dev] Python 2.7 failure from test_Tutorial.py, was: Compiling Biopython on 64 bit Windows In-Reply-To: References: Message-ID: Hi Peter, I just run this one from the Tests folder (the tests are there and don'tget moved in the installation) I got this: Running Tutorial doctests... ********************************************************************** File "Tests/test_Tutorial.py", line 153, in __main__.TutorialDocTestHolder.doctest_test_from_line_04424 Failed example: align_array.shape Expected: (7, 52) Got: (7L, 52L) ********************************************************************** 1 items had failures: 1 of 7 in __main__.TutorialDocTestHolder.doctest_test_from_line_04424 ***Test Failed*** 1 failures. C:\python27\lib\site-packages\Bio\Motif\__init__.py:26: BiopythonDeprecationWarning: The module Bio.Motif has been deprecated and will be removed in a future release of Biopython. Instead please use the new module Bio.motifs instead. Please be aware that though the functionality of Bio.Motif is retained (and extended) in Bio.motifs, usage may be different. BiopythonDeprecationWarning) Traceback (most recent call last): File "Tests/test_Tutorial.py", line 185, in RuntimeError: 1/806 tests failed Another Python2/3 compatibility problem... On Mon, Apr 21, 2014 at 6:47 PM, Peter Cock wrote: > Hi Manlio, > > Moving on to Python 2.7, apart from the TogoWS issue, > http://lists.open-bio.org/pipermail/biopython-dev/2014-April/011259.html > your system showed one other failure: > > ====================================================================== > ERROR: test_doctests (test_Tutorial.TutorialTestCase) > Run tutorial doctests. > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "test_Tutorial.py", line 167, in test_doctests > ValueError: 1 Tutorial doctests failed: test_from_line_04424 > > ---------------------------------------------------------------------- > > > You should be able to rerun the tutorial tests alone (from memory > only if Biopython is installed) with just: > > c:\python27\python test_Tutorial.py > > This is from the following doctest in the Tutorial.tex file, > > %This example fails under PyPy 2.0, https://bugs.pypy.org/issue1546 > %doctest examples lib:numpy > \begin{verbatim} >>>> import numpy as np >>>> from Bio import AlignIO >>>> alignment = AlignIO.read("PF05371_seed.sth", "stockholm") >>>> align_array = np.array([list(rec) for rec in alignment], np.character) >>>> align_array.shape > (7, 52) > \end{verbatim} > > Could you try running that example "by hand" and seeing what > happens? Note the doctest framework is very very fragile and > the slightest output difference will break a test. > > Thanks, > > Peter > > On Fri, Apr 18, 2014 at 7:40 PM, Manlio Calvi wrote: >> And at last of this forward I'm posting the build and tests logs (more a >> dump than a log) for those who want peek into it. >> As you'll see the only consistent error was the TogoWS' XHTML error. >> >> Cheers, >> Manlio >> >> >> >> On Fri, Apr 18, 2014 at 8:35 PM, Manlio Calvi wrote: >> >>> >>> >>> ---------- Forwarded message ---------- >>> From: Peter Cock >>> Date: Fri, Apr 18, 2014 at 12:01 PM >>> Subject: Re: Compiling Biopython on 64 bit Windows, was: Volunteer >>> buildslave machines? >>> To: Manlio Calvi >>> Cc: Tiago Antao >>> >>> Manlio >>> >>> Excellent work - well done :) >>> >>> We should add some of this to the "Installing from source on Windows" >>> section of the installation notes (which we also need to update regarding >>> newer versions of Python and which compiler works): >>> >>> >>> https://github.com/biopython/biopython/blob/master/Doc/install/Installation.tex >>> >>> Step 6 sounds familiar, I think there is a similar batch file to set the >>> environment variables for the 43bit MSVC compiler. >>> >>> Regarding step 7, Python should use separate subdirectoroes under >>> build/ for each version of Python, so that shouldn't be a needed. >>> However, if you do use the same folder some of our tests use >>> hard coded temporary filenames so it is best to only test one version >>> of Python at a time. >>> >>> Regarding step 8, normally on Windows I don't bother with setting the >>> $PATH environment variable and simply call the explicit version of >>> Python I am testing, e.g. >>> >>> C:\Python27\python.exe setup.py build >>> C:\Python27\python.exe setup.py test >>> C:\Python27\python.exe setup.py install >>> >>> (This is what the current buildbot instructions do where we >>> have automated the tests). >>> >>> As to what's text - let's try to fix these test failures :) >>> Can you post the test output please? >>> >>> TogoWS is an online service, so from time to time it does have >>> transient failures (so does the NCBI Entrez API). The fact this >>> broke under both Python 2.7 and 3.4 suggests this is not a >>> Python 3 issue. >>> >>> I am a little surprised you didn't see this problem with Python 3.4, >>> but Windows does unicode a little differently so that may be why? >>> https://github.com/biopython/biopython/issues/305 >>> >>> How did you install git & clone Biopython? Specifically, what >>> is it doing about text files (i.e. Unix versus DOS/Windows >>> new lines). This was something which has caused test >>> failures in the past - perhaps some new cases have arisen. >>> >>> Thanks! >>> >>> Peter >>> >>> >> >> _______________________________________________ >> Biopython-dev mailing list >> Biopython-dev at lists.open-bio.org >> http://lists.open-bio.org/mailman/listinfo/biopython-dev >> From p.j.a.cock at googlemail.com Tue Apr 22 04:32:32 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 09:32:32 +0100 Subject: [Biopython-dev] Python 2.7 failure from test_Tutorial.py, was: Compiling Biopython on 64 bit Windows In-Reply-To: References: Message-ID: On Tue, Apr 22, 2014 at 6:51 AM, Manlio Calvi wrote: > Hi Peter, > I just run this one from the Tests folder (the tests are there and > don'tget moved in the installation) Sorry, yes - I should have said that. > I got this: > > Running Tutorial doctests... > ********************************************************************** > File "Tests/test_Tutorial.py", line 153, in > __main__.TutorialDocTestHolder.doctest_test_from_line_04424 > Failed example: > align_array.shape > Expected: > (7, 52) > Got: > (7L, 52L) > ********************************************************************** > 1 items had failures: > 1 of 7 in __main__.TutorialDocTestHolder.doctest_test_from_line_04424 > ***Test Failed*** 1 failures. > C:\python27\lib\site-packages\Bio\Motif\__init__.py:26: > BiopythonDeprecationWarning: The module Bio.Motif has been deprecated > and will be removed in a future release of Biopython. Instead please > use the new module Bio.motifs instead. Please be aware that though the > functionality of Bio.Motif is retained (and extended) in Bio.motifs, > usage may be different. > BiopythonDeprecationWarning) > Traceback (most recent call last): > File "Tests/test_Tutorial.py", line 185, in > RuntimeError: 1/806 tests failed > > > Another Python2/3 compatibility problem... Well yes, but also a 32 bit versus 64 bit Windows issue. Python 3 hides the C level differences between ints and longs, while Python 2 can sometimes expose this - odd that this test didn't give longs on the other platforms, just 64 bit windows. I suggest we switch from: %This example fails under PyPy 2.0, https://bugs.pypy.org/issue1546 %doctest examples lib:numpy \begin{verbatim} >>> import numpy as np >>> from Bio import AlignIO >>> alignment = AlignIO.read("PF05371_seed. sth", "stockholm") >>> align_array = np.array([list(rec) for rec in alignment], np.character) >>> align_array.shape (7, 52) \end{verbatim} To: %This example fails under PyPy 2.0, https://bugs.pypy.org/issue1546 %doctest examples lib:numpy \begin{verbatim} >>> import numpy as np >>> from Bio import AlignIO >>> alignment = AlignIO.read("PF05371_seed.sth", "stockholm") >>> align_array = np.array([list(rec) for rec in alignment], np.character) >>> print("Array shape %i by %i" % align_array.shape) Array shape 7 by 52 \end{verbatim} That should be safe :) Peter From tra at popgen.net Tue Apr 22 05:23:16 2014 From: tra at popgen.net (Tiago Antao) Date: Tue, 22 Apr 2014 10:23:16 +0100 Subject: [Biopython-dev] Python 3.4 Message-ID: <20140422102316.5b049105@lnx> Dear all (especially Manlio and Peter ;) ), With Ubuntu trusty officially out, I am planning on doing a (Linux) docker file running Python 3.4 and attach it to buildbot. Hopefully today. This would allow us to have a stable testing environment of 3.4 against Linux... Which might give some clues to what is happening with 3.4 on Windows... Tiago From p.j.a.cock at googlemail.com Tue Apr 22 05:42:33 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 10:42:33 +0100 Subject: [Biopython-dev] Python 3.4 In-Reply-To: <20140422102316.5b049105@lnx> References: <20140422102316.5b049105@lnx> Message-ID: On Tue, Apr 22, 2014 at 10:23 AM, Tiago Antao wrote: > Dear all (especially Manlio and Peter ;) ), > > With Ubuntu trusty officially out, I am planning on doing a (Linux) > docker file running Python 3.4 and attach it to buildbot. Hopefully > today. This would allow us to have a stable testing environment of 3.4 > against Linux... Which might give some clues to what is happening with > 3.4 on Windows... > > Tiago Good plan :) I've already got Python 3.4 on three build slaves ready to be connected (Windows XP 32 bit, Linux 64 bit, and Mac OS X 64 bit), so we'll have the main platforms covered. Right now the only Python 3.4 specific issue I have seen is in Bio.trie https://github.com/biopython/biopython/issues/305 I am hoping we'll have everything ready on Python 3.4 for the next release of Biopython - aim for early May? Peter From p.j.a.cock at googlemail.com Tue Apr 22 05:57:23 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 10:57:23 +0100 Subject: [Biopython-dev] Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? In-Reply-To: References: <20140417124408.5af66518@lnx> Message-ID: Hi Manlio, There are several issues here - On Mon, Apr 21, 2014 at 10:51 PM, Manlio Calvi wrote: > OK test on WinPython concluded. Analyzing it we have, the expected > TogoWS problem, (which the TogoWS team will be fixing shortly) > a skipped test for a non-compiled C module, two tests really but > dependant from the same uncompiled module ( > ################################### > test_KDTree ... skipping. C module in Bio.KDTree not compiled > ################################### (plus the other C code modules not getting compiled/installed, which is a concern) > and this errors: > > ====================================================================== > ERROR: test_unicode_exception (test_Phylo.IOTests) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File ".\test_Phylo.py", line 57, in test_unicode_exception > tree = Phylo.read(EX_NEWICK_BOM, 'newick') > File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\_io.py", > line 65, in read > tree = next(tree_gen) > File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\_io.py", > line 53, in parse > for tree in getattr(supported_formats[format], 'parse')(fp, **kwargs): > File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\NewickIO.py", > line 107, in parse > for line in self.handle: > File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\lib\encodings\cp1252.py", > line 23, in decode > return codecs.charmap_decode(input,self.errors,decoding_table)[0] > UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position > 15: character maps to > As you suggested below, the test_unicode_exception failure from test_Phylo.py is from a recent commit, and likely has to do with subtle Windows unicode differences: https://github.com/biopython/biopython/commit/94d1e54b80df9e0fbdd3b56b30e40b860e48f26f The 'good news' is this also broke under 32 bit Windows XP, http://testing.open-bio.org/biopython/builders/Windows%20XP%20-%20Python%203.3/builds/557/steps/shell/logs/stdio I have filed a bug: https://github.com/biopython/biopython/issues/311 > ====================================================================== > ERROR: test_write_0 (test_Phylo_CDAO.WriterTests) > Write and re-parse the phylogenies in test.cdao. > ---------------------------------------------------------------------- > Traceback (most recent call last): > File ".\test_Phylo_CDAO.py", line 62, in test_write > CDAOIO.write([t1], outfile) > File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\CDAOIO.py", > line 74, in write > return Writer(trees).write(handle, plain=plain, **kwargs) > File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\CDAOIO.py", > line 268, in write > handle.write('@prefix %s: <%s> .\n' % (k, v)) > TypeError: 'str' does not support the buffer interface > That looks like a string versus bytes issue with the handle (text versus binary mode), but I don't recall any changes in that code... maybe a question for Eric? > ################################################# > > > The test output for CDAO reports a series of slightly messages (more > than an hundred) that looks like this: > file://C:\WinPython3.3\3.3 - Biopython > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Tests\CDAO\test.cdao/tree0000001#node0000001 > does not look like a valid URI, trying to serialize this will break. My hunch is this warning could be due to the mixed slashes (forward and back) which is typically fine with the windows API for filenames, but evidently is confusing the CDAO code - a question for Ben? Thanks, Peter From manlio.calvi at gmail.com Tue Apr 22 05:57:52 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Tue, 22 Apr 2014 11:57:52 +0200 Subject: [Biopython-dev] Python 3.4 In-Reply-To: <20140422102316.5b049105@lnx> References: <20140422102316.5b049105@lnx> Message-ID: This is a good news Tiago :) My problem now is try to mantain a "standard build" of 3.4 x64 and another more "experimental" and separate from the first. The only way I know to have it "native" is using the virtual env that came with Python 3 that I never used before, and I don't know how reliable is in a testing environment. Apparently none of the python distribution went to 3.4 now. Seems I should learn how to use virtual envs or remain with my 3.4 only. What I would like to "experiment" is build up all the sources (or maybe only the modules sources) from the ground up, specifically I want to see if there is different behaviours using the modules from Golhke's site or not. Manlio On Tue, Apr 22, 2014 at 11:23 AM, Tiago Antao wrote: > Dear all (especially Manlio and Peter ;) ), > > With Ubuntu trusty officially out, I am planning on doing a (Linux) > docker file running Python 3.4 and attach it to buildbot. Hopefully > today. This would allow us to have a stable testing environment of 3.4 > against Linux... Which might give some clues to what is happening with > 3.4 on Windows... > > Tiago From p.j.a.cock at googlemail.com Tue Apr 22 06:44:18 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 11:44:18 +0100 Subject: [Biopython-dev] Pickle problem on 64 bit Windows with Python 3.4 In-Reply-To: References: Message-ID: On Mon, Apr 21, 2014 at 6:45 PM, Manlio Calvi wrote: > From what I read here http://hg.python.org/cpython/rev/4a6b8f86b081 could be > a problem related to that file. Seems to me they stripped the check for a > quote that must be in, and looking at the pickle apparently isn't > OK, now things are more confusing - this seems to be working on a colleague's machine, so it may be something different on your setup. Are you using a self compiled Python 3.4? We installed the 64 bit version Python 3.4 on Windows 7 using the binary installed from the website (Windows x86-64 MSI installer), selecting for all users (which probably requires admin rights): https://www.python.org/ftp/python/3.4.0/python-3.4.0.amd64.msi We manually downloaded the pickle file via the raw link on GitHub, and tried the test code (as shown below), and it worked perfectly. Regards, Peter -- C:\Users\xxx\Desktop>ls *.pik acc_rep_mat.pik C:\Users\xxx\Desktop>c:\python34\python.exe -c "import pickle; h = open('acc_rep_mat.pik', 'rb'); m = pickle.load(h);h.close();print(m);" {('K', 'Q'): 108, ('A', 'F'): 122, ('P', 'W'): 18, ('D', 'F'): 63, ('S', 'Y'): 7 3, ('C', 'M'): 14, ('A', 'N'): 139, ('K', 'Y'): 66, ('I', 'V'): 310, ('G', 'V'): 131, ('T', 'Y'): 91, ('M', 'T'): 35, ('F', 'V'): 160, ('D', 'Q'): 116, ('H', 'Y '): 43, ('R', 'T'): 114, ('A', 'Q'): 129, ('I', 'P'): 67, ('N', 'V'): 93, ('C', 'R'): 15, ('E', 'F'): 50, ('A', 'R'): 152, ('D', 'S'): 167, ('L', 'T'): 183, ('L ', 'P'): 79, ('I', 'M'): 88, ('C', 'K'): 24, ('E', 'K'): 164, ('G', 'N'): 147, ( 'P', 'T'): 92, ('L', 'R'): 109, ('H', 'T'): 69, ('E', 'G'): 146, ('I', 'T'): 88, ('L', 'Y'): 173, ('F', 'W'): 43, ('T', 'T'): 84, ('D', 'W'): 22, ('M', 'V'): 95 , ('K', 'T'): 140, ('W', 'Y'): 51, ('E', 'N'): 115, ('R', 'V'): 88, ('R', 'Y'): 62, ('D', 'Y'): 64, ('G', 'Q'): 89, ('D', 'T'): 125, ('F', 'K'): 67, ('W', 'W'): 32, ('C', 'H'): 10, ('R', 'W'): 23, ('I', 'W'): 38, ('A', 'H'): 66, ('L', 'V'): 447, ('Q', 'Q'): 42, ('D', 'P'): 105, ('D', 'H'): 52, ('M', 'W'): 25, ('S', 'T' ): 172, ('D', 'R'): 115, ('L', 'W'): 74, ('F', 'I'): 151, ('C', 'G'): 39, ('H', 'Q'): 44, ('C', 'V'): 59, ('N', 'P'): 94, ('D', 'E'): 163, ('C', 'E'): 26, ('R', 'R'): 96, ('G', 'K'): 148, ('C', 'S'): 23, ('F', 'Q'): 39, ('G', 'H'): 53, ('R' , 'S'): 94, ('D', 'I'): 66, ('G', 'R'): 81, ('N', 'Q'): 77, ('M', 'Q'): 27, ('D' , 'N'): 163, ('C', 'Q'): 16, ('Q', 'W'): 15, ('A', 'T'): 195, ('Y', 'Y'): 82, (' C', 'F'): 32, ('E', 'H'): 38, ('F', 'S'): 92, ('M', 'P'): 20, ('D', 'L'): 120, ( 'A', 'M'): 79, ('C', 'I'): 42, ('I', 'N'): 65, ('E', 'W'): 21, ('F', 'L'): 236, ('G', 'L'): 154, ('A', 'E'): 207, ('G', 'W'): 35, ('H', 'H'): 56, ('I', 'Q'): 62 , ('L', 'L'): 320, ('H', 'N'): 57, ('G', 'G'): 321, ('A', 'C'): 49, ('H', 'I'): 31, ('N', 'W'): 29, ('A', 'I'): 202, ('C', 'L'): 62, ('A', 'W'): 47, ('L', 'S'): 159, ('K', 'S'): 144, ('D', 'D'): 151, ('C', 'P'): 21, ('A', 'D'): 171, ('E', ' L'): 133, ('S', 'S'): 116, ('K', 'R'): 151, ('Q', 'V'): 69, ('L', 'M'): 153, ('A ', 'Y'): 124, ('E', 'T'): 150, ('K', 'P'): 94, ('D', 'V'): 95, ('F', 'G'): 102, ('H', 'L'): 70, ('H', 'K'): 63, ('C', 'Y'): 19, ('K', 'M'): 32, ('F', 'F'): 79, ('H', 'R'): 44, ('P', 'R'): 72, ('M', 'Y'): 41, ('A', 'P'): 119, ('G', 'S'): 161 , ('H', 'P'): 23, ('D', 'G'): 143, ('E', 'M'): 33, ('K', 'N'): 139, ('G', 'I'): 92, ('K', 'W'): 21, ('I', 'S'): 84, ('C', 'T'): 30, ('D', 'K'): 169, ('P', 'Y'): 60, ('I', 'R'): 58, ('M', 'S'): 41, ('N', 'N'): 66, ('N', 'T'): 105, ('F', 'H') : 39, ('F', 'T'): 69, ('F', 'M'): 67, ('C', 'W'): 8, ('D', 'M'): 20, ('K', 'K'): 121, ('Q', 'Y'): 41, ('V', 'W'): 44, ('C', 'C'): 42, ('M', 'M'): 22, ('T', 'W') : 35, ('A', 'V'): 252, ('F', 'N'): 55, ('P', 'V'): 109, ('I', 'K'): 80, ('H', 'V '): 54, ('N', 'R'): 90, ('M', 'N'): 34, ('E', 'Q'): 105, ('A', 'S'): 199, ('G', 'M'): 49, ('H', 'W'): 13, ('N', 'S'): 115, ('T', 'V'): 145, ('E', 'I'): 63, ('F' , 'R'): 57, ('Q', 'S'): 101, ('K', 'V'): 131, ('K', 'L'): 112, ('A', 'L'): 307, ('Q', 'R'): 73, ('E', 'S'): 143, ('E', 'Y'): 60, ('A', 'K'): 191, ('A', 'G'): 22 9, ('V', 'Y'): 134, ('H', 'S'): 59, ('P', 'S'): 103, ('N', 'Y'): 75, ('G', 'T'): 153, ('G', 'P'): 108, ('I', 'Y'): 94, ('E', 'P'): 91, ('C', 'D'): 15, ('S', 'W' ): 22, ('S', 'V'): 114, ('L', 'N'): 101, ('E', 'E'): 119, ('Q', 'T'): 97, ('E', 'R'): 118, ('F', 'P'): 50, ('P', 'Q'): 68, ('I', 'I'): 145, ('H', 'M'): 16, ('G' , 'Y'): 101, ('F', 'Y'): 142, ('C', 'N'): 24, ('A', 'A'): 211, ('E', 'V'): 110, ('L', 'Q'): 79, ('P', 'P'): 120, ('I', 'L'): 348, ('M', 'R'): 40, ('V', 'V'): 23 9} C:\Users\xxx3\Desktop>c:\python34\python.exe -V Python 3.4.0 C:\Users\xxx\Desktop>c:\python34\python.exe Python 3.4.0 (v3.4.0:04f714765c13, Mar 16 2014, 19:25:23) [MSC v.1600 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> quit() So, this works perfectly here. Peter From manlio.calvi at gmail.com Tue Apr 22 07:09:31 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Tue, 22 Apr 2014 13:09:31 +0200 Subject: [Biopython-dev] Pickle problem on 64 bit Windows with Python 3.4 In-Reply-To: References: Message-ID: On Tue, Apr 22, 2014 at 12:44 PM, Peter Cock wrote: > On Mon, Apr 21, 2014 at 6:45 PM, Manlio Calvi wrote: >> From what I read here http://hg.python.org/cpython/rev/4a6b8f86b081 could be >> a problem related to that file. Seems to me they stripped the check for a >> quote that must be in, and looking at the pickle apparently isn't >> > > OK, now things are more confusing - this seems to be working on > a colleague's machine, so it may be something different on your > setup. Are you using a self compiled Python 3.4? > > We installed the 64 bit version Python 3.4 on Windows 7 using the > binary installed from the website (Windows x86-64 MSI installer), > selecting for all users (which probably requires admin rights): > https://www.python.org/ftp/python/3.4.0/python-3.4.0.amd64.msi Exactly as I did, I installed the dependencies (numpy and the like) for Biopython using Gohlke's ones. > We manually downloaded the pickle file via the raw link on GitHub, > and tried the test code (as shown below), and it worked perfectly. I've used the standard "git pull" command from the repository. Moreover I'm coming from a recent format and reinstall of windows in this machine. I'm a bit lost here... From p.j.a.cock at googlemail.com Tue Apr 22 08:36:23 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 13:36:23 +0100 Subject: [Biopython-dev] Pickle problem on 64 bit Windows with Python 3.4 In-Reply-To: References: Message-ID: On Tue, Apr 22, 2014 at 12:09 PM, Manlio Calvi wrote: > On Tue, Apr 22, 2014 at 12:44 PM, Peter Cock wrote: >> On Mon, Apr 21, 2014 at 6:45 PM, Manlio Calvi wrote: >>> From what I read here http://hg.python.org/cpython/rev/4a6b8f86b081 could be >>> a problem related to that file. Seems to me they stripped the check for a >>> quote that must be in, and looking at the pickle apparently isn't >>> >> >> OK, now things are more confusing - this seems to be working on >> a colleague's machine, so it may be something different on your >> setup. Are you using a self compiled Python 3.4? >> >> We installed the 64 bit version Python 3.4 on Windows 7 using the >> binary installed from the website (Windows x86-64 MSI installer), >> selecting for all users (which probably requires admin rights): >> https://www.python.org/ftp/python/3.4.0/python-3.4.0.amd64.msi > > Exactly as I did, I installed the dependencies (numpy and the like) > for Biopython using Gohlke's ones. > >> We manually downloaded the pickle file via the raw link on GitHub, >> and tried the test code (as shown below), and it worked perfectly. > > I've used the standard "git pull" command from the repository. > Moreover I'm coming from a recent format and reinstall of windows in > this machine. > I'm a bit lost here... OK, I had an idea over lunch which turned out to solve this :) First I checked that my pickle on Linux file uses Unix new lines, $ hexdump -C acc_rep_mat.pik | head 00000000 28 64 70 31 0a 28 53 27 4c 27 0a 53 27 52 27 0a |(dp1.(S'L'.S'R'.| 00000010 74 49 31 30 39 0a 73 28 53 27 49 27 0a 53 27 49 |tI109.s(S'I'.S'I| 00000020 27 0a 74 49 31 34 35 0a 73 28 53 27 51 27 0a 53 |'.tI145.s(S'Q'.S| 00000030 27 51 27 0a 74 49 34 32 0a 73 28 53 27 53 27 0a |'Q'.tI42.s(S'S'.| 00000040 53 27 54 27 0a 74 49 31 37 32 0a 73 28 53 27 48 |S'T'.tI172.s(S'H| 00000050 27 0a 53 27 54 27 0a 74 49 36 39 0a 73 28 53 27 |'.S'T'.tI69.s(S'| 00000060 51 27 0a 53 27 59 27 0a 74 49 34 31 0a 73 28 53 |Q'.S'Y'.tI41.s(S| 00000070 27 48 27 0a 53 27 50 27 0a 74 49 32 33 0a 73 28 |'H'.S'P'.tI23.s(| 00000080 53 27 4e 27 0a 53 27 59 27 0a 74 49 37 35 0a 73 |S'N'.S'Y'.tI75.s| 00000090 28 53 27 48 27 0a 53 27 4c 27 0a 74 49 37 30 0a |(S'H'.S'L'.tI70.| Then I converted it to DOS/Windows newlines (e.g. unix2dos is easy if you have that, or a few lines of Python if not - see below): $ hexdump -C acc_rep_mat.dos.pik | head 00000000 28 64 70 31 0d 0a 28 53 27 4c 27 0d 0a 53 27 52 |(dp1..(S'L'..S'R| 00000010 27 0d 0a 74 49 31 30 39 0d 0a 73 28 53 27 49 27 |'..tI109..s(S'I'| 00000020 0d 0a 53 27 49 27 0d 0a 74 49 31 34 35 0d 0a 73 |..S'I'..tI145..s| 00000030 28 53 27 51 27 0d 0a 53 27 51 27 0d 0a 74 49 34 |(S'Q'..S'Q'..tI4| 00000040 32 0d 0a 73 28 53 27 53 27 0d 0a 53 27 54 27 0d |2..s(S'S'..S'T'.| 00000050 0a 74 49 31 37 32 0d 0a 73 28 53 27 48 27 0d 0a |.tI172..s(S'H'..| 00000060 53 27 54 27 0d 0a 74 49 36 39 0d 0a 73 28 53 27 |S'T'..tI69..s(S'| 00000070 51 27 0d 0a 53 27 59 27 0d 0a 74 49 34 31 0d 0a |Q'..S'Y'..tI41..| 00000080 73 28 53 27 48 27 0d 0a 53 27 50 27 0d 0a 74 49 |s(S'H'..S'P'..tI| 00000090 32 33 0d 0a 73 28 53 27 4e 27 0d 0a 53 27 59 27 |23..s(S'N'..S'Y'| This increases the file size from 3658 bytes to 4289 bytes. $ python3.4 -c "import pickle; h=open('acc_rep_mat.pik', 'rb'); m=pickle.load(h); h.close(); print(m)" {('E', 'M'): 33, ..., ('D', 'V'): 95} $ python3.4 -c "import pickle; h=open('acc_rep_mat.dos.pik', 'rb'); m=pickle.load(h); h.close(); print(m)" Traceback (most recent call last): File "", line 1, in _pickle.UnpicklingError: the STRING opcode argument must be quoted So I can get the exact same error under Linux now :) I confirmed this on Windows where my copy of git is setup to use Unix newlines by default (I think), and the file has Unix newlines (and is 3658 bytes). C:\repositories\biopython\Tests\SubsMat>c:\python34\python Python 3.4.0 (v3.4.0:04f714765c13, Mar 16 2014, 19:24:06) [MSC v.1600 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> data = open("acc_rep_mat.pik", "rb").read() >>> with open("acc_rep_mat.dos.pik", "wb") as h: h.write(data.replace(b"\n", b"\r\n")) ... 4289 >>> quit() C:\repositories\biopython\Tests\SubsMat>c:\python34\python -c "import pickle; h=open('acc_rep_mat.pik', 'rb'); m=pickle.load(h); h.close(); print(m)" {('D', 'R'): 115, ..., ('H', 'Q'): 44} C:\repositories\biopython\Tests\SubsMat>c:\python34\python -c "import pickle; h=open('acc_rep_mat.dos.pik', 'rb'); m=pickle.load(h); h.close(); print(m)" Traceback (most recent call last): File "", line 1, in _pickle.UnpicklingError: the STRING opcode argument must be quoted So, the upshot is that this git setting change should fix it: https://github.com/biopython/biopython/commit/b7cc2fe199d22f794612d68e5554361413468372 Could you update your copy of the Biopython source code via git, and see if that solves this pickle? Thank you, Peter From p.j.a.cock at googlemail.com Tue Apr 22 08:57:34 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 13:57:34 +0100 Subject: [Biopython-dev] Python 2.7 failure from test_Tutorial.py, was: Compiling Biopython on 64 bit Windows In-Reply-To: References: Message-ID: On Tue, Apr 22, 2014 at 9:32 AM, Peter Cock wrote: > On Tue, Apr 22, 2014 at 6:51 AM, Manlio Calvi wrote: >> Hi Peter, >> I just run this one from the Tests folder (the tests are there and >> don'tget moved in the installation) > > Sorry, yes - I should have said that. > >> I got this: >> >> Running Tutorial doctests... >> ********************************************************************** >> File "Tests/test_Tutorial.py", line 153, in >> __main__.TutorialDocTestHolder.doctest_test_from_line_04424 >> Failed example: >> align_array.shape >> Expected: >> (7, 52) >> Got: >> (7L, 52L) >> ********************************************************************** >> 1 items had failures: >> 1 of 7 in __main__.TutorialDocTestHolder.doctest_test_from_line_04424 >> ***Test Failed*** 1 failures. >> C:\python27\lib\site-packages\Bio\Motif\__init__.py:26: >> BiopythonDeprecationWarning: The module Bio.Motif has been deprecated >> and will be removed in a future release of Biopython. Instead please >> use the new module Bio.motifs instead. Please be aware that though the >> functionality of Bio.Motif is retained (and extended) in Bio.motifs, >> usage may be different. >> BiopythonDeprecationWarning) >> Traceback (most recent call last): >> File "Tests/test_Tutorial.py", line 185, in >> RuntimeError: 1/806 tests failed >> >> >> Another Python2/3 compatibility problem... > > Well yes, but also a 32 bit versus 64 bit Windows issue. > Python 3 hides the C level differences between ints and longs, > while Python 2 can sometimes expose this - odd that this test > didn't give longs on the other platforms, just 64 bit windows. > > I suggest we switch from: > > %This example fails under PyPy 2.0, https://bugs.pypy.org/issue1546 > %doctest examples lib:numpy > \begin{verbatim} >>>> import numpy as np >>>> from Bio import AlignIO >>>> alignment = AlignIO.read("PF05371_seed. > sth", "stockholm") >>>> align_array = np.array([list(rec) for rec in alignment], np.character) >>>> align_array.shape > (7, 52) > \end{verbatim} > > To: > > %This example fails under PyPy 2.0, https://bugs.pypy.org/issue1546 > %doctest examples lib:numpy > \begin{verbatim} >>>> import numpy as np >>>> from Bio import AlignIO >>>> alignment = AlignIO.read("PF05371_seed.sth", "stockholm") >>>> align_array = np.array([list(rec) for rec in alignment], np.character) >>>> print("Array shape %i by %i" % align_array.shape) > Array shape 7 by 52 > \end{verbatim} > > That should be safe :) > > Peter Change applied to git, https://github.com/biopython/biopython/commit/0e26dfb40ab1ce78319eaf6893e35abda43c8738 Peter From p.j.a.cock at googlemail.com Tue Apr 22 09:08:40 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 14:08:40 +0100 Subject: [Biopython-dev] CDAO problems on Windows, was: Compiling Biopython on 64 bit Windows Message-ID: On Tue, Apr 22, 2014 at 10:57 AM, Peter Cock wrote: > On Mon, Apr 21, 2014 at 10:51 PM, Manlio Calvi wrote: >> ... >> >> ====================================================================== >> ERROR: test_write_0 (test_Phylo_CDAO.WriterTests) >> Write and re-parse the phylogenies in test.cdao. >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File ".\test_Phylo_CDAO.py", line 62, in test_write >> CDAOIO.write([t1], outfile) >> File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython >> test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\CDAOIO.py", >> line 74, in write >> return Writer(trees).write(handle, plain=plain, **kwargs) >> File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython >> test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\CDAOIO.py", >> line 268, in write >> handle.write('@prefix %s: <%s> .\n' % (k, v)) >> TypeError: 'str' does not support the buffer interface >> > > That looks like a string versus bytes issue with the handle > (text versus binary mode), but I don't recall any changes > in that code... maybe a question for Eric? I've not looked into this issue yet, however the reason the existing Windows buildbot never reported this is that I have not installed RDFlib on it (not even for Python 2.6 and 2.7). That was an oversight - and part of the reason to try and get more buildslaves - diversity. >> ################################################# >> >> >> The test output for CDAO reports a series of slightly messages (more >> than an hundred) that looks like this: >> file://C:\WinPython3.3\3.3 - Biopython >> test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Tests\CDAO\test.cdao/tree0000001#node0000001 >> does not look like a valid URI, trying to serialize this will break. > > My hunch is this warning could be due to the mixed slashes > (forward and back) which is typically fine with the windows > API for filenames, but evidently is confusing the CDAO > code - a question for Ben? According to the RDFLib docstring for their URI class, yes, they issue this warning if the URI contains a Windows-style forward slash: http://rdflib.readthedocs.org/en/latest/_modules/rdflib/term.html The relevant snippet of their code is: _invalid_uri_chars = '<>" {}|\\^`' def _is_valid_uri(uri): for c in _invalid_uri_chars: if c in uri: return False return True My suggestion would be to switch all the "\" into "/" here. Peter From p.j.a.cock at googlemail.com Tue Apr 22 12:11:33 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 17:11:33 +0100 Subject: [Biopython-dev] Fwd: [GSoC] Welcome aboard, GSoC 2014 students! In-Reply-To: References: Message-ID: Dear Biopythoneers, Please join me in congratulating this year's accepted students for Google Summer of Code under the Open Bioinformatics Foundation (OBF), and in particular welcome Evan Parker who will be working on Biopython with Bow and myself as co-mentors. As always the scheme has been very competitive, so our sympathies and commiserations to those students who were not accepted. Please do stay involved in Biopython or other open source projects - this would be a positive factor if you are eligible to apply for next year's GSoC. Thank you, Peter ---------- Forwarded message ---------- From: Eric Talevich Date: Tue, Apr 22, 2014 at 4:41 AM Subject: [GSoC] Welcome aboard, GSoC 2014 students! To: OBF GSoC Hi all, I'm pleased to announce the acceptance of OBF's 2010 Google Summer of Code students: Sarah Berkemer - "Open source high-performance BioHaskell" (Mentors: Christian H?ner zu Siederdissen, Ketil Malde) Loris Cro - "An ultra-fast scalable RESTful API to query large numbers of VCF datapoints" (Mentors: Francesco Strozzi, Raoul Bonnal & the BioRuby team) Victor Kofia - "JSBML: Redesign the implementation of mathematical formulas" (Mentors: Alex Thomas, Sarah Keating & the JSBML team) Evan Parker - "Addition of a lazy loading sequence parser to Biopython's SeqIO package" (Mentors: Wibowo Arindrarto, Peter Cock & the Biopython team) Ibrahim Vairabad - "Improving the Plug-in interface for CellDesigner" (Mentors: Andreas Dr?ger, Alex Thomas & the JSBML team) Leandro Watanabe - "Dynamic Modeling of Cellular Populations within JSBML" (Mentors: Nicolas Rodriguez, Chris Meyers & the JSBML team) Congratulations to our accepted students! Thanks very much to all the students who applied, we very much appreciate your hard work. Today marks the start of the Community Bonding Period. Official work starts on May 23rd, and until then, students should prepare for their projects: get on the project mailing lists, solidify your plans, figure out where all the version control repositories are and which branch or fork you'll be working on, and start doing preparatory work. Students: if you have not done so already, make sure you have subscribed to the OBF GSoC email list at: http://lists.open-bio.org/mailman/listinfo/gsoc This list is for discussions among students and mentors, and for administrative announcements from me or my co-administrators. Here's to a great 2014 Summer of Code, Eric & Raoul OBF GSoC 2014 Organization Administrators _______________________________________________ GSoC mailing list GSoC at lists.open-bio.org http://lists.open-bio.org/mailman/listinfo/gsoc From manlio.calvi at gmail.com Tue Apr 22 14:40:09 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Tue, 22 Apr 2014 20:40:09 +0200 Subject: [Biopython-dev] Python 2.7 failure from test_Tutorial.py, was: Compiling Biopython on 64 bit Windows In-Reply-To: References: Message-ID: OK tested, all good on my machine, the only failed test is the TogoWS that don't depend on us. Manlio On Tue, Apr 22, 2014 at 2:57 PM, Peter Cock wrote: > On Tue, Apr 22, 2014 at 9:32 AM, Peter Cock wrote: >> On Tue, Apr 22, 2014 at 6:51 AM, Manlio Calvi wrote: >>> Hi Peter, >>> I just run this one from the Tests folder (the tests are there and >>> don'tget moved in the installation) >> >> Sorry, yes - I should have said that. >> >>> I got this: >>> >>> Running Tutorial doctests... >>> ********************************************************************** >>> File "Tests/test_Tutorial.py", line 153, in >>> __main__.TutorialDocTestHolder.doctest_test_from_line_04424 >>> Failed example: >>> align_array.shape >>> Expected: >>> (7, 52) >>> Got: >>> (7L, 52L) >>> ********************************************************************** >>> 1 items had failures: >>> 1 of 7 in __main__.TutorialDocTestHolder.doctest_test_from_line_04424 >>> ***Test Failed*** 1 failures. >>> C:\python27\lib\site-packages\Bio\Motif\__init__.py:26: >>> BiopythonDeprecationWarning: The module Bio.Motif has been deprecated >>> and will be removed in a future release of Biopython. Instead please >>> use the new module Bio.motifs instead. Please be aware that though the >>> functionality of Bio.Motif is retained (and extended) in Bio.motifs, >>> usage may be different. >>> BiopythonDeprecationWarning) >>> Traceback (most recent call last): >>> File "Tests/test_Tutorial.py", line 185, in >>> RuntimeError: 1/806 tests failed >>> >>> >>> Another Python2/3 compatibility problem... >> >> Well yes, but also a 32 bit versus 64 bit Windows issue. >> Python 3 hides the C level differences between ints and longs, >> while Python 2 can sometimes expose this - odd that this test >> didn't give longs on the other platforms, just 64 bit windows. >> >> I suggest we switch from: >> >> %This example fails under PyPy 2.0, https://bugs.pypy.org/issue1546 >> %doctest examples lib:numpy >> \begin{verbatim} >>>>> import numpy as np >>>>> from Bio import AlignIO >>>>> alignment = AlignIO.read("PF05371_seed. >> sth", "stockholm") >>>>> align_array = np.array([list(rec) for rec in alignment], np.character) >>>>> align_array.shape >> (7, 52) >> \end{verbatim} >> >> To: >> >> %This example fails under PyPy 2.0, https://bugs.pypy.org/issue1546 >> %doctest examples lib:numpy >> \begin{verbatim} >>>>> import numpy as np >>>>> from Bio import AlignIO >>>>> alignment = AlignIO.read("PF05371_seed.sth", "stockholm") >>>>> align_array = np.array([list(rec) for rec in alignment], np.character) >>>>> print("Array shape %i by %i" % align_array.shape) >> Array shape 7 by 52 >> \end{verbatim} >> >> That should be safe :) >> >> Peter > > Change applied to git, > https://github.com/biopython/biopython/commit/0e26dfb40ab1ce78319eaf6893e35abda43c8738 > > Peter From manlio.calvi at gmail.com Tue Apr 22 15:09:54 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Tue, 22 Apr 2014 21:09:54 +0200 Subject: [Biopython-dev] Pickle problem on 64 bit Windows with Python 3.4 In-Reply-To: References: Message-ID: Uhm no, I get the same error as before... Seems this machine don't like conserved vegetables. Manlio On Tue, Apr 22, 2014 at 2:36 PM, Peter Cock wrote: > On Tue, Apr 22, 2014 at 12:09 PM, Manlio Calvi wrote: >> On Tue, Apr 22, 2014 at 12:44 PM, Peter Cock wrote: >>> On Mon, Apr 21, 2014 at 6:45 PM, Manlio Calvi wrote: >>>> From what I read here http://hg.python.org/cpython/rev/4a6b8f86b081 could be >>>> a problem related to that file. Seems to me they stripped the check for a >>>> quote that must be in, and looking at the pickle apparently isn't >>>> >>> >>> OK, now things are more confusing - this seems to be working on >>> a colleague's machine, so it may be something different on your >>> setup. Are you using a self compiled Python 3.4? >>> >>> We installed the 64 bit version Python 3.4 on Windows 7 using the >>> binary installed from the website (Windows x86-64 MSI installer), >>> selecting for all users (which probably requires admin rights): >>> https://www.python.org/ftp/python/3.4.0/python-3.4.0.amd64.msi >> >> Exactly as I did, I installed the dependencies (numpy and the like) >> for Biopython using Gohlke's ones. >> >>> We manually downloaded the pickle file via the raw link on GitHub, >>> and tried the test code (as shown below), and it worked perfectly. >> >> I've used the standard "git pull" command from the repository. >> Moreover I'm coming from a recent format and reinstall of windows in >> this machine. >> I'm a bit lost here... > > OK, I had an idea over lunch which turned out to solve this :) > > First I checked that my pickle on Linux file uses Unix new lines, > > $ hexdump -C acc_rep_mat.pik | head > 00000000 28 64 70 31 0a 28 53 27 4c 27 0a 53 27 52 27 0a |(dp1.(S'L'.S'R'.| > 00000010 74 49 31 30 39 0a 73 28 53 27 49 27 0a 53 27 49 |tI109.s(S'I'.S'I| > 00000020 27 0a 74 49 31 34 35 0a 73 28 53 27 51 27 0a 53 |'.tI145.s(S'Q'.S| > 00000030 27 51 27 0a 74 49 34 32 0a 73 28 53 27 53 27 0a |'Q'.tI42.s(S'S'.| > 00000040 53 27 54 27 0a 74 49 31 37 32 0a 73 28 53 27 48 |S'T'.tI172.s(S'H| > 00000050 27 0a 53 27 54 27 0a 74 49 36 39 0a 73 28 53 27 |'.S'T'.tI69.s(S'| > 00000060 51 27 0a 53 27 59 27 0a 74 49 34 31 0a 73 28 53 |Q'.S'Y'.tI41.s(S| > 00000070 27 48 27 0a 53 27 50 27 0a 74 49 32 33 0a 73 28 |'H'.S'P'.tI23.s(| > 00000080 53 27 4e 27 0a 53 27 59 27 0a 74 49 37 35 0a 73 |S'N'.S'Y'.tI75.s| > 00000090 28 53 27 48 27 0a 53 27 4c 27 0a 74 49 37 30 0a |(S'H'.S'L'.tI70.| > > Then I converted it to DOS/Windows newlines (e.g. unix2dos > is easy if you have that, or a few lines of Python if not - see below): > > $ hexdump -C acc_rep_mat.dos.pik | head > 00000000 28 64 70 31 0d 0a 28 53 27 4c 27 0d 0a 53 27 52 |(dp1..(S'L'..S'R| > 00000010 27 0d 0a 74 49 31 30 39 0d 0a 73 28 53 27 49 27 |'..tI109..s(S'I'| > 00000020 0d 0a 53 27 49 27 0d 0a 74 49 31 34 35 0d 0a 73 |..S'I'..tI145..s| > 00000030 28 53 27 51 27 0d 0a 53 27 51 27 0d 0a 74 49 34 |(S'Q'..S'Q'..tI4| > 00000040 32 0d 0a 73 28 53 27 53 27 0d 0a 53 27 54 27 0d |2..s(S'S'..S'T'.| > 00000050 0a 74 49 31 37 32 0d 0a 73 28 53 27 48 27 0d 0a |.tI172..s(S'H'..| > 00000060 53 27 54 27 0d 0a 74 49 36 39 0d 0a 73 28 53 27 |S'T'..tI69..s(S'| > 00000070 51 27 0d 0a 53 27 59 27 0d 0a 74 49 34 31 0d 0a |Q'..S'Y'..tI41..| > 00000080 73 28 53 27 48 27 0d 0a 53 27 50 27 0d 0a 74 49 |s(S'H'..S'P'..tI| > 00000090 32 33 0d 0a 73 28 53 27 4e 27 0d 0a 53 27 59 27 |23..s(S'N'..S'Y'| > > This increases the file size from 3658 bytes to 4289 bytes. > > $ python3.4 -c "import pickle; h=open('acc_rep_mat.pik', 'rb'); > m=pickle.load(h); h.close(); print(m)" > {('E', 'M'): 33, ..., ('D', 'V'): 95} > > $ python3.4 -c "import pickle; h=open('acc_rep_mat.dos.pik', 'rb'); > m=pickle.load(h); h.close(); print(m)" > Traceback (most recent call last): > File "", line 1, in > _pickle.UnpicklingError: the STRING opcode argument must be quoted > > So I can get the exact same error under Linux now :) > > I confirmed this on Windows where my copy of git is setup to use > Unix newlines by default (I think), and the file has Unix newlines > (and is 3658 bytes). > > C:\repositories\biopython\Tests\SubsMat>c:\python34\python > Python 3.4.0 (v3.4.0:04f714765c13, Mar 16 2014, 19:24:06) [MSC v.1600 > 32 bit (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. >>>> data = open("acc_rep_mat.pik", "rb").read() >>>> with open("acc_rep_mat.dos.pik", "wb") as h: h.write(data.replace(b"\n", b"\r\n")) > ... > 4289 >>>> quit() > > C:\repositories\biopython\Tests\SubsMat>c:\python34\python -c "import > pickle; h=open('acc_rep_mat.pik', 'rb'); m=pickle.load(h); h.close(); > print(m)" > {('D', 'R'): 115, ..., ('H', 'Q'): 44} > > C:\repositories\biopython\Tests\SubsMat>c:\python34\python -c "import > pickle; h=open('acc_rep_mat.dos.pik', 'rb'); m=pickle.load(h); > h.close(); print(m)" > Traceback (most recent call last): > File "", line 1, in > _pickle.UnpicklingError: the STRING opcode argument must be quoted > > So, the upshot is that this git setting change should fix it: > https://github.com/biopython/biopython/commit/b7cc2fe199d22f794612d68e5554361413468372 > > Could you update your copy of the Biopython source code via git, > and see if that solves this pickle? > > Thank you, > > Peter From p.j.a.cock at googlemail.com Tue Apr 22 15:14:25 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 20:14:25 +0100 Subject: [Biopython-dev] Pickle problem on 64 bit Windows with Python 3.4 In-Reply-To: References: Message-ID: On Tue, Apr 22, 2014 at 8:09 PM, Manlio Calvi wrote: > Uhm no, I get the same error as before... > Seems this machine don't like conserved vegetables. > > Manlio It may just be git didn't update the file with the Unix newlines. Can you check the file (e.g. size in bytes, or some Windows hexdump equivalent like opening it for reading in Python in raw/binary mode)? Perhaps deleting the file on disk and then getting get to restore it might do the trick? Peter From zruan1991 at gmail.com Tue Apr 22 17:18:58 2014 From: zruan1991 at gmail.com (Zheng Ruan) Date: Tue, 22 Apr 2014 17:18:58 -0400 Subject: [Biopython-dev] MSAprobs test fails Message-ID: Hi, The MASprobs wrapper fails when I run the test using the latest code. The problem seems to be the err.returncode is -11 rather than 139, at least in my machine (Linux 3.12 x86_64) python2.7 and 3.3 will both generate the error. Here is scratch of the err message: FAIL: test_single_sequence (test_MSAProbs_tool.MSAProbsTestErrorConditions) Test an input file containing a single sequence. ---------------------------------------------------------------------- Traceback (most recent call last): File "./test_MSAProbs_tool.py", line 86, in test_single_sequence stdout, stderr = cline() Bio.Application.ApplicationError: Command 'msaprobs Fasta/f001' returned non-zero exit status -11, '-------------------------------------' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "./test_MSAProbs_tool.py", line 88, in test_single_sequence self.assertEqual(err.returncode, 139) AssertionError: -11 != 139 Best, Zheng Ruan From manlio.calvi at gmail.com Tue Apr 22 17:31:53 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Tue, 22 Apr 2014 23:31:53 +0200 Subject: [Biopython-dev] Pickle problem on 64 bit Windows with Python 3.4 In-Reply-To: References: Message-ID: OK this one tastes good! This problems was solved but another two emerged (or I never take notice before), time for another post... Manlio On Tue, Apr 22, 2014 at 9:14 PM, Peter Cock wrote: > On Tue, Apr 22, 2014 at 8:09 PM, Manlio Calvi wrote: >> Uhm no, I get the same error as before... >> Seems this machine don't like conserved vegetables. >> >> Manlio > > It may just be git didn't update the file with the > Unix newlines. > > Can you check the file (e.g. size in bytes, or some > Windows hexdump equivalent like opening it for > reading in Python in raw/binary mode)? > > Perhaps deleting the file on disk and then getting > get to restore it might do the trick? > > Peter From manlio.calvi at gmail.com Tue Apr 22 17:36:27 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Tue, 22 Apr 2014 23:36:27 +0200 Subject: [Biopython-dev] Win7 x64 - python 3.4 - Phylo and searchIO Message-ID: I got those two new errors: ====================================================================== ERROR: test_unicode_exception (test_Phylo.IOTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Users\Public\BioPython3\biopython\Tests\test_Phylo.py", line 57, in test_unicode_exception tree = Phylo.read(EX_NEWICK_BOM, 'newick') File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", line 65, in read tree = next(tree_gen) File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", line 53, in parse for tree in getattr(supported_formats[format], 'parse')(fp, **kwargs): File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\NewickIO.py", line 107, in parse for line in self.handle: File "C:\Python34\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 15: character maps to ====================================================================== FAIL: test_xml_2226_blastn_006 (test_SearchIO_blast_xml.BlastXmlSpecialCases) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Users\Public\BioPython3\biopython\Tests\test_SearchIO_blast_xml.py", line 3127, in test_xml_2226_blastn_006 "Expected one BiopythonParserWarning, got %r" % w) AssertionError: 2 != 1 : Expected one BiopythonParserWarning, got [, ] Manlio From p.j.a.cock at googlemail.com Tue Apr 22 17:38:57 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 22:38:57 +0100 Subject: [Biopython-dev] Pickle problem on 64 bit Windows with Python 3.4 In-Reply-To: References: Message-ID: On Tue, Apr 22, 2014 at 10:31 PM, Manlio Calvi wrote: > OK this one tastes good! This problems was solved but another two > emerged (or I never take notice before), time for another post... > > Manlio So good news (this problem is solved) but bad news (you have noticed some other issues). I'll await your next email. Thanks, Peter From p.j.a.cock at googlemail.com Tue Apr 22 17:48:28 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 22:48:28 +0100 Subject: [Biopython-dev] MSAprobs test fails In-Reply-To: References: Message-ID: Hi Zheng, Negative error codes (from Python subprocess) are really Unix signals, https://docs.python.org/2/library/subprocess.html i.e. Linux killed the job with signal 11, probably SIGSEGV - Invalid memory reference. Can you run the tool by hand? Does it seg fault or something like that? Peter On Tue, Apr 22, 2014 at 10:18 PM, Zheng Ruan wrote: > Hi, > > The MASprobs wrapper fails when I run the test using the latest code. The > problem seems to be the err.returncode is -11 rather than 139, at least in > my machine (Linux 3.12 x86_64) python2.7 and 3.3 will both generate the > error. > > Here is scratch of the err message: > > FAIL: test_single_sequence (test_MSAProbs_tool.MSAProbsTestErrorConditions) > Test an input file containing a single sequence. > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "./test_MSAProbs_tool.py", line 86, in test_single_sequence > stdout, stderr = cline() > Bio.Application.ApplicationError: Command 'msaprobs Fasta/f001' returned > non-zero exit status -11, '-------------------------------------' > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "./test_MSAProbs_tool.py", line 88, in test_single_sequence > self.assertEqual(err.returncode, 139) > AssertionError: -11 != 139 > > > Best, > Zheng Ruan > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev From p.j.a.cock at googlemail.com Tue Apr 22 17:58:59 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 22:58:59 +0100 Subject: [Biopython-dev] Win7 x64 - python 3.4 - Phylo and searchIO In-Reply-To: References: Message-ID: On Tue, Apr 22, 2014 at 10:36 PM, Manlio Calvi wrote: > I got those two new errors: > > ====================================================================== > ERROR: test_unicode_exception (test_Phylo.IOTests) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Users\Public\BioPython3\biopython\Tests\test_Phylo.py", > line 57, in test_unicode_exception > tree = Phylo.read(EX_NEWICK_BOM, 'newick') > File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", > line 65, in read > tree = next(tree_gen) > File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", > line 53, in parse > for tree in getattr(supported_formats[format], 'parse')(fp, **kwargs): > File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\NewickIO.py", > line 107, in parse > for line in self.handle: > File "C:\Python34\lib\encodings\cp1252.py", line 23, in decode > return codecs.charmap_decode(input,self.errors,decoding_table)[0] > UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position > 15: character maps to > Yep, see https://github.com/biopython/biopython/issues/311 > ====================================================================== > FAIL: test_xml_2226_blastn_006 (test_SearchIO_blast_xml.BlastXmlSpecialCases) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Users\Public\BioPython3\biopython\Tests\test_SearchIO_blast_xml.py", > line 3127, in test_xml_2226_blastn_006 > "Expected one BiopythonParserWarning, got %r" % w) > AssertionError: 2 != 1 : Expected one BiopythonParserWarning, got > [, > ] > I suspect I have seen that before... must double check that. @Bow, I think this check needs to be relaxed slightly (e.g. at least one warning?) as it can potentially interact with global warning settings (even though the test framework tries to restore the filters to a known state before each test). Could you take a look? Thanks, Peter From zruan1991 at gmail.com Tue Apr 22 18:32:54 2014 From: zruan1991 at gmail.com (Zheng Ruan) Date: Tue, 22 Apr 2014 18:32:54 -0400 Subject: [Biopython-dev] MSAprobs test fails In-Reply-To: References: Message-ID: I see. I was just trying different alignment program. In this case, it might not be good to test the single sequence by the error code. Zheng On Tue, Apr 22, 2014 at 5:48 PM, Peter Cock wrote: > Hi Zheng, > > Negative error codes (from Python subprocess) are really Unix signals, > https://docs.python.org/2/library/subprocess.html > > i.e. Linux killed the job with signal 11, probably SIGSEGV - Invalid > memory reference. > > > Can you run the tool by hand? Does it seg fault or something like that? > > Peter > > On Tue, Apr 22, 2014 at 10:18 PM, Zheng Ruan wrote: > > Hi, > > > > The MASprobs wrapper fails when I run the test using the latest code. The > > problem seems to be the err.returncode is -11 rather than 139, at least > in > > my machine (Linux 3.12 x86_64) python2.7 and 3.3 will both generate the > > error. > > > > Here is scratch of the err message: > > > > FAIL: test_single_sequence > (test_MSAProbs_tool.MSAProbsTestErrorConditions) > > Test an input file containing a single sequence. > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File "./test_MSAProbs_tool.py", line 86, in test_single_sequence > > stdout, stderr = cline() > > Bio.Application.ApplicationError: Command 'msaprobs Fasta/f001' returned > > non-zero exit status -11, '-------------------------------------' > > > > During handling of the above exception, another exception occurred: > > > > Traceback (most recent call last): > > File "./test_MSAProbs_tool.py", line 88, in test_single_sequence > > self.assertEqual(err.returncode, 139) > > AssertionError: -11 != 139 > > > > > > Best, > > Zheng Ruan > > _______________________________________________ > > Biopython-dev mailing list > > Biopython-dev at lists.open-bio.org > > http://lists.open-bio.org/mailman/listinfo/biopython-dev > From manlio.calvi at gmail.com Wed Apr 23 00:51:31 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Wed, 23 Apr 2014 06:51:31 +0200 Subject: [Biopython-dev] Win7 x64 - python 3.4 - Phylo and searchIO In-Reply-To: References: Message-ID: Ops a bit of overzealousness here, yesterday night was a bit tired and I preferred to be sure than skip an issue. However the second one I raised on the first round of my tests but don't get addressed Manlio On Tue, Apr 22, 2014 at 11:58 PM, Peter Cock wrote: > On Tue, Apr 22, 2014 at 10:36 PM, Manlio Calvi wrote: >> I got those two new errors: >> >> ====================================================================== >> ERROR: test_unicode_exception (test_Phylo.IOTests) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "C:\Users\Public\BioPython3\biopython\Tests\test_Phylo.py", >> line 57, in test_unicode_exception >> tree = Phylo.read(EX_NEWICK_BOM, 'newick') >> File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", >> line 65, in read >> tree = next(tree_gen) >> File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", >> line 53, in parse >> for tree in getattr(supported_formats[format], 'parse')(fp, **kwargs): >> File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\NewickIO.py", >> line 107, in parse >> for line in self.handle: >> File "C:\Python34\lib\encodings\cp1252.py", line 23, in decode >> return codecs.charmap_decode(input,self.errors,decoding_table)[0] >> UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position >> 15: character maps to >> > > Yep, see https://github.com/biopython/biopython/issues/311 > >> ====================================================================== >> FAIL: test_xml_2226_blastn_006 (test_SearchIO_blast_xml.BlastXmlSpecialCases) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "C:\Users\Public\BioPython3\biopython\Tests\test_SearchIO_blast_xml.py", >> line 3127, in test_xml_2226_blastn_006 >> "Expected one BiopythonParserWarning, got %r" % w) >> AssertionError: 2 != 1 : Expected one BiopythonParserWarning, got >> [, >> ] >> > > I suspect I have seen that before... must double check that. > > @Bow, I think this check needs to be relaxed slightly (e.g. at > least one warning?) as it can potentially interact with global > warning settings (even though the test framework tries to > restore the filters to a known state before each test). Could > you take a look? > > Thanks, > > Peter From alexeymorozov1991 at gmail.com Wed Apr 23 04:09:40 2014 From: alexeymorozov1991 at gmail.com (Alexey Morozov) Date: Wed, 23 Apr 2014 17:09:40 +0900 Subject: [Biopython-dev] Codon usage module? Message-ID: Biopython currently supports only a limited set of operations for codon usage analysis, so I made my own module for stuff like counting codon usage in genes, clustering them based on codon usage, making some genome- or gene-wide statistics, and so on. If I want to get it to main release eventually, what are the rules? In particular, is NumPy dependency OK? Should all my classes inherit from something? Are there any limits on how data is to be internally stored and returned? -- Alexey Morozov, LIN SB RAS, bioinformatics group. Irkutsk, Russia. From zruan1991 at gmail.com Wed Apr 23 14:03:04 2014 From: zruan1991 at gmail.com (Zheng Ruan) Date: Wed, 23 Apr 2014 14:03:04 -0400 Subject: [Biopython-dev] [biopython] Code for Codon Alignment GSoC (#259) In-Reply-To: References: Message-ID: Hi, I integrate my document into the Tutorial ( https://github.com/zruan/biopython/tree/codonalignment/Doc) as chapter 18. Meanwhile, I split each chapter into the `chapter` directory and include them to the Tutorial.tex. The test_Tutorial.py is also updated accordingly, which is able to report per-chapter line numbers if the doctest fails. Best, Zheng On Mon, Apr 21, 2014 at 1:06 PM, Peter Cock wrote: > On Mon, Apr 21, 2014 at 6:02 PM, Zheng Ruan wrote: > > > > Hi Peter and Eric, > > > > I just made some changes to my repository. The old codonalignment branch > has been > > renamed to gsoc2013 for backup. I will write new code to the > codonalignment branch. > > OK. You could write a final GSoC blog post about this too ;) > > > Eric: > > I think the whole module should be marked as experimental like the > Bio.SearchIO > > module. I anticipate to enhance the codon alignment construction method > as well > > as add more methods for dn/ds ratio estimation and Bayes Empirical Bayes > this > > summer. > > > > I place a warning in the `__init__.py`. ( > https://github.com/zruan/biopython/commit/1c9e20c8a15febb7fd9219c632afb96127a006f1 > ) > > That sounds wise. > > > Peter: > > The NEWS file is updated ( > https://github.com/zruan/biopython/commit/0dcb39bcfd48cc009edc290ceaaa3b2c2e8f806f > ). > > I will update the wiki shortly and make the announcement. > > > > Do I need to make a separate pull request to make the above change to > biopython > > master? Or you may help me do that? > > I think Eric or I can do that from those links, Eric - do you want to do > this? > > > I am also planning to include my latex document to Tutorial.tex. I > typically use > > the Latex-Suite plugin to write latex in vim. However, it took a long > time to open the > > Tutorial.tex, which made it difficult to manage. Why not put the latex > code of each > > chapter into different tex file and use \include{} to import them into > the main tex file? > > > > Thank you! > > Zheng Ruan > > We could split the Tutorial.tex file like that - although it would mean a > bit of extra work on test_Tutorial.py (but with the benefit of getting > per-chapter line numbers). > > Thank you, > > Peter > From bow at bow.web.id Wed Apr 23 17:09:27 2014 From: bow at bow.web.id (Wibowo Arindrarto) Date: Wed, 23 Apr 2014 23:09:27 +0200 Subject: [Biopython-dev] Win7 x64 - python 3.4 - Phylo and searchIO In-Reply-To: References: Message-ID: Hi Peter, Manlio, Thanks for looking deeper into the tests :). For the SearchIO issue, I'm not too fond of relaxing the filter, as that could mean the test passes even though warnings for this particular case is never raised. After checking more, I actually found that this isn't due to the global warning filter settings. Apparently, the 'U' file read mode is deprecated in Python 3.4[1] and SearchIO is using the 'rU' mode to open its files. So the number of warnings in Python 3.4 is indeed 2: the rU DeprecationWarning and the BiopythonParserWarning we are checking. Since it looks like we can't remove 'rU' altogether from our codebase, the more appropriate fix for this case would be to check for len(w) == 2 when Python version is >= 3.4. Best, Bow [1] https://docs.python.org/3.5/whatsnew/3.4.html#deprecated-3-4 On Wed, Apr 23, 2014 at 6:51 AM, Manlio Calvi wrote: > Ops a bit of overzealousness here, yesterday night was a bit tired and > I preferred to be sure than skip an issue. > However the second one I raised on the first round of my tests but > don't get addressed > > Manlio > > On Tue, Apr 22, 2014 at 11:58 PM, Peter Cock wrote: >> On Tue, Apr 22, 2014 at 10:36 PM, Manlio Calvi wrote: >>> I got those two new errors: >>> >>> ====================================================================== >>> ERROR: test_unicode_exception (test_Phylo.IOTests) >>> ---------------------------------------------------------------------- >>> Traceback (most recent call last): >>> File "C:\Users\Public\BioPython3\biopython\Tests\test_Phylo.py", >>> line 57, in test_unicode_exception >>> tree = Phylo.read(EX_NEWICK_BOM, 'newick') >>> File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", >>> line 65, in read >>> tree = next(tree_gen) >>> File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", >>> line 53, in parse >>> for tree in getattr(supported_formats[format], 'parse')(fp, **kwargs): >>> File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\NewickIO.py", >>> line 107, in parse >>> for line in self.handle: >>> File "C:\Python34\lib\encodings\cp1252.py", line 23, in decode >>> return codecs.charmap_decode(input,self.errors,decoding_table)[0] >>> UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position >>> 15: character maps to >>> >> >> Yep, see https://github.com/biopython/biopython/issues/311 >> >>> ====================================================================== >>> FAIL: test_xml_2226_blastn_006 (test_SearchIO_blast_xml.BlastXmlSpecialCases) >>> ---------------------------------------------------------------------- >>> Traceback (most recent call last): >>> File "C:\Users\Public\BioPython3\biopython\Tests\test_SearchIO_blast_xml.py", >>> line 3127, in test_xml_2226_blastn_006 >>> "Expected one BiopythonParserWarning, got %r" % w) >>> AssertionError: 2 != 1 : Expected one BiopythonParserWarning, got >>> [, >>> ] >>> >> >> I suspect I have seen that before... must double check that. >> >> @Bow, I think this check needs to be relaxed slightly (e.g. at >> least one warning?) as it can potentially interact with global >> warning settings (even though the test framework tries to >> restore the filters to a known state before each test). Could >> you take a look? >> >> Thanks, >> >> Peter From p.j.a.cock at googlemail.com Wed Apr 23 18:02:55 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 23 Apr 2014 23:02:55 +0100 Subject: [Biopython-dev] Win7 x64 - python 3.4 - Phylo and searchIO In-Reply-To: References: Message-ID: Good work Bow - that sounds like a better plan :) Peter On Wednesday, April 23, 2014, Wibowo Arindrarto wrote: > Hi Peter, Manlio, > > Thanks for looking deeper into the tests :). For the SearchIO issue, > I'm not too fond of relaxing the filter, as that could mean the test > passes even though warnings for this particular case is never raised. > > After checking more, I actually found that this isn't due to the > global warning filter settings. Apparently, the 'U' file read mode is > deprecated in Python 3.4[1] and SearchIO is using the 'rU' mode to > open its files. So the number of warnings in Python 3.4 is indeed 2: > the rU DeprecationWarning and the BiopythonParserWarning we are > checking. > > Since it looks like we can't remove 'rU' altogether from our codebase, > the more appropriate fix for this case would be to check for len(w) == 2 > when > Python version is >= 3.4. > > Best, > Bow > > [1] https://docs.python.org/3.5/whatsnew/3.4.html#deprecated-3-4 > > On Wed, Apr 23, 2014 at 6:51 AM, Manlio Calvi > > wrote: > > Ops a bit of overzealousness here, yesterday night was a bit tired and > > I preferred to be sure than skip an issue. > > However the second one I raised on the first round of my tests but > > don't get addressed > > > > Manlio > > > > On Tue, Apr 22, 2014 at 11:58 PM, Peter Cock > > wrote: > >> On Tue, Apr 22, 2014 at 10:36 PM, Manlio Calvi > > wrote: > >>> I got those two new errors: > >>> > >>> ====================================================================== > >>> ERROR: test_unicode_exception (test_Phylo.IOTests) > >>> ---------------------------------------------------------------------- > >>> Traceback (most recent call last): > >>> File "C:\Users\Public\BioPython3\biopython\Tests\test_Phylo.py", > >>> line 57, in test_unicode_exception > >>> tree = Phylo.read(EX_NEWICK_BOM, 'newick') > >>> File > "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", > >>> line 65, in read > >>> tree = next(tree_gen) > >>> File > "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", > >>> line 53, in parse > >>> for tree in getattr(supported_formats[format], 'parse')(fp, > **kwargs): > >>> File > "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\NewickIO.py", > >>> line 107, in parse > >>> for line in self.handle: > >>> File "C:\Python34\lib\encodings\cp1252.py", line 23, in decode > >>> return codecs.charmap_decode(input,self.errors,decoding_table)[0] > >>> UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position > >>> 15: character maps to > >>> > >> > >> Yep, see https://github.com/biopython/biopython/issues/311 > >> > >>> ====================================================================== > >>> FAIL: test_xml_2226_blastn_006 > (test_SearchIO_blast_xml.BlastXmlSpecialCases) > >>> ---------------------------------------------------------------------- > >>> Traceback (most recent call last): > >>> File > "C:\Users\Public\BioPython3\biopython\Tests\test_SearchIO_blast_xml.py", > >>> line 3127, in test_xml_2226_blastn_006 > >>> "Expected one BiopythonParserWarning, got %r" % w) > >>> AssertionError: 2 != 1 : Expected one BiopythonParserWarning, got > >>> [, > >>> ] > >>> > >> > >> I suspect I have seen that before... must double check that. > >> > >> @Bow, I think this check needs to be relaxed slightly (e.g. at > >> least one warning?) as it can potentially interact with global > >> warning settings (even though the test framework tries to > >> restore the filters to a known state before each test). Could > >> you take a look? > >> > >> Thanks, > >> > >> Peter > From bow at bow.web.id Wed Apr 23 19:06:47 2014 From: bow at bow.web.id (Wibowo Arindrarto) Date: Thu, 24 Apr 2014 01:06:47 +0200 Subject: [Biopython-dev] Win7 x64 - python 3.4 - Phylo and searchIO In-Reply-To: References: Message-ID: Thanks, Peter. The change is here: https://github.com/biopython/biopython/commit/82c23cb6afe48864f3dcdb3099cac2116842e9b2. All test pass on my box (except for the trie bug). We can't see the results in Travis yet (Python 3.4 support is coming soon they say[1]), but hopefully it will say the same thing then :). Cheers, Bow [1] https://github.com/travis-ci/travis-ci/issues/1989 On Thu, Apr 24, 2014 at 12:02 AM, Peter Cock wrote: > Good work Bow - that sounds like a better plan :) > > Peter > > On Wednesday, April 23, 2014, Wibowo Arindrarto wrote: >> >> Hi Peter, Manlio, >> >> Thanks for looking deeper into the tests :). For the SearchIO issue, >> I'm not too fond of relaxing the filter, as that could mean the test >> passes even though warnings for this particular case is never raised. >> >> After checking more, I actually found that this isn't due to the >> global warning filter settings. Apparently, the 'U' file read mode is >> deprecated in Python 3.4[1] and SearchIO is using the 'rU' mode to >> open its files. So the number of warnings in Python 3.4 is indeed 2: >> the rU DeprecationWarning and the BiopythonParserWarning we are >> checking. >> >> Since it looks like we can't remove 'rU' altogether from our codebase, >> the more appropriate fix for this case would be to check for len(w) == 2 >> when >> Python version is >= 3.4. >> >> Best, >> Bow >> >> [1] https://docs.python.org/3.5/whatsnew/3.4.html#deprecated-3-4 >> >> On Wed, Apr 23, 2014 at 6:51 AM, Manlio Calvi >> wrote: >> > Ops a bit of overzealousness here, yesterday night was a bit tired and >> > I preferred to be sure than skip an issue. >> > However the second one I raised on the first round of my tests but >> > don't get addressed >> > >> > Manlio >> > >> > On Tue, Apr 22, 2014 at 11:58 PM, Peter Cock >> > wrote: >> >> On Tue, Apr 22, 2014 at 10:36 PM, Manlio Calvi >> >> wrote: >> >>> I got those two new errors: >> >>> >> >>> ====================================================================== >> >>> ERROR: test_unicode_exception (test_Phylo.IOTests) >> >>> ---------------------------------------------------------------------- >> >>> Traceback (most recent call last): >> >>> File "C:\Users\Public\BioPython3\biopython\Tests\test_Phylo.py", >> >>> line 57, in test_unicode_exception >> >>> tree = Phylo.read(EX_NEWICK_BOM, 'newick') >> >>> File >> >>> "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", >> >>> line 65, in read >> >>> tree = next(tree_gen) >> >>> File >> >>> "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", >> >>> line 53, in parse >> >>> for tree in getattr(supported_formats[format], 'parse')(fp, >> >>> **kwargs): >> >>> File >> >>> "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\NewickIO.py", >> >>> line 107, in parse >> >>> for line in self.handle: >> >>> File "C:\Python34\lib\encodings\cp1252.py", line 23, in decode >> >>> return codecs.charmap_decode(input,self.errors,decoding_table)[0] >> >>> UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position >> >>> 15: character maps to >> >>> >> >> >> >> Yep, see https://github.com/biopython/biopython/issues/311 >> >> >> >>> ====================================================================== >> >>> FAIL: test_xml_2226_blastn_006 >> >>> (test_SearchIO_blast_xml.BlastXmlSpecialCases) >> >>> ---------------------------------------------------------------------- >> >>> Traceback (most recent call last): >> >>> File >> >>> "C:\Users\Public\BioPython3\biopython\Tests\test_SearchIO_blast_xml.py", >> >>> line 3127, in test_xml_2226_blastn_006 >> >>> "Expected one BiopythonParserWarning, got %r" % w) >> >>> AssertionError: 2 != 1 : Expected one BiopythonParserWarning, got >> >>> [, >> >>> ] >> >>> >> >> >> >> I suspect I have seen that before... must double check that. >> >> >> >> @Bow, I think this check needs to be relaxed slightly (e.g. at >> >> least one warning?) as it can potentially interact with global >> >> warning settings (even though the test framework tries to >> >> restore the filters to a known state before each test). Could >> >> you take a look? >> >> >> >> Thanks, >> >> >> >> Peter From p.j.a.cock at googlemail.com Thu Apr 24 04:26:34 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Thu, 24 Apr 2014 09:26:34 +0100 Subject: [Biopython-dev] Win7 x64 - python 3.4 - Phylo and searchIO In-Reply-To: References: Message-ID: Great - Tiago and I need to add Python 3.4 to the buildbot, and as you say, fingers crossed TravisCI finish rolling out Python 3.4 support soon... Peter On Thursday, April 24, 2014, Wibowo Arindrarto wrote: > Thanks, Peter. The change is here: > > https://github.com/biopython/biopython/commit/82c23cb6afe48864f3dcdb3099cac2116842e9b2 > . > All test pass on my box (except for the trie bug). We can't see the > results in Travis yet > (Python 3.4 support is coming soon they say[1]), but > hopefully it will say the same thing then :). > > Cheers, > Bow > > [1] https://github.com/travis-ci/travis-ci/issues/1989 > > > From tra at popgen.net Thu Apr 24 04:47:33 2014 From: tra at popgen.net (Tiago Antao) Date: Thu, 24 Apr 2014 09:47:33 +0100 Subject: [Biopython-dev] Win7 x64 - python 3.4 - Phylo and searchIO In-Reply-To: References: Message-ID: <20140424094733.16dafbb1@grandao> On Thu, 24 Apr 2014 09:26:34 +0100 Peter Cock wrote: > Great - Tiago and I need to add Python 3.4 to the buildbot, > and as you say, fingers crossed TravisCI finish rolling out > Python 3.4 support soon... I have been silent on this because I was waiting for a docker image based on ubuntu trusty (which has 3.4 as the default Python). The image is now available but has a bug (one of the apt-sources is still saucy). My expectation is that this will be corrected very soon. I have all the machinery ready to test on 3.4 as soon as the docker image is out and fully working. Maybe even today (as the bug is trivial to sort out - lets just hope that they do it fast). Tiago From manlio.calvi at gmail.com Thu Apr 24 05:11:47 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Thu, 24 Apr 2014 11:11:47 +0200 Subject: [Biopython-dev] Win7 x64 - python 3.4 - Phylo and searchIO In-Reply-To: <20140424094733.16dafbb1@grandao> References: <20140424094733.16dafbb1@grandao> Message-ID: In the afternoon I'll count to rebuild/retest all on 2.7, 3.3 and the 3.4 on my Win7 x64 machine On Thu, Apr 24, 2014 at 10:47 AM, Tiago Antao wrote: > > On Thu, 24 Apr 2014 09:26:34 +0100 > Peter Cock wrote: > >> Great - Tiago and I need to add Python 3.4 to the buildbot, >> and as you say, fingers crossed TravisCI finish rolling out >> Python 3.4 support soon... > > I have been silent on this because I was waiting for a docker image > based on ubuntu trusty (which has 3.4 as the default Python). The image > is now available but has a bug (one of the apt-sources is still saucy). > My expectation is that this will be corrected very soon. > > I have all the machinery ready to test on 3.4 as soon as the docker > image is out and fully working. Maybe even today (as the bug is > trivial to sort out - lets just hope that they do it fast). > > Tiago > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev From tra at popgen.net Thu Apr 24 06:36:23 2014 From: tra at popgen.net (Tiago Antao) Date: Thu, 24 Apr 2014 11:36:23 +0100 Subject: [Biopython-dev] Next biopython release (was Re: Python 3.4) In-Reply-To: References: <20140422102316.5b049105@lnx> Message-ID: <20140424113623.2bdd2d9d@lnx> On Tue, 22 Apr 2014 10:42:33 +0100 Peter Cock wrote: > I am hoping we'll have everything ready on Python 3.4 > for the next release of Biopython - aim for early May? It would be good if we rotated release managers. But, if no one else volunteers, I can put my head for the chop. Tiago From manlio.calvi at gmail.com Thu Apr 24 12:34:20 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Thu, 24 Apr 2014 18:34:20 +0200 Subject: [Biopython-dev] Testings on Win 7 x64 Message-ID: OK I launched the tests on all my serpent's nest :) As some changes are made on .git folders and files and seeing how badly git on windows manage those I erased the biopython folders and started from scratch, that's the results: Python 3.4 x64 NO error (yay!): The only note here is the fact that bio.trieand KDtree get skipped because the C module refuse to compile. Python 3.3 (WinPython 3.3.3.3 x64 2 errors (one may be temporary): CDAO test fail with (an issue we already seen): ====================================================================== ERROR: test_write_0 (test_Phylo_CDAO.WriterTests) Write and re-parse the phylogenies in test.cdao. ---------------------------------------------------------------------- Traceback (most recent call last): File ".\test_Phylo_CDAO.py", line 62, in test_write CDAOIO.write([t1], outfile) File "C:\Users\xxx\3.3 - Biopython test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\build\lib.win-amd64-3.3\Bio\Phylo\CDAOIO.py", line 74, in write return Writer(trees).write(handle, plain=plain, **kwargs) File "C:\Users\xxx\3.3 - Biopython test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\build\lib.win-amd64-3.3\Bio\Phylo\CDAOIO.py", line 268, in write handle.write('@prefix %s: <%s> .\n' % (k, v)) TypeError: 'str' does not support the buffer interface --------------------------------------- Also during the test this line popped over and over file://C:\Users\xxx\3.3 - Biopython test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Tests\CDAO\test.cdao/tree0000001#edge_annotation0000071 does not look like a valid URI, trying to serialize this will break. --------------------------------? The other isssue is related with NCBI qblast but being NCBI probably was I hit a server down. ###################################### Python 2.7 x64 NO issues! Adding to that I hit a "strangeness" I don't understand because the mentioned scipy module IS installed, the test pass but not with "flying colours": test_CodonAlign ... test_CodonAlign.py:156: UserWarning: Importing scipy.linalg.expm failed. Skip testing ML method for dN/dS estimation warnings.warn('Importing scipy.linalg.expm failed. Skip testing ML method for dN/dS estimation') test_CodonAlign.py:164: UserWarning: Importing scipy.optimize.minimize failed. Skip testing ML method for dN/dS estimation warnings.warn('Importing scipy.optimize.minimize failed. Skip testing ML method for dN/dS estimation') ok Second a more general signalation, I've seen some test depends on PIL but PIL library get discontinued and substituted by one called Pillow that is said compatible with it predecessor (but I'm not sure how much it is). This is a more general issue because require an scan of the entire codebase. Looking around seems to me the NLTK package is in a situation like this, too. Manlio From p.j.a.cock at googlemail.com Thu Apr 24 12:49:38 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Thu, 24 Apr 2014 17:49:38 +0100 Subject: [Biopython-dev] Testings on Win 7 x64 In-Reply-To: References: Message-ID: On Thu, Apr 24, 2014 at 5:34 PM, Manlio Calvi wrote: > OK I launched the tests on all my serpent's nest :) > > As some changes are made on .git folders and files and seeing how > badly git on windows manage those I erased the biopython folders and > started from scratch, that's the results: > > Python 3.4 x64 NO error (yay!): > The only note here is the fact that bio.trieand KDtree get skipped > because the C module refuse to compile. It seems none of the compiled C code is getting tested :( http://lists.open-bio.org/pipermail/biopython-dev/2014-April/011270.html > Python 3.3 (WinPython 3.3.3.3 x64 2 errors (one may be temporary): > CDAO test fail with (an issue we already seen): > > ====================================================================== > ERROR: test_write_0 (test_Phylo_CDAO.WriterTests) > Write and re-parse the phylogenies in test.cdao. > ---------------------------------------------------------------------- > Traceback (most recent call last): > File ".\test_Phylo_CDAO.py", line 62, in test_write > CDAOIO.write([t1], outfile) > File "C:\Users\xxx\3.3 - Biopython > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\build\lib.win-amd64-3.3\Bio\Phylo\CDAOIO.py", > line 74, in write > return Writer(trees).write(handle, plain=plain, **kwargs) > File "C:\Users\xxx\3.3 - Biopython > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\build\lib.win-amd64-3.3\Bio\Phylo\CDAOIO.py", > line 268, in write > handle.write('@prefix %s: <%s> .\n' % (k, v)) > TypeError: 'str' does not support the buffer interface > --------------------------------------- > > Also during the test this line popped over and over > > file://C:\Users\xxx\3.3 - Biopython > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Tests\CDAO\test.cdao/tree0000001#edge_annotation0000071 > does not look like a valid URI, trying to serialize this will break. > > -------------------------------- See this thread - no replies yet so maybe we'd better note this as a GitHub issue, http://lists.open-bio.org/pipermail/biopython-dev/2014-April/011294.html > > The other isssue is related with NCBI qblast but being NCBI probably > was I hit a server down. I've noticed the NCBI blast website is currently very slow :( > ###################################### > > Python 2.7 x64 NO issues! > Excellent - we are making progress (slowly). > > Adding to that I hit a "strangeness" I don't understand because the > mentioned scipy module IS installed, the test pass but not with > "flying colours": > test_CodonAlign ... test_CodonAlign.py:156: UserWarning: Importing > scipy.linalg.expm failed. Skip testing ML method for dN/dS estimation > warnings.warn('Importing scipy.linalg.expm failed. Skip testing ML > method for dN/dS estimation') > test_CodonAlign.py:164: UserWarning: Importing scipy.optimize.minimize > failed. Skip testing ML method for dN/dS estimation > warnings.warn('Importing scipy.optimize.minimize failed. Skip > testing ML method for dN/dS estimation') > ok The CodonAlign stuff is new, this might need looking at: https://github.com/biopython/biopython/pull/259 http://lists.open-bio.org/pipermail/biopython-dev/2014-April/011278.html > Second a more general signalation, I've seen some test depends on PIL > but PIL library get discontinued and substituted by one called Pillow > that is said compatible with it predecessor (but I'm not sure how much > it is). This is a more general issue because require an scan of the > entire codebase. Looking around seems to me the NLTK package is in a > situation like this, too. That probably refers to test_GraphicsBitmaps.py which is trying to check if ReportLab will be able to make PNG files. If ReportLab can now use Pillow or PIL, we should allow for that... https://github.com/python-imaging/Pillow (As an aside ReportLab's Python 3 support is still quite new - but I don't recall seeing any problems from Biopython's usage of it.) Peter From zruan1991 at gmail.com Thu Apr 24 12:53:53 2014 From: zruan1991 at gmail.com (Zheng Ruan) Date: Thu, 24 Apr 2014 12:53:53 -0400 Subject: [Biopython-dev] Testings on Win 7 x64 In-Reply-To: References: Message-ID: Hi Manlio, As for the test_CodonAlign warning, what's the scipy version you installed? Are you able to manually import `scipy.linalg.expm` and ` scipy.optimize.minimize` manually? scipy.optimize.minimize is added to scipy since v0.11 (http://docs.scipy.org/doc/scipy-0.11.0/reference/tutorial/optimize.html). I'm not sure why `scipy.linalg.expm` is not able to be import, since it's available in the earliest version of scipy ( http://docs.scipy.org/doc/scipy-0.7.x/reference/linalg.html). Thank you! Zheng On Thu, Apr 24, 2014 at 12:34 PM, Manlio Calvi wrote: > OK I launched the tests on all my serpent's nest :) > > As some changes are made on .git folders and files and seeing how > badly git on windows manage those I erased the biopython folders and > started from scratch, that's the results: > > Python 3.4 x64 NO error (yay!): > The only note here is the fact that bio.trieand KDtree get skipped > because the C module refuse to compile. > > Python 3.3 (WinPython 3.3.3.3 x64 2 errors (one may be temporary): > CDAO test fail with (an issue we already seen): > > ====================================================================== > ERROR: test_write_0 (test_Phylo_CDAO.WriterTests) > Write and re-parse the phylogenies in test.cdao. > ---------------------------------------------------------------------- > Traceback (most recent call last): > File ".\test_Phylo_CDAO.py", line 62, in test_write > CDAOIO.write([t1], outfile) > File "C:\Users\xxx\3.3 - Biopython > > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\build\lib.win-amd64-3.3\Bio\Phylo\CDAOIO.py", > line 74, in write > return Writer(trees).write(handle, plain=plain, **kwargs) > File "C:\Users\xxx\3.3 - Biopython > > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\build\lib.win-amd64-3.3\Bio\Phylo\CDAOIO.py", > line 268, in write > handle.write('@prefix %s: <%s> .\n' % (k, v)) > TypeError: 'str' does not support the buffer interface > --------------------------------------- > > Also during the test this line popped over and over > > file://C:\Users\xxx\3.3 - Biopython > > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Tests\CDAO\test.cdao/tree0000001#edge_annotation0000071 > does not look like a valid URI, trying to serialize this will break. > > --------------------------------? > > The other isssue is related with NCBI qblast but being NCBI probably > was I hit a server down. > > ###################################### > > Python 2.7 x64 NO issues! > > > Adding to that I hit a "strangeness" I don't understand because the > mentioned scipy module IS installed, the test pass but not with > "flying colours": > test_CodonAlign ... test_CodonAlign.py:156: UserWarning: Importing > scipy.linalg.expm failed. Skip testing ML method for dN/dS estimation > warnings.warn('Importing scipy.linalg.expm failed. Skip testing ML > method for dN/dS estimation') > test_CodonAlign.py:164: UserWarning: Importing scipy.optimize.minimize > failed. Skip testing ML method for dN/dS estimation > warnings.warn('Importing scipy.optimize.minimize failed. Skip > testing ML method for dN/dS estimation') > ok > > Second a more general signalation, I've seen some test depends on PIL > but PIL library get discontinued and substituted by one called Pillow > that is said compatible with it predecessor (but I'm not sure how much > it is). This is a more general issue because require an scan of the > entire codebase. Looking around seems to me the NLTK package is in a > situation like this, too. > > Manlio > > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev > From p.j.a.cock at googlemail.com Thu Apr 24 14:34:16 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Thu, 24 Apr 2014 19:34:16 +0100 Subject: [Biopython-dev] Win7 x64 - python 3.4 - Phylo and searchIO In-Reply-To: References: Message-ID: On Wed, Apr 23, 2014 at 11:02 PM, Peter Cock wrote: > Good work Bow - that sounds like a better plan :) > > Peter Sadly it failed on one of the Linux machines, http://testing.open-bio.org/biopython/builders/OS%20X%20-%20Python%203.4/builds/2 I've fixed this by removing the mode U under Python 3 instead: https://github.com/biopython/biopython/commit/d308194cfc9f9535dd98b2a004e4becb52096090 And tweaked your test to got back to expecting one warning only: https://github.com/biopython/biopython/commit/3d07450abf345c62c2377e18755d96b7ae7b9263 There are other uses of mode U which we may want to change if the warnings become too visible. Peter From p.j.a.cock at googlemail.com Thu Apr 24 15:53:34 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Thu, 24 Apr 2014 20:53:34 +0100 Subject: [Biopython-dev] Versions of PyPy to support? Message-ID: Hello all, PyPy 2.3 is due out shortly, which prompts me to ask which versions of PyPy are people using Biopython with? PyPy is an alternative implementation of Python, which can often be much faster - see http://pypy.org/ We're currently testing with PyPy 1.8, 1.9, 2.0, 2.1 and 2.2 but I would like to suggest we drop at least PyPy 1.8 and 1.9. Is that OK? Thanks! Peter From manlio.calvi at gmail.com Thu Apr 24 16:13:07 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Thu, 24 Apr 2014 22:13:07 +0200 Subject: [Biopython-dev] Testings on Win 7 x64 In-Reply-To: References: Message-ID: Hi Zheng, A failed manual import of scipy make me control what happened to my distro, the scipy was there (infact for some other math computation script it works) but for some reason wasn't used by biopython, however I reinstalled the last version available, rebuilt and retest and the problem disappeared. Peter, the C modules instead I have no idea to how make them works, they're compiled and installed in place but seems that my snake's nest don't want to load it, I don't know why. Apparently they don't looks like an importable python module. Manlio On Thu, Apr 24, 2014 at 6:53 PM, Zheng Ruan wrote: > Hi Manlio, > > As for the test_CodonAlign warning, what's the scipy version you installed? > Are > you able to manually import `scipy.linalg.expm` and > `scipy.optimize.minimize` > manually? scipy.optimize.minimize is added to scipy since v0.11 > (http://docs.scipy.org/doc/scipy-0.11.0/reference/tutorial/optimize.html). > I'm not sure why `scipy.linalg.expm` is not able to be import, since it's > available > in the earliest version of scipy > (http://docs.scipy.org/doc/scipy-0.7.x/reference/linalg.html). > > Thank you! > Zheng > > > On Thu, Apr 24, 2014 at 12:34 PM, Manlio Calvi > wrote: >> >> OK I launched the tests on all my serpent's nest :) >> >> As some changes are made on .git folders and files and seeing how >> badly git on windows manage those I erased the biopython folders and >> started from scratch, that's the results: >> >> Python 3.4 x64 NO error (yay!): >> The only note here is the fact that bio.trieand KDtree get skipped >> because the C module refuse to compile. >> >> Python 3.3 (WinPython 3.3.3.3 x64 2 errors (one may be temporary): >> CDAO test fail with (an issue we already seen): >> >> ====================================================================== >> ERROR: test_write_0 (test_Phylo_CDAO.WriterTests) >> Write and re-parse the phylogenies in test.cdao. >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File ".\test_Phylo_CDAO.py", line 62, in test_write >> CDAOIO.write([t1], outfile) >> File "C:\Users\xxx\3.3 - Biopython >> >> test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\build\lib.win-amd64-3.3\Bio\Phylo\CDAOIO.py", >> line 74, in write >> return Writer(trees).write(handle, plain=plain, **kwargs) >> File "C:\Users\xxx\3.3 - Biopython >> >> test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\build\lib.win-amd64-3.3\Bio\Phylo\CDAOIO.py", >> line 268, in write >> handle.write('@prefix %s: <%s> .\n' % (k, v)) >> TypeError: 'str' does not support the buffer interface >> --------------------------------------- >> >> Also during the test this line popped over and over >> >> file://C:\Users\xxx\3.3 - Biopython >> >> test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Tests\CDAO\test.cdao/tree0000001#edge_annotation0000071 >> does not look like a valid URI, trying to serialize this will break. >> >> --------------------------------? >> >> The other isssue is related with NCBI qblast but being NCBI probably >> was I hit a server down. >> >> ###################################### >> >> Python 2.7 x64 NO issues! >> >> >> Adding to that I hit a "strangeness" I don't understand because the >> mentioned scipy module IS installed, the test pass but not with >> "flying colours": >> test_CodonAlign ... test_CodonAlign.py:156: UserWarning: Importing >> scipy.linalg.expm failed. Skip testing ML method for dN/dS estimation >> warnings.warn('Importing scipy.linalg.expm failed. Skip testing ML >> method for dN/dS estimation') >> test_CodonAlign.py:164: UserWarning: Importing scipy.optimize.minimize >> failed. Skip testing ML method for dN/dS estimation >> warnings.warn('Importing scipy.optimize.minimize failed. Skip >> testing ML method for dN/dS estimation') >> ok >> >> Second a more general signalation, I've seen some test depends on PIL >> but PIL library get discontinued and substituted by one called Pillow >> that is said compatible with it predecessor (but I'm not sure how much >> it is). This is a more general issue because require an scan of the >> entire codebase. Looking around seems to me the NLTK package is in a >> situation like this, too. >> >> Manlio >> >> _______________________________________________ >> Biopython-dev mailing list >> Biopython-dev at lists.open-bio.org >> http://lists.open-bio.org/mailman/listinfo/biopython-dev > > From manlio.calvi at gmail.com Thu Apr 24 17:04:47 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Thu, 24 Apr 2014 23:04:47 +0200 Subject: [Biopython-dev] Versions of PyPy to support? In-Reply-To: References: Message-ID: I'm thinking about run a PyPy test on my machine, something specific to watch out? I've seen they have a beta on pypy for 3, probably a bit experimental at the moment but they saying it mostly works (as they say of PyPy in general) Manlio On Thu, Apr 24, 2014 at 9:53 PM, Peter Cock wrote: > Hello all, > > PyPy 2.3 is due out shortly, which prompts me to ask which > versions of PyPy are people using Biopython with? > > PyPy is an alternative implementation of Python, which > can often be much faster - see http://pypy.org/ > > We're currently testing with PyPy 1.8, 1.9, 2.0, 2.1 and 2.2 > but I would like to suggest we drop at least PyPy 1.8 and 1.9. > > Is that OK? > > Thanks! > > Peter > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev From p.j.a.cock at googlemail.com Thu Apr 24 17:07:54 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Thu, 24 Apr 2014 22:07:54 +0100 Subject: [Biopython-dev] Versions of PyPy to support? In-Reply-To: References: Message-ID: Try PyPy 2.2 first, and if that works you can try their experimental Python 3 support? Peter On Thursday, April 24, 2014, Manlio Calvi wrote: > I'm thinking about run a PyPy test on my machine, something specific > to watch out? > > I've seen they have a beta on pypy for 3, probably a bit experimental > at the moment but they saying it mostly works (as they say of PyPy in > general) > > Manlio > > On Thu, Apr 24, 2014 at 9:53 PM, Peter Cock > > wrote: > > Hello all, > > > > PyPy 2.3 is due out shortly, which prompts me to ask which > > versions of PyPy are people using Biopython with? > > > > PyPy is an alternative implementation of Python, which > > can often be much faster - see http://pypy.org/ > > > > We're currently testing with PyPy 1.8, 1.9, 2.0, 2.1 and 2.2 > > but I would like to suggest we drop at least PyPy 1.8 and 1.9. > > > > Is that OK? > > > > Thanks! > > > > Peter > > _______________________________________________ > > Biopython-dev mailing list > > Biopython-dev at lists.open-bio.org > > http://lists.open-bio.org/mailman/listinfo/biopython-dev > From p.j.a.cock at googlemail.com Thu Apr 24 17:31:27 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Thu, 24 Apr 2014 22:31:27 +0100 Subject: [Biopython-dev] Testings on Win 7 x64 In-Reply-To: References: Message-ID: On Thu, Apr 24, 2014 at 9:13 PM, Manlio Calvi wrote: > Hi Zheng, > > A failed manual import of scipy make me control what happened to my > distro, the scipy was there (infact for some other math computation > script it works) but for some reason wasn't used by biopython, however > I reinstalled the last version available, rebuilt and retest and the > problem disappeared. Good :) > Peter, the C modules instead I have no idea to how make them works, > they're compiled and installed in place but seems that my snake's nest > don't want to load it, I don't know why. Apparently they don't looks > like an importable python module. So Python 2.7 is working, but to confirm the C code is not working for either Python 3.3 or Python 3.4 on your 64 bit Windows 7 machine? Strange... I think we will need to wait for someone else to have a go, perhaps Tiago - or I might be able to borrow a machine... Still, I would be pleased if we can have at least Python 2.7 running on your desktop as a buildslave? Thanks, Peter From manlio.calvi at gmail.com Thu Apr 24 17:42:25 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Thu, 24 Apr 2014 23:42:25 +0200 Subject: [Biopython-dev] Testings on Win 7 x64 In-Reply-To: References: Message-ID: Confirmed, Python 2.7 miss only the test who needs external code now. I have to understand why that's don't works...Could you pass those windows compiled .pyd files to me in a separate mail? I would like to understand if it my compiled that produced malformated code(but then why only that file?) or if I've look to something else in my building environments Manlio On Thu, Apr 24, 2014 at 11:31 PM, Peter Cock wrote: > On Thu, Apr 24, 2014 at 9:13 PM, Manlio Calvi wrote: >> Hi Zheng, >> >> A failed manual import of scipy make me control what happened to my >> distro, the scipy was there (infact for some other math computation >> script it works) but for some reason wasn't used by biopython, however >> I reinstalled the last version available, rebuilt and retest and the >> problem disappeared. > > Good :) > >> Peter, the C modules instead I have no idea to how make them works, >> they're compiled and installed in place but seems that my snake's nest >> don't want to load it, I don't know why. Apparently they don't looks >> like an importable python module. > > So Python 2.7 is working, but to confirm the C code is not > working for either Python 3.3 or Python 3.4 on your 64 bit > Windows 7 machine? Strange... I think we will need to wait > for someone else to have a go, perhaps Tiago - or I might > be able to borrow a machine... > > Still, I would be pleased if we can have at least Python 2.7 > running on your desktop as a buildslave? > > Thanks, > > Peter From manlio.calvi at gmail.com Thu Apr 24 17:57:56 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Thu, 24 Apr 2014 23:57:56 +0200 Subject: [Biopython-dev] Win7 x64 - python 3.4 - Phylo and searchIO In-Reply-To: References: Message-ID: OK recompiled 3.4 and nothing to report there And I can confirm an all clear in 2.7 too apart the some warning and that compilation problem. Python 3.3 reported the same error on CDAO as before. Manlio On Thu, Apr 24, 2014 at 8:34 PM, Peter Cock wrote: > On Wed, Apr 23, 2014 at 11:02 PM, Peter Cock wrote: >> Good work Bow - that sounds like a better plan :) >> >> Peter > > Sadly it failed on one of the Linux machines, > http://testing.open-bio.org/biopython/builders/OS%20X%20-%20Python%203.4/builds/2 > > I've fixed this by removing the mode U under Python 3 instead: > https://github.com/biopython/biopython/commit/d308194cfc9f9535dd98b2a004e4becb52096090 > > And tweaked your test to got back to expecting one warning only: > https://github.com/biopython/biopython/commit/3d07450abf345c62c2377e18755d96b7ae7b9263 > > There are other uses of mode U which we may want to change > if the warnings become too visible. > > Peter > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev From anaryin at gmail.com Thu Apr 24 19:27:29 2014 From: anaryin at gmail.com (=?UTF-8?Q?Jo=C3=A3o_Rodrigues?=) Date: Fri, 25 Apr 2014 01:27:29 +0200 Subject: [Biopython-dev] SearchIO.parse HMMER3 issue Message-ID: Hi all, I downloaded HMMER3.1b today and tried to use the SearchIO module to parse the results. I ran into a weird error, something to do with a regex. The problem seems to be that after the '//' line, there is another line with '[ok]' that breaks the parser. I'm running HMMER with pretty much default options so I guess something might have changed in the latest version or something weird is happening here with mine? Cheers, Jo?o Traceback (most recent call last): File "do_hmmer.py", line 99, in results = SearchIO.read(ali_fname, 'hmmer3-text') File "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", line 363, in read second = next(generator) File "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", line 316, in parse yield qresult File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__ self.gen.throw(type, value, traceback) File "/home/software/python-libs/lib64/python2.6/site-packages/Bio/File.py", line 77, in as_handle yield fp File "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", line 315, in parse for qresult in generator: File "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/HmmerIO/hmmer3_text.py", line 47, in __iter__ for qresult in self._parse_qresult(): File "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/HmmerIO/hmmer3_text.py", line 110, in _parse_qresult qid = regx.group(1).strip() AttributeError: 'NoneType' object has no attribute 'group' From w.arindrarto at gmail.com Thu Apr 24 19:36:07 2014 From: w.arindrarto at gmail.com (Wibowo Arindrarto) Date: Fri, 25 Apr 2014 01:36:07 +0200 Subject: [Biopython-dev] SearchIO.parse HMMER3 issue In-Reply-To: References: Message-ID: Hi Joao, Yes, this is a known issue with version 3.1b1. And indeed, the cause is the '[ok]' line (just one last line in the file). I haven't finished testing some cases (and I intended to add proper support for 3.1 ~ a.ka. nhmmer + nhmmscan ~ with the update), so my updates for this has not been up (probably sometime next week, not guaranteeing anything though..). But if you need a quick solution, removing the '[ok]' line fixes the problems in my case. Which reminds me, I think I need to properly track this issue in our issue tracker :). There is an earlier post reporting this issue (http://permalink.gmane.org/gmane.comp.python.bio.general/8027), but I think we ended up emailing privately (where I mentioned this issue). Anyway, thanks for the report. Cheers, Bow On Fri, Apr 25, 2014 at 1:27 AM, Jo?o Rodrigues wrote: > Hi all, > > I downloaded HMMER3.1b today and tried to use the SearchIO module to parse > the results. I ran into a weird error, something to do with a regex. > > The problem seems to be that after the '//' line, there is another line > with '[ok]' that breaks the parser. I'm running HMMER with pretty much > default options so I guess something might have changed in the latest > version or something weird is happening here with mine? > > Cheers, > > Jo?o > > Traceback (most recent call last): > File "do_hmmer.py", line 99, in > results = SearchIO.read(ali_fname, 'hmmer3-text') > File > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", > line 363, in read > second = next(generator) > File > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", > line 316, in parse > yield qresult > File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__ > self.gen.throw(type, value, traceback) > File > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/File.py", > line 77, in as_handle > yield fp > File > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", > line 315, in parse > for qresult in generator: > File > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/HmmerIO/hmmer3_text.py", > line 47, in __iter__ > for qresult in self._parse_qresult(): > File > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/HmmerIO/hmmer3_text.py", > line 110, in _parse_qresult > qid = regx.group(1).strip() > AttributeError: 'NoneType' object has no attribute 'group' > > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev From anaryin at gmail.com Thu Apr 24 19:38:54 2014 From: anaryin at gmail.com (=?UTF-8?Q?Jo=C3=A3o_Rodrigues?=) Date: Fri, 25 Apr 2014 01:38:54 +0200 Subject: [Biopython-dev] SearchIO.parse HMMER3 issue In-Reply-To: References: Message-ID: Hey Bow, Thanks, didn't know if it was known or not and I usually don't use these sequence parsers so.. :) Removing it did the trick indeed, seemed the most practical option. Cheers, Jo?o 2014-04-25 1:36 GMT+02:00 Wibowo Arindrarto : > Hi Joao, > > Yes, this is a known issue with version 3.1b1. And indeed, the cause > is the '[ok]' line (just one last line in the file). I haven't > finished testing some cases (and I intended to add proper support for > 3.1 ~ a.ka. nhmmer + nhmmscan ~ with the update), so my updates for > this has not been up (probably sometime next week, not guaranteeing > anything though..). But if you need a quick solution, removing the > '[ok]' line fixes the problems in my case. > > Which reminds me, I think I need to properly track this issue in our > issue tracker :). There is an earlier post reporting this issue > (http://permalink.gmane.org/gmane.comp.python.bio.general/8027), but I > think we ended up emailing privately (where I mentioned this issue). > > Anyway, thanks for the report. > > Cheers, > Bow > > On Fri, Apr 25, 2014 at 1:27 AM, Jo?o Rodrigues wrote: > > Hi all, > > > > I downloaded HMMER3.1b today and tried to use the SearchIO module to > parse > > the results. I ran into a weird error, something to do with a regex. > > > > The problem seems to be that after the '//' line, there is another line > > with '[ok]' that breaks the parser. I'm running HMMER with pretty much > > default options so I guess something might have changed in the latest > > version or something weird is happening here with mine? > > > > Cheers, > > > > Jo?o > > > > Traceback (most recent call last): > > File "do_hmmer.py", line 99, in > > results = SearchIO.read(ali_fname, 'hmmer3-text') > > File > > > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", > > line 363, in read > > second = next(generator) > > File > > > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", > > line 316, in parse > > yield qresult > > File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__ > > self.gen.throw(type, value, traceback) > > File > > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/File.py", > > line 77, in as_handle > > yield fp > > File > > > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", > > line 315, in parse > > for qresult in generator: > > File > > > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/HmmerIO/hmmer3_text.py", > > line 47, in __iter__ > > for qresult in self._parse_qresult(): > > File > > > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/HmmerIO/hmmer3_text.py", > > line 110, in _parse_qresult > > qid = regx.group(1).strip() > > AttributeError: 'NoneType' object has no attribute 'group' > > > > _______________________________________________ > > Biopython-dev mailing list > > Biopython-dev at lists.open-bio.org > > http://lists.open-bio.org/mailman/listinfo/biopython-dev > From w.arindrarto at gmail.com Thu Apr 24 19:45:12 2014 From: w.arindrarto at gmail.com (Wibowo Arindrarto) Date: Fri, 25 Apr 2014 01:45:12 +0200 Subject: [Biopython-dev] SearchIO.parse HMMER3 issue In-Reply-To: References: Message-ID: Hey Joao, No problem. The discussion with the user went to private exchanges anyway, so it was not indeed publicly searchable. In any case, I posted this in our issue tracker (also to remind myself about this :) ): https://github.com/biopython/biopython/issues/313. If you find anything else breaking or something you'd like to have, feel free to post there directly. Cheers, Bow On Fri, Apr 25, 2014 at 1:38 AM, Jo?o Rodrigues wrote: > Hey Bow, > > Thanks, didn't know if it was known or not and I usually don't use these > sequence parsers so.. :) Removing it did the trick indeed, seemed the most > practical option. > > Cheers, > > Jo?o > > > 2014-04-25 1:36 GMT+02:00 Wibowo Arindrarto : > >> Hi Joao, >> >> Yes, this is a known issue with version 3.1b1. And indeed, the cause >> is the '[ok]' line (just one last line in the file). I haven't >> finished testing some cases (and I intended to add proper support for >> 3.1 ~ a.ka. nhmmer + nhmmscan ~ with the update), so my updates for >> this has not been up (probably sometime next week, not guaranteeing >> anything though..). But if you need a quick solution, removing the >> '[ok]' line fixes the problems in my case. >> >> Which reminds me, I think I need to properly track this issue in our >> issue tracker :). There is an earlier post reporting this issue >> (http://permalink.gmane.org/gmane.comp.python.bio.general/8027), but I >> think we ended up emailing privately (where I mentioned this issue). >> >> Anyway, thanks for the report. >> >> Cheers, >> Bow >> >> On Fri, Apr 25, 2014 at 1:27 AM, Jo?o Rodrigues wrote: >> > Hi all, >> > >> > I downloaded HMMER3.1b today and tried to use the SearchIO module to >> > parse >> > the results. I ran into a weird error, something to do with a regex. >> > >> > The problem seems to be that after the '//' line, there is another line >> > with '[ok]' that breaks the parser. I'm running HMMER with pretty much >> > default options so I guess something might have changed in the latest >> > version or something weird is happening here with mine? >> > >> > Cheers, >> > >> > Jo?o >> > >> > Traceback (most recent call last): >> > File "do_hmmer.py", line 99, in >> > results = SearchIO.read(ali_fname, 'hmmer3-text') >> > File >> > >> > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", >> > line 363, in read >> > second = next(generator) >> > File >> > >> > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", >> > line 316, in parse >> > yield qresult >> > File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__ >> > self.gen.throw(type, value, traceback) >> > File >> > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/File.py", >> > line 77, in as_handle >> > yield fp >> > File >> > >> > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", >> > line 315, in parse >> > for qresult in generator: >> > File >> > >> > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/HmmerIO/hmmer3_text.py", >> > line 47, in __iter__ >> > for qresult in self._parse_qresult(): >> > File >> > >> > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/HmmerIO/hmmer3_text.py", >> > line 110, in _parse_qresult >> > qid = regx.group(1).strip() >> > AttributeError: 'NoneType' object has no attribute 'group' >> > >> > _______________________________________________ >> > Biopython-dev mailing list >> > Biopython-dev at lists.open-bio.org >> > http://lists.open-bio.org/mailman/listinfo/biopython-dev > > From w.arindrarto at gmail.com Thu Apr 24 20:05:51 2014 From: w.arindrarto at gmail.com (Wibowo Arindrarto) Date: Fri, 25 Apr 2014 02:05:51 +0200 Subject: [Biopython-dev] Test files leftovers Message-ID: Hello everyone, I was running full tests when I noticed that some of our test cases seem to create files for testing but not delete them afterwards. I have these from my tests: BWA/1.sai BWA/1.sam BWA/2.sai BWA/human_g1k_v37_truncated.fasta.amb BWA/human_g1k_v37_truncated.fasta.ann BWA/human_g1k_v37_truncated.fasta.bwt BWA/human_g1k_v37_truncated.fasta.pac BWA/human_g1k_v37_truncated.fasta.sa BWA/out.sam SamBam/bam1.bam.bai TreeConstruction/bootstrap_consensus.tre TreeConstruction/pars1.tre TreeConstruction/pars2.tre bam1.bam Now, I could just add these files to .gitignore, but I think it's more proper if we at least try to remove them after every test. In some cases, I saw that the creation of these files is actually the test. Which means having them before the test (i.e. from a previous run) may not be ideal. But I could be wrong, so I'm sending this first to ask whether these files should have been deleted or not :). Cheers, Bow From saketkc at gmail.com Thu Apr 24 20:11:26 2014 From: saketkc at gmail.com (Saket Choudhary) Date: Fri, 25 Apr 2014 05:41:26 +0530 Subject: [Biopython-dev] Test files leftovers In-Reply-To: References: Message-ID: Hi Bow, These are generated by the test functions written for BWA wrapper which I wrote. I could send a PR which does a 'cleanup' in tearDown()? Saket On 25 April 2014 05:35, Wibowo Arindrarto wrote: > Hello everyone, > > I was running full tests when I noticed that some of our test cases > seem to create files for testing but not delete them afterwards. > > I have these from my tests: > > BWA/1.sai > BWA/1.sam > BWA/2.sai > BWA/human_g1k_v37_truncated.fasta.amb > BWA/human_g1k_v37_truncated.fasta.ann > BWA/human_g1k_v37_truncated.fasta.bwt > BWA/human_g1k_v37_truncated.fasta.pac > BWA/human_g1k_v37_truncated.fasta.sa > BWA/out.sam > SamBam/bam1.bam.bai > TreeConstruction/bootstrap_consensus.tre > TreeConstruction/pars1.tre > TreeConstruction/pars2.tre > bam1.bam > > Now, I could just add these files to .gitignore, but I think it's more > proper if we at least try to remove them after every test. In some > cases, I saw that the creation of these files is actually the test. > Which means having them before the test (i.e. from a previous run) may > not be ideal. > > But I could be wrong, so I'm sending this first to ask whether these > files should have been deleted or not :). > > Cheers, > Bow > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev From saketkc at gmail.com Thu Apr 24 21:27:10 2014 From: saketkc at gmail.com (Saket Choudhary) Date: Fri, 25 Apr 2014 06:57:10 +0530 Subject: [Biopython-dev] Test files leftovers In-Reply-To: References: Message-ID: Maybe, I need some help here: A simple tearDown() approach won't work as I need the indexed files to be present for 'test_samse' and 'test_sampe' to be run. What I need is a tearDownClass() approach but it is not supported on python2.6. Any suggestions? Saket On 25 April 2014 05:41, Saket Choudhary wrote: > Hi Bow, > > These are generated by the test functions written for BWA wrapper > which I wrote. I could send a PR which does a 'cleanup' in > tearDown()? > > Saket > > On 25 April 2014 05:35, Wibowo Arindrarto wrote: >> Hello everyone, >> >> I was running full tests when I noticed that some of our test cases >> seem to create files for testing but not delete them afterwards. >> >> I have these from my tests: >> >> BWA/1.sai >> BWA/1.sam >> BWA/2.sai >> BWA/human_g1k_v37_truncated.fasta.amb >> BWA/human_g1k_v37_truncated.fasta.ann >> BWA/human_g1k_v37_truncated.fasta.bwt >> BWA/human_g1k_v37_truncated.fasta.pac >> BWA/human_g1k_v37_truncated.fasta.sa >> BWA/out.sam >> SamBam/bam1.bam.bai >> TreeConstruction/bootstrap_consensus.tre >> TreeConstruction/pars1.tre >> TreeConstruction/pars2.tre >> bam1.bam >> >> Now, I could just add these files to .gitignore, but I think it's more >> proper if we at least try to remove them after every test. In some >> cases, I saw that the creation of these files is actually the test. >> Which means having them before the test (i.e. from a previous run) may >> not be ideal. >> >> But I could be wrong, so I'm sending this first to ask whether these >> files should have been deleted or not :). >> >> Cheers, >> Bow >> _______________________________________________ >> Biopython-dev mailing list >> Biopython-dev at lists.open-bio.org >> http://lists.open-bio.org/mailman/listinfo/biopython-dev From saketkc at gmail.com Fri Apr 25 00:43:42 2014 From: saketkc at gmail.com (Saket Choudhary) Date: Fri, 25 Apr 2014 10:13:42 +0530 Subject: [Biopython-dev] Test files leftovers In-Reply-To: References: Message-ID: Apparently I have polluted the master branch of my biopython clone: https://github.com/saketkc/biopython PR on a fake clone: https://github.com/saketkc/biopython-2/pull/1/files On 25 April 2014 06:57, Saket Choudhary wrote: > Maybe, I need some help here: > > A simple tearDown() approach won't work as I need the indexed files to > be present for 'test_samse' and 'test_sampe' to be run. > > What I need is a tearDownClass() approach but it is not supported on python2.6. > > Any suggestions? > > Saket > > On 25 April 2014 05:41, Saket Choudhary wrote: >> Hi Bow, >> >> These are generated by the test functions written for BWA wrapper >> which I wrote. I could send a PR which does a 'cleanup' in >> tearDown()? >> >> Saket >> >> On 25 April 2014 05:35, Wibowo Arindrarto wrote: >>> Hello everyone, >>> >>> I was running full tests when I noticed that some of our test cases >>> seem to create files for testing but not delete them afterwards. >>> >>> I have these from my tests: >>> >>> BWA/1.sai >>> BWA/1.sam >>> BWA/2.sai >>> BWA/human_g1k_v37_truncated.fasta.amb >>> BWA/human_g1k_v37_truncated.fasta.ann >>> BWA/human_g1k_v37_truncated.fasta.bwt >>> BWA/human_g1k_v37_truncated.fasta.pac >>> BWA/human_g1k_v37_truncated.fasta.sa >>> BWA/out.sam >>> SamBam/bam1.bam.bai >>> TreeConstruction/bootstrap_consensus.tre >>> TreeConstruction/pars1.tre >>> TreeConstruction/pars2.tre >>> bam1.bam >>> >>> Now, I could just add these files to .gitignore, but I think it's more >>> proper if we at least try to remove them after every test. In some >>> cases, I saw that the creation of these files is actually the test. >>> Which means having them before the test (i.e. from a previous run) may >>> not be ideal. >>> >>> But I could be wrong, so I'm sending this first to ask whether these >>> files should have been deleted or not :). >>> >>> Cheers, >>> Bow >>> _______________________________________________ >>> Biopython-dev mailing list >>> Biopython-dev at lists.open-bio.org >>> http://lists.open-bio.org/mailman/listinfo/biopython-dev From manlio.calvi at gmail.com Fri Apr 25 02:27:31 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Fri, 25 Apr 2014 08:27:31 +0200 Subject: [Biopython-dev] Test files leftovers In-Reply-To: References: Message-ID: If the problem is delete the files at the end of the tests maybe the last test in BWA test suite search for all those files with one or more regex and call os.remove() method to delete it, you don't need the files after all tests on them runs out. Another cleaner way, although need more work, is create a folder test inside the BWA test suite and put there the files created, run there the tests and then remove the folder at the end, this mean rewrite the path to that test folder. Manlio On Fri, Apr 25, 2014 at 6:43 AM, Saket Choudhary wrote: > Apparently I have polluted the master branch of my biopython clone: > https://github.com/saketkc/biopython > > PR on a fake clone: https://github.com/saketkc/biopython-2/pull/1/files > > On 25 April 2014 06:57, Saket Choudhary wrote: >> Maybe, I need some help here: >> >> A simple tearDown() approach won't work as I need the indexed files to >> be present for 'test_samse' and 'test_sampe' to be run. >> >> What I need is a tearDownClass() approach but it is not supported on python2.6. >> >> Any suggestions? >> >> Saket >> >> On 25 April 2014 05:41, Saket Choudhary wrote: >>> Hi Bow, >>> >>> These are generated by the test functions written for BWA wrapper >>> which I wrote. I could send a PR which does a 'cleanup' in >>> tearDown()? >>> >>> Saket >>> >>> On 25 April 2014 05:35, Wibowo Arindrarto wrote: >>>> Hello everyone, >>>> >>>> I was running full tests when I noticed that some of our test cases >>>> seem to create files for testing but not delete them afterwards. >>>> >>>> I have these from my tests: >>>> >>>> BWA/1.sai >>>> BWA/1.sam >>>> BWA/2.sai >>>> BWA/human_g1k_v37_truncated.fasta.amb >>>> BWA/human_g1k_v37_truncated.fasta.ann >>>> BWA/human_g1k_v37_truncated.fasta.bwt >>>> BWA/human_g1k_v37_truncated.fasta.pac >>>> BWA/human_g1k_v37_truncated.fasta.sa >>>> BWA/out.sam >>>> SamBam/bam1.bam.bai >>>> TreeConstruction/bootstrap_consensus.tre >>>> TreeConstruction/pars1.tre >>>> TreeConstruction/pars2.tre >>>> bam1.bam >>>> >>>> Now, I could just add these files to .gitignore, but I think it's more >>>> proper if we at least try to remove them after every test. In some >>>> cases, I saw that the creation of these files is actually the test. >>>> Which means having them before the test (i.e. from a previous run) may >>>> not be ideal. >>>> >>>> But I could be wrong, so I'm sending this first to ask whether these >>>> files should have been deleted or not :). >>>> >>>> Cheers, >>>> Bow >>>> _______________________________________________ >>>> Biopython-dev mailing list >>>> Biopython-dev at lists.open-bio.org >>>> http://lists.open-bio.org/mailman/listinfo/biopython-dev > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev From p.j.a.cock at googlemail.com Fri Apr 25 03:08:49 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Fri, 25 Apr 2014 08:08:49 +0100 Subject: [Biopython-dev] Test files leftovers In-Reply-To: References: Message-ID: On Fri, Apr 25, 2014 at 2:27 AM, Saket Choudhary wrote: > Maybe, I need some help here: > > A simple tearDown() approach won't work as I need > the indexed files to be present for 'test_samse' and > 'test_sampe' to be run. Suggestion 1: Could you fix these test methods to be self contained? i.e. include building the index > What I need is a tearDownClass() approach but it > is not supported on python2.6. Suggestion 2: Not ideal, but use that and comment that it doesn't work on Python 2.6? > Any suggestions? > > Saket Peter From p.j.a.cock at googlemail.com Fri Apr 25 03:15:10 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Fri, 25 Apr 2014 08:15:10 +0100 Subject: [Biopython-dev] Test files leftovers In-Reply-To: References: Message-ID: On Fri, Apr 25, 2014 at 7:27 AM, Manlio Calvi wrote: > If the problem is delete the files at the end of the tests maybe the > last test in BWA test suite search for all those files with one or > more regex and call os.remove() method to delete it, you don't need > the files after all tests on them runs out. > Another cleaner way, although need more work, is create a folder test > inside the BWA test suite and put there the files created, run there > the tests and then remove the folder at the end, this mean rewrite the > path to that test folder. > > Manlio Same problem - ensuring that the folder/files are removed after all the tests. This is what the Python 2.7+ method tearDownClass() is for. Peter From p.j.a.cock at googlemail.com Fri Apr 25 03:21:27 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Fri, 25 Apr 2014 08:21:27 +0100 Subject: [Biopython-dev] Python 3.4 In-Reply-To: References: <20140422102316.5b049105@lnx> Message-ID: On Tue, Apr 22, 2014 at 10:42 AM, Peter Cock wrote: > On Tue, Apr 22, 2014 at 10:23 AM, Tiago Antao wrote: >> Dear all (especially Manlio and Peter ;) ), >> >> With Ubuntu trusty officially out, I am planning on doing a (Linux) >> docker file running Python 3.4 and attach it to buildbot. Hopefully >> today. This would allow us to have a stable testing environment of 3.4 >> against Linux... Which might give some clues to what is happening with >> 3.4 on Windows... >> >> Tiago > > Good plan :) > > I've already got Python 3.4 on three build slaves ready to > be connected (Windows XP 32 bit, Linux 64 bit, and Mac > OS X 64 bit), so we'll have the main platforms covered. I've asked the buildbot master to run tests for Python 3.4 on the Mac now, and got another 64bit Linux machine ready to run the tests. Oddly the buildbot server has become quite sluggish, perhaps we have too much old data and/or too many build targets? Removing the older PyPy versions from the matrix should help: http://lists.open-bio.org/pipermail/biopython/2014-April/009110.html > Right now the only Python 3.4 specific issue I have seen is > in Bio.trie https://github.com/biopython/biopython/issues/305 Provisional fix checked in. Peter From tra at popgen.net Fri Apr 25 05:05:03 2014 From: tra at popgen.net (Tiago Antao) Date: Fri, 25 Apr 2014 10:05:03 +0100 Subject: [Biopython-dev] Python 3.4 In-Reply-To: References: <20140422102316.5b049105@lnx> Message-ID: <20140425100503.4939bd34@lnx> On Fri, 25 Apr 2014 08:21:27 +0100 Peter Cock wrote: > I've asked the buildbot master to run tests for Python 3.4 > on the Mac now, and got another 64bit Linux machine > ready to run the tests. I hope to lunch mine this morning (or early afternoon - meetings...). The docker work is finished. > Oddly the buildbot server has become quite sluggish, > perhaps we have too much old data and/or too many > build targets? Removing the older PyPy versions > from the matrix should help: > http://lists.open-bio.org/pipermail/biopython/2014-April/009110.html I think we need to purge the old data. I do not think we have anything resembling too much build targets. I will try to purge the old data myself today. From tra at popgen.net Fri Apr 25 08:57:36 2014 From: tra at popgen.net (Tiago Antao) Date: Fri, 25 Apr 2014 13:57:36 +0100 Subject: [Biopython-dev] Python 3.4 and buildbot Message-ID: <20140425135736.7bfb4a8b@grandao> Dear all, Just a summary feedback from my part regarding buildbot and Python 3.4 Buildbot: I have cleanup the history to accelarate things. I might have cleaned too much and we lost all the past information. I guess it is not important, but it was an historical record of sorts... Python 3.4: I have set-up a couple of docker images: Biopython out of the box: https://github.com/tiagoantao/my-containers/blob/master/biopython/Biopython3 For buildbot: https://github.com/tiagoantao/my-containers/blob/master/biopython/Biopython3-Test The buildbot results are now available: http://testing.open-bio.org/biopython/builders/Linux%2064%20-%20Python%203.4/builds/1/steps/shell/logs/stdio [There seem to be a few problems] I would like to notice that it seems that there is no mysqldb connector for python3 (??!!) and thus we cannot test that. Tiago From p.j.a.cock at googlemail.com Sat Apr 26 13:19:05 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Sat, 26 Apr 2014 18:19:05 +0100 Subject: [Biopython-dev] TogoWS test failure, was: Compiling Biopython on 64 bit Windows In-Reply-To: References: Message-ID: On Sat, Apr 19, 2014 at 1:23 PM, Peter Cock wrote: > The TogoWS team confirmed this is a problem with the > server and expect it to be fixed shortly. > > Peter For the record, this is working now :) Peter From p.j.a.cock at googlemail.com Sun Apr 27 08:43:23 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Sun, 27 Apr 2014 13:43:23 +0100 Subject: [Biopython-dev] BWA command line change? Message-ID: Hi Saket, Could you have a look at this BWA problem Andreas found while testing the Debian package? I have just reordered the error message in Bio.Application to put the return code earlier in case the exception text is truncated (as here): https://github.com/biopython/biopython/commit/9860973c05aeb102f82e217ff3a88e241a3ba620 I've also switched to relative paths which will also shorten the error message: https://github.com/biopython/biopython/commit/a03d23760783c157ff7c30f483ace55beda31c09 I just tested with BWA 0.6.1 (in the Debian repository on my old desktop), and test_BWA_tool.py was fine. Thanks, Peter ---------- Forwarded message ---------- From: Andreas Tille Date: Tue, Mar 4, 2014 at 8:18 PM Subject: Re: Adding python3-biopython To: Peter Cock Cc: Debian Med Project List , Philipp Benner Hi Peter, I can confirm that I'm working on the python3 packages. I'm not finished yet but I included more verbose testing into the packaging and when I did so I realised some incompatibilities with bwa: ERROR: test_sampe (test_BWA_tool.BwaTestCase) Test for generating samfile by paired end sequencing ---------------------------------------------------------------------- Traceback (most recent call last): File "test_BWA_tool.py", line 113, in test_sampe self.do_aln(self.infile1, self.saifile1) File "test_BWA_tool.py", line 90, in do_aln stdout, stderr = cmdline(stdout=out_file) File "/tmp/buildd/python-biopython-1.63/Bio/Application/__init__.py", line 511, in __call__ stdout_str, stderr_str) ApplicationError: Command 'bwa aln /tmp/buildd/python-biopython-1.63/Tests/BWA/human_g1k_v37_truncated.fasta /tmp/buildd/python-biopython-1.63/Tests/BWA/HNSCC1_1_truncated.fastq' returne ====================================================================== ERROR: test_samse (test_BWA_tool.BwaTestCase) Test for single end sequencing ---------------------------------------------------------------------- Traceback (most recent call last): File "test_BWA_tool.py", line 102, in test_samse stdout, stderr = cmdline(stdout=self.samfile1) File "/tmp/buildd/python-biopython-1.63/Bio/Application/__init__.py", line 511, in __call__ stdout_str, stderr_str) ApplicationError: Command 'bwa samse /tmp/buildd/python-biopython-1.63/Tests/BWA/human_g1k_v37_truncated.fasta /tmp/buildd/python-biopython-1.63/Tests/BWA/1.sai /tmp/buildd/python-biopyt The used bwa says: $ bwa Program: bwa (alignment via Burrows-Wheeler transformation) Version: 0.7.6a-r433 Contact: Heng Li Usage: bwa [options] Command: index index sequences in the FASTA format mem BWA-MEM algorithm fastmap identify super-maximal exact matches fa2pac convert FASTA to PAC format pac2bwt generate BWT from PAC pac2bwtgen alternative algorithm for generating BWT bwtupdate update .bwt to the new format bwt2sa generate SA from BWT and Occ Note: To use BWA, you need to first index the genome with `bwa index'. There are three alignment algorithms in BWA: `mem', `bwasw', and `aln/samse/sampe'. If you are not sure which to use, try `bwa mem' first. Please `man ./bwa.1' for the manual. I think you should adjust your test somehow to make sure that also the latest bwa version will succeed. Kind regards Andreas. On Mon, Mar 03, 2014 at 02:08:52PM +0000, Peter Cock wrote: > Hello Philipp & DebianMed, > > The last few releases of Biopython supported Python 3 > via 2to3, but as of Biopython 1.63 (released December > 2013) we have a single codebase which works on both > Python 2.6/2.7 and Python 3.3+. ... -- http://fam-tille.de From manlio.calvi at gmail.com Mon Apr 28 04:09:54 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 28 Apr 2014 10:09:54 +0200 Subject: [Biopython-dev] Pypy2 on Win7 x64, numpy problem Message-ID: Hi all, Yesterday, after a short break on compiling, I tried to build my pypy 2 stack to run the biopython tests on it, the problem is on numpy, for every thing I tried to compile their numpy version I failed, out of curiosity I tried to compile numpy on pypy3 but get even worse. In the end both don't compile, at this point I stopped after looking my pypy envs are sane. Manlio From p.j.a.cock at googlemail.com Mon Apr 28 08:51:10 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 28 Apr 2014 13:51:10 +0100 Subject: [Biopython-dev] Pypy2 on Win7 x64, numpy problem In-Reply-To: References: Message-ID: On Mon, Apr 28, 2014 at 9:09 AM, Manlio Calvi wrote: > Hi all, > > Yesterday, after a short break on compiling, I tried to build my pypy > 2 stack to run the biopython tests on it, the problem is on numpy, for > every thing I tried to compile their numpy version I failed, out of > curiosity I tried to compile numpy on pypy3 but get even worse. In the > end both don't compile, at this point I stopped after looking my pypy > envs are sane. > > Manlio NumPy has a ton of C code, and will not work on PyPy. Attempting to compile/install NumPy under PyPy is a waste of time. However, the PyPy team are reimplementing (some of) NumPy themselves as the NumPyPy project - see e.g. http://buildbot.pypy.org/numpy-status/latest.html Currently Biopython ignores NumPy (and NumPyPy) under PyPy - last time I did a proper test there were still too many things missing to make it worth the effort. However, this is something we should review - I would hope by now that at least the PDB parsing code might work - even if things like SVD and numpy.linalg are not available on PyPy. Similarly, NumPy won't work under Jython. Peter From manlio.calvi at gmail.com Mon Apr 28 09:02:34 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 28 Apr 2014 15:02:34 +0200 Subject: [Biopython-dev] Pypy2 on Win7 x64, numpy problem In-Reply-To: References: Message-ID: I tried to install their Numpy (the Numpypy project) because I read about their difficulties about porting Cpython Numpy but even with that I get errors everywhere and don't compile it. OK giving the tests ignore Numpy(py) I'll try to build biopython against it. On Mon, Apr 28, 2014 at 2:51 PM, Peter Cock wrote: > On Mon, Apr 28, 2014 at 9:09 AM, Manlio Calvi wrote: >> Hi all, >> >> Yesterday, after a short break on compiling, I tried to build my pypy >> 2 stack to run the biopython tests on it, the problem is on numpy, for >> every thing I tried to compile their numpy version I failed, out of >> curiosity I tried to compile numpy on pypy3 but get even worse. In the >> end both don't compile, at this point I stopped after looking my pypy >> envs are sane. >> >> Manlio > > NumPy has a ton of C code, and will not work on PyPy. > Attempting to compile/install NumPy under PyPy is a waste > of time. > > However, the PyPy team are reimplementing (some of) > NumPy themselves as the NumPyPy project - see e.g. > http://buildbot.pypy.org/numpy-status/latest.html > > Currently Biopython ignores NumPy (and NumPyPy) under > PyPy - last time I did a proper test there were still too many > things missing to make it worth the effort. > > However, this is something we should review - I would hope > by now that at least the PDB parsing code might work - even > if things like SVD and numpy.linalg are not available on PyPy. > > Similarly, NumPy won't work under Jython. > > Peter From manlio.calvi at gmail.com Mon Apr 28 14:53:43 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 28 Apr 2014 20:53:43 +0200 Subject: [Biopython-dev] Pypy x64 tests Was: Pypy2 on Win7 x64, numpy problem Message-ID: OK the Tests on Pypy 2 passed but a lot was skipped for one reason to another. After this mail I'll send another with builds and tests log for archive and lookups Manlio On Mon, Apr 28, 2014 at 3:02 PM, Manlio Calvi wrote: > I tried to install their Numpy (the Numpypy project) because I read > about their difficulties about porting Cpython Numpy but even with > that I get errors everywhere and don't compile it. > OK giving the tests ignore Numpy(py) I'll try to build biopython against it. > > > > On Mon, Apr 28, 2014 at 2:51 PM, Peter Cock wrote: >> On Mon, Apr 28, 2014 at 9:09 AM, Manlio Calvi wrote: >>> Hi all, >>> >>> Yesterday, after a short break on compiling, I tried to build my pypy >>> 2 stack to run the biopython tests on it, the problem is on numpy, for >>> every thing I tried to compile their numpy version I failed, out of >>> curiosity I tried to compile numpy on pypy3 but get even worse. In the >>> end both don't compile, at this point I stopped after looking my pypy >>> envs are sane. >>> >>> Manlio >> >> NumPy has a ton of C code, and will not work on PyPy. >> Attempting to compile/install NumPy under PyPy is a waste >> of time. >> >> However, the PyPy team are reimplementing (some of) >> NumPy themselves as the NumPyPy project - see e.g. >> http://buildbot.pypy.org/numpy-status/latest.html >> >> Currently Biopython ignores NumPy (and NumPyPy) under >> PyPy - last time I did a proper test there were still too many >> things missing to make it worth the effort. >> >> However, this is something we should review - I would hope >> by now that at least the PDB parsing code might work - even >> if things like SVD and numpy.linalg are not available on PyPy. >> >> Similarly, NumPy won't work under Jython. >> >> Peter From manlio.calvi at gmail.com Mon Apr 28 15:07:56 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 28 Apr 2014 21:07:56 +0200 Subject: [Biopython-dev] Tests on x64 Win 7 Message-ID: Hi people, I rebuild and retest the last biopython commit, no test failed so I here signal some recurrent warning in python3.4, nothing too bad but there is a lot of file left not closed. This problem could be related to the one with the remaing files after the tests? Seems to me was the same files. If that the case it could be a class who open and don't the files after the read. PS after this mail I'll send another mail with the logs. Please Peter, acknowledge that mail (and the one before that on Pypy subject). Manlio From manlio.calvi at gmail.com Mon Apr 28 14:55:33 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 28 Apr 2014 20:55:33 +0200 Subject: [Biopython-dev] Pypy x64 tests Was: Pypy2 on Win7 x64, numpy problem In-Reply-To: References: Message-ID: Here's the logs, enjoy! Manlio On Mon, Apr 28, 2014 at 8:53 PM, Manlio Calvi wrote: > OK the Tests on Pypy 2 passed but a lot was skipped for one reason to another. > After this mail I'll send another with builds and tests log for > archive and lookups > > Manlio > > > On Mon, Apr 28, 2014 at 3:02 PM, Manlio Calvi wrote: >> I tried to install their Numpy (the Numpypy project) because I read >> about their difficulties about porting Cpython Numpy but even with >> that I get errors everywhere and don't compile it. >> OK giving the tests ignore Numpy(py) I'll try to build biopython against it. >> >> >> >> On Mon, Apr 28, 2014 at 2:51 PM, Peter Cock wrote: >>> On Mon, Apr 28, 2014 at 9:09 AM, Manlio Calvi wrote: >>>> Hi all, >>>> >>>> Yesterday, after a short break on compiling, I tried to build my pypy >>>> 2 stack to run the biopython tests on it, the problem is on numpy, for >>>> every thing I tried to compile their numpy version I failed, out of >>>> curiosity I tried to compile numpy on pypy3 but get even worse. In the >>>> end both don't compile, at this point I stopped after looking my pypy >>>> envs are sane. >>>> >>>> Manlio >>> >>> NumPy has a ton of C code, and will not work on PyPy. >>> Attempting to compile/install NumPy under PyPy is a waste >>> of time. >>> >>> However, the PyPy team are reimplementing (some of) >>> NumPy themselves as the NumPyPy project - see e.g. >>> http://buildbot.pypy.org/numpy-status/latest.html >>> >>> Currently Biopython ignores NumPy (and NumPyPy) under >>> PyPy - last time I did a proper test there were still too many >>> things missing to make it worth the effort. >>> >>> However, this is something we should review - I would hope >>> by now that at least the PDB parsing code might work - even >>> if things like SVD and numpy.linalg are not available on PyPy. >>> >>> Similarly, NumPy won't work under Jython. >>> >>> Peter -------------- next part -------------- A non-text attachment was scrubbed... Name: buildpypy.log Type: application/octet-stream Size: 41621 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: testpypy.log Type: application/octet-stream Size: 14772 bytes Desc: not available URL: From manlio.calvi at gmail.com Mon Apr 28 15:09:43 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 28 Apr 2014 21:09:43 +0200 Subject: [Biopython-dev] Tests on x64 Win 7 In-Reply-To: References: Message-ID: Here's the files Manlio On Mon, Apr 28, 2014 at 9:07 PM, Manlio Calvi wrote: > Hi people, > > I rebuild and retest the last biopython commit, no test failed so I > here signal some recurrent warning in python3.4, nothing too bad but > there is a lot of file left not closed. This problem could be related > to the one with the remaing files after the tests? Seems to me was the > same files. > If that the case it could be a class who open and don't the files > after the read. > > PS after this mail I'll send another mail with the logs. Please Peter, > acknowledge that mail (and the one before that on Pypy subject). > > Manlio -------------- next part -------------- A non-text attachment was scrubbed... Name: build27.log Type: application/octet-stream Size: 63700 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test27.log Type: application/octet-stream Size: 10789 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: build34.log Type: application/octet-stream Size: 63796 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test34.log Type: application/octet-stream Size: 24973 bytes Desc: not available URL: From p.j.a.cock at googlemail.com Mon Apr 28 15:49:16 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 28 Apr 2014 20:49:16 +0100 Subject: [Biopython-dev] Tests on x64 Win 7 In-Reply-To: References: Message-ID: On Mon, Apr 28, 2014 at 8:07 PM, Manlio Calvi wrote: > Hi people, > > I rebuild and retest the last biopython commit, no test failed so I > here signal some recurrent warning in python3.4, nothing too bad but > there is a lot of file left not closed. This problem could be related > to the one with the remaing files after the tests? Seems to me was the > same files. > If that the case it could be a class who open and don't the files > after the read. The problem is lots of (old) Python code tends to do something like this: handle = open(filename) handle.read() etc #handle.close() missing This relies on C Python doing garbage collection when the handle variable goes out of scope, which will close the handle. This is not so predictable on alternative Python implementations like Jython and PyPy so the handle may not get closed for some time. That can cause problems, for example the OS imposes limits on the number of open files, and Windows will refuse to delete a file if there are handles open to it. The solution is to either close the handle explicitly, or use a context manager (with statement). Now in some senses the warning is harmless, see: http://emptysqua.re/blog/against-resourcewarnings-in-python-3/ By default the ResourceWarning is silent, but it seems something sometimes goes wrong with our test framework and this starts to show up (due to a possible design flaw, the Python warnings filters are global): http://lists.open-bio.org/pipermail/biopython-dev/2013-April/010521.html How we test for warnings and/or silence them is something in need of an overhaul - perhaps the warnings.catch_warnings context manager would help (new in Python 2.6, so we can use that)? > PS after this mail I'll send another mail with the logs. Please Peter, > acknowledge that mail (and the one before that on Pypy subject). I approved those emails with attachments which were waiting for moderation. Peter From saketkc at gmail.com Mon Apr 28 16:07:43 2014 From: saketkc at gmail.com (Saket Choudhary) Date: Tue, 29 Apr 2014 01:37:43 +0530 Subject: [Biopython-dev] BWA command line change? In-Reply-To: References: Message-ID: This is turning out to be an issue with this particular version of bwa[0.7.6a-r433];l $ bwa7.6a aln [main] unrecognized command 'aln' However the later two versions do support it, and the tests do pass: $ bwa7.8 aln Usage: bwa aln [options] $ bwa7.7 aln Usage: bwa aln [options] I have bwa7.5[ 0.7.5a-r405] installed which I used for testing and it seems to be working as well. Unfortunately, I was not able to locate this error being reported on the bwa mailing list[The reason probably being more people using 'mem' now] Andrew, can you check this with the latest version of bwa?[0.7.8] Thanks, Saket On 27 April 2014 18:13, Peter Cock wrote: > Hi Saket, > > Could you have a look at this BWA problem Andreas found > while testing the Debian package? > > I have just reordered the error message in Bio.Application > to put the return code earlier in case the exception text is > truncated (as here): > https://github.com/biopython/biopython/commit/9860973c05aeb102f82e217ff3a88e241a3ba620 > > I've also switched to relative paths which will also shorten > the error message: > https://github.com/biopython/biopython/commit/a03d23760783c157ff7c30f483ace55beda31c09 > > I just tested with BWA 0.6.1 (in the Debian repository on > my old desktop), and test_BWA_tool.py was fine. > > Thanks, > > Peter > > > ---------- Forwarded message ---------- > From: Andreas Tille > Date: Tue, Mar 4, 2014 at 8:18 PM > Subject: Re: Adding python3-biopython > To: Peter Cock > Cc: Debian Med Project List , Philipp > Benner > > > Hi Peter, > > I can confirm that I'm working on the python3 packages. I'm not > finished yet but I included more verbose testing into the packaging and > when I did so I realised some incompatibilities with bwa: > > > ERROR: test_sampe (test_BWA_tool.BwaTestCase) > Test for generating samfile by paired end sequencing > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "test_BWA_tool.py", line 113, in test_sampe > self.do_aln(self.infile1, self.saifile1) > File "test_BWA_tool.py", line 90, in do_aln > stdout, stderr = cmdline(stdout=out_file) > File "/tmp/buildd/python-biopython-1.63/Bio/Application/__init__.py", > line 511, in __call__ > stdout_str, stderr_str) > ApplicationError: Command 'bwa aln > /tmp/buildd/python-biopython-1.63/Tests/BWA/human_g1k_v37_truncated.fasta > /tmp/buildd/python-biopython-1.63/Tests/BWA/HNSCC1_1_truncated.fastq' > returne > > ====================================================================== > ERROR: test_samse (test_BWA_tool.BwaTestCase) > Test for single end sequencing > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "test_BWA_tool.py", line 102, in test_samse > stdout, stderr = cmdline(stdout=self.samfile1) > File "/tmp/buildd/python-biopython-1.63/Bio/Application/__init__.py", > line 511, in __call__ > stdout_str, stderr_str) > ApplicationError: Command 'bwa samse > /tmp/buildd/python-biopython-1.63/Tests/BWA/human_g1k_v37_truncated.fasta > /tmp/buildd/python-biopython-1.63/Tests/BWA/1.sai > /tmp/buildd/python-biopyt > > > > The used bwa says: > > $ bwa > > Program: bwa (alignment via Burrows-Wheeler transformation) > Version: 0.7.6a-r433 > Contact: Heng Li > > Usage: bwa [options] > > Command: index index sequences in the FASTA format > mem BWA-MEM algorithm > fastmap identify super-maximal exact matches > > fa2pac convert FASTA to PAC format > pac2bwt generate BWT from PAC > pac2bwtgen alternative algorithm for generating BWT > bwtupdate update .bwt to the new format > bwt2sa generate SA from BWT and Occ > > Note: To use BWA, you need to first index the genome with `bwa index'. > There are three alignment algorithms in BWA: `mem', `bwasw', and > `aln/samse/sampe'. If you are not sure which to use, try `bwa mem' > first. Please `man ./bwa.1' for the manual. > > > I think you should adjust your test somehow to make sure that > also the latest bwa version will succeed. > > Kind regards > > Andreas. > > > On Mon, Mar 03, 2014 at 02:08:52PM +0000, Peter Cock wrote: >> Hello Philipp & DebianMed, >> >> The last few releases of Biopython supported Python 3 >> via 2to3, but as of Biopython 1.63 (released December >> 2013) we have a single codebase which works on both >> Python 2.6/2.7 and Python 3.3+. > ... > > -- > http://fam-tille.de From p.j.a.cock at googlemail.com Mon Apr 28 16:23:51 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 28 Apr 2014 21:23:51 +0100 Subject: [Biopython-dev] BWA command line change? In-Reply-To: References: Message-ID: On Mon, Apr 28, 2014 at 9:07 PM, Saket Choudhary wrote: > This is turning out to be an issue with this particular version of > bwa[0.7.6a-r433];l > $ bwa7.6a aln > [main] unrecognized command 'aln' > > However the later two versions do support it, and the tests do pass: The simplest solution may be to check for that error during the header where we see if bwa is installed, and if it is missing the aln command skip the tests? Peter From saketkc at gmail.com Mon Apr 28 17:17:35 2014 From: saketkc at gmail.com (Saket Choudhary) Date: Tue, 29 Apr 2014 02:47:35 +0530 Subject: [Biopython-dev] BWA command line change? In-Reply-To: References: Message-ID: Commited a fix here: https://github.com/saketkc/biopython/commit/d884ee6e12a798b974b3a3996e22b5292ab79be6 If it looks good, I can send a PR? Unfortunately my commit history is a bit muddled up on master. On 29 April 2014 01:53, Peter Cock wrote: > On Mon, Apr 28, 2014 at 9:07 PM, Saket Choudhary wrote: >> This is turning out to be an issue with this particular version of >> bwa[0.7.6a-r433];l >> $ bwa7.6a aln >> [main] unrecognized command 'aln' >> >> However the later two versions do support it, and the tests do pass: > > The simplest solution may be to check for that error during > the header where we see if bwa is installed, and if it is > missing the aln command skip the tests? > > Peter From saketkc at gmail.com Tue Apr 29 08:45:24 2014 From: saketkc at gmail.com (Saket Choudhary) Date: Tue, 29 Apr 2014 18:15:24 +0530 Subject: [Biopython-dev] BioPython Debian packages uploaded (Was: BWA command line change?) In-Reply-To: <20140429091359.GC20235@an3as.eu> References: <20140429091359.GC20235@an3as.eu> Message-ID: Thanks Andreas, Just for the record, I have now submitted this as a pull request: https://github.com/biopython/biopython/pull/315 Saket On 29 April 2014 14:43, Andreas Tille wrote: > Hi, > > I have included the patch below in the just uploaded BioPython package. > The package will need to passe the new queue due to the new packages for > Python 3.3 and 3.4 so it might need some days to become available in > Debian unstable. > > Besides the Python 3 support the other major change is that we try to > run as many tests of the test suite as possible at package build time > and there is also an autopkgtest added which means that in a regular > basis the package is installed automatically on a clean system and the > test suite is runned on the installation. > > As an additional hint for you as developers: Here you can find all > patches applied to the package: > > http://anonscm.debian.org/viewvc/debian-med/trunk/packages/python-biopython/trunk/debian/patches/ > > I think the only patch which I did not reported is > biopython-from-pil-import-image.patch which I'd recommend to apply > upstream as well (description is included). > > Kind regards and thanks for providing BioPython as free software > > Andreas. > > On Tue, Apr 29, 2014 at 02:47:35AM +0530, Saket Choudhary wrote: >> Commited a fix here: >> https://github.com/saketkc/biopython/commit/d884ee6e12a798b974b3a3996e22b5292ab79be6 >> >> If it looks good, I can send a PR? >> Unfortunately my commit history is a bit muddled up on master. >> >> On 29 April 2014 01:53, Peter Cock wrote: >> > On Mon, Apr 28, 2014 at 9:07 PM, Saket Choudhary wrote: >> >> This is turning out to be an issue with this particular version of >> >> bwa[0.7.6a-r433];l >> >> $ bwa7.6a aln >> >> [main] unrecognized command 'aln' >> >> >> >> However the later two versions do support it, and the tests do pass: >> > >> > The simplest solution may be to check for that error during >> > the header where we see if bwa is installed, and if it is >> > missing the aln command skip the tests? >> > >> > Peter >> > > -- > http://fam-tille.de From p.j.a.cock at googlemail.com Wed Apr 30 06:02:52 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 30 Apr 2014 11:02:52 +0100 Subject: [Biopython-dev] Python 3.4 - UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 Message-ID: Hi all, One of the 64 bit Linux buildslaves is showing a unicode problem on Python 3.4 in test_SeqIO_SeqXML.py and test_Phylo.py - perhaps locale related? e.g. http://testing.open-bio.org/biopython/builders/Linux%2064%20-%20Python%203.4/builds/9/steps/shell/logs/stdio UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 797: ordinal not in range(128) Tiago, this is on your new docker3 buildslave - can you reproduce this error 'by hand' and look at the locale settings please? These seem to be the "problem" files: $ hexdump -C PhyloXML/phyloxml_examples.xml | grep " c3 " 00003320 20 5a c3 bc 72 69 63 68 3c 2f 64 65 73 63 3e 0d | Z..rich.| $ hexdump -C SeqXML/rna_example.xml | grep " c3 " 00000310 3c 64 65 73 63 72 69 70 74 69 6f 6e 3e c3 a5 c3 |...| 00000320 85 c3 bc c3 b6 c3 96 c3 9f c3 b8 c3 a4 c2 a2 c2 |................| These are deliberate tests of accented characters (and other non-ASCII text) in a unicode description. Both XML files says they are using UTF-8 as the encoding. Thanks, Peter From p.j.a.cock at googlemail.com Wed Apr 30 06:06:22 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 30 Apr 2014 11:06:22 +0100 Subject: [Biopython-dev] Python 3.4 In-Reply-To: <20140425100503.4939bd34@lnx> References: <20140422102316.5b049105@lnx> <20140425100503.4939bd34@lnx> Message-ID: On Fri, Apr 25, 2014 at 10:05 AM, Tiago Antao wrote: > On Fri, 25 Apr 2014 08:21:27 +0100 Peter Cock wrote: > >> I've asked the buildbot master to run tests for Python 3.4 >> on the Mac now, and got another 64bit Linux machine >> ready to run the tests. > > I hope to lunch mine this morning (or early afternoon - meetings...). > The docker work is finished. I can see it as a buildslave now - well done :) >> Oddly the buildbot server has become quite sluggish, >> perhaps we have too much old data and/or too many >> build targets? Removing the older PyPy versions >> from the matrix should help: >> http://lists.open-bio.org/pipermail/biopython/2014-April/009110.html > > > I think we need to purge the old data. I do not think we have anything > resembling too much build targets. I will try to purge the old data > myself today. Thanks Tiago - its a shame to lose the old builds with error logs, but it was of limited value now. Thanks for doing that - the buildbot webpages are very responsive now :) Now some good news for Python 3.4 - this is now supported by TravisCI, and our tests pass: https://github.com/biopython/biopython/commit/e61afb73fdc6cf5721e16299414e6b4b161a5099 Bad news - another unicode error under Python 3.4: http://lists.open-bio.org/pipermail/biopython-dev/2014-April/011355.html Regards, Peter From p.j.a.cock at googlemail.com Wed Apr 30 06:25:41 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 30 Apr 2014 11:25:41 +0100 Subject: [Biopython-dev] Python 3.4 In-Reply-To: References: <20140422102316.5b049105@lnx> <20140425100503.4939bd34@lnx> Message-ID: For the README file we currently say: We currently recommend using Python 2.7 from http://www.python.org which is the final version of Python 2. Early adopters are encouraged to try Biopython under Python 3.3 as well. How about this?: Most of the third party Python libraries Biopython can use are now supported under Python 3 (e.g. NumPy and ReportLab), but not all (e.g. MySQLdb). Overall we therefore still recommend using Python 2.7 from http://www.python.org which is the final version of Python 2. Peter From tra at popgen.net Wed Apr 30 06:33:11 2014 From: tra at popgen.net (Tiago Antao) Date: Wed, 30 Apr 2014 11:33:11 +0100 Subject: [Biopython-dev] Python 3.4 In-Reply-To: References: <20140422102316.5b049105@lnx> <20140425100503.4939bd34@lnx> Message-ID: <20140430113311.34cff5e1@lnx> I am wondering if it would make sense to replace mysqldb with the pure python mysql connector? That would cut a non-python dependency out whilst making everything python 3 compliant. I would be willing to volunteer time (after the next release) for this. On Wed, 30 Apr 2014 11:25:41 +0100 Peter Cock wrote: > For the README file we currently say: > > We currently recommend using Python 2.7 from http://www.python.org > which is the final version of Python 2. Early adopters are encouraged > to try Biopython under Python 3.3 as well. > > How about this?: > > Most of the third party Python libraries Biopython can use are now > supported under Python 3 (e.g. NumPy and ReportLab), but not all > (e.g. MySQLdb). Overall we therefore still recommend using Python 2.7 > from http://www.python.org which is the final version of Python 2. > > Peter > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev From p.j.a.cock at googlemail.com Wed Apr 30 06:38:29 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 30 Apr 2014 11:38:29 +0100 Subject: [Biopython-dev] Python 3.4 In-Reply-To: <20140430113311.34cff5e1@lnx> References: <20140422102316.5b049105@lnx> <20140425100503.4939bd34@lnx> <20140430113311.34cff5e1@lnx> Message-ID: Since mysqldb isn't being actively maintained, that seems like a good plan - thanks :) Note several very old bits of Biopython use MySQL (not just BioSQL), but I suspect they are not in wide use, and could be targets for deprecation. Peter On Wed, Apr 30, 2014 at 11:33 AM, Tiago Antao wrote: > I am wondering if it would make sense to replace mysqldb with the pure > python mysql connector? That would cut a non-python dependency out > whilst making everything python 3 compliant. > > I would be willing to volunteer time (after the next release) for > this. > > On Wed, 30 Apr 2014 11:25:41 +0100 > Peter Cock wrote: > >> For the README file we currently say: >> >> We currently recommend using Python 2.7 from http://www.python.org >> which is the final version of Python 2. Early adopters are encouraged >> to try Biopython under Python 3.3 as well. >> >> How about this?: >> >> Most of the third party Python libraries Biopython can use are now >> supported under Python 3 (e.g. NumPy and ReportLab), but not all >> (e.g. MySQLdb). Overall we therefore still recommend using Python 2.7 >> from http://www.python.org which is the final version of Python 2. >> >> Peter >> _______________________________________________ >> Biopython-dev mailing list >> Biopython-dev at lists.open-bio.org >> http://lists.open-bio.org/mailman/listinfo/biopython-dev > From p.j.a.cock at googlemail.com Wed Apr 30 09:24:42 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 30 Apr 2014 14:24:42 +0100 Subject: [Biopython-dev] Bug#746484: file not distributable In-Reply-To: <20140430131703.GD758@an3as.eu> References: <20140430131703.GD758@an3as.eu> Message-ID: Hi Andreas, Are you specifically asking about Biopython 1.63 here? I think you can reasonable exclude this DTD file (and any others under the Bio/Entrez/DTD file). Biopython 1.63 will warn if they are missing but attempt to download them automatically. We're looking at dropping all the NCBI Entrez related DTD files, since the next Biopython release (v1.64) will automatically download AND cache them - see recent discussion, e.g. http://lists.open-bio.org/pipermail/biopython-dev/2014-March/011205.html We haven't actually removed the files on GitHub yet, but this might be an incentive to do so. Thanks, Peter On Wed, Apr 30, 2014 at 2:17 PM, Andreas Tille wrote: > Hello, > > our ftpmaster has detected an issue with one of the DTDs which are > distributed with BioPython source. > > I can confirm that after applying the following patch > > > --- a/Bio/Entrez/DTDs/modules.ent > +++ b/Bio/Entrez/DTDs/modules.ent > @@ -350,13 +350,6 @@ Version Reason/Occasion > "mathmlsetup.ent" > > > > - > - - PUBLIC > -"-//W3C//ENTITIES MathML 2.0 Qualified Names 1.0//EN" > -"mathml/mathml2-qname-1.mod" > > - > - > > "-//W3C//DTD MathML 2.0//EN" > > > the file in question can be removed from the archive without breaking > the build (including the test suite). I would like to suggest to drop > the file in question from your distribution tarball in case my analysis > that it is not needed is correct. > > Kind regards > > Andreas. > > On Wed, Apr 30, 2014 at 02:25:09PM +0200, Thorsten Alteholz wrote: >> Package: python-biopython >> Version: 1.63-2 >> Severity: serious >> User: alteholz at debian.org >> Usertags: ftp >> X-Debbugs-CC: ftpmaster at ftp-master.debian.org >> thanks >> >> Dear Maintainer, >> >> according to: >> http://www.w3.org/Consortium/Legal/2002/copyright-documents-20021231 >> the file >> biopython-1.63\Bio\Entrez\DTDs\mathml2-qname-1.mod >> may not be modified and such this file is not distributable in main. >> >> Thorsten >> >> _______________________________________________ >> Debian-med-packaging mailing list >> Debian-med-packaging at lists.alioth.debian.org >> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/debian-med-packaging >> > > -- > http://fam-tille.de From p.j.a.cock at googlemail.com Wed Apr 30 09:38:11 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 30 Apr 2014 14:38:11 +0100 Subject: [Biopython-dev] Bug#746484: file not distributable In-Reply-To: <20140430133016.GE758@an3as.eu> References: <20140430131703.GD758@an3as.eu> <20140430133016.GE758@an3as.eu> Message-ID: On Wed, Apr 30, 2014 at 2:30 PM, Andreas Tille wrote: > Hi Peter, > > thanks for your super-fast response. > > On Wed, Apr 30, 2014 at 02:24:42PM +0100, Peter Cock wrote: >> Are you specifically asking about Biopython 1.63 here? > > Yes. Since I have added python3 binary packages Biopython 1.63 went > through manual inspection by ftpmaster and this issue was noticed. Very through of them - thanks! Also thank you for doing the Debian Python3 packaging of Biopython :) >> I think you >> can reasonable exclude this DTD file (and any others under the >> Bio/Entrez/DTD file). Biopython 1.63 will warn if they are missing >> but attempt to download them automatically. > > OK. > >> We're looking at dropping all the NCBI Entrez related DTD files, >> since the next Biopython release (v1.64) will automatically download >> AND cache them - see recent discussion, e.g. >> >> http://lists.open-bio.org/pipermail/biopython-dev/2014-March/011205.html > > That's fine. > >> We haven't actually removed the files on GitHub yet, but this >> might be an incentive to do so. > > OK, meanwhile (as long as 1.64 is not yet released) I will remove the > file from the Debian archive. > > Thanks for the clarification > > Andreas. Great, Peter P.S. I'm skimming over the Debian patches to see what we can fix: http://anonscm.debian.org/viewvc/debian-med/trunk/packages/python-biopython/trunk/debian/patches/ e.g. https://github.com/biopython/biopython/commit/2f098ac5311e0eec3d6737f4fff60e18c50b9481 From p.j.a.cock at googlemail.com Wed Apr 30 10:05:51 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 30 Apr 2014 15:05:51 +0100 Subject: [Biopython-dev] PIL/Pillow import, was: Testings on Win 7 x64 Message-ID: On Thu, Apr 24, 2014 at 5:49 PM, Peter Cock wrote: > On Thu, Apr 24, 2014 at 5:34 PM, Manlio Calvi wrote: >> >> Second a more general signalation, I've seen some test depends on PIL >> but PIL library get discontinued and substituted by one called Pillow >> that is said compatible with it predecessor (but I'm not sure how much >> it is). This is a more general issue because require an scan of the >> entire codebase. Looking around seems to me the NLTK package is in a >> situation like this, too. > > That probably refers to test_GraphicsBitmaps.py which is trying to > check if ReportLab will be able to make PNG files. If ReportLab > can now use Pillow or PIL, we should allow for that... > https://github.com/python-imaging/Pillow > > (As an aside ReportLab's Python 3 support is still quite new - > but I don't recall seeing any problems from Biopython's usage > of it.) During Debian packaging Andreas Tille had to tweak Biopython 1.63 to work nicely with Pillow (which does seem to have effectively taken over from the older PIL project): http://anonscm.debian.org/viewvc/debian-med/trunk/packages/python-biopython/trunk/debian/patches/biopython-from-pil-import-image.patch?revision=16855&view=markup I've made the following (hopefully) backwards compatible change: https://github.com/biopython/biopython/commit/3c2a2f683d766d43964e7271e90c28c16be3b06e Peter From krishnaroskin at gmail.com Tue Apr 1 06:41:04 2014 From: krishnaroskin at gmail.com (Krishna Roskin) Date: Mon, 31 Mar 2014 23:41:04 -0700 Subject: [Biopython-dev] interest module for sequence based clustering tools? Message-ID: Hey all, Long time fan, taking my first crack at contributing. I've built a basic module to run and parse the result of sequence based clustering tools such as DNACLUST and CD-HIT. I've written subclasses of AbstractCommandline to run dnaclust and cd-hit. I've also written classes to store the clusters and their members and loaders for the output formats used by those programs. I posting here to gauge interest and get some feedback and maybe some beta testers. My code is available at: https://github.com/krishnaroskin/biopython.git under the seqcluster branch. I've started writing some test code at: Tests/seqcluster/test_seqcluster.py that also severs as example code. I've pasted that at the end of this message so people can get an idea of how it works without having to checkout code. If there is interest, I'm planning on adding a seqclust.cluster function that takes a list of SeqRecords and returns a clustering using one of the supported tools. I envision that function being the main interface to this module. I also want to write a something that will map the cluster membership (given by ids) back to collections of SeqRecords. Other to-dos: Test that all the flavors of CD-HIT work (there are many) Add support for other sequence based clustering tools (suggestions?) Documentation Tutorial Test code -krish #!/usr/bin/env python from __future__ import print_function import StringIO from Bio.seqcluster.applications import DNAClustCommandline from Bio.seqcluster import DNAClustIterator from Bio.seqcluster.applications import CDHITCommandline from Bio.seqcluster import CDHITClustIterator cmd = DNAClustCommandline(similarity=0.8, header=True, threads=2, inputfile= "test_sequences.fasta") stdout, stderr = cmd() clusters = DNAClustIterator(StringIO.StringIO(stdout)) for cluster in clusters: print(cluster.name) for member in cluster: if member == cluster.representative: print("\t" + member.name + "*") else: print("\t" + member.name) print() # blank line cmd = CDHITCommandline(cutoff=0.8, threads=2, inputfile= "test_sequences.fasta", outputfile="tmp") stdout, stderr = cmd() clusters = CDHITClustIterator(open("tmp.clstr", "r")) for cluster in clusters: print(cluster.name) for member in cluster: if member == cluster.representative: print("\t" + member.name + "*") else: print("\t" + member.name) From tra at popgen.net Tue Apr 1 10:46:20 2014 From: tra at popgen.net (Tiago Antao) Date: Tue, 1 Apr 2014 11:46:20 +0100 Subject: [Biopython-dev] Fwd: [Numpy-discussion] EuroSciPy 2014 Call for Abstracts In-Reply-To: References: <20140320114815.70210fd5@grandao> <87txathtux.fsf@chupacabra.windows.ebi.ac.uk> Message-ID: <20140401114620.67490302@lnx> Hi, On Mon, 31 Mar 2014 17:30:21 +0100 Peter Cock wrote: > I'm glad to see there are several people interested in EuroSciPy. > One one of you like to submit a Biopython talk? The deadline > is 14 April: https://www.euroscipy.org/2014/calls/abstracts/ I know I have fuelled all this, but I will only know next week if I am in the US or not. I could offer to do the presentation, if I am around. In any case, I still believe it might be a good geographical place for some Biopython developers to meet. Brandon: We can talk face-to-face on Thursday, as we will be in the same place ;) Tiago From p.j.a.cock at googlemail.com Tue Apr 1 13:08:45 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 1 Apr 2014 14:08:45 +0100 Subject: [Biopython-dev] Fwd: [Numpy-discussion] EuroSciPy 2014 Call for Abstracts In-Reply-To: <20140401114620.67490302@lnx> References: <20140320114815.70210fd5@grandao> <87txathtux.fsf@chupacabra.windows.ebi.ac.uk> <20140401114620.67490302@lnx> Message-ID: On Tue, Apr 1, 2014 at 11:46 AM, Tiago Antao wrote: > Hi, > > On Mon, 31 Mar 2014 17:30:21 +0100 > Peter Cock wrote: > >> I'm glad to see there are several people interested in EuroSciPy. >> One one of you like to submit a Biopython talk? The deadline >> is 14 April: https://www.euroscipy.org/2014/calls/abstracts/ > > > I know I have fuelled all this, but I will only know next week if I am > in the US or not. I could offer to do the presentation, if I am around. > In any case, I still believe it might be a good geographical place for > some Biopython developers to meet. > > Brandon: We can talk face-to-face on Thursday, as we will be in the > same place ;) > > Tiago OK, let's sort this out next week then - this week is the BOSC deadline (4 April), and Bow has volunteered to present the Biopython talk. Thanks, Peter From p.j.a.cock at googlemail.com Tue Apr 1 21:37:45 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 1 Apr 2014 22:37:45 +0100 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: References: Message-ID: On Mon, Mar 31, 2014 at 10:08 PM, Wibowo Arindrarto wrote: > Hi Peter, everyone, > > A LaTeX template would be great :). I'm still preparing the abstract, > should be ready for everyone to check soon. > > Cheers, > Bow LaTeX template sent. I will be travelling the next few days and may be largely out of email contact - so if you don't hear from me, don't worry and submit the abstract anyway. I'm confident the others will have good advice and suggestions (and I'm not the only one who does email list moderation - but it might be simpler to send round plain text and a link to the PDF if you want to show it and not worry about the attachment triggering moderation). As usual this will get reviewed by the BOSC panel, and you'll have a chance to revise the abstract at that point - for the project updates is it OK if the initial text has a couple of details "to be confirmed", like a summary of the forthcoming release Biopython 1.64 (time for another thread, maybe we can aim for the end of this month?). Thanks Bow, Peter From clements at galaxyproject.org Wed Apr 2 20:06:56 2014 From: clements at galaxyproject.org (Dave Clements) Date: Wed, 2 Apr 2014 13:06:56 -0700 Subject: [Biopython-dev] Talk Abstracts Due April 4: 2014 Galaxy Community Conference (GCC2014) Message-ID: Hello all, This is a reminder that talk abstract submissions for the 2014 Galaxy Community Conference *are due this Friday, April 4*. See http://bit.ly/gcc2014abs for details. This is a great opportunity to share your work in high-throughput biology. Accepted abstracts will be considered for the *GigaScience * "Galaxy: Data Intensive and Reproducible Research " series (announced for GCC2013)*. * BGI is also continuing to cover the article processing charges until the end of the year Poster abstracts are being accepted until April 25. Early registration (save a bundle, really) is also open. Thanks, and we do hope you'll join us in Baltimore! Dave C, on behalf of the GCC2014 Organizing Committee On Sat, Feb 22, 2014 at 3:21 PM, Dave Clements wrote: > *2014 Galaxy Community Conference (GCC2014) > * > http://galaxyproject.org/GCC2014 > June 30 - July 2, 2014 > Homewood Campus > > Johns Hopkins University > Baltimore, Maryland , United > States > > ------ > The *2014 Galaxy Community Conference *(*GCC2014*, > http://galaxyproject.org/GCC2014) features two full days of > presentations, discussions, poster sessions, lightning talks and > birds-of-a-feather, *all about data-intensive biology and the tools that > support it*. GCC2014 also includes a Training Day with > five concurrent tracks and in-depth coverage of thirteen different topics. > > GCC2014 will be held at the Homewood > Campus > of Johns Hopkins University , in Baltimore, Maryland, > United States, from June 30 through July 2, 2014. > > Galaxy is an easily extensible data > integration and analysis platform for life sciences research that supports > hundreds of bioinformatics analysis tools. Galaxy is open-source and can be > locally installed or run on the cloud. There are hundreds of local > installs, and over 50 publicly accessible serversaround the world. > > *Early registration > * is now open. > Early combined registration (Training Day + > main meeting) starts at $140 for post-docs and students. Registration is > capped this year at 250 participants, *and we expect to hit that limit*. > Registering early assures you a place at the conference and also a spot in > the Training Day workshops you want to attend. > > You can also book affordable conference housing at the same time you > register. See the conference Logistics page for > details on this and other housing options. > > *Abstract submission > * for both oral > presentations and posters is also open. Abstract submission for oral > presentations closes April 4, and poster submission closes April 25. The *GigaScience > * "Galaxy: Data Intensive and > Reproducible Research " > series (announced for GCC2013) *is continuing to take submissions for > this year's meeting and beyond. * BGI is also continuing to cover the > article processing charges until the end of the year, and for more > information see their latest update > . > > Thanks, and hope to see you in Baltimore! > > The GCC2014 Organizing Committee > -- http://galaxyproject.org/GCC2014 http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ https://wiki.galaxyproject.org/ From w.arindrarto at gmail.com Wed Apr 2 21:54:21 2014 From: w.arindrarto at gmail.com (Wibowo Arindrarto) Date: Wed, 2 Apr 2014 23:54:21 +0200 Subject: [Biopython-dev] Fwd: [Open-bio-l] Proposed BLAST XML Changes In-Reply-To: References: Message-ID: Dear all, Regarding the update on querying SearchIO hits with multiple keys, I've just added some (not so) small changes to the submodule here: https://github.com/biopython/biopython/pull/307. The proposed changes affect our current SearchIO core object model, so I suppose it's better if more than one person take a look at this first :). Cheers, Bow On Tue, Mar 18, 2014 at 12:48 PM, Wibowo Arindrarto wrote: > On Tue, Mar 18, 2014 at 11:58 AM, Peter Cock wrote: >> On Tue, Mar 18, 2014 at 10:33 AM, Wibowo Arindrarto >> wrote: >>> On Tue, Mar 18, 2014 at 11:17 AM, Peter Cock wrote: >>>> On Tue, Mar 18, 2014 at 9:52 AM, Wibowo Arindrarto >>>> wrote: >>>>> Hi Peter, everyone, >>>>> >>>>> Thanks for the heads up. If implemented as it is, the updates will >>>>> change our underlying SearchIO model (aside from the blast-xml parser >>>>> itself), by allowing a Hit retrieval using multiple different keys. >>>> >>>> Could you clarify what you mean by multiple keys here? >>> >>> Currently, we can retrieve hits from a query using its ID, aside from >>> its numeric index. With their proposed changes to the Hit element >>> here: ftp://ftp.ncbi.nlm.nih.gov/blast/documents/NEWXML/ProposedBLASTXMLChanges.pdf, >>> it means that a given Hit can now be annotated with more than one ID. >> >> But this happens already in the current output from merged entries >> in databases like NR - we effectively use the first alternative ID as >> the hit ID. See for example the nasty > separated entries in >> the legacy BLAST XML's tag where only the first ID >> appears in the tag: >> >> http://blastedbio.blogspot.co.uk/2012/05/blast-tabular-missing-descriptions.html >> >> See also the new optional fields in the tabular output which >> explicitly list all the aliases for the merge record (e.g. sallseqid). > > In the BLAST outputs, yes. However, there's no explicit support yet in > SearchIO for this. Currently we only parse whatever is in as > the ID and as the description. If the tag has is > separated by semicolons / has more than one IDs, the current parser > does not try to split it into multiple IDs. Instead it takes the whole > string as the ID. > > Also, in the blast tabular format, even though sallseqid is parsed, > it's merely stored as an attribute of the hit object, not something > that can be used to retrieve Hits from the QueryResult object. > >>> Ideally, this should also be reflected in the QueryResult object: a >>> hit item should be retrievable using any of the IDs it has. >>> >>> This will also affect membership checking on the QueryResult object. >> >> This looks like something we should review anyway, regardless >> of the new BLAST XML format. > > Of course :). > >>>>> I have a feeling it will be difficult to jam all the new changes into >>>>> a backwards-compatible parser. One way to make it transparent to users >>>>> is to use the underlying DTD to do validation before parsing (for the >>>>> two BLAST DTDs, use the one which the file can be validated against). >>>>> However, this comes at a price. Since the standard library-bundled >>>>> elementtree doesn't seem to support validation, we have to use another >>>>> library (lxml is my choice). This means adding 3rd party dependency >>>>> which require compiling (lxml is also partly written in C). >>>> >>>> We can probably tell by sniffing the first few lines... but how >>>> to do that without using a handle seek to rewind may be >>>> tricky (desirable to support parsing streams, e.g. stdin). >>> >>> Ah yes. We have a rewindable file seek object in Bio.File, don't we >>> :)? I'll have to play around with some real datasets first, I think. >> >> Yes, the UndoHandle in Bio.File might be the best solution >> here for auto-detection. But two explicit formats is probably better. >> >>> The other thing we should take into account is the Xinclude tag. Would >>> we want to make it possible to query *either* the single query XML >>> results or the master Xinclude document (point 2 of the proposed >>> change)? Or should we restrict our parser only to the single query >>> files? >> >> I think single files is a reasonable restriction... assuming BLAST >> will still have the option of producing a big multi-query XML? >> Probably we should ask the NCBI about that... > > In a way, the Xinclude file is the file containing multi-query XML. I > have a feeling that if Xinclude is proposed, producing multi-output > BLAST XML files will not be an option anymore (otherwise it seems > redundant). But yes, NCBI should has more info about this. > >> I would hope the Bio.SearchIO.index_db(...) approach could >> be used on a colloection of little XML files, one for each query. >> >>>>> The other option is to introduce a new format name (e.g. >>>>> 'blast-xml2'), which makes the user responsible for knowing which >>>>> BLAST XML he/she is parsing. It feels more explicit this way, so I am >>>>> leaning towards this option, despite 'blast-xml2' not sounding very >>>>> nice to me ;). >>>>> >>>>> Any other thoughts? >>>>> >>>>> Best, >>>>> Bow >>>> >>>> I agree for the SearchIO interface, two format names makes >>>> sense - unless there is a neat way to auto-detect this on input. >>>> >>>> Using "blast-xml2" would work, or maybe something like >>>> "blast-xml-2014" (too long?). >>>> >>>> We could even go for "blast-xml-old" and "blast-xml" perhaps? >>> >>> Hmm..'blast-xml-old', may make it difficult to adapt for future XML >>> schema changes. How about renaming the current parser to >>> 'blast-xml-legacy', and the new one to just 'blast-xml'? >> >> A possible downside of 'blast-xml-legacy' over 'blast-xml-old' >> is this may be confused with the "legacy" BLAST in C to the >> current BLAST+ in C++ move (which happened well before >> this XML format change). > > Hmm. In this case then I am leaning to 'blast-xml2', I think. It's the > shortest and most future-proof (subsequent changes to the XML format > could be denoted as 'blast-xml3'). But it does make it slightly > inconsistent with the names we have for HMMER (i.e. 'hmmer2-text' is > for HMMER version 2 text output, 'hmmer3-text' is for HMMER version 3 > text output). > > Cheers, > Bow From w.arindrarto at gmail.com Thu Apr 3 07:06:29 2014 From: w.arindrarto at gmail.com (Wibowo Arindrarto) Date: Thu, 3 Apr 2014 09:06:29 +0200 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: References: Message-ID: Hi everyone, I'm forwarding the previous message to the mailing list now. I just realized that my message did not get through yesterday due to the attachment :(. Apologies for the quite late check, should've checked this earlier. I have now reuploaded our abstract here: http://bit.ly/biopython_bosc_2014. As noted below, please let me know if there are additions / deletions you would like me to make before the abstract is submitted. Best, Bow On Wed, Apr 2, 2014 at 11:52 AM, Wibowo Arindrarto wrote: > Hello everyone, > > I've finished the first draft of our update (attached). Please let me > know if there are additions / deletions you would like me to make > before I submit the abstract. > > Also, some things I would like to note: > > * I am missing the formal affiliations of everybody in the authors > list except for me and Peter. Could you please send me your most > recent affiliation? > > * I'm not sure about whether to include Tiago's recent Biopython > docker container (I think it would be nice). Tiago, do we have the > container up and running for our tests? > > Cheers, > Bow > > On Tue, Apr 1, 2014 at 11:37 PM, Peter Cock wrote: >> On Mon, Mar 31, 2014 at 10:08 PM, Wibowo Arindrarto >> wrote: >>> Hi Peter, everyone, >>> >>> A LaTeX template would be great :). I'm still preparing the abstract, >>> should be ready for everyone to check soon. >>> >>> Cheers, >>> Bow >> >> LaTeX template sent. >> >> I will be travelling the next few days and may be largely out of >> email contact - so if you don't hear from me, don't worry and >> submit the abstract anyway. I'm confident the others will have >> good advice and suggestions (and I'm not the only one who >> does email list moderation - but it might be simpler to send >> round plain text and a link to the PDF if you want to show it >> and not worry about the attachment triggering moderation). >> >> As usual this will get reviewed by the BOSC panel, and you'll >> have a chance to revise the abstract at that point - for the >> project updates is it OK if the initial text has a couple of details >> "to be confirmed", like a summary of the forthcoming release >> Biopython 1.64 (time for another thread, maybe we can aim >> for the end of this month?). >> >> Thanks Bow, >> >> Peter From eric.talevich at gmail.com Thu Apr 3 22:18:01 2014 From: eric.talevich at gmail.com (Eric Talevich) Date: Thu, 3 Apr 2014 15:18:01 -0700 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: References: Message-ID: Hi Bow, The abstract looks good. It's great to see you're carrying the torch this year. In the second paragraph, the sentence beginning "New features since version 1.62" confused me. Should this be "New features *in* version 1.62"? My recollection is that the listed features were actually introduced after 1.63, in which case that sentence would move to the end of the paragraph as "new features since version *1.63*". I'm looking at: http://news.open-bio.org/news/2013/08/biopython-1-62-released/ http://news.open-bio.org/news/2013/12/biopython-1-63-released/ Maybe you could streamline that paragraph to say that since BOSC 2013 Biopython has successfully supported Python 2 and Python 3 (and Pypy and Jython 2.7) with a single codebase, and leave the details of the transition for the talk. Then describe the other changes/enhancements. Also, in the same paragraph: "various enhancements" could be swapped for "other significant enhancements". Cheers, Eric On Thu, Apr 3, 2014 at 12:06 AM, Wibowo Arindrarto wrote: > Hi everyone, > > I'm forwarding the previous message to the mailing list now. I just > realized that my message did not get through yesterday due to the > attachment :(. Apologies for the quite late check, should've checked > this earlier. > > I have now reuploaded our abstract here: http://bit.ly/biopython_bosc_2014 > . > > As noted below, please let me know if there are additions / deletions > you would like me to make before the abstract is submitted. > > Best, > Bow > > On Wed, Apr 2, 2014 at 11:52 AM, Wibowo Arindrarto > wrote: > > Hello everyone, > > > > I've finished the first draft of our update (attached). Please let me > > know if there are additions / deletions you would like me to make > > before I submit the abstract. > > > > Also, some things I would like to note: > > > > * I am missing the formal affiliations of everybody in the authors > > list except for me and Peter. Could you please send me your most > > recent affiliation? > > > > * I'm not sure about whether to include Tiago's recent Biopython > > docker container (I think it would be nice). Tiago, do we have the > > container up and running for our tests? > > > > Cheers, > > Bow > > > > On Tue, Apr 1, 2014 at 11:37 PM, Peter Cock > wrote: > >> On Mon, Mar 31, 2014 at 10:08 PM, Wibowo Arindrarto > >> wrote: > >>> Hi Peter, everyone, > >>> > >>> A LaTeX template would be great :). I'm still preparing the abstract, > >>> should be ready for everyone to check soon. > >>> > >>> Cheers, > >>> Bow > >> > >> LaTeX template sent. > >> > >> I will be travelling the next few days and may be largely out of > >> email contact - so if you don't hear from me, don't worry and > >> submit the abstract anyway. I'm confident the others will have > >> good advice and suggestions (and I'm not the only one who > >> does email list moderation - but it might be simpler to send > >> round plain text and a link to the PDF if you want to show it > >> and not worry about the attachment triggering moderation). > >> > >> As usual this will get reviewed by the BOSC panel, and you'll > >> have a chance to revise the abstract at that point - for the > >> project updates is it OK if the initial text has a couple of details > >> "to be confirmed", like a summary of the forthcoming release > >> Biopython 1.64 (time for another thread, maybe we can aim > >> for the end of this month?). > >> > >> Thanks Bow, > >> > >> Peter > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev > From w.arindrarto at gmail.com Fri Apr 4 00:57:07 2014 From: w.arindrarto at gmail.com (Wibowo Arindrarto) Date: Fri, 4 Apr 2014 02:57:07 +0200 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: References: Message-ID: Hi Eric, Thanks for the feedback. I have updated the abstract and you can access it here: http://bit.ly/biopython_bosc_2014_02. In addition to the changes you suggested, I added Michiel in the authors list as well. May I also ask what I should put as your affiliation :)? I think I only have mine and Peter's at the moment, and I'm not sure if I could get the most recent affiliations by scouring the web ;). Best, Bow On Fri, Apr 4, 2014 at 12:18 AM, Eric Talevich wrote: > Hi Bow, > > The abstract looks good. It's great to see you're carrying the torch this > year. > > In the second paragraph, the sentence beginning "New features since version > 1.62" confused me. Should this be "New features *in* version 1.62"? My > recollection is that the listed features were actually introduced after > 1.63, in which case that sentence would move to the end of the paragraph as > "new features since version *1.63*". I'm looking at: > http://news.open-bio.org/news/2013/08/biopython-1-62-released/ > http://news.open-bio.org/news/2013/12/biopython-1-63-released/ > > Maybe you could streamline that paragraph to say that since BOSC 2013 > Biopython has successfully supported Python 2 and Python 3 (and Pypy and > Jython 2.7) with a single codebase, and leave the details of the transition > for the talk. Then describe the other changes/enhancements. Also, in the > same paragraph: "various enhancements" could be swapped for "other > significant enhancements". > > Cheers, > Eric > > > On Thu, Apr 3, 2014 at 12:06 AM, Wibowo Arindrarto > wrote: >> >> Hi everyone, >> >> I'm forwarding the previous message to the mailing list now. I just >> realized that my message did not get through yesterday due to the >> attachment :(. Apologies for the quite late check, should've checked >> this earlier. >> >> I have now reuploaded our abstract here: >> http://bit.ly/biopython_bosc_2014. >> >> As noted below, please let me know if there are additions / deletions >> you would like me to make before the abstract is submitted. >> >> Best, >> Bow >> >> On Wed, Apr 2, 2014 at 11:52 AM, Wibowo Arindrarto >> wrote: >> > Hello everyone, >> > >> > I've finished the first draft of our update (attached). Please let me >> > know if there are additions / deletions you would like me to make >> > before I submit the abstract. >> > >> > Also, some things I would like to note: >> > >> > * I am missing the formal affiliations of everybody in the authors >> > list except for me and Peter. Could you please send me your most >> > recent affiliation? >> > >> > * I'm not sure about whether to include Tiago's recent Biopython >> > docker container (I think it would be nice). Tiago, do we have the >> > container up and running for our tests? >> > >> > Cheers, >> > Bow >> > >> > On Tue, Apr 1, 2014 at 11:37 PM, Peter Cock >> > wrote: >> >> On Mon, Mar 31, 2014 at 10:08 PM, Wibowo Arindrarto >> >> wrote: >> >>> Hi Peter, everyone, >> >>> >> >>> A LaTeX template would be great :). I'm still preparing the abstract, >> >>> should be ready for everyone to check soon. >> >>> >> >>> Cheers, >> >>> Bow >> >> >> >> LaTeX template sent. >> >> >> >> I will be travelling the next few days and may be largely out of >> >> email contact - so if you don't hear from me, don't worry and >> >> submit the abstract anyway. I'm confident the others will have >> >> good advice and suggestions (and I'm not the only one who >> >> does email list moderation - but it might be simpler to send >> >> round plain text and a link to the PDF if you want to show it >> >> and not worry about the attachment triggering moderation). >> >> >> >> As usual this will get reviewed by the BOSC panel, and you'll >> >> have a chance to revise the abstract at that point - for the >> >> project updates is it OK if the initial text has a couple of details >> >> "to be confirmed", like a summary of the forthcoming release >> >> Biopython 1.64 (time for another thread, maybe we can aim >> >> for the end of this month?). >> >> >> >> Thanks Bow, >> >> >> >> Peter >> _______________________________________________ >> Biopython-dev mailing list >> Biopython-dev at lists.open-bio.org >> http://lists.open-bio.org/mailman/listinfo/biopython-dev > > From w.arindrarto at gmail.com Fri Apr 4 16:05:36 2014 From: w.arindrarto at gmail.com (Wibowo Arindrarto) Date: Fri, 4 Apr 2014 18:05:36 +0200 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: References: Message-ID: Hello everyone, Our abstract have been submitted. You can access the submitted version here: http://bit.ly/biopython_bosc_2014_submitted. There are still some parts we need to update during the abstract review (hopefully including a newly released 1.64 :) ), but they are minimum. Have a nice weekend, Bow On Fri, Apr 4, 2014 at 2:57 AM, Wibowo Arindrarto wrote: > Hi Eric, > > Thanks for the feedback. I have updated the abstract and you can > access it here: http://bit.ly/biopython_bosc_2014_02. In addition to > the changes you suggested, I added Michiel in the authors list as > well. > > May I also ask what I should put as your affiliation :)? I think I > only have mine and Peter's at the moment, and I'm not sure if I could > get the most recent affiliations by scouring the web ;). > > Best, > Bow > > On Fri, Apr 4, 2014 at 12:18 AM, Eric Talevich wrote: >> Hi Bow, >> >> The abstract looks good. It's great to see you're carrying the torch this >> year. >> >> In the second paragraph, the sentence beginning "New features since version >> 1.62" confused me. Should this be "New features *in* version 1.62"? My >> recollection is that the listed features were actually introduced after >> 1.63, in which case that sentence would move to the end of the paragraph as >> "new features since version *1.63*". I'm looking at: >> http://news.open-bio.org/news/2013/08/biopython-1-62-released/ >> http://news.open-bio.org/news/2013/12/biopython-1-63-released/ >> >> Maybe you could streamline that paragraph to say that since BOSC 2013 >> Biopython has successfully supported Python 2 and Python 3 (and Pypy and >> Jython 2.7) with a single codebase, and leave the details of the transition >> for the talk. Then describe the other changes/enhancements. Also, in the >> same paragraph: "various enhancements" could be swapped for "other >> significant enhancements". >> >> Cheers, >> Eric >> >> >> On Thu, Apr 3, 2014 at 12:06 AM, Wibowo Arindrarto >> wrote: >>> >>> Hi everyone, >>> >>> I'm forwarding the previous message to the mailing list now. I just >>> realized that my message did not get through yesterday due to the >>> attachment :(. Apologies for the quite late check, should've checked >>> this earlier. >>> >>> I have now reuploaded our abstract here: >>> http://bit.ly/biopython_bosc_2014. >>> >>> As noted below, please let me know if there are additions / deletions >>> you would like me to make before the abstract is submitted. >>> >>> Best, >>> Bow >>> >>> On Wed, Apr 2, 2014 at 11:52 AM, Wibowo Arindrarto >>> wrote: >>> > Hello everyone, >>> > >>> > I've finished the first draft of our update (attached). Please let me >>> > know if there are additions / deletions you would like me to make >>> > before I submit the abstract. >>> > >>> > Also, some things I would like to note: >>> > >>> > * I am missing the formal affiliations of everybody in the authors >>> > list except for me and Peter. Could you please send me your most >>> > recent affiliation? >>> > >>> > * I'm not sure about whether to include Tiago's recent Biopython >>> > docker container (I think it would be nice). Tiago, do we have the >>> > container up and running for our tests? >>> > >>> > Cheers, >>> > Bow >>> > >>> > On Tue, Apr 1, 2014 at 11:37 PM, Peter Cock >>> > wrote: >>> >> On Mon, Mar 31, 2014 at 10:08 PM, Wibowo Arindrarto >>> >> wrote: >>> >>> Hi Peter, everyone, >>> >>> >>> >>> A LaTeX template would be great :). I'm still preparing the abstract, >>> >>> should be ready for everyone to check soon. >>> >>> >>> >>> Cheers, >>> >>> Bow >>> >> >>> >> LaTeX template sent. >>> >> >>> >> I will be travelling the next few days and may be largely out of >>> >> email contact - so if you don't hear from me, don't worry and >>> >> submit the abstract anyway. I'm confident the others will have >>> >> good advice and suggestions (and I'm not the only one who >>> >> does email list moderation - but it might be simpler to send >>> >> round plain text and a link to the PDF if you want to show it >>> >> and not worry about the attachment triggering moderation). >>> >> >>> >> As usual this will get reviewed by the BOSC panel, and you'll >>> >> have a chance to revise the abstract at that point - for the >>> >> project updates is it OK if the initial text has a couple of details >>> >> "to be confirmed", like a summary of the forthcoming release >>> >> Biopython 1.64 (time for another thread, maybe we can aim >>> >> for the end of this month?). >>> >> >>> >> Thanks Bow, >>> >> >>> >> Peter >>> _______________________________________________ >>> Biopython-dev mailing list >>> Biopython-dev at lists.open-bio.org >>> http://lists.open-bio.org/mailman/listinfo/biopython-dev >> >> From w.arindrarto at gmail.com Wed Apr 2 09:52:07 2014 From: w.arindrarto at gmail.com (Wibowo Arindrarto) Date: Wed, 2 Apr 2014 11:52:07 +0200 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: References: Message-ID: Hello everyone, I've finished the first draft of our update (attached). Please let me know if there are additions / deletions you would like me to make before I submit the abstract. Also, some things I would like to note: * I am missing the formal affiliations of everybody in the authors list except for me and Peter. Could you please send me your most recent affiliation? * I'm not sure about whether to include Tiago's recent Biopython docker container (I think it would be nice). Tiago, do we have the container up and running for our tests? Cheers, Bow On Tue, Apr 1, 2014 at 11:37 PM, Peter Cock wrote: > On Mon, Mar 31, 2014 at 10:08 PM, Wibowo Arindrarto > wrote: >> Hi Peter, everyone, >> >> A LaTeX template would be great :). I'm still preparing the abstract, >> should be ready for everyone to check soon. >> >> Cheers, >> Bow > > LaTeX template sent. > > I will be travelling the next few days and may be largely out of > email contact - so if you don't hear from me, don't worry and > submit the abstract anyway. I'm confident the others will have > good advice and suggestions (and I'm not the only one who > does email list moderation - but it might be simpler to send > round plain text and a link to the PDF if you want to show it > and not worry about the attachment triggering moderation). > > As usual this will get reviewed by the BOSC panel, and you'll > have a chance to revise the abstract at that point - for the > project updates is it OK if the initial text has a couple of details > "to be confirmed", like a summary of the forthcoming release > Biopython 1.64 (time for another thread, maybe we can aim > for the end of this month?). > > Thanks Bow, > > Peter -------------- next part -------------- A non-text attachment was scrubbed... Name: Biopython_BOSC_abstract_2014.pdf Type: application/pdf Size: 258281 bytes Desc: not available URL: From tra at popgen.net Mon Apr 7 10:41:51 2014 From: tra at popgen.net (Tiago Antao) Date: Mon, 7 Apr 2014 11:41:51 +0100 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: References: Message-ID: <20140407114151.23947d6b@lnx> Hi, 1. Shouldn't Peter be the first author on the list? 2. We have 2 containers: 1. a container with all (almost) the dependencies, which can be used to deploy biopython on the fly (for normal usage): https://github.com/tiagoantao/my-containers/blob/master/biopython/Biopython 2. A buildbot test container (that is in used for builbot testing): https://github.com/tiagoantao/my-containers/blob/master/biopython/Biopython-Test 3. My address: Vector Biology Department, Liverpool School of Tropical Medicine, Pembroke Place, Liverpool, United Kingdom On Wed, 2 Apr 2014 11:52:07 +0200 Wibowo Arindrarto wrote: > Hello everyone, > > I've finished the first draft of our update (attached). Please let me > know if there are additions / deletions you would like me to make > before I submit the abstract. > > Also, some things I would like to note: > > * I am missing the formal affiliations of everybody in the authors > list except for me and Peter. Could you please send me your most > recent affiliation? > > * I'm not sure about whether to include Tiago's recent Biopython > docker container (I think it would be nice). Tiago, do we have the > container up and running for our tests? > > Cheers, > Bow > > On Tue, Apr 1, 2014 at 11:37 PM, Peter Cock > wrote: > > On Mon, Mar 31, 2014 at 10:08 PM, Wibowo Arindrarto > > wrote: > >> Hi Peter, everyone, > >> > >> A LaTeX template would be great :). I'm still preparing the > >> abstract, should be ready for everyone to check soon. > >> > >> Cheers, > >> Bow > > > > LaTeX template sent. > > > > I will be travelling the next few days and may be largely out of > > email contact - so if you don't hear from me, don't worry and > > submit the abstract anyway. I'm confident the others will have > > good advice and suggestions (and I'm not the only one who > > does email list moderation - but it might be simpler to send > > round plain text and a link to the PDF if you want to show it > > and not worry about the attachment triggering moderation). > > > > As usual this will get reviewed by the BOSC panel, and you'll > > have a chance to revise the abstract at that point - for the > > project updates is it OK if the initial text has a couple of details > > "to be confirmed", like a summary of the forthcoming release > > Biopython 1.64 (time for another thread, maybe we can aim > > for the end of this month?). > > > > Thanks Bow, > > > > Peter From w.arindrarto at gmail.com Mon Apr 7 11:00:42 2014 From: w.arindrarto at gmail.com (Wibowo Arindrarto) Date: Mon, 7 Apr 2014 13:00:42 +0200 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: <20140407114151.23947d6b@lnx> References: <20140407114151.23947d6b@lnx> Message-ID: Hi Tiago, 1. I have no objections to that. Could we change it post submission? If you noticed, I listed the authors initially alphabetically based on last name. After talking with Eric, I updated the order with the last one submitted. But in general, I'm ok with any ordering so long as we all agree. 2. I didn't have chance to add this prior to submission, unfortunately :/. But adding this would be nice. Is it possible to do this during abstract review? 3. Noted, thanks :). Best, Bow On Mon, Apr 7, 2014 at 12:41 PM, Tiago Antao wrote: > Hi, > > 1. Shouldn't Peter be the first author on the list? > > 2. We have 2 containers: > 1. a container with all (almost) the dependencies, which can be used > to deploy biopython on the fly (for normal usage): > https://github.com/tiagoantao/my-containers/blob/master/biopython/Biopython > 2. A buildbot test container (that is in used for builbot testing): > https://github.com/tiagoantao/my-containers/blob/master/biopython/Biopython-Test > > 3. My address: Vector Biology Department, Liverpool School of Tropical > Medicine, Pembroke Place, Liverpool, United Kingdom > > On Wed, 2 Apr 2014 11:52:07 +0200 > Wibowo Arindrarto wrote: > >> Hello everyone, >> >> I've finished the first draft of our update (attached). Please let me >> know if there are additions / deletions you would like me to make >> before I submit the abstract. >> >> Also, some things I would like to note: >> >> * I am missing the formal affiliations of everybody in the authors >> list except for me and Peter. Could you please send me your most >> recent affiliation? >> >> * I'm not sure about whether to include Tiago's recent Biopython >> docker container (I think it would be nice). Tiago, do we have the >> container up and running for our tests? >> >> Cheers, >> Bow >> >> On Tue, Apr 1, 2014 at 11:37 PM, Peter Cock >> wrote: >> > On Mon, Mar 31, 2014 at 10:08 PM, Wibowo Arindrarto >> > wrote: >> >> Hi Peter, everyone, >> >> >> >> A LaTeX template would be great :). I'm still preparing the >> >> abstract, should be ready for everyone to check soon. >> >> >> >> Cheers, >> >> Bow >> > >> > LaTeX template sent. >> > >> > I will be travelling the next few days and may be largely out of >> > email contact - so if you don't hear from me, don't worry and >> > submit the abstract anyway. I'm confident the others will have >> > good advice and suggestions (and I'm not the only one who >> > does email list moderation - but it might be simpler to send >> > round plain text and a link to the PDF if you want to show it >> > and not worry about the attachment triggering moderation). >> > >> > As usual this will get reviewed by the BOSC panel, and you'll >> > have a chance to revise the abstract at that point - for the >> > project updates is it OK if the initial text has a couple of details >> > "to be confirmed", like a summary of the forthcoming release >> > Biopython 1.64 (time for another thread, maybe we can aim >> > for the end of this month?). >> > >> > Thanks Bow, >> > >> > Peter > From p.j.a.cock at googlemail.com Mon Apr 7 12:48:03 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 7 Apr 2014 13:48:03 +0100 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: References: <20140407114151.23947d6b@lnx> Message-ID: Hi all, Thanks for handling this Bow, and I'm sorry I didn't see the emails while I was away - thank you to Eric, Tiago and anyone else who had commented directly. Regarding (1), authorship: Bow is presenting, so I think he should be first author and corresponding author. But Tiago has raised an important point: Past BOSC talk authorship has been largely unplanned - often just named presenter and anyone directly contributing to the talk slides. Last year I'd just put Peter Cock et al. which was not very explicit. We could and perhaps should use something like "W. Arindrarato, ... and the Biopython Consortium" or "Team" or "Contributors"? (The phrase "Biopython Consortium" is what it says on our PyPy page, but it sounds very formal to me.) Regarding (2), yes - assuming the reviewers don't reject the abstract completely (which would be a shock!), we will have a chance to revise the abstract. This is usually to address reviewers' comments, but for the project update talks it helps ensure that the abstract is as up to date as possible. Regards, Peter On Mon, Apr 7, 2014 at 12:00 PM, Wibowo Arindrarto wrote: > Hi Tiago, > > 1. I have no objections to that. Could we change it post submission? > If you noticed, I listed the authors initially alphabetically based on > last name. After talking with Eric, I updated the order with the last > one submitted. But in general, I'm ok with any ordering so long as we > all agree. > > 2. I didn't have chance to add this prior to submission, unfortunately > :/. But adding this would be nice. Is it possible to do this during > abstract review? > > 3. Noted, thanks :). > > Best, > Bow > > On Mon, Apr 7, 2014 at 12:41 PM, Tiago Antao wrote: >> Hi, >> >> 1. Shouldn't Peter be the first author on the list? >> >> 2. We have 2 containers: >> 1. a container with all (almost) the dependencies, which can be used >> to deploy biopython on the fly (for normal usage): >> https://github.com/tiagoantao/my-containers/blob/master/biopython/Biopython >> 2. A buildbot test container (that is in used for builbot testing): >> https://github.com/tiagoantao/my-containers/blob/master/biopython/Biopython-Test >> >> 3. My address: Vector Biology Department, Liverpool School of Tropical >> Medicine, Pembroke Place, Liverpool, United Kingdom >> >> On Wed, 2 Apr 2014 11:52:07 +0200 >> Wibowo Arindrarto wrote: >> >>> Hello everyone, >>> >>> I've finished the first draft of our update (attached). Please let me >>> know if there are additions / deletions you would like me to make >>> before I submit the abstract. >>> >>> Also, some things I would like to note: >>> >>> * I am missing the formal affiliations of everybody in the authors >>> list except for me and Peter. Could you please send me your most >>> recent affiliation? >>> >>> * I'm not sure about whether to include Tiago's recent Biopython >>> docker container (I think it would be nice). Tiago, do we have the >>> container up and running for our tests? >>> >>> Cheers, >>> Bow >>> >>> On Tue, Apr 1, 2014 at 11:37 PM, Peter Cock >>> wrote: >>> > On Mon, Mar 31, 2014 at 10:08 PM, Wibowo Arindrarto >>> > wrote: >>> >> Hi Peter, everyone, >>> >> >>> >> A LaTeX template would be great :). I'm still preparing the >>> >> abstract, should be ready for everyone to check soon. >>> >> >>> >> Cheers, >>> >> Bow >>> > >>> > LaTeX template sent. >>> > >>> > I will be travelling the next few days and may be largely out of >>> > email contact - so if you don't hear from me, don't worry and >>> > submit the abstract anyway. I'm confident the others will have >>> > good advice and suggestions (and I'm not the only one who >>> > does email list moderation - but it might be simpler to send >>> > round plain text and a link to the PDF if you want to show it >>> > and not worry about the attachment triggering moderation). >>> > >>> > As usual this will get reviewed by the BOSC panel, and you'll >>> > have a chance to revise the abstract at that point - for the >>> > project updates is it OK if the initial text has a couple of details >>> > "to be confirmed", like a summary of the forthcoming release >>> > Biopython 1.64 (time for another thread, maybe we can aim >>> > for the end of this month?). >>> > >>> > Thanks Bow, >>> > >>> > Peter >> From mjldehoon at yahoo.com Wed Apr 9 02:35:54 2014 From: mjldehoon at yahoo.com (Michiel de Hoon) Date: Tue, 8 Apr 2014 19:35:54 -0700 (PDT) Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: Message-ID: <1397010954.15323.YahooMailBasic@web164006.mail.gq1.yahoo.com> -------------------------------------------- On Mon, 4/7/14, Peter Cock wrote: > We could and perhaps should use something > like "W. Arindrarato, ... and the Biopython Consortium" This sounds good to me. Perhaps it is a bit formal, but it also implies that we are serious about Biopython. I would suggest to write consortium without a capital letter though, since it is not an official name. Best, -Michiel. From w.arindrarto at gmail.com Wed Apr 9 08:46:36 2014 From: w.arindrarto at gmail.com (Wibowo Arindrarto) Date: Wed, 9 Apr 2014 10:46:36 +0200 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: References: <20140407114151.23947d6b@lnx> Message-ID: Hi Peter, everyone, > Thanks for handling this Bow, and I'm sorry I didn't see the emails > while I was away - thank you to Eric, Tiago and anyone else who > had commented directly. Happy to help :). > Regarding (1), authorship: Bow is presenting, so I think he should > be first author and corresponding author. But Tiago has raised > an important point: > > Past BOSC talk authorship has been largely unplanned - often > just named presenter and anyone directly contributing to the > talk slides. Last year I'd just put Peter Cock et al. which was not > very explicit. We could and perhaps should use something > like "W. Arindrarato, ... and the Biopython Consortium" or > "Team" or "Contributors"? > > (The phrase "Biopython Consortium" is what it says on our > PyPy page, but it sounds very formal to me.) I have no strong preference for this, but "Biopython developers" seems more appealing and appropriate than "Biopython consortium" I think. If not developers, perhaps "Biopython contributors" is ok. > Regarding (2), yes - assuming the reviewers don't reject the > abstract completely (which would be a shock!), we will have > a chance to revise the abstract. This is usually to address > reviewers' comments, but for the project update talks it helps > ensure that the abstract is as up to date as possible. Ok, I'll post the latest updates on our abstract along the way then. Having it up to date makes sense ~ which reminds me of something Peter mentioned some emails ago: do we have date target on releasing 1.64?. Cheers, Bow From tra at popgen.net Wed Apr 9 08:56:52 2014 From: tra at popgen.net (Tiago Antao) Date: Wed, 9 Apr 2014 09:56:52 +0100 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: References: <20140407114151.23947d6b@lnx> Message-ID: <20140409095652.13ee5258@lnx> On Wed, 9 Apr 2014 10:46:36 +0200 Wibowo Arindrarto wrote: > I have no strong preference for this, but "Biopython developers" > seems more appealing and appropriate than "Biopython consortium" I > think. If not developers, > perhaps "Biopython contributors" is ok. Another opinion that consortium is over-formal. Developers or contributors sounds more appealing for me... From p.j.a.cock at googlemail.com Wed Apr 9 09:02:41 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 9 Apr 2014 10:02:41 +0100 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: <20140409095652.13ee5258@lnx> References: <20140407114151.23947d6b@lnx> <20140409095652.13ee5258@lnx> Message-ID: On Wed, Apr 9, 2014 at 9:56 AM, Tiago Antao wrote: > On Wed, 9 Apr 2014 10:46:36 +0200 > Wibowo Arindrarto wrote: > >> I have no strong preference for this, but "Biopython developers" >> seems more appealing and appropriate than "Biopython consortium" I >> think. If not developers, perhaps "Biopython contributors" is ok. > > Another opinion that consortium is over-formal. Developers > or contributors sounds more appealing for me... > I agree Biopython Consortium is too formal - should we change the name on PyPI as well? Time for a little vote? (a) The Biopython Consortium (as on PyPI now) (b) (The) Biopython consortium (lower case c) (c) (The) Biopython developers (suggestive of a core team) (d) (The) Biopython contributors (even more inclusive) Or any other ideas? Biopython Team? Or getting to informal, Biopythoneers or Biopythonistas? ;) Regards, Peter From tra at popgen.net Wed Apr 9 09:09:31 2014 From: tra at popgen.net (Tiago Antao) Date: Wed, 9 Apr 2014 10:09:31 +0100 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: References: <20140407114151.23947d6b@lnx> <20140409095652.13ee5258@lnx> Message-ID: <20140409100931.1fd685b8@lnx> On Wed, 9 Apr 2014 10:02:41 +0100 Peter Cock wrote: > (a) The Biopython Consortium (as on PyPI now) > (b) (The) Biopython consortium (lower case c) > (c) (The) Biopython developers (suggestive of a core team) > (d) (The) Biopython contributors (even more inclusive) My vote: d>c>>b=a The reason I sligthly prefer contributors to developers is to suggest that there are many ways to help that are not development (e.g. documentation writing). From christian at brueffer.de Wed Apr 9 09:52:30 2014 From: christian at brueffer.de (Christian Brueffer) Date: Wed, 09 Apr 2014 11:52:30 +0200 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: <20140409100931.1fd685b8@lnx> References: <20140407114151.23947d6b@lnx> <20140409095652.13ee5258@lnx> <20140409100931.1fd685b8@lnx> Message-ID: <5345185E.6060102@brueffer.de> On 4/9/14 11:09 AM, Tiago Antao wrote: > On Wed, 9 Apr 2014 10:02:41 +0100 > Peter Cock wrote: > >> (a) The Biopython Consortium (as on PyPI now) >> (b) (The) Biopython consortium (lower case c) >> (c) (The) Biopython developers (suggestive of a core team) >> (d) (The) Biopython contributors (even more inclusive) > > My vote: > d>c>>b=a > > The reason I sligthly prefer contributors to developers is to suggest > that there are many ways to help that are not development (e.g. > documentation writing). +1 Chris From anaryin at gmail.com Thu Apr 10 07:36:27 2014 From: anaryin at gmail.com (=?UTF-8?Q?Jo=C3=A3o_Rodrigues?=) Date: Thu, 10 Apr 2014 09:36:27 +0200 Subject: [Biopython-dev] Biopython update talk at BOSC 2014 In-Reply-To: <5345185E.6060102@brueffer.de> References: <20140407114151.23947d6b@lnx> <20140409095652.13ee5258@lnx> <20140409100931.1fd685b8@lnx> <5345185E.6060102@brueffer.de> Message-ID: +1 too for contributors. Sorry for the time it took to answer, but I was on a trip. Bow, thanks for the abstract and volunteering, looks great. Jo?o 2014-04-09 11:52 GMT+02:00 Christian Brueffer : > On 4/9/14 11:09 AM, Tiago Antao wrote: > > On Wed, 9 Apr 2014 10:02:41 +0100 > > Peter Cock wrote: > > > >> (a) The Biopython Consortium (as on PyPI now) > >> (b) (The) Biopython consortium (lower case c) > >> (c) (The) Biopython developers (suggestive of a core team) > >> (d) (The) Biopython contributors (even more inclusive) > > > > My vote: > > d>c>>b=a > > > > The reason I sligthly prefer contributors to developers is to suggest > > that there are many ways to help that are not development (e.g. > > documentation writing). > > +1 > > Chris > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev > From arklenna at gmail.com Sun Apr 13 20:16:13 2014 From: arklenna at gmail.com (Lenna Peterson) Date: Sun, 13 Apr 2014 16:16:13 -0400 Subject: [Biopython-dev] Coordinate Mapper pull request Message-ID: Hi all, I've finally submitted a pull request for my coordinate mapper (many thanks to Reece Hart for the original code). An example of its usage is on the cookbook: http://biopython.org/wiki/Coordinate_mapping The example code can also be found in this gist: https://gist.github.com/lennax/10600113 Comments are welcome on the list or the pull request: https://github.com/biopython/biopython/pull/309 Cheers, Lenna From p.j.a.cock at googlemail.com Mon Apr 14 09:20:25 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 14 Apr 2014 10:20:25 +0100 Subject: [Biopython-dev] interest module for sequence based clustering tools? In-Reply-To: References: Message-ID: Hi Krishna, This looks interesting, although it is not something I can remember doing myself with Python before. Do you think the framework would be general enough to cover other sequence clustering tools like UCLUST http://drive5.com/usearch/manual/uclust_algo.html or OrthoMCL? Peter On Tue, Apr 1, 2014 at 7:41 AM, Krishna Roskin wrote: > Hey all, > > Long time fan, taking my first crack at contributing. > > I've built a basic module to run and parse the result of sequence based > clustering tools such as DNACLUST and CD-HIT. I've written subclasses of > AbstractCommandline to run dnaclust and cd-hit. I've also written classes > to store the clusters and their members and loaders for the output formats > used by those programs. > > I posting here to gauge interest and get some feedback and maybe some beta > testers. > > My code is available at: > > https://github.com/krishnaroskin/biopython.git > > under the seqcluster branch. I've started writing some test code at: > > Tests/seqcluster/test_seqcluster.py > > that also severs as example code. I've pasted that at the end of this > message so people can get an idea of how it works without having to > checkout code. > > If there is interest, I'm planning on adding a seqclust.cluster function > that takes a list of SeqRecords and returns a clustering using one of the > supported tools. I envision that function being the main interface to this > module. I also want to write a something that will map the cluster > membership (given by ids) back to collections of SeqRecords. > > Other to-dos: > > Test that all the flavors of CD-HIT work (there are many) > Add support for other sequence based clustering tools (suggestions?) > Documentation > Tutorial > Test code > > -krish > > #!/usr/bin/env python > > > from __future__ import print_function > > > import StringIO > > > from Bio.seqcluster.applications import DNAClustCommandline > > from Bio.seqcluster import DNAClustIterator > > > from Bio.seqcluster.applications import CDHITCommandline > > from Bio.seqcluster import CDHITClustIterator > > > cmd = DNAClustCommandline(similarity=0.8, header=True, threads=2, inputfile= > "test_sequences.fasta") > > stdout, stderr = cmd() > > clusters = DNAClustIterator(StringIO.StringIO(stdout)) > > for cluster in clusters: > > print(cluster.name) > > for member in cluster: > > if member == cluster.representative: > > print("\t" + member.name + "*") > > else: > > print("\t" + member.name) > > > print() # blank line > > > cmd = CDHITCommandline(cutoff=0.8, threads=2, inputfile= > "test_sequences.fasta", outputfile="tmp") > > stdout, stderr = cmd() > > clusters = CDHITClustIterator(open("tmp.clstr", "r")) > > for cluster in clusters: > > print(cluster.name) > > for member in cluster: > > if member == cluster.representative: > > print("\t" + member.name + "*") > > else: > > print("\t" + member.name) > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev From p.j.a.cock at googlemail.com Mon Apr 14 11:36:23 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 14 Apr 2014 12:36:23 +0100 Subject: [Biopython-dev] Problem on Mac OS X since update Xcode 5.1 In-Reply-To: References: Message-ID: On Sun, Mar 16, 2014 at 9:15 PM, Peter Cock wrote: > On Sun, Mar 16, 2014 at 8:53 PM, Lenna Peterson wrote: >> On Wed, Mar 12, 2014 at 5:59 AM, Peter Cock >> wrote: >>> >>> ... >>> >>> This looks relevant, especially this reply from Paul Kehrer which >>> suggests this is entirely Apple's fault for shipping a Python and >>> clang compiler which don't get along with the default settings: >>> >>> >>> http://stackoverflow.com/questions/22313407/clang-error-unknown-argument-mno-fused-madd-psycopg2-installation-failure >>> >> >>> The suggested workaround seems to do the trick, >>> >>> $ export CFLAGS=-Qunused-arguments >>> $ export CPPFLAGS=-Qunused-arguments >>> >> >> I encountered the same problem (clean install of Mavericks, vanilla Python, >> latest XCode from App Store). >> >> One answer [1] suggests this is not a guaranteed solution but offers a >> different flag (which I did not test). >> >> I chose to edit system python files [2] which is definitely not the best >> option for most users. >> >> [1]: http://stackoverflow.com/a/22315129 >> [2]: http://stackoverflow.com/a/22322068 >> >>> Perhaps we can add this hack to our setup.py on Mac OS X... >>> it seems harmless under gcc (e.g. my locally compiled version >>> of Python 3.3 used gcc rather than clang)? >> >> Do you mean editing environment variables with `os.environ`? I don't know >> enough about the details of how packages are built to know what will work >> with both compiling from source, easy_install, pip, etc. > > Yes, I was thinking about editing the environment variables in > setup.py via the os module. I agree there are potential risks > with 3rd party installers, but adding -Qunused-arguments to > any existing CFLAGS (within the scope of the Biopython install) > is hopefully low risk... > >>> Or it could be done via the buildbot setup, or on this buildslave >>> directly (e.g. the ~/.bash_profile). >> >> It's a dilemma, because asking users to edit their .bashrc or .bash_profile >> before installation is annoying and easy to overlook, but modifying them in >> setup.py feels hacky (i.e. how long will this solution work?). Crossing my >> fingers and hoping Apple fixes this in an update... >> > > Fingers crossed Apple pushes another update in the next few > weeks to resolve this... > > Peter I just retested with XCode 5.1.1 (released last week and updated via the App Store), and the problem persists. $ cc -v Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn) Target: x86_64-apple-darwin13.1.0 Thread model: posix $ which python-config /Library/Frameworks/Python.framework/Versions/Current/bin/python-config $ python-config --cflags -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 i.e. Apple's current Python still says to use C compliler options which Apple's current clang compiler rejects :( The good news is the environment variable trick still works. It appears Apple are not going to fix this, so perhaps we do need to add a hack to our setup.py file? Peter From krishnaroskin at gmail.com Mon Apr 14 17:00:32 2014 From: krishnaroskin at gmail.com (Krishna Roskin) Date: Mon, 14 Apr 2014 10:00:32 -0700 Subject: [Biopython-dev] interest module for sequence based clustering tools? In-Reply-To: References: Message-ID: Peter, It should be general enough. UCLUST was on my list of programs to support but I back-burnered it because it's not free software/open source. But I'll take a look and checkout OrthoMCL as well. I've since added tools that take an iterable of SeqRecords and returns a clustering. Writing to a temporary FASTA file, running the program, and parsing the results are all taken care of. I'm also working on some documentation. I'm planning on having examples of doing sequence deduplication, building motifs for each cluster, and recalculating cluster medoids using multiple sequence alignments. Any other suggestions are more than welcome. -krish On Mon, Apr 14, 2014 at 2:20 AM, Peter Cock wrote: > Hi Krishna, > > This looks interesting, although it is not something I can remember > doing myself with Python before. Do you think the framework would > be general enough to cover other sequence clustering tools like > UCLUST http://drive5.com/usearch/manual/uclust_algo.html > or OrthoMCL? > > Peter > > On Tue, Apr 1, 2014 at 7:41 AM, Krishna Roskin > wrote: > > Hey all, > > > > Long time fan, taking my first crack at contributing. > > > > I've built a basic module to run and parse the result of sequence based > > clustering tools such as DNACLUST and CD-HIT. I've written subclasses of > > AbstractCommandline to run dnaclust and cd-hit. I've also written classes > > to store the clusters and their members and loaders for the output > formats > > used by those programs. > > > > I posting here to gauge interest and get some feedback and maybe some > beta > > testers. > > > > My code is available at: > > > > https://github.com/krishnaroskin/biopython.git > > > > under the seqcluster branch. I've started writing some test code at: > > > > Tests/seqcluster/test_seqcluster.py > > > > that also severs as example code. I've pasted that at the end of this > > message so people can get an idea of how it works without having to > > checkout code. > > > > If there is interest, I'm planning on adding a seqclust.cluster function > > that takes a list of SeqRecords and returns a clustering using one of the > > supported tools. I envision that function being the main interface to > this > > module. I also want to write a something that will map the cluster > > membership (given by ids) back to collections of SeqRecords. > > > > Other to-dos: > > > > Test that all the flavors of CD-HIT work (there are many) > > Add support for other sequence based clustering tools (suggestions?) > > Documentation > > Tutorial > > Test code > > > > -krish > > > > #!/usr/bin/env python > > > > > > from __future__ import print_function > > > > > > import StringIO > > > > > > from Bio.seqcluster.applications import DNAClustCommandline > > > > from Bio.seqcluster import DNAClustIterator > > > > > > from Bio.seqcluster.applications import CDHITCommandline > > > > from Bio.seqcluster import CDHITClustIterator > > > > > > cmd = DNAClustCommandline(similarity=0.8, header=True, threads=2, > inputfile= > > "test_sequences.fasta") > > > > stdout, stderr = cmd() > > > > clusters = DNAClustIterator(StringIO.StringIO(stdout)) > > > > for cluster in clusters: > > > > print(cluster.name) > > > > for member in cluster: > > > > if member == cluster.representative: > > > > print("\t" + member.name + "*") > > > > else: > > > > print("\t" + member.name) > > > > > > print() # blank line > > > > > > cmd = CDHITCommandline(cutoff=0.8, threads=2, inputfile= > > "test_sequences.fasta", outputfile="tmp") > > > > stdout, stderr = cmd() > > > > clusters = CDHITClustIterator(open("tmp.clstr", "r")) > > > > for cluster in clusters: > > > > print(cluster.name) > > > > for member in cluster: > > > > if member == cluster.representative: > > > > print("\t" + member.name + "*") > > > > else: > > > > print("\t" + member.name) > > _______________________________________________ > > Biopython-dev mailing list > > Biopython-dev at lists.open-bio.org > > http://lists.open-bio.org/mailman/listinfo/biopython-dev > From arklenna at gmail.com Mon Apr 14 17:32:00 2014 From: arklenna at gmail.com (Lenna Peterson) Date: Mon, 14 Apr 2014 13:32:00 -0400 Subject: [Biopython-dev] Problem on Mac OS X since update Xcode 5.1 In-Reply-To: References: Message-ID: On Apr 14, 2014 7:36 AM, "Peter Cock" wrote: > > On Sun, Mar 16, 2014 at 9:15 PM, Peter Cock wrote: > > On Sun, Mar 16, 2014 at 8:53 PM, Lenna Peterson wrote: > >> On Wed, Mar 12, 2014 at 5:59 AM, Peter Cock > >> wrote: > >>> > >>> ... > >>> > >>> This looks relevant, especially this reply from Paul Kehrer which > >>> suggests this is entirely Apple's fault for shipping a Python and > >>> clang compiler which don't get along with the default settings: > >>> > >>> > >>> http://stackoverflow.com/questions/22313407/clang-error-unknown-argument-mno-fused-madd-psycopg2-installation-failure > >>> > >> > >>> The suggested workaround seems to do the trick, > >>> > >>> $ export CFLAGS=-Qunused-arguments > >>> $ export CPPFLAGS=-Qunused-arguments > >>> > >> > >> I encountered the same problem (clean install of Mavericks, vanilla Python, > >> latest XCode from App Store). > >> > >> One answer [1] suggests this is not a guaranteed solution but offers a > >> different flag (which I did not test). > >> > >> I chose to edit system python files [2] which is definitely not the best > >> option for most users. > >> > >> [1]: http://stackoverflow.com/a/22315129 > >> [2]: http://stackoverflow.com/a/22322068 > >> > >>> Perhaps we can add this hack to our setup.py on Mac OS X... > >>> it seems harmless under gcc (e.g. my locally compiled version > >>> of Python 3.3 used gcc rather than clang)? > >> > >> Do you mean editing environment variables with `os.environ`? I don't know > >> enough about the details of how packages are built to know what will work > >> with both compiling from source, easy_install, pip, etc. > > > > Yes, I was thinking about editing the environment variables in > > setup.py via the os module. I agree there are potential risks > > with 3rd party installers, but adding -Qunused-arguments to > > any existing CFLAGS (within the scope of the Biopython install) > > is hopefully low risk... > > > >>> Or it could be done via the buildbot setup, or on this buildslave > >>> directly (e.g. the ~/.bash_profile). > >> > >> It's a dilemma, because asking users to edit their .bashrc or .bash_profile > >> before installation is annoying and easy to overlook, but modifying them in > >> setup.py feels hacky (i.e. how long will this solution work?). Crossing my > >> fingers and hoping Apple fixes this in an update... > >> > > > > Fingers crossed Apple pushes another update in the next few > > weeks to resolve this... > > > > Peter > > I just retested with XCode 5.1.1 (released last week and updated > via the App Store), and the problem persists. > > $ cc -v > Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn) > Target: x86_64-apple-darwin13.1.0 > Thread model: posix > > $ which python-config > /Library/Frameworks/Python.framework/Versions/Current/bin/python-config > > $ python-config --cflags > -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 > -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 > -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk > -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd > -fno-common -dynamic -DNDEBUG -g -O3 > > i.e. Apple's current Python still says to use C compliler options which > Apple's current clang compiler rejects :( But Python is distributed with the OS, not XCode. I'm still hoping the next update to Mavericks will fix it. > > The good news is the environment variable trick still works. > > It appears Apple are not going to fix this, so perhaps we do need > to add a hack to our setup.py file? I suppose modifying env variables is easier when it doesn't need to be cross platform! > > Peter From manlio.calvi at gmail.com Tue Apr 15 08:59:16 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Tue, 15 Apr 2014 10:59:16 +0200 Subject: [Biopython-dev] I could lend a little help Message-ID: Hello devs, I'm Manlio and I'm a MSc in medicinal chemistry with some stints in python coding and a budding interest in bioinformatics. At the moment don't think I could really help you coding, I'm picking up my skills in python really, but I want to help in some manner here, maybe some testing, documentation or some other related task and in meantime learn more of either python and its application in this field(s) I'm interested in. So, I would help but this is my first project I collaborate in and I don't really know how it works and what I could really do here, a pointer or two on where to start working on could be useful. Thank you in advance to all. Cheers, Manlio From tra at popgen.net Wed Apr 16 09:03:49 2014 From: tra at popgen.net (Tiago Antao) Date: Wed, 16 Apr 2014 10:03:49 +0100 Subject: [Biopython-dev] Fwd: [Numpy-discussion] EuroSciPy 2014 Call for Abstracts In-Reply-To: References: <20140320114815.70210fd5@grandao> <87txathtux.fsf@chupacabra.windows.ebi.ac.uk> <20140401114620.67490302@lnx> Message-ID: <20140416100349.5bd8e4a9@lnx> Dear all, On Tue, 1 Apr 2014 14:08:45 +0100 Peter Cock wrote: > OK, let's sort this out next week then - this week is the BOSC > deadline (4 April), and Bow has volunteered to present the > Biopython talk. The abstract deadline for euroscipy has just been extended. I am not in the UK at that time, unfortunately. Is there still anyone interested on going? From p.j.a.cock at googlemail.com Wed Apr 16 12:58:04 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 16 Apr 2014 13:58:04 +0100 Subject: [Biopython-dev] Volunteer buildslave machines? e.g. Windows & 32 bit Linux In-Reply-To: References: Message-ID: On Mon, Mar 24, 2014 at 11:57 AM, Peter Cock wrote: > Hello all, > > Tiago and I have been looking after a range of machines > covering different operating systems and Python versions, > running as volunteer buildslaves for Biopython using buildbot: > http://testing.open-bio.org/biopython/tgrid > > Does anyone else have a lab/home server which could be > setup to run nightly Biopython tests for us via buildbot? > Ideally the machine needs to be online overnight (European > time) when the server is currently setup to schedule tests: > > http://www.biopython.org/wiki/Continuous_integration > > Our elderly 32 bit Linux desktop which has been running > as a Biopython buildslave for the last few years is finally > failing (hard drive problem). > > I would particularly like to see new buildslaves for: > > * 32 bit Linux > * 64 bit Windows > * Windows 7 or 8 (we have a 32 bit XP machine) > > If you think you might be able to help, the first hurdle is > verifying you can checkout Biopython from github, and > then compile the source (this is non-trivial on Windows, > especially for 64 bit Windows). > > Note that this is separate from the continuous integration > testing done for use via TravisCI whenever the GitHub > repository is updated - this is very useful but currently > only covers Linux: > https://travis-ci.org/biopython/biopython/builds > > The key benefit of the buildbot server is cross platform > testing - but this requires a range of volunteer machines. > > Thanks, > > Peter > > Anyone? Right now I'm particularly interesting in Windows 32bit, since my own machine has started showing test failures with SQLite and I would like to know if this is unique or a common issue (e.g. due to a Windows update): http://lists.open-bio.org/pipermail/biopython-dev/2014-March/011169.html Also a couple of people have suggested using VMs, which would be a neat solution but some physically redundancy seems helpful. Thanks, Peter From manlio.calvi at gmail.com Wed Apr 16 13:30:26 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Wed, 16 Apr 2014 15:30:26 +0200 Subject: [Biopython-dev] Volunteer buildslave machines? e.g. Windows & 32 bit Linux In-Reply-To: References: Message-ID: I have a question, exactly what it takes? I mean what kind of machine you need, must be a server? Long story short, could be a common home box?My PC in this case (a mid/mid-high range gaming machine) and what entail exactly as software installed(apart biopython and dependencies, of course) and internet access? I'm keeping general because I believe that knowing what is exactly needed in a detailed way could be useful here. In all cases my machine is a Win7 x64 box, I'm currently running python 2.7 x64. Cheers, Manlio On Wed, Apr 16, 2014 at 2:58 PM, Peter Cock wrote: > On Mon, Mar 24, 2014 at 11:57 AM, Peter Cock wrote: > >> Hello all, >> >> Tiago and I have been looking after a range of machines >> covering different operating systems and Python versions, >> running as volunteer buildslaves for Biopython using buildbot: >> http://testing.open-bio.org/biopython/tgrid >> >> Does anyone else have a lab/home server which could be >> setup to run nightly Biopython tests for us via buildbot? >> Ideally the machine needs to be online overnight (European >> time) when the server is currently setup to schedule tests: >> >> http://www.biopython.org/wiki/Continuous_integration >> >> Our elderly 32 bit Linux desktop which has been running >> as a Biopython buildslave for the last few years is finally >> failing (hard drive problem). >> >> I would particularly like to see new buildslaves for: >> >> * 32 bit Linux >> * 64 bit Windows >> * Windows 7 or 8 (we have a 32 bit XP machine) >> >> If you think you might be able to help, the first hurdle is >> verifying you can checkout Biopython from github, and >> then compile the source (this is non-trivial on Windows, >> especially for 64 bit Windows). >> >> Note that this is separate from the continuous integration >> testing done for use via TravisCI whenever the GitHub >> repository is updated - this is very useful but currently >> only covers Linux: >> https://travis-ci.org/biopython/biopython/builds >> >> The key benefit of the buildbot server is cross platform >> testing - but this requires a range of volunteer machines. >> >> Thanks, >> >> Peter >> >> > Anyone? > > Right now I'm particularly interesting in Windows 32bit, > since my own machine has started showing test failures > with SQLite and I would like to know if this is unique or > a common issue (e.g. due to a Windows update): > http://lists.open-bio.org/pipermail/biopython-dev/2014-March/011169.html > > Also a couple of people have suggested using VMs, > which would be a neat solution but some physically > redundancy seems helpful. > > Thanks, > > Peter > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev From p.j.a.cock at googlemail.com Wed Apr 16 13:51:13 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 16 Apr 2014 14:51:13 +0100 Subject: [Biopython-dev] Volunteer buildslave machines? e.g. Windows & 32 bit Linux In-Reply-To: References: Message-ID: Hi Manlio, >From a hardware point of view, that sounds fine. The Biopython tests are deliberately not computationally heavy - each full test suite should take less than 10 minutes. One of the BuildBots is my old desktop machine at home, Ideally your machine would be turned on, with internet access, all the time - but in particular overnight European time when the nightly tests are currently triggered automatically. See: http://www.biopython.org/wiki/Continuous_integration Have you ever compiled Biopython from source on Windows? That would be the first step (without worrying about the buildbot side of things for now), although ... if that is too much ... we could maybe just run the PyPy tests on your machine (no need for a C compiler which makes it much simpler)? Regards, Peter On Wed, Apr 16, 2014 at 2:30 PM, Manlio Calvi wrote: > > I have a question, > exactly what it takes? I mean what kind of machine you need, must be a > server? Long story short, could be a common home box?My PC in this > case (a mid/mid-high range gaming machine) and what entail exactly as > software installed(apart biopython and dependencies, of course) and > internet access? > I'm keeping general because I believe that knowing what is exactly > needed in a detailed way could be useful here. > > In all cases my machine is a Win7 x64 box, I'm currently running python 2.7 x64. > > Cheers, > Manlio > > > On Wed, Apr 16, 2014 at 2:58 PM, Peter Cock wrote: > > On Mon, Mar 24, 2014 at 11:57 AM, Peter Cock wrote: > > > >> Hello all, > >> > >> Tiago and I have been looking after a range of machines > >> covering different operating systems and Python versions, > >> running as volunteer buildslaves for Biopython using buildbot: > >> http://testing.open-bio.org/biopython/tgrid > >> > >> Does anyone else have a lab/home server which could be > >> setup to run nightly Biopython tests for us via buildbot? > >> Ideally the machine needs to be online overnight (European > >> time) when the server is currently setup to schedule tests: > >> > >> http://www.biopython.org/wiki/Continuous_integration > >> > >> Our elderly 32 bit Linux desktop which has been running > >> as a Biopython buildslave for the last few years is finally > >> failing (hard drive problem). > >> > >> I would particularly like to see new buildslaves for: > >> > >> * 32 bit Linux > >> * 64 bit Windows > >> * Windows 7 or 8 (we have a 32 bit XP machine) > >> > >> If you think you might be able to help, the first hurdle is > >> verifying you can checkout Biopython from github, and > >> then compile the source (this is non-trivial on Windows, > >> especially for 64 bit Windows). > >> > >> Note that this is separate from the continuous integration > >> testing done for use via TravisCI whenever the GitHub > >> repository is updated - this is very useful but currently > >> only covers Linux: > >> https://travis-ci.org/biopython/biopython/builds > >> > >> The key benefit of the buildbot server is cross platform > >> testing - but this requires a range of volunteer machines. > >> > >> Thanks, > >> > >> Peter > >> > >> > > Anyone? > > > > Right now I'm particularly interesting in Windows 32bit, > > since my own machine has started showing test failures > > with SQLite and I would like to know if this is unique or > > a common issue (e.g. due to a Windows update): > > http://lists.open-bio.org/pipermail/biopython-dev/2014-March/011169.html > > > > Also a couple of people have suggested using VMs, > > which would be a neat solution but some physically > > redundancy seems helpful. > > > > Thanks, > > > > Peter > > _______________________________________________ > > Biopython-dev mailing list > > Biopython-dev at lists.open-bio.org > > http://lists.open-bio.org/mailman/listinfo/biopython-dev From manlio.calvi at gmail.com Wed Apr 16 14:01:55 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Wed, 16 Apr 2014 16:01:55 +0200 Subject: [Biopython-dev] Volunteer buildslave machines? e.g. Windows & 32 bit Linux In-Reply-To: References: Message-ID: Hi Peter, No as I said my (bio)python distibution is from binary, if you could say me what tools is needed to compile it I think I could manage (or try to) build up a python enviroment from source(I have some past stints in building linuxes from source). Let me know Best, Manlio On Wed, Apr 16, 2014 at 3:51 PM, Peter Cock wrote: > Hi Manlio, > > From a hardware point of view, that sounds fine. The Biopython > tests are deliberately not computationally heavy - each full test > suite should take less than 10 minutes. One of the BuildBots > is my old desktop machine at home, > > Ideally your machine would be turned on, with internet access, > all the time - but in particular overnight European time when the > nightly tests are currently triggered automatically. > > See: http://www.biopython.org/wiki/Continuous_integration > > Have you ever compiled Biopython from source on Windows? > That would be the first step (without worrying about the buildbot > side of things for now), although ... if that is too much ... we > could maybe just run the PyPy tests on your machine (no > need for a C compiler which makes it much simpler)? > > Regards, > > Peter > > > On Wed, Apr 16, 2014 at 2:30 PM, Manlio Calvi wrote: >> >> I have a question, >> exactly what it takes? I mean what kind of machine you need, must be a >> server? Long story short, could be a common home box?My PC in this >> case (a mid/mid-high range gaming machine) and what entail exactly as >> software installed(apart biopython and dependencies, of course) and >> internet access? >> I'm keeping general because I believe that knowing what is exactly >> needed in a detailed way could be useful here. >> >> In all cases my machine is a Win7 x64 box, I'm currently running python 2.7 x64. >> >> Cheers, >> Manlio >> >> >> On Wed, Apr 16, 2014 at 2:58 PM, Peter Cock wrote: >> > On Mon, Mar 24, 2014 at 11:57 AM, Peter Cock wrote: >> > >> >> Hello all, >> >> >> >> Tiago and I have been looking after a range of machines >> >> covering different operating systems and Python versions, >> >> running as volunteer buildslaves for Biopython using buildbot: >> >> http://testing.open-bio.org/biopython/tgrid >> >> >> >> Does anyone else have a lab/home server which could be >> >> setup to run nightly Biopython tests for us via buildbot? >> >> Ideally the machine needs to be online overnight (European >> >> time) when the server is currently setup to schedule tests: >> >> >> >> http://www.biopython.org/wiki/Continuous_integration >> >> >> >> Our elderly 32 bit Linux desktop which has been running >> >> as a Biopython buildslave for the last few years is finally >> >> failing (hard drive problem). >> >> >> >> I would particularly like to see new buildslaves for: >> >> >> >> * 32 bit Linux >> >> * 64 bit Windows >> >> * Windows 7 or 8 (we have a 32 bit XP machine) >> >> >> >> If you think you might be able to help, the first hurdle is >> >> verifying you can checkout Biopython from github, and >> >> then compile the source (this is non-trivial on Windows, >> >> especially for 64 bit Windows). >> >> >> >> Note that this is separate from the continuous integration >> >> testing done for use via TravisCI whenever the GitHub >> >> repository is updated - this is very useful but currently >> >> only covers Linux: >> >> https://travis-ci.org/biopython/biopython/builds >> >> >> >> The key benefit of the buildbot server is cross platform >> >> testing - but this requires a range of volunteer machines. >> >> >> >> Thanks, >> >> >> >> Peter >> >> >> >> >> > Anyone? >> > >> > Right now I'm particularly interesting in Windows 32bit, >> > since my own machine has started showing test failures >> > with SQLite and I would like to know if this is unique or >> > a common issue (e.g. due to a Windows update): >> > http://lists.open-bio.org/pipermail/biopython-dev/2014-March/011169.html >> > >> > Also a couple of people have suggested using VMs, >> > which would be a neat solution but some physically >> > redundancy seems helpful. >> > >> > Thanks, >> > >> > Peter >> > _______________________________________________ >> > Biopython-dev mailing list >> > Biopython-dev at lists.open-bio.org >> > http://lists.open-bio.org/mailman/listinfo/biopython-dev From p.j.a.cock at googlemail.com Wed Apr 16 14:13:41 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 16 Apr 2014 15:13:41 +0100 Subject: [Biopython-dev] Volunteer buildslave machines? e.g. Windows & 32 bit Linux In-Reply-To: References: Message-ID: Hi Manlio, If you are willing to try getting Biopython to compile under Windows 7 with 64 bit Python that would be great. I hope Tiago will be able to share some pointers to get started. The slightly less ambitious plan is compiling with 32 bit Python. My experience with 32bit Windows XP should help here. The least ambitious plan is avoiding compiling as required for C Python, and only testing under PyPy and Jython - or maybe try IronPython if that interests you? Peter On Wed, Apr 16, 2014 at 3:01 PM, Manlio Calvi wrote: > Hi Peter, > > No as I said my (bio)python distibution is from binary, if you could > say me what tools is needed to compile it I think I could manage (or > try to) build up a python enviroment from source(I have some past > stints in building linuxes from source). > Let me know > > Best, > Manlio > > > On Wed, Apr 16, 2014 at 3:51 PM, Peter Cock wrote: >> Hi Manlio, >> >> From a hardware point of view, that sounds fine. The Biopython >> tests are deliberately not computationally heavy - each full test >> suite should take less than 10 minutes. One of the BuildBots >> is my old desktop machine at home, >> >> Ideally your machine would be turned on, with internet access, >> all the time - but in particular overnight European time when the >> nightly tests are currently triggered automatically. >> >> See: http://www.biopython.org/wiki/Continuous_integration >> >> Have you ever compiled Biopython from source on Windows? >> That would be the first step (without worrying about the buildbot >> side of things for now), although ... if that is too much ... we >> could maybe just run the PyPy tests on your machine (no >> need for a C compiler which makes it much simpler)? >> >> Regards, >> >> Peter >> >> >> On Wed, Apr 16, 2014 at 2:30 PM, Manlio Calvi wrote: >>> >>> I have a question, >>> exactly what it takes? I mean what kind of machine you need, must be a >>> server? Long story short, could be a common home box?My PC in this >>> case (a mid/mid-high range gaming machine) and what entail exactly as >>> software installed(apart biopython and dependencies, of course) and >>> internet access? >>> I'm keeping general because I believe that knowing what is exactly >>> needed in a detailed way could be useful here. >>> >>> In all cases my machine is a Win7 x64 box, I'm currently running python 2.7 x64. >>> >>> Cheers, >>> Manlio >>> >>> >>> On Wed, Apr 16, 2014 at 2:58 PM, Peter Cock wrote: >>> > On Mon, Mar 24, 2014 at 11:57 AM, Peter Cock wrote: >>> > >>> >> Hello all, >>> >> >>> >> Tiago and I have been looking after a range of machines >>> >> covering different operating systems and Python versions, >>> >> running as volunteer buildslaves for Biopython using buildbot: >>> >> http://testing.open-bio.org/biopython/tgrid >>> >> >>> >> Does anyone else have a lab/home server which could be >>> >> setup to run nightly Biopython tests for us via buildbot? >>> >> Ideally the machine needs to be online overnight (European >>> >> time) when the server is currently setup to schedule tests: >>> >> >>> >> http://www.biopython.org/wiki/Continuous_integration >>> >> >>> >> Our elderly 32 bit Linux desktop which has been running >>> >> as a Biopython buildslave for the last few years is finally >>> >> failing (hard drive problem). >>> >> >>> >> I would particularly like to see new buildslaves for: >>> >> >>> >> * 32 bit Linux >>> >> * 64 bit Windows >>> >> * Windows 7 or 8 (we have a 32 bit XP machine) >>> >> >>> >> If you think you might be able to help, the first hurdle is >>> >> verifying you can checkout Biopython from github, and >>> >> then compile the source (this is non-trivial on Windows, >>> >> especially for 64 bit Windows). >>> >> >>> >> Note that this is separate from the continuous integration >>> >> testing done for use via TravisCI whenever the GitHub >>> >> repository is updated - this is very useful but currently >>> >> only covers Linux: >>> >> https://travis-ci.org/biopython/biopython/builds >>> >> >>> >> The key benefit of the buildbot server is cross platform >>> >> testing - but this requires a range of volunteer machines. >>> >> >>> >> Thanks, >>> >> >>> >> Peter >>> >> >>> >> >>> > Anyone? >>> > >>> > Right now I'm particularly interesting in Windows 32bit, >>> > since my own machine has started showing test failures >>> > with SQLite and I would like to know if this is unique or >>> > a common issue (e.g. due to a Windows update): >>> > http://lists.open-bio.org/pipermail/biopython-dev/2014-March/011169.html >>> > >>> > Also a couple of people have suggested using VMs, >>> > which would be a neat solution but some physically >>> > redundancy seems helpful. >>> > >>> > Thanks, >>> > >>> > Peter >>> > _______________________________________________ >>> > Biopython-dev mailing list >>> > Biopython-dev at lists.open-bio.org >>> > http://lists.open-bio.org/mailman/listinfo/biopython-dev From tra at popgen.net Wed Apr 16 14:17:40 2014 From: tra at popgen.net (Tiago Antao) Date: Wed, 16 Apr 2014 15:17:40 +0100 Subject: [Biopython-dev] Volunteer buildslave machines? e.g. Windows & 32 bit Linux In-Reply-To: References: Message-ID: <20140416151740.3a7d6c10@lnx> On Wed, 16 Apr 2014 15:13:41 +0100 Peter Cock wrote: > If you are willing to try getting Biopython to compile under > Windows 7 with 64 bit Python that would be great. I hope Tiago > will be able to share some pointers to get started. I agree with you: 32 bits is a better place to start. Regarding 64 bits, the only thing that I have to share is pain and gried. When I tried (twice in the long past) I always failed. It is obviously possible, as there are unofficial 64-bit distributions - but above my pay grade. The process now might be easier as things are maturing a bit... Tiago From manlio.calvi at gmail.com Wed Apr 16 14:33:20 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Wed, 16 Apr 2014 16:33:20 +0200 Subject: [Biopython-dev] Volunteer buildslave machines? e.g. Windows & 32 bit Linux In-Reply-To: <20140416151740.3a7d6c10@lnx> References: <20140416151740.3a7d6c10@lnx> Message-ID: Well I'm currently running on the WinPython 64bit binary distribution with all the py packages from Gohlke. I could try to build from scratch but what I don't understand is if I need to build every single piece of python and dependencies of biopython from scratch or we could start with a binary distro and some toolchain for building from C. For the other thing PyPy, Jithon and IronPython, I never used it but that means nothing as I could get interested in it in future. I guess for PyPy is only a battery of tests against it and as far I understood things either Jython and IronPython needs their toolchain to be tested.At the moment I could build only one thing though. On Wed, Apr 16, 2014 at 4:17 PM, Tiago Antao wrote: > On Wed, 16 Apr 2014 15:13:41 +0100 > Peter Cock wrote: > >> If you are willing to try getting Biopython to compile under >> Windows 7 with 64 bit Python that would be great. I hope Tiago >> will be able to share some pointers to get started. > > > I agree with you: 32 bits is a better place to start. > Regarding 64 bits, the only thing that I have to share is pain and > gried. When I tried (twice in the long past) I always failed. It is > obviously possible, as there are unofficial 64-bit distributions - but > above my pay grade. The process now might be easier as things are > maturing a bit... > > Tiago From p.j.a.cock at googlemail.com Wed Apr 16 14:41:12 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 16 Apr 2014 15:41:12 +0100 Subject: [Biopython-dev] Volunteer buildslave machines? e.g. Windows & 32 bit Linux In-Reply-To: References: <20140416151740.3a7d6c10@lnx> Message-ID: On Wed, Apr 16, 2014 at 3:33 PM, Manlio Calvi wrote: > Well I'm currently running on the WinPython 64bit binary distribution > with all the py packages from Gohlke. > I could try to build from scratch but what I don't understand is if I > need to build every single piece of python and dependencies of > biopython from scratch or we could start with a binary distro and some > toolchain for building from C. You should be OK using the Python binary installer, plus the matching NumPy binary installer from Christoph Gohlke, and a C compiler tool chain matching what he uses: http://www.lfd.uci.edu/~gohlke/pythonlibs/ > For the other thing PyPy, Jithon and IronPython, I never used it but > that means nothing as I could get interested in it in future. I guess > for PyPy is only a battery of tests against it and as far I understood > things either Jython and IronPython needs their toolchain to be > tested.At the moment I could build only one thing though. I've only tried IronPython briefly, getting Biopython to work nicely there is a project in itself. If anyone is interested, great, otherwise this is not a personal priority. Testing under PyPy should be trivial - download and install one of the provided binaries for PyPy, put them somewhere like C:\pypy-2.2, and then from the Biopython source do something like: C:\pypy2.2\pypy setup.py build C:\pypy2.2\pypy setup.py test C:\pypy2.2\pypy setup.py install Jython 2.7 ought to be similar, however the current release is only a beta and if I recall correctly I had to manually fix at least one bug in Jython for the Biopython tests to pass. Peter From manlio.calvi at gmail.com Wed Apr 16 15:03:16 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Wed, 16 Apr 2014 17:03:16 +0200 Subject: [Biopython-dev] Volunteer buildslave machines? e.g. Windows & 32 bit Linux In-Reply-To: References: <20140416151740.3a7d6c10@lnx> Message-ID: OK one step at time, giving that WinPython is a self contained distro(or so is claimed) I'm downloading that and all the binaries Christoph Gohlke use of compiling his tools (this mean all the dependencies of biopython are built against the Gohlke's toolchain) I'm downloading two of them one contain python 2.7 and the other 3.3, I'll try to build biopython against those two when all the downloading and installing went well. Then I'll report here. Only I don't know if could to all today Manlio On Wed, Apr 16, 2014 at 4:41 PM, Peter Cock wrote: > On Wed, Apr 16, 2014 at 3:33 PM, Manlio Calvi wrote: >> Well I'm currently running on the WinPython 64bit binary distribution >> with all the py packages from Gohlke. >> I could try to build from scratch but what I don't understand is if I >> need to build every single piece of python and dependencies of >> biopython from scratch or we could start with a binary distro and some >> toolchain for building from C. > > You should be OK using the Python binary installer, plus the > matching NumPy binary installer from Christoph Gohlke, and > a C compiler tool chain matching what he uses: > http://www.lfd.uci.edu/~gohlke/pythonlibs/ > >> For the other thing PyPy, Jithon and IronPython, I never used it but >> that means nothing as I could get interested in it in future. I guess >> for PyPy is only a battery of tests against it and as far I understood >> things either Jython and IronPython needs their toolchain to be >> tested.At the moment I could build only one thing though. > > I've only tried IronPython briefly, getting Biopython to work > nicely there is a project in itself. If anyone is interested, great, > otherwise this is not a personal priority. > > Testing under PyPy should be trivial - download and install > one of the provided binaries for PyPy, put them somewhere > like C:\pypy-2.2, and then from the Biopython source do > something like: > > C:\pypy2.2\pypy setup.py build > C:\pypy2.2\pypy setup.py test > C:\pypy2.2\pypy setup.py install > > Jython 2.7 ought to be similar, however the current release > is only a beta and if I recall correctly I had to manually fix at > least one bug in Jython for the Biopython tests to pass. > > Peter From p.j.a.cock at googlemail.com Wed Apr 16 16:03:37 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 16 Apr 2014 17:03:37 +0100 Subject: [Biopython-dev] Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? Message-ID: On Wed, Apr 16, 2014 at 4:03 PM, Manlio Calvi wrote: > OK one step at time, giving that WinPython is a self contained > distro(or so is claimed) I'm downloading that and all the binaries > Christoph Gohlke use of compiling his tools (this mean all the > dependencies of biopython are built against the Gohlke's toolchain) > I'm downloading two of them one contain python 2.7 and the other 3.3, > I'll try to build biopython against those two when all the downloading > and installing went well. > Then I'll report here. > Only I don't know if could to all today > > Manlio Good luck - even if we don't reach the buildbot stage, this should help establish proper Windows 64 bit support for Biopython :) Thank you, Peter From marco.galardini at unifi.it Wed Apr 16 17:11:25 2014 From: marco.galardini at unifi.it (Marco Galardini) Date: Wed, 16 Apr 2014 19:11:25 +0200 Subject: [Biopython-dev] Interested in a Phenotype Microarray parser? In-Reply-To: <533A0164.9010700@unifi.it> References: <1389160399.57779.YahooMailBasic@web164004.mail.gq1.yahoo.com> <52CD2948.7050102@unifi.it> <533213FC.8010304@unifi.it> <20140326142642.9o3hmuipcocsw8g8@webmail.unifi.it> <533A0164.9010700@unifi.it> Message-ID: <20140416191125.ysfq235mtc44ksg4@webmail.unifi.it> Hi, regarding further additions to the Bio.Phenotype module I was considering the following solution to add support for sigmoid curve fitting and parameters extraction (which is of interest when analysing this kind of data). Since the easiest way to do the curve fitting is by using the scipy package, a solution may be to implement it as a "optional" feature, like the ability to draw trees with the Phylo module using matplotlib. An exception would be raised if the "function" is called with no scipy installed. Would it that be ok? Alternatively some other way to perform curve fitting may be found, but the maintenance may become very difficult. Marco ----- Messaggio da marco.galardini at unifi.it --------- Data: Tue, 01 Apr 2014 00:59:32 +0100 Da: Marco Galardini Rispondi-A:Marco Galardini Oggetto: Re: [Biopython-dev] Interested in a Phenotype Microarray parser? A: Peter Cock Cc: Biopython-Dev Mailing List > Hi, > > as suggested, I've made a few changes to the proposed Bio.Phenotype > module (apart from the less-omics name). > > The PlateRecord object now can be indexed in a similar fashion as > AlignIO multiple alignments: it is still possible to use the WellRecord > identifier as an index, but when integers or slices are used, new > sub-plates or single wells are returned. The system uses the well > identifier as a mean to divide the plate into rows/column. Thanks for > pointing out the AlignIO system, it has been very useful. > I've left the two getColumns and getRows functions, since for some > people it may still be useful to use the wells identifiers. If you feel > like they are too confusing I can remove them. > > The updated branch is here: > https://github.com/mgalardini/biopython/tree/phenomics > > Kind regards, > Marco > > > On 26/03/2014 13:26, Marco Galardini wrote: >> Hi, >> >> many thanks for your comments, below some replies: >> >> ----- Messaggio da p.j.a.cock at googlemail.com --------- >> Data: Wed, 26 Mar 2014 10:14:53 +0000 >> Da: Peter Cock >> Rispondi-A:Peter Cock >> Oggetto: Re: [Biopython-dev] Interested in a Phenotype Microarray parser? >> A: Marco Galardini >> Cc: Biopython-Dev Mailing List >> >> >>> On Tue, Mar 25, 2014 at 11:40 PM, Marco Galardini >>> wrote: >>>> Hi all, >>>> >>>> following your suggestions (as well as the other modules implementations) >>>> I've just committed a couple of commits to my biopython fork, >>>> featuring the >>>> Bio.Phenomics module. >>>> The module capabilities are limited to reading/writing Phenotype >>>> Microarray >>>> files and basic operations on the PlateRecord/WellRecord objects. >>>> The module >>>> requires numpy to interpolate the signal when the user request a >>>> time point >>>> that wasn't in the input file (this way the WellRecord object can >>>> be queried >>>> with slices). >>>> I'm thinking on how to implement the parameters extraction from WellRecord >>>> objects without the use of scipy. >>>> >>>> Here's the link to my branch: >>>> https://github.com/mgalardini/biopython/tree/phenomics >>>> The module and functions have been documented taking inspiration from the >>>> other modules: hope they are clear enough for you to try it out. >>>> Some example files can be found in Tests/Phenomics. >>>> >>>> Marco >>> >>> Hi Marco, >>> >>> I've not worked with kind of data so my comments are not on >>> the application specifics. But I'm pleased to see unit tests :) >>> >>> One thought was while you define (Java like?) getRow and getColumn >>> methods, your __getitem__ does not support (NumPy like) access, >>> which is something we do for multiple sequence alignments. I guess >>> while most plates are laid out in a grid, the row/column for each >>> sample is not the most important thing - the sample identifier is? >>> >>> Thinking out loud, would properties `rows` and `columns` etc be >>> nicer than `getRow` and `getColumn`, supporting iteration over >>> the rows/columns/etc and indexing? >> >> Yeah, absolutely: I'll work on some changes to have a more >> straightforward way to select multiple WellRecords on row/column >> basis. >> >>> >>> Minor: Your longer function docstrings do not follow PEP257, >>> specifically starting with a one line summary, then a blank line, >>> then the details. Also you are using triple single-quotes, rather >>> than triple double-quotes (like the rest of Biopthon). >>> http://legacy.python.org/dev/peps/pep-0257/ >> >> Whoops, I'll change it, thanks >> >>> >>> Peter >>> >>> P.S. Also, I'm not very keen on the module name, phenomics - >>> I wonder if it would earn Biopython a badomics award? ;) >>> http://dx.doi.org/10.1186/2047-217X-1-6 >> >> That's meta-omics right? :p >> What about 'Phenotype' then? Maybe it's too general, but future >> extensions may include other phenotypic readouts. >> >> Marco >>> _______________________________________________ >>> Biopython-dev mailing list >>> Biopython-dev at lists.open-bio.org >>> http://lists.open-bio.org/mailman/listinfo/biopython-dev >>> >> >> >> ----- Fine del messaggio da p.j.a.cock at googlemail.com ----- >> >> >> >> Marco Galardini >> Postdoctoral Fellow >> EMBL-EBI - European Bioinformatics Institute >> Wellcome Trust Genome Campus >> Hinxton, Cambridge CB10 1SD, UK >> Phone: +44 (0)1223 49 2547 >> >> >> _______________________________________________ >> Biopython-dev mailing list >> Biopython-dev at lists.open-bio.org >> http://lists.open-bio.org/mailman/listinfo/biopython-dev > > -- > ------------------------------------------------- > Marco Galardini, PhD > Dipartimento di Biologia > Via Madonna del Piano, 6 - 50019 Sesto Fiorentino (FI) > > e-mail: marco.galardini at unifi.it > www: http://www.unifi.it/dblage/CMpro-v-p-51.html > phone: +39 055 4574737 > mobile: +39 340 2808041 > ------------------------------------------------- > > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev ----- Fine del messaggio da marco.galardini at unifi.it ----- Marco Galardini Postdoctoral Fellow EMBL-EBI - European Bioinformatics Institute Wellcome Trust Genome Campus Hinxton, Cambridge CB10 1SD, UK Phone: +44 (0)1223 49 2547 From arklenna at gmail.com Wed Apr 16 17:28:46 2014 From: arklenna at gmail.com (Lenna Peterson) Date: Wed, 16 Apr 2014 13:28:46 -0400 Subject: [Biopython-dev] Coordinate Mapper pull request In-Reply-To: References: Message-ID: On Sun, Apr 13, 2014 at 4:16 PM, Lenna Peterson wrote: > Hi all, > > I've finally submitted a pull request for my coordinate mapper (many > thanks to Reece Hart for the original code). > > An example of its usage is on the cookbook: > http://biopython.org/wiki/Coordinate_mapping > The example code can also be found in this gist: > https://gist.github.com/lennax/10600113 > > Comments are welcome on the list or the pull request: > > https://github.com/biopython/biopython/pull/309 > I fixed it for Python 3 and caught a logic error, but it's now passing all tests on Travis CI. > > > Cheers, > > Lenna > From manlio.calvi at gmail.com Thu Apr 17 06:57:58 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Thu, 17 Apr 2014 08:57:58 +0200 Subject: [Biopython-dev] Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? In-Reply-To: References: Message-ID: Update from the first round of fights :) Apparently even using the ginormous windows toolchains available online is the compilers hit a snags on architectures in the build.py script (isn't a build.py problem, I think), that's because every main version of python needs a different toolchain in windows and must be the FULL windows toolchain because they put in limitation to the compilers on the x64 architectures, the only (legal) way to have the full win tools is buy it and I can't afford them. So tl;dr I'm going to try mingw-64 toolchains, hoping those goes well. Bye to all, Manlio On Wed, Apr 16, 2014 at 6:03 PM, Peter Cock wrote: > On Wed, Apr 16, 2014 at 4:03 PM, Manlio Calvi wrote: >> OK one step at time, giving that WinPython is a self contained >> distro(or so is claimed) I'm downloading that and all the binaries >> Christoph Gohlke use of compiling his tools (this mean all the >> dependencies of biopython are built against the Gohlke's toolchain) >> I'm downloading two of them one contain python 2.7 and the other 3.3, >> I'll try to build biopython against those two when all the downloading >> and installing went well. >> Then I'll report here. >> Only I don't know if could to all today >> >> Manlio > > Good luck - even if we don't reach the buildbot stage, this > should help establish proper Windows 64 bit support for > Biopython :) > > Thank you, > > Peter From reece at harts.net Fri Apr 18 05:57:23 2014 From: reece at harts.net (Reece Hart) Date: Thu, 17 Apr 2014 22:57:23 -0700 Subject: [Biopython-dev] Coordinate Mapper pull request In-Reply-To: References: Message-ID: Lenna- I hope that grad school is treating you well. As the diagnostic sequencing space has heated up, accurate transcript-genome mapping has become ever more relevant to genome interpretations. The mapping code from that original post, now 4 years old I think, was quite primitive. My recollection is that the original post didn't even handle minus strand transcripts. Perhaps you've improved it since then. A team of us at Invitae have recently released a much more robust HGVS parser, formatter, and mapper (Apache 2.0 licensed). It's available at https://bitbucket.org/invitae/hgvs. This package currently relies on transcripts from the Universal Transcript Archive, which has current and recent historical transcripts from multiple sources and alignments to reference genomes and patches using splign and blat. UTA is available via postgresql/libpq at uta.invitae.com:5432; code is at https://bitbucket.org/invitae/uta, but only required for loading the database). This pair is reasonably accurate, but does have shortcomings. Users are advised to read through issues to understand limitations and the development roadmap. We are actively working on improving these tools. Patches are certainly welcome. (For those going to HVP in May, I'll be speaking about it there and would love to connect with users.) -Reece From p.j.a.cock at googlemail.com Fri Apr 18 18:26:00 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Fri, 18 Apr 2014 19:26:00 +0100 Subject: [Biopython-dev] TogoWS test failure, was: Compiling Biopython on 64 bit Windows Message-ID: On Thu, Apr 17, 2014 at 7:57 AM, Manlio Calvi wrote: > Update from the first round of fights :) > Following a successful build (the emails didn't get CC'd to the list - can we forward them?), Manlio reported some test failures including with TogoWS (both Python 2.7 and 3.4), pasted at the end of this email. Something is amiss in the call to Bio.TogoWS.search_count(...) where we expect TogoWS to return a plain text file with an integer count in it. Somehow some XHTML is being returned instead. Has anyone else seen this before? I wonder if it is a proxy problem or something like that? Thanks, Peter -- ====================================================================== ERROR: test_pubmed_search_bioruby (test_TogoWS.TogoSearch) Bio.TogoWS.search_iter("pubmed", "BioRuby") etc ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", line 425, in test_pubmed_search_bioruby "20739307", "20015970", "14693808"]) File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", line 463, in check search_count = TogoWS.search_count(database, search_term) File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\TogoWS\__init__.py", line 167, in search_count count = int(handle.read().strip()) ValueError: invalid literal for int() with base 10: '\n\n\n \n\n\n \n\n\n References: <20140417124408.5af66518@lnx> Message-ID: ---------- Forwarded message ---------- From: Manlio Calvi Date: Thu, Apr 17, 2014 at 10:11 PM Subject: Re: Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? To: Tiago Antao Cc: Peter Cock People, announcement! :) I'm successful! (well, mostly) I compiled on Win7 x64 against both native Python 2.7.6 x64 and Python 3.4.0 x64 How I did: 1. Installed MS Visual Studio 2008 express 2. Installed Windows SDK for Windows Server 2008 and .NET Framework 3.5 because the x64 toolchains was placed there 3. Run windows update to fix and patch things in those packages 4. Installed both Python native binary for last 2.7 and 3.4 (I discovered that is better put in this order) 5. Install from C.Gohlke all the x64 deps biopython could have + SciPy just to be sure. A pair of caveat here, I tried to produce the same configuration for both the python versions and in that site some optional deps for 3.4 are missing. I did an exception later for networkX, I downloaded from git and compiled (I'll tell you when in this process) Now coming the tricky part: 6. I found the only way to make the compilation start is to call a shell with this command - *cmd /k ""C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\vcvars64.bat""* - this set the enviroment variable for that specific shell to the x64 toolchain, otherwise don't compile (I opened two of those, one for major python version) 7. For this experiments I created two separate copies of biopython in two different dir where I cloned the reps. 8. I set the python env variable, for the Python 2.7 dedicated shell - SET PATH=C:\python27\;%PATH% - and for Python 3.4 dedicated shell - SET PATH=C:\python34\;%PATH; - of course. If you don't do that is possible you compile against the wrong python version, and in my specific case it compile (that strange but that's how happened to me) 9. Now I downloaded and installed networkx from git for test the "assembly", it worked 10. Cloned, build both the package "versions" flawlessy or so seems 11. Tested the two, a lot of skipping for missing soft dependencies. A Note here, I see a lot packages skipped because missing optional packages (in and out python) a lot MORE than the one listed on the biopython site. The results of the test are: Biopython x64 - Python 3.4 Ran 224 test and failed 3 on SearchIO_blast_xml.py, on SubsMat and on TogoWS(more than one kind of error generated here) Biopython x64 - Python 2.7.6 Ran 225 test and failed 2 on TogoWS(The same errors seems at first look) and a Docstring test on test_Tutorial.py OK that's all folks for now. What's next? :) Manlio From manlio.calvi at gmail.com Fri Apr 18 18:35:42 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Fri, 18 Apr 2014 20:35:42 +0200 Subject: [Biopython-dev] Fwd: Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? In-Reply-To: References: <20140417124408.5af66518@lnx> Message-ID: ---------- Forwarded message ---------- From: Peter Cock Date: Fri, Apr 18, 2014 at 12:01 PM Subject: Re: Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? To: Manlio Calvi Cc: Tiago Antao Manlio Excellent work - well done :) We should add some of this to the "Installing from source on Windows" section of the installation notes (which we also need to update regarding newer versions of Python and which compiler works): https://github.com/biopython/biopython/blob/master/Doc/install/Installation.tex Step 6 sounds familiar, I think there is a similar batch file to set the environment variables for the 43bit MSVC compiler. Regarding step 7, Python should use separate subdirectoroes under build/ for each version of Python, so that shouldn't be a needed. However, if you do use the same folder some of our tests use hard coded temporary filenames so it is best to only test one version of Python at a time. Regarding step 8, normally on Windows I don't bother with setting the $PATH environment variable and simply call the explicit version of Python I am testing, e.g. C:\Python27\python.exe setup.py build C:\Python27\python.exe setup.py test C:\Python27\python.exe setup.py install (This is what the current buildbot instructions do where we have automated the tests). As to what's text - let's try to fix these test failures :) Can you post the test output please? TogoWS is an online service, so from time to time it does have transient failures (so does the NCBI Entrez API). The fact this broke under both Python 2.7 and 3.4 suggests this is not a Python 3 issue. I am a little surprised you didn't see this problem with Python 3.4, but Windows does unicode a little differently so that may be why? https://github.com/biopython/biopython/issues/305 How did you install git & clone Biopython? Specifically, what is it doing about text files (i.e. Unix versus DOS/Windows new lines). This was something which has caused test failures in the past - perhaps some new cases have arisen. Thanks! Peter From arklenna at gmail.com Fri Apr 18 18:53:56 2014 From: arklenna at gmail.com (Lenna Peterson) Date: Fri, 18 Apr 2014 14:53:56 -0400 Subject: [Biopython-dev] TogoWS test failure, was: Compiling Biopython on 64 bit Windows In-Reply-To: References: Message-ID: I see similar errors (Ubuntu 12.04, Python 2.7.3, university network). I can test whether the same thing happens while I'm off-campus. Lenna On Fri, Apr 18, 2014 at 2:26 PM, Peter Cock wrote: > On Thu, Apr 17, 2014 at 7:57 AM, Manlio Calvi > wrote: > > Update from the first round of fights :) > > > > Following a successful build (the emails didn't get CC'd to > the list - can we forward them?), Manlio reported some test > failures including with TogoWS (both Python 2.7 and 3.4), > pasted at the end of this email. > > Something is amiss in the call to Bio.TogoWS.search_count(...) > where we expect TogoWS to return a plain text file with an > integer count in it. > > Somehow some XHTML is being returned instead. > > Has anyone else seen this before? I wonder if it is a proxy > problem or something like that? > > Thanks, > > Peter > > -- > > > ====================================================================== > ERROR: test_pubmed_search_bioruby (test_TogoWS.TogoSearch) > Bio.TogoWS.search_iter("pubmed", "BioRuby") etc > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > line 425, in test_pubmed_search_bioruby > "20739307", "20015970", "14693808"]) > File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > line 463, in check > search_count = TogoWS.search_count(database, search_term) > File > "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\TogoWS\__init__.py", > line 167, in search_count > count = int(handle.read().strip()) > ValueError: invalid literal for int() with base 10: ' PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" > "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n xmlns="http://www.w3.org/1999/xhtml">\n\n http-equiv="Content-Type" c > > ====================================================================== > ERROR: test_pubmed_search_porin (test_TogoWS.TogoSearch) > Bio.TogoWS.search_iter("pubmed", "human porin") etc > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > line 434, in test_pubmed_search_porin > self.check("pubmed", "human porin", ["21189321", "21835183"]) > File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > line 463, in check > search_count = TogoWS.search_count(database, search_term) > File > "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\TogoWS\__init__.py", > line 167, in search_count > count = int(handle.read().strip()) > ValueError: invalid literal for int() with base 10: ' PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" > "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n xmlns="http://www.w3.org/1999/xhtml">\n\n http-equiv="Content-Type" c > > ====================================================================== > ERROR: test_pubmed_search_togows (test_TogoWS.TogoSearch) > Bio.TogoWS.search_iter("pubmed", "TogoWS") etc > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > line 420, in test_pubmed_search_togows > self.check("pubmed", "TogoWS", ["20472643"]) > File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > line 463, in check > search_count = TogoWS.search_count(database, search_term) > File > "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\TogoWS\__init__.py", > line 167, in search_count > count = int(handle.read().strip()) > ValueError: invalid literal for int() with base 10: ' PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" > "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n xmlns="http://www.w3.org/1999/xhtml">\n\n http-equiv="Content-Type" c > > ---------------------------------------------------------------------- > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev > From p.j.a.cock at googlemail.com Fri Apr 18 20:24:53 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Fri, 18 Apr 2014 21:24:53 +0100 Subject: [Biopython-dev] TogoWS test failure, was: Compiling Biopython on 64 bit Windows In-Reply-To: References: Message-ID: Thanks Lenna, Strange - although I was sure it was working perfectly on Linux at work this afternoon, I can reproduce this as well now. This commit adds a more informative error message: https://github.com/biopython/biopython/commit/e5c656e7ce7496db0291f0055da61c492c5054aa e.g. ValueError: Expected an integer from URL http://togows.dbcls.jp/search/pubmed/TogoWS/count, got: '\n\n\n \n\n\n
1
\n\n\n\n' i.e. Rather than plain text, it is HTML with the value inside a
tag. This may be a content negotiation problem, but I will raise this with the TogoWS team. Manlio - we can ignore this for now for 64 bit Windows testing, Thanks, Peter On Fri, Apr 18, 2014 at 7:53 PM, Lenna Peterson wrote: > I see similar errors (Ubuntu 12.04, Python 2.7.3, university network). > > I can test whether the same thing happens while I'm off-campus. > > Lenna > > > On Fri, Apr 18, 2014 at 2:26 PM, Peter Cock > wrote: >> >> On Thu, Apr 17, 2014 at 7:57 AM, Manlio Calvi >> wrote: >> > Update from the first round of fights :) >> > >> >> Following a successful build (the emails didn't get CC'd to >> the list - can we forward them?), Manlio reported some test >> failures including with TogoWS (both Python 2.7 and 3.4), >> pasted at the end of this email. >> >> Something is amiss in the call to Bio.TogoWS.search_count(...) >> where we expect TogoWS to return a plain text file with an >> integer count in it. >> >> Somehow some XHTML is being returned instead. >> >> Has anyone else seen this before? I wonder if it is a proxy >> problem or something like that? >> >> Thanks, >> >> Peter >> >> -- >> >> >> ====================================================================== >> ERROR: test_pubmed_search_bioruby (test_TogoWS.TogoSearch) >> Bio.TogoWS.search_iter("pubmed", "BioRuby") etc >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", >> line 425, in test_pubmed_search_bioruby >> "20739307", "20015970", "14693808"]) >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", >> line 463, in check >> search_count = TogoWS.search_count(database, search_term) >> File >> "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\TogoWS\__init__.py", >> line 167, in search_count >> count = int(handle.read().strip()) >> ValueError: invalid literal for int() with base 10: '> PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" >> "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n> xmlns="http://www.w3.org/1999/xhtml">\n\n > http-equiv="Content-Type" c >> >> ====================================================================== >> ERROR: test_pubmed_search_porin (test_TogoWS.TogoSearch) >> Bio.TogoWS.search_iter("pubmed", "human porin") etc >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", >> line 434, in test_pubmed_search_porin >> self.check("pubmed", "human porin", ["21189321", "21835183"]) >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", >> line 463, in check >> search_count = TogoWS.search_count(database, search_term) >> File >> "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\TogoWS\__init__.py", >> line 167, in search_count >> count = int(handle.read().strip()) >> ValueError: invalid literal for int() with base 10: '> PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" >> "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n> xmlns="http://www.w3.org/1999/xhtml">\n\n > http-equiv="Content-Type" c >> >> ====================================================================== >> ERROR: test_pubmed_search_togows (test_TogoWS.TogoSearch) >> Bio.TogoWS.search_iter("pubmed", "TogoWS") etc >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", >> line 420, in test_pubmed_search_togows >> self.check("pubmed", "TogoWS", ["20472643"]) >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", >> line 463, in check >> search_count = TogoWS.search_count(database, search_term) >> File >> "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\TogoWS\__init__.py", >> line 167, in search_count >> count = int(handle.read().strip()) >> ValueError: invalid literal for int() with base 10: '> PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" >> "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n> xmlns="http://www.w3.org/1999/xhtml">\n\n > http-equiv="Content-Type" c >> >> ---------------------------------------------------------------------- >> _______________________________________________ >> Biopython-dev mailing list >> Biopython-dev at lists.open-bio.org >> http://lists.open-bio.org/mailman/listinfo/biopython-dev > > From manlio.calvi at gmail.com Fri Apr 18 20:57:41 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Fri, 18 Apr 2014 22:57:41 +0200 Subject: [Biopython-dev] TogoWS test failure, was: Compiling Biopython on 64 bit Windows In-Reply-To: References: Message-ID: Maybe they did a slight change to the output and thought was so minor that no one get disturbed by it. However if remain in this form I think it's not too difficult extract it with a regex. I haven't time to look to the other issues in the Win 64 build raised, they are different from python 2.7 to 3.4. I'll build manually more or less once a day until we get the buildbot in order. If something else get raised up, I'll post here. Manlio On Fri, Apr 18, 2014 at 10:24 PM, Peter Cock wrote: > Thanks Lenna, > > Strange - although I was sure it was working perfectly on Linux > at work this afternoon, I can reproduce this as well now. > > This commit adds a more informative error message: > > https://github.com/biopython/biopython/commit/e5c656e7ce7496db0291f0055da61c492c5054aa > > e.g. > > ValueError: Expected an integer from URL > http://togows.dbcls.jp/search/pubmed/TogoWS/count, got: ' html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" > "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n xmlns="http://www.w3.org/1999/xhtml">\n\n http-equiv="Content-Type" content="text/html; charset=UTF-8" > />\n\n\n
1
\n\n\n\n' > > i.e. Rather than plain text, it is HTML with the value > inside a
tag. This may be a content negotiation > problem, but I will raise this with the TogoWS team. > > Manlio - we can ignore this for now for 64 bit Windows > testing, > > Thanks, > > Peter > > On Fri, Apr 18, 2014 at 7:53 PM, Lenna Peterson > wrote: > > I see similar errors (Ubuntu 12.04, Python 2.7.3, university network). > > > > I can test whether the same thing happens while I'm off-campus. > > > > Lenna > > > > > > On Fri, Apr 18, 2014 at 2:26 PM, Peter Cock > > wrote: > >> > >> On Thu, Apr 17, 2014 at 7:57 AM, Manlio Calvi > >> wrote: > >> > Update from the first round of fights :) > >> > > >> > >> Following a successful build (the emails didn't get CC'd to > >> the list - can we forward them?), Manlio reported some test > >> failures including with TogoWS (both Python 2.7 and 3.4), > >> pasted at the end of this email. > >> > >> Something is amiss in the call to Bio.TogoWS.search_count(...) > >> where we expect TogoWS to return a plain text file with an > >> integer count in it. > >> > >> Somehow some XHTML is being returned instead. > >> > >> Has anyone else seen this before? I wonder if it is a proxy > >> problem or something like that? > >> > >> Thanks, > >> > >> Peter > >> > >> -- > >> > >> > >> ====================================================================== > >> ERROR: test_pubmed_search_bioruby (test_TogoWS.TogoSearch) > >> Bio.TogoWS.search_iter("pubmed", "BioRuby") etc > >> ---------------------------------------------------------------------- > >> Traceback (most recent call last): > >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > >> line 425, in test_pubmed_search_bioruby > >> "20739307", "20015970", "14693808"]) > >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > >> line 463, in check > >> search_count = TogoWS.search_count(database, search_term) > >> File > >> > "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\TogoWS\__init__.py", > >> line 167, in search_count > >> count = int(handle.read().strip()) > >> ValueError: invalid literal for int() with base 10: ' >> PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" > >> "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n >> xmlns="http://www.w3.org/1999/xhtml">\n\n >> http-equiv="Content-Type" c > >> > >> ====================================================================== > >> ERROR: test_pubmed_search_porin (test_TogoWS.TogoSearch) > >> Bio.TogoWS.search_iter("pubmed", "human porin") etc > >> ---------------------------------------------------------------------- > >> Traceback (most recent call last): > >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > >> line 434, in test_pubmed_search_porin > >> self.check("pubmed", "human porin", ["21189321", "21835183"]) > >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > >> line 463, in check > >> search_count = TogoWS.search_count(database, search_term) > >> File > >> > "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\TogoWS\__init__.py", > >> line 167, in search_count > >> count = int(handle.read().strip()) > >> ValueError: invalid literal for int() with base 10: ' >> PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" > >> "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n >> xmlns="http://www.w3.org/1999/xhtml">\n\n >> http-equiv="Content-Type" c > >> > >> ====================================================================== > >> ERROR: test_pubmed_search_togows (test_TogoWS.TogoSearch) > >> Bio.TogoWS.search_iter("pubmed", "TogoWS") etc > >> ---------------------------------------------------------------------- > >> Traceback (most recent call last): > >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > >> line 420, in test_pubmed_search_togows > >> self.check("pubmed", "TogoWS", ["20472643"]) > >> File "C:\Users\Public\BioPython3\biopython\Tests\test_TogoWS.py", > >> line 463, in check > >> search_count = TogoWS.search_count(database, search_term) > >> File > >> > "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\TogoWS\__init__.py", > >> line 167, in search_count > >> count = int(handle.read().strip()) > >> ValueError: invalid literal for int() with base 10: ' >> PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" > >> "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n >> xmlns="http://www.w3.org/1999/xhtml">\n\n >> http-equiv="Content-Type" c > >> > >> ---------------------------------------------------------------------- > >> _______________________________________________ > >> Biopython-dev mailing list > >> Biopython-dev at lists.open-bio.org > >> http://lists.open-bio.org/mailman/listinfo/biopython-dev > > > > > From manlio.calvi at gmail.com Fri Apr 18 18:40:43 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Fri, 18 Apr 2014 20:40:43 +0200 Subject: [Biopython-dev] Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? In-Reply-To: References: <20140417124408.5af66518@lnx> Message-ID: And at last of this forward I'm posting the build and tests logs (more a dump than a log) for those who want peek into it. As you'll see the only consistent error was the TogoWS' XHTML error. Cheers, Manlio On Fri, Apr 18, 2014 at 8:35 PM, Manlio Calvi wrote: > > > ---------- Forwarded message ---------- > From: Peter Cock > Date: Fri, Apr 18, 2014 at 12:01 PM > Subject: Re: Compiling Biopython on 64 bit Windows, was: Volunteer > buildslave machines? > To: Manlio Calvi > Cc: Tiago Antao > > Manlio > > Excellent work - well done :) > > We should add some of this to the "Installing from source on Windows" > section of the installation notes (which we also need to update regarding > newer versions of Python and which compiler works): > > > https://github.com/biopython/biopython/blob/master/Doc/install/Installation.tex > > Step 6 sounds familiar, I think there is a similar batch file to set the > environment variables for the 43bit MSVC compiler. > > Regarding step 7, Python should use separate subdirectoroes under > build/ for each version of Python, so that shouldn't be a needed. > However, if you do use the same folder some of our tests use > hard coded temporary filenames so it is best to only test one version > of Python at a time. > > Regarding step 8, normally on Windows I don't bother with setting the > $PATH environment variable and simply call the explicit version of > Python I am testing, e.g. > > C:\Python27\python.exe setup.py build > C:\Python27\python.exe setup.py test > C:\Python27\python.exe setup.py install > > (This is what the current buildbot instructions do where we > have automated the tests). > > As to what's text - let's try to fix these test failures :) > Can you post the test output please? > > TogoWS is an online service, so from time to time it does have > transient failures (so does the NCBI Entrez API). The fact this > broke under both Python 2.7 and 3.4 suggests this is not a > Python 3 issue. > > I am a little surprised you didn't see this problem with Python 3.4, > but Windows does unicode a little differently so that may be why? > https://github.com/biopython/biopython/issues/305 > > How did you install git & clone Biopython? Specifically, what > is it doing about text files (i.e. Unix versus DOS/Windows > new lines). This was something which has caused test > failures in the past - perhaps some new cases have arisen. > > Thanks! > > Peter > > -------------- next part -------------- A non-text attachment was scrubbed... Name: test27.log Type: application/octet-stream Size: 13944 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: build27.log Type: application/octet-stream Size: 63254 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test34.log Type: application/octet-stream Size: 29282 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: build34.log Type: application/octet-stream Size: 63350 bytes Desc: not available URL: From p.j.a.cock at googlemail.com Sat Apr 19 12:23:35 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Sat, 19 Apr 2014 13:23:35 +0100 Subject: [Biopython-dev] TogoWS test failure, was: Compiling Biopython on 64 bit Windows In-Reply-To: References: Message-ID: The TogoWS team confirmed this is a problem with the server and expect it to be fixed shortly. Peter On Fri, Apr 18, 2014 at 9:57 PM, Manlio Calvi wrote: > Maybe they did a slight change to the output and thought was so minor that > no one get disturbed by it. However if remain in this form I think it's not > too difficult extract it with a regex. > > I haven't time to look to the other issues in the Win 64 build raised, they > are different from python 2.7 to 3.4. > > I'll build manually more or less once a day until we get the buildbot in > order. If something else get raised up, I'll post here. > > Manlio > > > On Fri, Apr 18, 2014 at 10:24 PM, Peter Cock > wrote: >> >> Thanks Lenna, >> >> Strange - although I was sure it was working perfectly on Linux >> at work this afternoon, I can reproduce this as well now. >> >> This commit adds a more informative error message: >> >> https://github.com/biopython/biopython/commit/e5c656e7ce7496db0291f0055da61c492c5054aa >> >> e.g. >> >> ValueError: Expected an integer from URL >> http://togows.dbcls.jp/search/pubmed/TogoWS/count, got: '> html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" >> "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n> xmlns="http://www.w3.org/1999/xhtml">\n\n > http-equiv="Content-Type" content="text/html; charset=UTF-8" >> />\n\n\n
1
\n\n\n\n' >> >> i.e. Rather than plain text, it is HTML with the value >> inside a
tag. This may be a content negotiation >> problem, but I will raise this with the TogoWS team. >> >> Manlio - we can ignore this for now for 64 bit Windows >> testing, >> >> Thanks, >> >> Peter >> From p.j.a.cock at googlemail.com Mon Apr 21 13:37:54 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 21 Apr 2014 14:37:54 +0100 Subject: [Biopython-dev] C code review - Bio.trie fix for Python 3.4 Message-ID: Hi all, Do we have any Python 3 bytes/unicode C API experts lurking on the list who could review my patch here please? https://github.com/biopython/biopython/issues/305 Thanks, Peter From manlio.calvi at gmail.com Mon Apr 21 14:41:39 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 21 Apr 2014 16:41:39 +0200 Subject: [Biopython-dev] C code review - Bio.trie fix for Python 3.4 In-Reply-To: References: Message-ID: I don't know if it is related (but probably is) but I understood why I don't get a trie error in my Test config... simply the module don't compile itself! Looking at the build dump seems to be compiled (with a lot of warning and deprecations) My test machine is Win7 x64, Python 3.4 x64 binary from Python.org That's is the command I enter and the copy of the test dump I obtained ############################################################## C:\Users\Public\BioPython3\biopython>python Tests/run_tests.py test_trie.py > TestTrie.log 2>&1 test_trie ... skipping. Could not import Bio.trie, check C code was compiled. ---------------------------------------------------------------------- Ran 1 test in 0.006 seconds Python version: 3.4.0 (v3.4.0:04f714765c13, Mar 16 2014, 19:25:23) [MSC v.1600 64 bit (AMD64)] Operating system: nt win32 ################################## What follows is part of the build dump related to the trie c module,instead ################################################ building 'Bio.trie' extension C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IBio -IC:\Python34\include -IC:\Python34\include /TcBio/triemodule.c /Fobuild\temp.win-amd64-3.4\Release\Bio/triemodule.obj triemodule.c Bio/triemodule.c(578) : warning C4133: 'initializing' : incompatible types - from 'long (__cdecl *)(PyObject *)' to 'hashfunc' Bio/triemodule.c(750) : warning C4244: 'function' : conversion from 'Py_ssize_t' to 'const int', possible loss of data C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IBio -IC:\Python34\include -IC:\Python34\include /TcBio/trie.c /Fobuild\temp.win-amd64-3.4\Release\Bio/trie.obj trie.c Bio/trie.c(11) : warning C4996: 'strcpy': This function or variable may be unsafe. Consider using strcpy_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details. Bio/trie.c(161) : warning C4996: 'strncpy': This function or variable may be unsafe. Consider using strncpy_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details. C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE\string.h(157) : see declaration of 'strncpy' Bio/trie.c(294) : warning C4267: 'initializing' : conversion from 'size_t' to 'int', possible loss of data Bio/trie.c(310) : warning C4996: 'strncat': This function or variable may be unsafe. Consider using strncat_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details. C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE\string.h(144) : see declaration of 'strncat' Bio/trie.c(379) : warning C4267: 'initializing' : conversion from 'size_t' to 'int', possible loss of data Bio/trie.c(382) : warning C4996: 'strcat': This function or variable may be unsafe. Consider using strcat_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details. Bio/trie.c(395) : warning C4267: 'function' : conversion from 'size_t' to 'const int', possible loss of data Bio/trie.c(466) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data Bio/trie.c(467) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data Bio/trie.c(494) : warning C4267: 'initializing' : conversion from 'size_t' to 'int', possible loss of data Bio/trie.c(501) : warning C4996: 'strcat': This function or variable may be unsafe. Consider using strcat_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details. Bio/trie.c(547) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data Bio/trie.c(548) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data Bio/trie.c(575) : warning C4267: 'initializing' : conversion from 'size_t' to 'int', possible loss of data Bio/trie.c(581) : warning C4996: 'strncat': This function or variable may be unsafe. Consider using strncat_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details. C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\INCLUDE\string.h(144) : see declaration of 'strncat' Bio/trie.c(669) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Python34\libs /LIBPATH:C:\Python34\PCbuild\amd64 /EXPORT:PyInit_trie build\temp.win-amd64-3.4\Release\Bio/triemodule.obj build\temp.win-amd64-3.4\Release\Bio/trie.obj /OUT:build\lib.win-amd64-3.4\Bio\trie.pyd /IMPLIB:build\temp.win-amd64-3.4\Release\Bio\trie.lib /MANIFESTFILE:build\temp.win-amd64-3.4\Release\Bio\trie.pyd.manifest Creating library build\temp.win-amd64-3.4\Release\Bio\trie.lib and object build\temp.win-amd64-3.4\Release\Bio\trie.exp On Mon, Apr 21, 2014 at 3:37 PM, Peter Cock wrote: > Hi all, > > Do we have any Python 3 bytes/unicode C API experts > lurking on the list who could review my patch here please? > > https://github.com/biopython/biopython/issues/305 > > Thanks, > > Peter > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev > From p.j.a.cock at googlemail.com Mon Apr 21 14:53:50 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 21 Apr 2014 15:53:50 +0100 Subject: [Biopython-dev] C code review - Bio.trie fix for Python 3.4 In-Reply-To: References: Message-ID: On Mon, Apr 21, 2014 at 3:41 PM, Manlio Calvi wrote: > I don't know if it is related (but probably is) ... but I understood why I don't > get a trie error in my Test config... simply the module don't compile > itself! > Looking at the build dump seems to be compiled (with a lot of warning and > deprecations) That is strange - all I can see is warnings, no errors. > My test machine is Win7 x64, Python 3.4 x64 binary from Python.org > Referring to your test log for Python 3.4 [*], it looks like other C code extensions are also not working (but they are for Python 2.7): running test test_Ace ... ok ... test_Cluster ... skipping. If you want to use Bio.Cluster, install NumPy first and then reinstall Biopython ... test_KDTree ... skipping. C module in Bio.KDTree not compiled ... test_PDB_KDTree ... skipping. C module in Bio.KDTree not compiled ... test_pairwise2 ... ok ... test_trie ... skipping. Could not import Bio.trie, check C code was compiled. Let's continue this on the other thread - its a bigger issue than the Bio.trie unicode problem under Python 3.4 :( Thanks, Peter [*] available on the archive here, http://lists.open-bio.org/pipermail/biopython-dev/2014-April/011265.html From p.j.a.cock at googlemail.com Mon Apr 21 14:59:46 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 21 Apr 2014 15:59:46 +0100 Subject: [Biopython-dev] Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? In-Reply-To: References: <20140417124408.5af66518@lnx> Message-ID: On Fri, Apr 18, 2014 at 7:40 PM, Manlio Calvi wrote: > And at last of this forward I'm posting the build and tests logs (more a > dump than a log) for those who want peek into it. > As you'll see the only consistent error was the TogoWS' XHTML error. > > Cheers, > Manlio Thanks Manlio, Following the discussion here, I've looked at your Python 3.4 test log in more detail - and sadly it looks like none of the C code extensions are also not working (but they are for Python 2.7): running test test_Ace ... ok ... test_Cluster ... skipping. If you want to use Bio.Cluster, install NumPy first and then reinstall Biopython ... test_KDTree ... skipping. C module in Bio.KDTree not compiled ... test_PDB_KDTree ... skipping. C module in Bio.KDTree not compiled ... test_pairwise2 ... ok ... test_trie ... skipping. Could not import Bio.trie, check C code was compiled. Note that test_pairwise2 is probably testing the pure Python fallback here (assuming its C code also didn't get used). My hunch in the absence of compilation errors, is something is amiss with Python 3.4 finding the compiled modules when we run the tests. Can you try installing Biopython under Python 3.4, and trying to import any of the C code modules? e.g. [Mac] $ python3.4 -c "from Bio import trie; print(trie.__file__)" /Users/pjcock/lib/python3.4/site-packages/Bio/trie.so [Linux] $ python3.4 -c "from Bio import trie; print(trie.__file__)" /home_local/buildslave/lib/python3.4/site-packages/Bio/trie.cpython-34m.so [32 bit Windows] > c:\python34\python -c "from Bio import trie; print(trie.__file__)" c:\python34\lib\site-packages\Bio\trie.pyd Thanks, Peter [*] available on the archive here, http://lists.open-bio.org/pipermail/biopython-dev/2014-April/011265.html From manlio.calvi at gmail.com Mon Apr 21 15:45:40 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 21 Apr 2014 17:45:40 +0200 Subject: [Biopython-dev] Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? In-Reply-To: References: <20140417124408.5af66518@lnx> Message-ID: > test_Cluster ... skipping. If you want to use Bio.Cluster, install NumPy first and then reinstall Biopython And that's strange, Numpy IS installed in my case and usable by other scripts... I tried to reinstall and rebuild, all seems the same as before. Apart from Numpy, seems the .pyd file is there but isn't importable by python as module as it raise an "ImportError" kind of error. This is the dump: #################### Traceback (most recent call last): File "", line 1, in ImportError: cannot import name 'trie' #################### Now, seems strange to me that File thing Manlio On Mon, Apr 21, 2014 at 4:59 PM, Peter Cock wrote: > On Fri, Apr 18, 2014 at 7:40 PM, Manlio Calvi wrote: >> And at last of this forward I'm posting the build and tests logs (more a >> dump than a log) for those who want peek into it. >> As you'll see the only consistent error was the TogoWS' XHTML error. >> >> Cheers, >> Manlio > > Thanks Manlio, > > Following the discussion here, I've looked at your Python 3.4 test > log in more detail - and sadly it looks like none of the C code > extensions are also not working (but they are for Python 2.7): > > running test > test_Ace ... ok > ... > test_Cluster ... skipping. If you want to use Bio.Cluster, install > NumPy first and then reinstall Biopython > ... > test_KDTree ... skipping. C module in Bio.KDTree not compiled > ... > test_PDB_KDTree ... skipping. C module in Bio.KDTree not compiled > ... > test_pairwise2 ... ok > ... > test_trie ... skipping. Could not import Bio.trie, check C code was compiled. > > > Note that test_pairwise2 is probably testing the pure Python > fallback here (assuming its C code also didn't get used). > > My hunch in the absence of compilation errors, is something is > amiss with Python 3.4 finding the compiled modules when we > run the tests. > > Can you try installing Biopython under Python 3.4, and trying > to import any of the C code modules? e.g. > > [Mac] $ python3.4 -c "from Bio import trie; print(trie.__file__)" > /Users/pjcock/lib/python3.4/site-packages/Bio/trie.so > > [Linux] $ python3.4 -c "from Bio import trie; print(trie.__file__)" > /home_local/buildslave/lib/python3.4/site-packages/Bio/trie.cpython-34m.so > > [32 bit Windows] > c:\python34\python -c "from Bio import trie; > print(trie.__file__)" > c:\python34\lib\site-packages\Bio\trie.pyd > > Thanks, > > Peter > > [*] available on the archive here, > http://lists.open-bio.org/pipermail/biopython-dev/2014-April/011265.html From p.j.a.cock at googlemail.com Mon Apr 21 16:12:20 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 21 Apr 2014 17:12:20 +0100 Subject: [Biopython-dev] Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? In-Reply-To: References: <20140417124408.5af66518@lnx> Message-ID: On Mon, Apr 21, 2014 at 4:45 PM, Manlio Calvi wrote: >> test_Cluster ... skipping. If you want to use Bio.Cluster, install NumPy first and then reinstall Biopython > > And that's strange, Numpy IS installed in my case and usable by other > scripts... I tried to reinstall and rebuild, all seems the same as > before. The error message is just triggered if "from Bio import Cluster" fails with an ImportError - much like all the other C code modules' imports are failing on your Python 3.4 setup. > Apart from Numpy, seems the .pyd file is there but isn't importable by > python as module as it raise an "ImportError" kind of error. > > This is the dump: > #################### > Traceback (most recent call last): > File "", line 1, in > ImportError: cannot import name 'trie' > #################### > > Now, seems strange to me that File thing > > Manlio What does this do? python3.4 -c "import numpy; print(numpy.__file__)" If it works, can you see any compiled modules in that folder, and how they are named? Peter From manlio.calvi at gmail.com Mon Apr 21 16:25:40 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 21 Apr 2014 18:25:40 +0200 Subject: [Biopython-dev] Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? In-Reply-To: References: <20140417124408.5af66518@lnx> Message-ID: Works as expected because I have Numpy modules running in other scripts (dumps the path to __init__.py) In C:\Python34\Lib\site-packages\Bio the non-py files are two: cpairwise2.pyd trie.pyd The same are in the folder I used to build them, and by the extension should be callable/importable. I tried to import cpairwise2 but (don't) work the same way, same error raised as before. On Mon, Apr 21, 2014 at 6:12 PM, Peter Cock wrote: > On Mon, Apr 21, 2014 at 4:45 PM, Manlio Calvi wrote: >>> test_Cluster ... skipping. If you want to use Bio.Cluster, install NumPy first and then reinstall Biopython >> >> And that's strange, Numpy IS installed in my case and usable by other >> scripts... I tried to reinstall and rebuild, all seems the same as >> before. > > The error message is just triggered if "from Bio import Cluster" > fails with an ImportError - much like all the other C code modules' > imports are failing on your Python 3.4 setup. > >> Apart from Numpy, seems the .pyd file is there but isn't importable by >> python as module as it raise an "ImportError" kind of error. >> >> This is the dump: >> #################### >> Traceback (most recent call last): >> File "", line 1, in >> ImportError: cannot import name 'trie' >> #################### >> >> Now, seems strange to me that File thing >> >> Manlio > > What does this do? > > python3.4 -c "import numpy; print(numpy.__file__)" > > If it works, can you see any compiled modules in that > folder, and how they are named? > > Peter From p.j.a.cock at googlemail.com Mon Apr 21 16:27:57 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 21 Apr 2014 17:27:57 +0100 Subject: [Biopython-dev] Pickle problem on 64 bit Windows with Python 3.4 Message-ID: Hi Manlio, One of the new errors you reported under Python 3.4 on Windows 7 was: ====================================================================== ERROR: test_SubsMat ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Users\Public\BioPython3\biopython\Tests\run_tests.py", line 408, in runTest suite = unittest.TestLoader().loadTestsFromName(name) File "C:\Python34\lib\unittest\loader.py", line 105, in loadTestsFromName module = __import__('.'.join(parts_copy)) File "C:\Users\Public\BioPython3\biopython\Tests\test_SubsMat.py", line 38, in acc_rep_mat = pickle.load(handle) _pickle.UnpicklingError: the STRING opcode argument must be quoted Except copied from test log here: http://lists.open-bio.org/pipermail/biopython-dev/2014-April/011265.html I suspect this could be a problem in Python itself, since pickle was changed in Python 3.4: https://docs.python.org/3.4/whatsnew/2.3.html#pep-307-pickle-enhancements http://www.python.org/dev/peps/pep-0307 Can you try this example on your machine - from my Mac: $ cd Tests/SubsMat $ python3.4 -c "import pickle; h = open('acc_rep_mat.pik', 'rb'); m = pickle.load(h); h.close(); print(m)" {('D', 'W'): 22, ..., ('A', 'E'): 207} >From 32 bit Windows XP, it also works for me: C:\repositories\biopython\Tests\SubsMat>c:\python34\python -c "import pickle; h = open('acc_rep_mat.pik', 'rb'); m = pickle.load(h); h.close(); print(m)" {('H', 'S'): 59, ..., ('K', 'P'): 94} Note the dictionary order is expected to change. Also I am not using a with statement in order to make a single command line friendly test you can copy & paste into the Windows terminal. It would be more work, but if you could try Python 3.3 that would be useful (and helpful if we get to run your desktop as a buildslave). Thanks, Peter From p.j.a.cock at googlemail.com Mon Apr 21 16:33:03 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 21 Apr 2014 17:33:03 +0100 Subject: [Biopython-dev] Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? In-Reply-To: References: <20140417124408.5af66518@lnx> Message-ID: On Mon, Apr 21, 2014 at 5:25 PM, Manlio Calvi wrote: > Works as expected because I have Numpy modules running in other > scripts (dumps the path to __init__.py) > > In C:\Python34\Lib\site-packages\Bio the non-py files are two: > cpairwise2.pyd > trie.pyd > > The same are in the folder I used to build them, and by the extension > should be callable/importable. > I tried to import cpairwise2 but (don't) work the same way, same error > raised as before. > So, if you've done an install & can't import the C modules, that tells me this isn't a simply problem in our test harness (run_tests.py does some magic with the Python path to try to test using the just built files rather than any installed files). This is a deeper problem with building/installing under Python 3.4 on your setup. I have for the moment run out of ideas :( (Other than trying Python 3.3 which may behave differently - and will be useful anyway for checking the other test failures and if they are in part due to the new Python version). Thanks, Peter From p.j.a.cock at googlemail.com Mon Apr 21 16:47:02 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 21 Apr 2014 17:47:02 +0100 Subject: [Biopython-dev] Python 2.7 failure from test_Tutorial.py, was: Compiling Biopython on 64 bit Windows Message-ID: Hi Manlio, Moving on to Python 2.7, apart from the TogoWS issue, http://lists.open-bio.org/pipermail/biopython-dev/2014-April/011259.html your system showed one other failure: ====================================================================== ERROR: test_doctests (test_Tutorial.TutorialTestCase) Run tutorial doctests. ---------------------------------------------------------------------- Traceback (most recent call last): File "test_Tutorial.py", line 167, in test_doctests ValueError: 1 Tutorial doctests failed: test_from_line_04424 ---------------------------------------------------------------------- You should be able to rerun the tutorial tests alone (from memory only if Biopython is installed) with just: c:\python27\python test_Tutorial.py This is from the following doctest in the Tutorial.tex file, %This example fails under PyPy 2.0, https://bugs.pypy.org/issue1546 %doctest examples lib:numpy \begin{verbatim} >>> import numpy as np >>> from Bio import AlignIO >>> alignment = AlignIO.read("PF05371_seed.sth", "stockholm") >>> align_array = np.array([list(rec) for rec in alignment], np.character) >>> align_array.shape (7, 52) \end{verbatim} Could you try running that example "by hand" and seeing what happens? Note the doctest framework is very very fragile and the slightest output difference will break a test. Thanks, Peter On Fri, Apr 18, 2014 at 7:40 PM, Manlio Calvi wrote: > And at last of this forward I'm posting the build and tests logs (more a > dump than a log) for those who want peek into it. > As you'll see the only consistent error was the TogoWS' XHTML error. > > Cheers, > Manlio > > > > On Fri, Apr 18, 2014 at 8:35 PM, Manlio Calvi wrote: > >> >> >> ---------- Forwarded message ---------- >> From: Peter Cock >> Date: Fri, Apr 18, 2014 at 12:01 PM >> Subject: Re: Compiling Biopython on 64 bit Windows, was: Volunteer >> buildslave machines? >> To: Manlio Calvi >> Cc: Tiago Antao >> >> Manlio >> >> Excellent work - well done :) >> >> We should add some of this to the "Installing from source on Windows" >> section of the installation notes (which we also need to update regarding >> newer versions of Python and which compiler works): >> >> >> https://github.com/biopython/biopython/blob/master/Doc/install/Installation.tex >> >> Step 6 sounds familiar, I think there is a similar batch file to set the >> environment variables for the 43bit MSVC compiler. >> >> Regarding step 7, Python should use separate subdirectoroes under >> build/ for each version of Python, so that shouldn't be a needed. >> However, if you do use the same folder some of our tests use >> hard coded temporary filenames so it is best to only test one version >> of Python at a time. >> >> Regarding step 8, normally on Windows I don't bother with setting the >> $PATH environment variable and simply call the explicit version of >> Python I am testing, e.g. >> >> C:\Python27\python.exe setup.py build >> C:\Python27\python.exe setup.py test >> C:\Python27\python.exe setup.py install >> >> (This is what the current buildbot instructions do where we >> have automated the tests). >> >> As to what's text - let's try to fix these test failures :) >> Can you post the test output please? >> >> TogoWS is an online service, so from time to time it does have >> transient failures (so does the NCBI Entrez API). The fact this >> broke under both Python 2.7 and 3.4 suggests this is not a >> Python 3 issue. >> >> I am a little surprised you didn't see this problem with Python 3.4, >> but Windows does unicode a little differently so that may be why? >> https://github.com/biopython/biopython/issues/305 >> >> How did you install git & clone Biopython? Specifically, what >> is it doing about text files (i.e. Unix versus DOS/Windows >> new lines). This was something which has caused test >> failures in the past - perhaps some new cases have arisen. >> >> Thanks! >> >> Peter >> >> > > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev > From manlio.calvi at gmail.com Mon Apr 21 17:00:34 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 21 Apr 2014 19:00:34 +0200 Subject: [Biopython-dev] Pickle problem on 64 bit Windows with Python 3.4 In-Reply-To: References: Message-ID: I think we hit a stumbling block here, and maybe a serious one (but we'll be sure with more tests), because this following error suggest me the way we used the pickling in the code changed or something strange happened in my config. Apart from installing a 3.3, I'll try to go experimental with 3.4, I want to understand if it is a problem with my build of 3.4 or if it is a general issue of 3.4 with some piece of BioPython. C:\Users\Public\BioPython3\biopython\Tests\SubsMat>python -c "import pickle; h = open('acc_rep_mat.pik', 'rb'); m = pickle.load(h);h.close();print(m); Traceback (most recent call last): File "", line 1, in _pickle.UnpicklingError: the STRING opcode argument must be quoted On Mon, Apr 21, 2014 at 6:27 PM, Peter Cock wrote: > Hi Manlio, > > One of the new errors you reported under Python 3.4 on Windows 7 was: > > ====================================================================== > ERROR: test_SubsMat > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Users\Public\BioPython3\biopython\Tests\run_tests.py", line > 408, in runTest > suite = unittest.TestLoader().loadTestsFromName(name) > File "C:\Python34\lib\unittest\loader.py", line 105, in loadTestsFromName > module = __import__('.'.join(parts_copy)) > File "C:\Users\Public\BioPython3\biopython\Tests\test_SubsMat.py", > line 38, in > acc_rep_mat = pickle.load(handle) > _pickle.UnpicklingError: the STRING opcode argument must be quoted > > Except copied from test log here: > http://lists.open-bio.org/pipermail/biopython-dev/2014-April/011265.html > > I suspect this could be a problem in Python itself, since pickle was > changed in Python 3.4: > > https://docs.python.org/3.4/whatsnew/2.3.html#pep-307-pickle-enhancements > http://www.python.org/dev/peps/pep-0307 > > Can you try this example on your machine - from my Mac: > > $ cd Tests/SubsMat > $ python3.4 -c "import pickle; h = open('acc_rep_mat.pik', 'rb'); m = > pickle.load(h); h.close(); print(m)" > {('D', 'W'): 22, ..., ('A', 'E'): 207} > > From 32 bit Windows XP, it also works for me: > > C:\repositories\biopython\Tests\SubsMat>c:\python34\python -c "import > pickle; h = open('acc_rep_mat.pik', 'rb'); m = pickle.load(h); > h.close(); print(m)" > {('H', 'S'): 59, ..., ('K', 'P'): 94} > > Note the dictionary order is expected to change. Also I am not > using a with statement in order to make a single command line > friendly test you can copy & paste into the Windows terminal. > > It would be more work, but if you could try Python 3.3 that would be > useful (and helpful if we get to run your desktop as a buildslave). > > Thanks, > > Peter From zruan1991 at gmail.com Mon Apr 21 17:02:36 2014 From: zruan1991 at gmail.com (Zheng Ruan) Date: Mon, 21 Apr 2014 13:02:36 -0400 Subject: [Biopython-dev] [biopython] Code for Codon Alignment GSoC (#259) In-Reply-To: References: Message-ID: Hi Peter and Eric, I just made some changes to my repository. The old codonalignment branch has been renamed to gsoc2013 for backup. I will write new code to the codonalignment branch. Eric: I think the whole module should be marked as experimental like the Bio.SearchIO module. I anticipate to enhance the codon alignment construction method as well as add more methods for dn/ds ratio estimation and Bayes Empirical Bayes this summer. I place a warning in the `__init__.py`. ( https://github.com/zruan/biopython/commit/1c9e20c8a15febb7fd9219c632afb96127a006f1 ) Peter: The NEWS file is updated ( https://github.com/zruan/biopython/commit/0dcb39bcfd48cc009edc290ceaaa3b2c2e8f806f ). I will update the wiki shortly and make the announcement. Do I need to make a separate pull request to make the above change to biopython master? Or you may help me do that? I am also planning to include my latex document to Tutorial.tex. I typically use the Latex-Suite plugin to write latex in vim. However, it took a long time to open the Tutorial.tex, which made it difficult to manage. Why not put the latex code of each chapter into different tex file and use \include{} to import them into the main tex file? Thank you! Zheng Ruan On Mon, Apr 21, 2014 at 6:41 AM, Peter Cock wrote: > Congratulations all - could you also add something to the NEWS file, email > the main list to announce this, and update the wiki as needed? > > ? > Reply to this email directly or view it on GitHub > . > From manlio.calvi at gmail.com Mon Apr 21 17:02:47 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 21 Apr 2014 19:02:47 +0200 Subject: [Biopython-dev] Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? In-Reply-To: References: <20140417124408.5af66518@lnx> Message-ID: I'm running the tests against a WinPython x64 3.3.3.3 at the moment (so a non-Python.org distro) and will see if that happens there too, I'm planning to install a native one(if the time permits). Manlio On Mon, Apr 21, 2014 at 6:33 PM, Peter Cock wrote: > On Mon, Apr 21, 2014 at 5:25 PM, Manlio Calvi wrote: >> Works as expected because I have Numpy modules running in other >> scripts (dumps the path to __init__.py) >> >> In C:\Python34\Lib\site-packages\Bio the non-py files are two: >> cpairwise2.pyd >> trie.pyd >> >> The same are in the folder I used to build them, and by the extension >> should be callable/importable. >> I tried to import cpairwise2 but (don't) work the same way, same error >> raised as before. >> > > So, if you've done an install & can't import the C modules, that > tells me this isn't a simply problem in our test harness (run_tests.py > does some magic with the Python path to try to test using the just > built files rather than any installed files). > > This is a deeper problem with building/installing under Python 3.4 > on your setup. I have for the moment run out of ideas :( > > (Other than trying Python 3.3 which may behave differently - and > will be useful anyway for checking the other test failures and if they > are in part due to the new Python version). > > Thanks, > > Peter From p.j.a.cock at googlemail.com Mon Apr 21 17:06:25 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 21 Apr 2014 18:06:25 +0100 Subject: [Biopython-dev] [biopython] Code for Codon Alignment GSoC (#259) In-Reply-To: References: Message-ID: On Mon, Apr 21, 2014 at 6:02 PM, Zheng Ruan wrote: > > Hi Peter and Eric, > > I just made some changes to my repository. The old codonalignment branch has been > renamed to gsoc2013 for backup. I will write new code to the codonalignment branch. OK. You could write a final GSoC blog post about this too ;) > Eric: > I think the whole module should be marked as experimental like the Bio.SearchIO > module. I anticipate to enhance the codon alignment construction method as well > as add more methods for dn/ds ratio estimation and Bayes Empirical Bayes this > summer. > > I place a warning in the `__init__.py`. (https://github.com/zruan/biopython/commit/1c9e20c8a15febb7fd9219c632afb96127a006f1) That sounds wise. > Peter: > The NEWS file is updated (https://github.com/zruan/biopython/commit/0dcb39bcfd48cc009edc290ceaaa3b2c2e8f806f). > I will update the wiki shortly and make the announcement. > > Do I need to make a separate pull request to make the above change to biopython > master? Or you may help me do that? I think Eric or I can do that from those links, Eric - do you want to do this? > I am also planning to include my latex document to Tutorial.tex. I typically use > the Latex-Suite plugin to write latex in vim. However, it took a long time to open the > Tutorial.tex, which made it difficult to manage. Why not put the latex code of each > chapter into different tex file and use \include{} to import them into the main tex file? > > Thank you! > Zheng Ruan We could split the Tutorial.tex file like that - although it would mean a bit of extra work on test_Tutorial.py (but with the benefit of getting per-chapter line numbers). Thank you, Peter From p.j.a.cock at googlemail.com Mon Apr 21 17:12:34 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 21 Apr 2014 18:12:34 +0100 Subject: [Biopython-dev] Pickle problem on 64 bit Windows with Python 3.4 In-Reply-To: References: Message-ID: On Mon, Apr 21, 2014 at 6:00 PM, Manlio Calvi wrote: > I think we hit a stumbling block here, and maybe a serious one (but > we'll be sure with more tests), because this following error suggest > me the way we used the pickling in the code changed or something > strange happened in my config. > Apart from installing a 3.3, I'll try to go experimental with 3.4, I > want to understand if it is a problem with my build of 3.4 or if it is > a general issue of 3.4 with some piece of BioPython. > > C:\Users\Public\BioPython3\biopython\Tests\SubsMat>python -c "import pickle; h = > open('acc_rep_mat.pik', 'rb'); m = pickle.load(h);h.close();print(m); > > Traceback (most recent call last): > File "", line 1, in > _pickle.UnpicklingError: the STRING opcode argument must be quoted > Good - I was hoping that would happen - we have a clear test case separate from the Biopython code now. The pickle file acc_rep_mat.pik was created long ago (15 Nov 2000) with Python 2, and I would have hoped it would be forward compatible. This could be a regression in Python 3.4 itself (under Windows 64 bit only)? If anyone else has a 64 bit Windows machine and Python 3.4, please try this and let us know what happens. Thanks, Peter From manlio.calvi at gmail.com Mon Apr 21 17:45:43 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 21 Apr 2014 19:45:43 +0200 Subject: [Biopython-dev] Pickle problem on 64 bit Windows with Python 3.4 In-Reply-To: References: Message-ID: >From what I read here http://hg.python.org/cpython/rev/4a6b8f86b081 could be a problem related to that file. Seems to me they stripped the check for a quote that must be in, and looking at the pickle apparently isn't On Mon, Apr 21, 2014 at 7:12 PM, Peter Cock wrote: > On Mon, Apr 21, 2014 at 6:00 PM, Manlio Calvi wrote: >> I think we hit a stumbling block here, and maybe a serious one (but >> we'll be sure with more tests), because this following error suggest >> me the way we used the pickling in the code changed or something >> strange happened in my config. >> Apart from installing a 3.3, I'll try to go experimental with 3.4, I >> want to understand if it is a problem with my build of 3.4 or if it is >> a general issue of 3.4 with some piece of BioPython. >> >> C:\Users\Public\BioPython3\biopython\Tests\SubsMat>python -c "import pickle; h = >> open('acc_rep_mat.pik', 'rb'); m = pickle.load(h);h.close();print(m); >> >> Traceback (most recent call last): >> File "", line 1, in >> _pickle.UnpicklingError: the STRING opcode argument must be quoted >> > > Good - I was hoping that would happen - we have a clear test > case separate from the Biopython code now. > > The pickle file acc_rep_mat.pik was created long ago (15 Nov 2000) > with Python 2, and I would have hoped it would be forward compatible. > This could be a regression in Python 3.4 itself (under Windows 64 bit > only)? > > If anyone else has a 64 bit Windows machine and Python 3.4, > please try this and let us know what happens. > > Thanks, > > Peter From manlio.calvi at gmail.com Mon Apr 21 21:51:28 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 21 Apr 2014 23:51:28 +0200 Subject: [Biopython-dev] Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? In-Reply-To: References: <20140417124408.5af66518@lnx> Message-ID: OK test on WinPython concluded. Analyzing it we have, the expected TogoWS problem, a skipped test for a non-compiled C module, two tests really but dependant from the same uncompiled module ( ################################### test_KDTree ... skipping. C module in Bio.KDTree not compiled ################################### and this errors: ====================================================================== ERROR: test_unicode_exception (test_Phylo.IOTests) ---------------------------------------------------------------------- Traceback (most recent call last): File ".\test_Phylo.py", line 57, in test_unicode_exception tree = Phylo.read(EX_NEWICK_BOM, 'newick') File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\_io.py", line 65, in read tree = next(tree_gen) File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\_io.py", line 53, in parse for tree in getattr(supported_formats[format], 'parse')(fp, **kwargs): File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\NewickIO.py", line 107, in parse for line in self.handle: File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 15: character maps to ====================================================================== ERROR: test_write_0 (test_Phylo_CDAO.WriterTests) Write and re-parse the phylogenies in test.cdao. ---------------------------------------------------------------------- Traceback (most recent call last): File ".\test_Phylo_CDAO.py", line 62, in test_write CDAOIO.write([t1], outfile) File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\CDAOIO.py", line 74, in write return Writer(trees).write(handle, plain=plain, **kwargs) File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\CDAOIO.py", line 268, in write handle.write('@prefix %s: <%s> .\n' % (k, v)) TypeError: 'str' does not support the buffer interface ################################################# The test output for CDAO reports a series of slightly messages (more than an hundred) that looks like this: file://C:\WinPython3.3\3.3 - Biopython test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Tests\CDAO\test.cdao/tree0000001#node0000001 does not look like a valid URI, trying to serialize this will break. Phylo.py related files was a recent commit, right? Manlio On Mon, Apr 21, 2014 at 7:02 PM, Manlio Calvi wrote: > I'm running the tests against a WinPython x64 3.3.3.3 at the moment > (so a non-Python.org distro) and will see if that happens there too, > I'm planning to install a native one(if the time permits). > > Manlio > > On Mon, Apr 21, 2014 at 6:33 PM, Peter Cock wrote: >> On Mon, Apr 21, 2014 at 5:25 PM, Manlio Calvi wrote: >>> Works as expected because I have Numpy modules running in other >>> scripts (dumps the path to __init__.py) >>> >>> In C:\Python34\Lib\site-packages\Bio the non-py files are two: >>> cpairwise2.pyd >>> trie.pyd >>> >>> The same are in the folder I used to build them, and by the extension >>> should be callable/importable. >>> I tried to import cpairwise2 but (don't) work the same way, same error >>> raised as before. >>> >> >> So, if you've done an install & can't import the C modules, that >> tells me this isn't a simply problem in our test harness (run_tests.py >> does some magic with the Python path to try to test using the just >> built files rather than any installed files). >> >> This is a deeper problem with building/installing under Python 3.4 >> on your setup. I have for the moment run out of ideas :( >> >> (Other than trying Python 3.3 which may behave differently - and >> will be useful anyway for checking the other test failures and if they >> are in part due to the new Python version). >> >> Thanks, >> >> Peter From manlio.calvi at gmail.com Tue Apr 22 05:51:45 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Tue, 22 Apr 2014 07:51:45 +0200 Subject: [Biopython-dev] Python 2.7 failure from test_Tutorial.py, was: Compiling Biopython on 64 bit Windows In-Reply-To: References: Message-ID: Hi Peter, I just run this one from the Tests folder (the tests are there and don'tget moved in the installation) I got this: Running Tutorial doctests... ********************************************************************** File "Tests/test_Tutorial.py", line 153, in __main__.TutorialDocTestHolder.doctest_test_from_line_04424 Failed example: align_array.shape Expected: (7, 52) Got: (7L, 52L) ********************************************************************** 1 items had failures: 1 of 7 in __main__.TutorialDocTestHolder.doctest_test_from_line_04424 ***Test Failed*** 1 failures. C:\python27\lib\site-packages\Bio\Motif\__init__.py:26: BiopythonDeprecationWarning: The module Bio.Motif has been deprecated and will be removed in a future release of Biopython. Instead please use the new module Bio.motifs instead. Please be aware that though the functionality of Bio.Motif is retained (and extended) in Bio.motifs, usage may be different. BiopythonDeprecationWarning) Traceback (most recent call last): File "Tests/test_Tutorial.py", line 185, in RuntimeError: 1/806 tests failed Another Python2/3 compatibility problem... On Mon, Apr 21, 2014 at 6:47 PM, Peter Cock wrote: > Hi Manlio, > > Moving on to Python 2.7, apart from the TogoWS issue, > http://lists.open-bio.org/pipermail/biopython-dev/2014-April/011259.html > your system showed one other failure: > > ====================================================================== > ERROR: test_doctests (test_Tutorial.TutorialTestCase) > Run tutorial doctests. > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "test_Tutorial.py", line 167, in test_doctests > ValueError: 1 Tutorial doctests failed: test_from_line_04424 > > ---------------------------------------------------------------------- > > > You should be able to rerun the tutorial tests alone (from memory > only if Biopython is installed) with just: > > c:\python27\python test_Tutorial.py > > This is from the following doctest in the Tutorial.tex file, > > %This example fails under PyPy 2.0, https://bugs.pypy.org/issue1546 > %doctest examples lib:numpy > \begin{verbatim} >>>> import numpy as np >>>> from Bio import AlignIO >>>> alignment = AlignIO.read("PF05371_seed.sth", "stockholm") >>>> align_array = np.array([list(rec) for rec in alignment], np.character) >>>> align_array.shape > (7, 52) > \end{verbatim} > > Could you try running that example "by hand" and seeing what > happens? Note the doctest framework is very very fragile and > the slightest output difference will break a test. > > Thanks, > > Peter > > On Fri, Apr 18, 2014 at 7:40 PM, Manlio Calvi wrote: >> And at last of this forward I'm posting the build and tests logs (more a >> dump than a log) for those who want peek into it. >> As you'll see the only consistent error was the TogoWS' XHTML error. >> >> Cheers, >> Manlio >> >> >> >> On Fri, Apr 18, 2014 at 8:35 PM, Manlio Calvi wrote: >> >>> >>> >>> ---------- Forwarded message ---------- >>> From: Peter Cock >>> Date: Fri, Apr 18, 2014 at 12:01 PM >>> Subject: Re: Compiling Biopython on 64 bit Windows, was: Volunteer >>> buildslave machines? >>> To: Manlio Calvi >>> Cc: Tiago Antao >>> >>> Manlio >>> >>> Excellent work - well done :) >>> >>> We should add some of this to the "Installing from source on Windows" >>> section of the installation notes (which we also need to update regarding >>> newer versions of Python and which compiler works): >>> >>> >>> https://github.com/biopython/biopython/blob/master/Doc/install/Installation.tex >>> >>> Step 6 sounds familiar, I think there is a similar batch file to set the >>> environment variables for the 43bit MSVC compiler. >>> >>> Regarding step 7, Python should use separate subdirectoroes under >>> build/ for each version of Python, so that shouldn't be a needed. >>> However, if you do use the same folder some of our tests use >>> hard coded temporary filenames so it is best to only test one version >>> of Python at a time. >>> >>> Regarding step 8, normally on Windows I don't bother with setting the >>> $PATH environment variable and simply call the explicit version of >>> Python I am testing, e.g. >>> >>> C:\Python27\python.exe setup.py build >>> C:\Python27\python.exe setup.py test >>> C:\Python27\python.exe setup.py install >>> >>> (This is what the current buildbot instructions do where we >>> have automated the tests). >>> >>> As to what's text - let's try to fix these test failures :) >>> Can you post the test output please? >>> >>> TogoWS is an online service, so from time to time it does have >>> transient failures (so does the NCBI Entrez API). The fact this >>> broke under both Python 2.7 and 3.4 suggests this is not a >>> Python 3 issue. >>> >>> I am a little surprised you didn't see this problem with Python 3.4, >>> but Windows does unicode a little differently so that may be why? >>> https://github.com/biopython/biopython/issues/305 >>> >>> How did you install git & clone Biopython? Specifically, what >>> is it doing about text files (i.e. Unix versus DOS/Windows >>> new lines). This was something which has caused test >>> failures in the past - perhaps some new cases have arisen. >>> >>> Thanks! >>> >>> Peter >>> >>> >> >> _______________________________________________ >> Biopython-dev mailing list >> Biopython-dev at lists.open-bio.org >> http://lists.open-bio.org/mailman/listinfo/biopython-dev >> From p.j.a.cock at googlemail.com Tue Apr 22 08:32:32 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 09:32:32 +0100 Subject: [Biopython-dev] Python 2.7 failure from test_Tutorial.py, was: Compiling Biopython on 64 bit Windows In-Reply-To: References: Message-ID: On Tue, Apr 22, 2014 at 6:51 AM, Manlio Calvi wrote: > Hi Peter, > I just run this one from the Tests folder (the tests are there and > don'tget moved in the installation) Sorry, yes - I should have said that. > I got this: > > Running Tutorial doctests... > ********************************************************************** > File "Tests/test_Tutorial.py", line 153, in > __main__.TutorialDocTestHolder.doctest_test_from_line_04424 > Failed example: > align_array.shape > Expected: > (7, 52) > Got: > (7L, 52L) > ********************************************************************** > 1 items had failures: > 1 of 7 in __main__.TutorialDocTestHolder.doctest_test_from_line_04424 > ***Test Failed*** 1 failures. > C:\python27\lib\site-packages\Bio\Motif\__init__.py:26: > BiopythonDeprecationWarning: The module Bio.Motif has been deprecated > and will be removed in a future release of Biopython. Instead please > use the new module Bio.motifs instead. Please be aware that though the > functionality of Bio.Motif is retained (and extended) in Bio.motifs, > usage may be different. > BiopythonDeprecationWarning) > Traceback (most recent call last): > File "Tests/test_Tutorial.py", line 185, in > RuntimeError: 1/806 tests failed > > > Another Python2/3 compatibility problem... Well yes, but also a 32 bit versus 64 bit Windows issue. Python 3 hides the C level differences between ints and longs, while Python 2 can sometimes expose this - odd that this test didn't give longs on the other platforms, just 64 bit windows. I suggest we switch from: %This example fails under PyPy 2.0, https://bugs.pypy.org/issue1546 %doctest examples lib:numpy \begin{verbatim} >>> import numpy as np >>> from Bio import AlignIO >>> alignment = AlignIO.read("PF05371_seed. sth", "stockholm") >>> align_array = np.array([list(rec) for rec in alignment], np.character) >>> align_array.shape (7, 52) \end{verbatim} To: %This example fails under PyPy 2.0, https://bugs.pypy.org/issue1546 %doctest examples lib:numpy \begin{verbatim} >>> import numpy as np >>> from Bio import AlignIO >>> alignment = AlignIO.read("PF05371_seed.sth", "stockholm") >>> align_array = np.array([list(rec) for rec in alignment], np.character) >>> print("Array shape %i by %i" % align_array.shape) Array shape 7 by 52 \end{verbatim} That should be safe :) Peter From tra at popgen.net Tue Apr 22 09:23:16 2014 From: tra at popgen.net (Tiago Antao) Date: Tue, 22 Apr 2014 10:23:16 +0100 Subject: [Biopython-dev] Python 3.4 Message-ID: <20140422102316.5b049105@lnx> Dear all (especially Manlio and Peter ;) ), With Ubuntu trusty officially out, I am planning on doing a (Linux) docker file running Python 3.4 and attach it to buildbot. Hopefully today. This would allow us to have a stable testing environment of 3.4 against Linux... Which might give some clues to what is happening with 3.4 on Windows... Tiago From p.j.a.cock at googlemail.com Tue Apr 22 09:42:33 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 10:42:33 +0100 Subject: [Biopython-dev] Python 3.4 In-Reply-To: <20140422102316.5b049105@lnx> References: <20140422102316.5b049105@lnx> Message-ID: On Tue, Apr 22, 2014 at 10:23 AM, Tiago Antao wrote: > Dear all (especially Manlio and Peter ;) ), > > With Ubuntu trusty officially out, I am planning on doing a (Linux) > docker file running Python 3.4 and attach it to buildbot. Hopefully > today. This would allow us to have a stable testing environment of 3.4 > against Linux... Which might give some clues to what is happening with > 3.4 on Windows... > > Tiago Good plan :) I've already got Python 3.4 on three build slaves ready to be connected (Windows XP 32 bit, Linux 64 bit, and Mac OS X 64 bit), so we'll have the main platforms covered. Right now the only Python 3.4 specific issue I have seen is in Bio.trie https://github.com/biopython/biopython/issues/305 I am hoping we'll have everything ready on Python 3.4 for the next release of Biopython - aim for early May? Peter From p.j.a.cock at googlemail.com Tue Apr 22 09:57:23 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 10:57:23 +0100 Subject: [Biopython-dev] Compiling Biopython on 64 bit Windows, was: Volunteer buildslave machines? In-Reply-To: References: <20140417124408.5af66518@lnx> Message-ID: Hi Manlio, There are several issues here - On Mon, Apr 21, 2014 at 10:51 PM, Manlio Calvi wrote: > OK test on WinPython concluded. Analyzing it we have, the expected > TogoWS problem, (which the TogoWS team will be fixing shortly) > a skipped test for a non-compiled C module, two tests really but > dependant from the same uncompiled module ( > ################################### > test_KDTree ... skipping. C module in Bio.KDTree not compiled > ################################### (plus the other C code modules not getting compiled/installed, which is a concern) > and this errors: > > ====================================================================== > ERROR: test_unicode_exception (test_Phylo.IOTests) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File ".\test_Phylo.py", line 57, in test_unicode_exception > tree = Phylo.read(EX_NEWICK_BOM, 'newick') > File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\_io.py", > line 65, in read > tree = next(tree_gen) > File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\_io.py", > line 53, in parse > for tree in getattr(supported_formats[format], 'parse')(fp, **kwargs): > File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\NewickIO.py", > line 107, in parse > for line in self.handle: > File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\lib\encodings\cp1252.py", > line 23, in decode > return codecs.charmap_decode(input,self.errors,decoding_table)[0] > UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position > 15: character maps to > As you suggested below, the test_unicode_exception failure from test_Phylo.py is from a recent commit, and likely has to do with subtle Windows unicode differences: https://github.com/biopython/biopython/commit/94d1e54b80df9e0fbdd3b56b30e40b860e48f26f The 'good news' is this also broke under 32 bit Windows XP, http://testing.open-bio.org/biopython/builders/Windows%20XP%20-%20Python%203.3/builds/557/steps/shell/logs/stdio I have filed a bug: https://github.com/biopython/biopython/issues/311 > ====================================================================== > ERROR: test_write_0 (test_Phylo_CDAO.WriterTests) > Write and re-parse the phylogenies in test.cdao. > ---------------------------------------------------------------------- > Traceback (most recent call last): > File ".\test_Phylo_CDAO.py", line 62, in test_write > CDAOIO.write([t1], outfile) > File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\CDAOIO.py", > line 74, in write > return Writer(trees).write(handle, plain=plain, **kwargs) > File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\CDAOIO.py", > line 268, in write > handle.write('@prefix %s: <%s> .\n' % (k, v)) > TypeError: 'str' does not support the buffer interface > That looks like a string versus bytes issue with the handle (text versus binary mode), but I don't recall any changes in that code... maybe a question for Eric? > ################################################# > > > The test output for CDAO reports a series of slightly messages (more > than an hundred) that looks like this: > file://C:\WinPython3.3\3.3 - Biopython > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Tests\CDAO\test.cdao/tree0000001#node0000001 > does not look like a valid URI, trying to serialize this will break. My hunch is this warning could be due to the mixed slashes (forward and back) which is typically fine with the windows API for filenames, but evidently is confusing the CDAO code - a question for Ben? Thanks, Peter From manlio.calvi at gmail.com Tue Apr 22 09:57:52 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Tue, 22 Apr 2014 11:57:52 +0200 Subject: [Biopython-dev] Python 3.4 In-Reply-To: <20140422102316.5b049105@lnx> References: <20140422102316.5b049105@lnx> Message-ID: This is a good news Tiago :) My problem now is try to mantain a "standard build" of 3.4 x64 and another more "experimental" and separate from the first. The only way I know to have it "native" is using the virtual env that came with Python 3 that I never used before, and I don't know how reliable is in a testing environment. Apparently none of the python distribution went to 3.4 now. Seems I should learn how to use virtual envs or remain with my 3.4 only. What I would like to "experiment" is build up all the sources (or maybe only the modules sources) from the ground up, specifically I want to see if there is different behaviours using the modules from Golhke's site or not. Manlio On Tue, Apr 22, 2014 at 11:23 AM, Tiago Antao wrote: > Dear all (especially Manlio and Peter ;) ), > > With Ubuntu trusty officially out, I am planning on doing a (Linux) > docker file running Python 3.4 and attach it to buildbot. Hopefully > today. This would allow us to have a stable testing environment of 3.4 > against Linux... Which might give some clues to what is happening with > 3.4 on Windows... > > Tiago From p.j.a.cock at googlemail.com Tue Apr 22 10:44:18 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 11:44:18 +0100 Subject: [Biopython-dev] Pickle problem on 64 bit Windows with Python 3.4 In-Reply-To: References: Message-ID: On Mon, Apr 21, 2014 at 6:45 PM, Manlio Calvi wrote: > From what I read here http://hg.python.org/cpython/rev/4a6b8f86b081 could be > a problem related to that file. Seems to me they stripped the check for a > quote that must be in, and looking at the pickle apparently isn't > OK, now things are more confusing - this seems to be working on a colleague's machine, so it may be something different on your setup. Are you using a self compiled Python 3.4? We installed the 64 bit version Python 3.4 on Windows 7 using the binary installed from the website (Windows x86-64 MSI installer), selecting for all users (which probably requires admin rights): https://www.python.org/ftp/python/3.4.0/python-3.4.0.amd64.msi We manually downloaded the pickle file via the raw link on GitHub, and tried the test code (as shown below), and it worked perfectly. Regards, Peter -- C:\Users\xxx\Desktop>ls *.pik acc_rep_mat.pik C:\Users\xxx\Desktop>c:\python34\python.exe -c "import pickle; h = open('acc_rep_mat.pik', 'rb'); m = pickle.load(h);h.close();print(m);" {('K', 'Q'): 108, ('A', 'F'): 122, ('P', 'W'): 18, ('D', 'F'): 63, ('S', 'Y'): 7 3, ('C', 'M'): 14, ('A', 'N'): 139, ('K', 'Y'): 66, ('I', 'V'): 310, ('G', 'V'): 131, ('T', 'Y'): 91, ('M', 'T'): 35, ('F', 'V'): 160, ('D', 'Q'): 116, ('H', 'Y '): 43, ('R', 'T'): 114, ('A', 'Q'): 129, ('I', 'P'): 67, ('N', 'V'): 93, ('C', 'R'): 15, ('E', 'F'): 50, ('A', 'R'): 152, ('D', 'S'): 167, ('L', 'T'): 183, ('L ', 'P'): 79, ('I', 'M'): 88, ('C', 'K'): 24, ('E', 'K'): 164, ('G', 'N'): 147, ( 'P', 'T'): 92, ('L', 'R'): 109, ('H', 'T'): 69, ('E', 'G'): 146, ('I', 'T'): 88, ('L', 'Y'): 173, ('F', 'W'): 43, ('T', 'T'): 84, ('D', 'W'): 22, ('M', 'V'): 95 , ('K', 'T'): 140, ('W', 'Y'): 51, ('E', 'N'): 115, ('R', 'V'): 88, ('R', 'Y'): 62, ('D', 'Y'): 64, ('G', 'Q'): 89, ('D', 'T'): 125, ('F', 'K'): 67, ('W', 'W'): 32, ('C', 'H'): 10, ('R', 'W'): 23, ('I', 'W'): 38, ('A', 'H'): 66, ('L', 'V'): 447, ('Q', 'Q'): 42, ('D', 'P'): 105, ('D', 'H'): 52, ('M', 'W'): 25, ('S', 'T' ): 172, ('D', 'R'): 115, ('L', 'W'): 74, ('F', 'I'): 151, ('C', 'G'): 39, ('H', 'Q'): 44, ('C', 'V'): 59, ('N', 'P'): 94, ('D', 'E'): 163, ('C', 'E'): 26, ('R', 'R'): 96, ('G', 'K'): 148, ('C', 'S'): 23, ('F', 'Q'): 39, ('G', 'H'): 53, ('R' , 'S'): 94, ('D', 'I'): 66, ('G', 'R'): 81, ('N', 'Q'): 77, ('M', 'Q'): 27, ('D' , 'N'): 163, ('C', 'Q'): 16, ('Q', 'W'): 15, ('A', 'T'): 195, ('Y', 'Y'): 82, (' C', 'F'): 32, ('E', 'H'): 38, ('F', 'S'): 92, ('M', 'P'): 20, ('D', 'L'): 120, ( 'A', 'M'): 79, ('C', 'I'): 42, ('I', 'N'): 65, ('E', 'W'): 21, ('F', 'L'): 236, ('G', 'L'): 154, ('A', 'E'): 207, ('G', 'W'): 35, ('H', 'H'): 56, ('I', 'Q'): 62 , ('L', 'L'): 320, ('H', 'N'): 57, ('G', 'G'): 321, ('A', 'C'): 49, ('H', 'I'): 31, ('N', 'W'): 29, ('A', 'I'): 202, ('C', 'L'): 62, ('A', 'W'): 47, ('L', 'S'): 159, ('K', 'S'): 144, ('D', 'D'): 151, ('C', 'P'): 21, ('A', 'D'): 171, ('E', ' L'): 133, ('S', 'S'): 116, ('K', 'R'): 151, ('Q', 'V'): 69, ('L', 'M'): 153, ('A ', 'Y'): 124, ('E', 'T'): 150, ('K', 'P'): 94, ('D', 'V'): 95, ('F', 'G'): 102, ('H', 'L'): 70, ('H', 'K'): 63, ('C', 'Y'): 19, ('K', 'M'): 32, ('F', 'F'): 79, ('H', 'R'): 44, ('P', 'R'): 72, ('M', 'Y'): 41, ('A', 'P'): 119, ('G', 'S'): 161 , ('H', 'P'): 23, ('D', 'G'): 143, ('E', 'M'): 33, ('K', 'N'): 139, ('G', 'I'): 92, ('K', 'W'): 21, ('I', 'S'): 84, ('C', 'T'): 30, ('D', 'K'): 169, ('P', 'Y'): 60, ('I', 'R'): 58, ('M', 'S'): 41, ('N', 'N'): 66, ('N', 'T'): 105, ('F', 'H') : 39, ('F', 'T'): 69, ('F', 'M'): 67, ('C', 'W'): 8, ('D', 'M'): 20, ('K', 'K'): 121, ('Q', 'Y'): 41, ('V', 'W'): 44, ('C', 'C'): 42, ('M', 'M'): 22, ('T', 'W') : 35, ('A', 'V'): 252, ('F', 'N'): 55, ('P', 'V'): 109, ('I', 'K'): 80, ('H', 'V '): 54, ('N', 'R'): 90, ('M', 'N'): 34, ('E', 'Q'): 105, ('A', 'S'): 199, ('G', 'M'): 49, ('H', 'W'): 13, ('N', 'S'): 115, ('T', 'V'): 145, ('E', 'I'): 63, ('F' , 'R'): 57, ('Q', 'S'): 101, ('K', 'V'): 131, ('K', 'L'): 112, ('A', 'L'): 307, ('Q', 'R'): 73, ('E', 'S'): 143, ('E', 'Y'): 60, ('A', 'K'): 191, ('A', 'G'): 22 9, ('V', 'Y'): 134, ('H', 'S'): 59, ('P', 'S'): 103, ('N', 'Y'): 75, ('G', 'T'): 153, ('G', 'P'): 108, ('I', 'Y'): 94, ('E', 'P'): 91, ('C', 'D'): 15, ('S', 'W' ): 22, ('S', 'V'): 114, ('L', 'N'): 101, ('E', 'E'): 119, ('Q', 'T'): 97, ('E', 'R'): 118, ('F', 'P'): 50, ('P', 'Q'): 68, ('I', 'I'): 145, ('H', 'M'): 16, ('G' , 'Y'): 101, ('F', 'Y'): 142, ('C', 'N'): 24, ('A', 'A'): 211, ('E', 'V'): 110, ('L', 'Q'): 79, ('P', 'P'): 120, ('I', 'L'): 348, ('M', 'R'): 40, ('V', 'V'): 23 9} C:\Users\xxx3\Desktop>c:\python34\python.exe -V Python 3.4.0 C:\Users\xxx\Desktop>c:\python34\python.exe Python 3.4.0 (v3.4.0:04f714765c13, Mar 16 2014, 19:25:23) [MSC v.1600 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> quit() So, this works perfectly here. Peter From manlio.calvi at gmail.com Tue Apr 22 11:09:31 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Tue, 22 Apr 2014 13:09:31 +0200 Subject: [Biopython-dev] Pickle problem on 64 bit Windows with Python 3.4 In-Reply-To: References: Message-ID: On Tue, Apr 22, 2014 at 12:44 PM, Peter Cock wrote: > On Mon, Apr 21, 2014 at 6:45 PM, Manlio Calvi wrote: >> From what I read here http://hg.python.org/cpython/rev/4a6b8f86b081 could be >> a problem related to that file. Seems to me they stripped the check for a >> quote that must be in, and looking at the pickle apparently isn't >> > > OK, now things are more confusing - this seems to be working on > a colleague's machine, so it may be something different on your > setup. Are you using a self compiled Python 3.4? > > We installed the 64 bit version Python 3.4 on Windows 7 using the > binary installed from the website (Windows x86-64 MSI installer), > selecting for all users (which probably requires admin rights): > https://www.python.org/ftp/python/3.4.0/python-3.4.0.amd64.msi Exactly as I did, I installed the dependencies (numpy and the like) for Biopython using Gohlke's ones. > We manually downloaded the pickle file via the raw link on GitHub, > and tried the test code (as shown below), and it worked perfectly. I've used the standard "git pull" command from the repository. Moreover I'm coming from a recent format and reinstall of windows in this machine. I'm a bit lost here... From p.j.a.cock at googlemail.com Tue Apr 22 12:36:23 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 13:36:23 +0100 Subject: [Biopython-dev] Pickle problem on 64 bit Windows with Python 3.4 In-Reply-To: References: Message-ID: On Tue, Apr 22, 2014 at 12:09 PM, Manlio Calvi wrote: > On Tue, Apr 22, 2014 at 12:44 PM, Peter Cock wrote: >> On Mon, Apr 21, 2014 at 6:45 PM, Manlio Calvi wrote: >>> From what I read here http://hg.python.org/cpython/rev/4a6b8f86b081 could be >>> a problem related to that file. Seems to me they stripped the check for a >>> quote that must be in, and looking at the pickle apparently isn't >>> >> >> OK, now things are more confusing - this seems to be working on >> a colleague's machine, so it may be something different on your >> setup. Are you using a self compiled Python 3.4? >> >> We installed the 64 bit version Python 3.4 on Windows 7 using the >> binary installed from the website (Windows x86-64 MSI installer), >> selecting for all users (which probably requires admin rights): >> https://www.python.org/ftp/python/3.4.0/python-3.4.0.amd64.msi > > Exactly as I did, I installed the dependencies (numpy and the like) > for Biopython using Gohlke's ones. > >> We manually downloaded the pickle file via the raw link on GitHub, >> and tried the test code (as shown below), and it worked perfectly. > > I've used the standard "git pull" command from the repository. > Moreover I'm coming from a recent format and reinstall of windows in > this machine. > I'm a bit lost here... OK, I had an idea over lunch which turned out to solve this :) First I checked that my pickle on Linux file uses Unix new lines, $ hexdump -C acc_rep_mat.pik | head 00000000 28 64 70 31 0a 28 53 27 4c 27 0a 53 27 52 27 0a |(dp1.(S'L'.S'R'.| 00000010 74 49 31 30 39 0a 73 28 53 27 49 27 0a 53 27 49 |tI109.s(S'I'.S'I| 00000020 27 0a 74 49 31 34 35 0a 73 28 53 27 51 27 0a 53 |'.tI145.s(S'Q'.S| 00000030 27 51 27 0a 74 49 34 32 0a 73 28 53 27 53 27 0a |'Q'.tI42.s(S'S'.| 00000040 53 27 54 27 0a 74 49 31 37 32 0a 73 28 53 27 48 |S'T'.tI172.s(S'H| 00000050 27 0a 53 27 54 27 0a 74 49 36 39 0a 73 28 53 27 |'.S'T'.tI69.s(S'| 00000060 51 27 0a 53 27 59 27 0a 74 49 34 31 0a 73 28 53 |Q'.S'Y'.tI41.s(S| 00000070 27 48 27 0a 53 27 50 27 0a 74 49 32 33 0a 73 28 |'H'.S'P'.tI23.s(| 00000080 53 27 4e 27 0a 53 27 59 27 0a 74 49 37 35 0a 73 |S'N'.S'Y'.tI75.s| 00000090 28 53 27 48 27 0a 53 27 4c 27 0a 74 49 37 30 0a |(S'H'.S'L'.tI70.| Then I converted it to DOS/Windows newlines (e.g. unix2dos is easy if you have that, or a few lines of Python if not - see below): $ hexdump -C acc_rep_mat.dos.pik | head 00000000 28 64 70 31 0d 0a 28 53 27 4c 27 0d 0a 53 27 52 |(dp1..(S'L'..S'R| 00000010 27 0d 0a 74 49 31 30 39 0d 0a 73 28 53 27 49 27 |'..tI109..s(S'I'| 00000020 0d 0a 53 27 49 27 0d 0a 74 49 31 34 35 0d 0a 73 |..S'I'..tI145..s| 00000030 28 53 27 51 27 0d 0a 53 27 51 27 0d 0a 74 49 34 |(S'Q'..S'Q'..tI4| 00000040 32 0d 0a 73 28 53 27 53 27 0d 0a 53 27 54 27 0d |2..s(S'S'..S'T'.| 00000050 0a 74 49 31 37 32 0d 0a 73 28 53 27 48 27 0d 0a |.tI172..s(S'H'..| 00000060 53 27 54 27 0d 0a 74 49 36 39 0d 0a 73 28 53 27 |S'T'..tI69..s(S'| 00000070 51 27 0d 0a 53 27 59 27 0d 0a 74 49 34 31 0d 0a |Q'..S'Y'..tI41..| 00000080 73 28 53 27 48 27 0d 0a 53 27 50 27 0d 0a 74 49 |s(S'H'..S'P'..tI| 00000090 32 33 0d 0a 73 28 53 27 4e 27 0d 0a 53 27 59 27 |23..s(S'N'..S'Y'| This increases the file size from 3658 bytes to 4289 bytes. $ python3.4 -c "import pickle; h=open('acc_rep_mat.pik', 'rb'); m=pickle.load(h); h.close(); print(m)" {('E', 'M'): 33, ..., ('D', 'V'): 95} $ python3.4 -c "import pickle; h=open('acc_rep_mat.dos.pik', 'rb'); m=pickle.load(h); h.close(); print(m)" Traceback (most recent call last): File "", line 1, in _pickle.UnpicklingError: the STRING opcode argument must be quoted So I can get the exact same error under Linux now :) I confirmed this on Windows where my copy of git is setup to use Unix newlines by default (I think), and the file has Unix newlines (and is 3658 bytes). C:\repositories\biopython\Tests\SubsMat>c:\python34\python Python 3.4.0 (v3.4.0:04f714765c13, Mar 16 2014, 19:24:06) [MSC v.1600 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> data = open("acc_rep_mat.pik", "rb").read() >>> with open("acc_rep_mat.dos.pik", "wb") as h: h.write(data.replace(b"\n", b"\r\n")) ... 4289 >>> quit() C:\repositories\biopython\Tests\SubsMat>c:\python34\python -c "import pickle; h=open('acc_rep_mat.pik', 'rb'); m=pickle.load(h); h.close(); print(m)" {('D', 'R'): 115, ..., ('H', 'Q'): 44} C:\repositories\biopython\Tests\SubsMat>c:\python34\python -c "import pickle; h=open('acc_rep_mat.dos.pik', 'rb'); m=pickle.load(h); h.close(); print(m)" Traceback (most recent call last): File "", line 1, in _pickle.UnpicklingError: the STRING opcode argument must be quoted So, the upshot is that this git setting change should fix it: https://github.com/biopython/biopython/commit/b7cc2fe199d22f794612d68e5554361413468372 Could you update your copy of the Biopython source code via git, and see if that solves this pickle? Thank you, Peter From p.j.a.cock at googlemail.com Tue Apr 22 12:57:34 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 13:57:34 +0100 Subject: [Biopython-dev] Python 2.7 failure from test_Tutorial.py, was: Compiling Biopython on 64 bit Windows In-Reply-To: References: Message-ID: On Tue, Apr 22, 2014 at 9:32 AM, Peter Cock wrote: > On Tue, Apr 22, 2014 at 6:51 AM, Manlio Calvi wrote: >> Hi Peter, >> I just run this one from the Tests folder (the tests are there and >> don'tget moved in the installation) > > Sorry, yes - I should have said that. > >> I got this: >> >> Running Tutorial doctests... >> ********************************************************************** >> File "Tests/test_Tutorial.py", line 153, in >> __main__.TutorialDocTestHolder.doctest_test_from_line_04424 >> Failed example: >> align_array.shape >> Expected: >> (7, 52) >> Got: >> (7L, 52L) >> ********************************************************************** >> 1 items had failures: >> 1 of 7 in __main__.TutorialDocTestHolder.doctest_test_from_line_04424 >> ***Test Failed*** 1 failures. >> C:\python27\lib\site-packages\Bio\Motif\__init__.py:26: >> BiopythonDeprecationWarning: The module Bio.Motif has been deprecated >> and will be removed in a future release of Biopython. Instead please >> use the new module Bio.motifs instead. Please be aware that though the >> functionality of Bio.Motif is retained (and extended) in Bio.motifs, >> usage may be different. >> BiopythonDeprecationWarning) >> Traceback (most recent call last): >> File "Tests/test_Tutorial.py", line 185, in >> RuntimeError: 1/806 tests failed >> >> >> Another Python2/3 compatibility problem... > > Well yes, but also a 32 bit versus 64 bit Windows issue. > Python 3 hides the C level differences between ints and longs, > while Python 2 can sometimes expose this - odd that this test > didn't give longs on the other platforms, just 64 bit windows. > > I suggest we switch from: > > %This example fails under PyPy 2.0, https://bugs.pypy.org/issue1546 > %doctest examples lib:numpy > \begin{verbatim} >>>> import numpy as np >>>> from Bio import AlignIO >>>> alignment = AlignIO.read("PF05371_seed. > sth", "stockholm") >>>> align_array = np.array([list(rec) for rec in alignment], np.character) >>>> align_array.shape > (7, 52) > \end{verbatim} > > To: > > %This example fails under PyPy 2.0, https://bugs.pypy.org/issue1546 > %doctest examples lib:numpy > \begin{verbatim} >>>> import numpy as np >>>> from Bio import AlignIO >>>> alignment = AlignIO.read("PF05371_seed.sth", "stockholm") >>>> align_array = np.array([list(rec) for rec in alignment], np.character) >>>> print("Array shape %i by %i" % align_array.shape) > Array shape 7 by 52 > \end{verbatim} > > That should be safe :) > > Peter Change applied to git, https://github.com/biopython/biopython/commit/0e26dfb40ab1ce78319eaf6893e35abda43c8738 Peter From p.j.a.cock at googlemail.com Tue Apr 22 13:08:40 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 14:08:40 +0100 Subject: [Biopython-dev] CDAO problems on Windows, was: Compiling Biopython on 64 bit Windows Message-ID: On Tue, Apr 22, 2014 at 10:57 AM, Peter Cock wrote: > On Mon, Apr 21, 2014 at 10:51 PM, Manlio Calvi wrote: >> ... >> >> ====================================================================== >> ERROR: test_write_0 (test_Phylo_CDAO.WriterTests) >> Write and re-parse the phylogenies in test.cdao. >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File ".\test_Phylo_CDAO.py", line 62, in test_write >> CDAOIO.write([t1], outfile) >> File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython >> test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\CDAOIO.py", >> line 74, in write >> return Writer(trees).write(handle, plain=plain, **kwargs) >> File "C:\Users\CasuallyChaotic\Desktop\3.3 - Biopython >> test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Bio\Phylo\CDAOIO.py", >> line 268, in write >> handle.write('@prefix %s: <%s> .\n' % (k, v)) >> TypeError: 'str' does not support the buffer interface >> > > That looks like a string versus bytes issue with the handle > (text versus binary mode), but I don't recall any changes > in that code... maybe a question for Eric? I've not looked into this issue yet, however the reason the existing Windows buildbot never reported this is that I have not installed RDFlib on it (not even for Python 2.6 and 2.7). That was an oversight - and part of the reason to try and get more buildslaves - diversity. >> ################################################# >> >> >> The test output for CDAO reports a series of slightly messages (more >> than an hundred) that looks like this: >> file://C:\WinPython3.3\3.3 - Biopython >> test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Tests\CDAO\test.cdao/tree0000001#node0000001 >> does not look like a valid URI, trying to serialize this will break. > > My hunch is this warning could be due to the mixed slashes > (forward and back) which is typically fine with the windows > API for filenames, but evidently is confusing the CDAO > code - a question for Ben? According to the RDFLib docstring for their URI class, yes, they issue this warning if the URI contains a Windows-style forward slash: http://rdflib.readthedocs.org/en/latest/_modules/rdflib/term.html The relevant snippet of their code is: _invalid_uri_chars = '<>" {}|\\^`' def _is_valid_uri(uri): for c in _invalid_uri_chars: if c in uri: return False return True My suggestion would be to switch all the "\" into "/" here. Peter From p.j.a.cock at googlemail.com Tue Apr 22 16:11:33 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 17:11:33 +0100 Subject: [Biopython-dev] Fwd: [GSoC] Welcome aboard, GSoC 2014 students! In-Reply-To: References: Message-ID: Dear Biopythoneers, Please join me in congratulating this year's accepted students for Google Summer of Code under the Open Bioinformatics Foundation (OBF), and in particular welcome Evan Parker who will be working on Biopython with Bow and myself as co-mentors. As always the scheme has been very competitive, so our sympathies and commiserations to those students who were not accepted. Please do stay involved in Biopython or other open source projects - this would be a positive factor if you are eligible to apply for next year's GSoC. Thank you, Peter ---------- Forwarded message ---------- From: Eric Talevich Date: Tue, Apr 22, 2014 at 4:41 AM Subject: [GSoC] Welcome aboard, GSoC 2014 students! To: OBF GSoC Hi all, I'm pleased to announce the acceptance of OBF's 2010 Google Summer of Code students: Sarah Berkemer - "Open source high-performance BioHaskell" (Mentors: Christian H?ner zu Siederdissen, Ketil Malde) Loris Cro - "An ultra-fast scalable RESTful API to query large numbers of VCF datapoints" (Mentors: Francesco Strozzi, Raoul Bonnal & the BioRuby team) Victor Kofia - "JSBML: Redesign the implementation of mathematical formulas" (Mentors: Alex Thomas, Sarah Keating & the JSBML team) Evan Parker - "Addition of a lazy loading sequence parser to Biopython's SeqIO package" (Mentors: Wibowo Arindrarto, Peter Cock & the Biopython team) Ibrahim Vairabad - "Improving the Plug-in interface for CellDesigner" (Mentors: Andreas Dr?ger, Alex Thomas & the JSBML team) Leandro Watanabe - "Dynamic Modeling of Cellular Populations within JSBML" (Mentors: Nicolas Rodriguez, Chris Meyers & the JSBML team) Congratulations to our accepted students! Thanks very much to all the students who applied, we very much appreciate your hard work. Today marks the start of the Community Bonding Period. Official work starts on May 23rd, and until then, students should prepare for their projects: get on the project mailing lists, solidify your plans, figure out where all the version control repositories are and which branch or fork you'll be working on, and start doing preparatory work. Students: if you have not done so already, make sure you have subscribed to the OBF GSoC email list at: http://lists.open-bio.org/mailman/listinfo/gsoc This list is for discussions among students and mentors, and for administrative announcements from me or my co-administrators. Here's to a great 2014 Summer of Code, Eric & Raoul OBF GSoC 2014 Organization Administrators _______________________________________________ GSoC mailing list GSoC at lists.open-bio.org http://lists.open-bio.org/mailman/listinfo/gsoc From manlio.calvi at gmail.com Tue Apr 22 18:40:09 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Tue, 22 Apr 2014 20:40:09 +0200 Subject: [Biopython-dev] Python 2.7 failure from test_Tutorial.py, was: Compiling Biopython on 64 bit Windows In-Reply-To: References: Message-ID: OK tested, all good on my machine, the only failed test is the TogoWS that don't depend on us. Manlio On Tue, Apr 22, 2014 at 2:57 PM, Peter Cock wrote: > On Tue, Apr 22, 2014 at 9:32 AM, Peter Cock wrote: >> On Tue, Apr 22, 2014 at 6:51 AM, Manlio Calvi wrote: >>> Hi Peter, >>> I just run this one from the Tests folder (the tests are there and >>> don'tget moved in the installation) >> >> Sorry, yes - I should have said that. >> >>> I got this: >>> >>> Running Tutorial doctests... >>> ********************************************************************** >>> File "Tests/test_Tutorial.py", line 153, in >>> __main__.TutorialDocTestHolder.doctest_test_from_line_04424 >>> Failed example: >>> align_array.shape >>> Expected: >>> (7, 52) >>> Got: >>> (7L, 52L) >>> ********************************************************************** >>> 1 items had failures: >>> 1 of 7 in __main__.TutorialDocTestHolder.doctest_test_from_line_04424 >>> ***Test Failed*** 1 failures. >>> C:\python27\lib\site-packages\Bio\Motif\__init__.py:26: >>> BiopythonDeprecationWarning: The module Bio.Motif has been deprecated >>> and will be removed in a future release of Biopython. Instead please >>> use the new module Bio.motifs instead. Please be aware that though the >>> functionality of Bio.Motif is retained (and extended) in Bio.motifs, >>> usage may be different. >>> BiopythonDeprecationWarning) >>> Traceback (most recent call last): >>> File "Tests/test_Tutorial.py", line 185, in >>> RuntimeError: 1/806 tests failed >>> >>> >>> Another Python2/3 compatibility problem... >> >> Well yes, but also a 32 bit versus 64 bit Windows issue. >> Python 3 hides the C level differences between ints and longs, >> while Python 2 can sometimes expose this - odd that this test >> didn't give longs on the other platforms, just 64 bit windows. >> >> I suggest we switch from: >> >> %This example fails under PyPy 2.0, https://bugs.pypy.org/issue1546 >> %doctest examples lib:numpy >> \begin{verbatim} >>>>> import numpy as np >>>>> from Bio import AlignIO >>>>> alignment = AlignIO.read("PF05371_seed. >> sth", "stockholm") >>>>> align_array = np.array([list(rec) for rec in alignment], np.character) >>>>> align_array.shape >> (7, 52) >> \end{verbatim} >> >> To: >> >> %This example fails under PyPy 2.0, https://bugs.pypy.org/issue1546 >> %doctest examples lib:numpy >> \begin{verbatim} >>>>> import numpy as np >>>>> from Bio import AlignIO >>>>> alignment = AlignIO.read("PF05371_seed.sth", "stockholm") >>>>> align_array = np.array([list(rec) for rec in alignment], np.character) >>>>> print("Array shape %i by %i" % align_array.shape) >> Array shape 7 by 52 >> \end{verbatim} >> >> That should be safe :) >> >> Peter > > Change applied to git, > https://github.com/biopython/biopython/commit/0e26dfb40ab1ce78319eaf6893e35abda43c8738 > > Peter From manlio.calvi at gmail.com Tue Apr 22 19:09:54 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Tue, 22 Apr 2014 21:09:54 +0200 Subject: [Biopython-dev] Pickle problem on 64 bit Windows with Python 3.4 In-Reply-To: References: Message-ID: Uhm no, I get the same error as before... Seems this machine don't like conserved vegetables. Manlio On Tue, Apr 22, 2014 at 2:36 PM, Peter Cock wrote: > On Tue, Apr 22, 2014 at 12:09 PM, Manlio Calvi wrote: >> On Tue, Apr 22, 2014 at 12:44 PM, Peter Cock wrote: >>> On Mon, Apr 21, 2014 at 6:45 PM, Manlio Calvi wrote: >>>> From what I read here http://hg.python.org/cpython/rev/4a6b8f86b081 could be >>>> a problem related to that file. Seems to me they stripped the check for a >>>> quote that must be in, and looking at the pickle apparently isn't >>>> >>> >>> OK, now things are more confusing - this seems to be working on >>> a colleague's machine, so it may be something different on your >>> setup. Are you using a self compiled Python 3.4? >>> >>> We installed the 64 bit version Python 3.4 on Windows 7 using the >>> binary installed from the website (Windows x86-64 MSI installer), >>> selecting for all users (which probably requires admin rights): >>> https://www.python.org/ftp/python/3.4.0/python-3.4.0.amd64.msi >> >> Exactly as I did, I installed the dependencies (numpy and the like) >> for Biopython using Gohlke's ones. >> >>> We manually downloaded the pickle file via the raw link on GitHub, >>> and tried the test code (as shown below), and it worked perfectly. >> >> I've used the standard "git pull" command from the repository. >> Moreover I'm coming from a recent format and reinstall of windows in >> this machine. >> I'm a bit lost here... > > OK, I had an idea over lunch which turned out to solve this :) > > First I checked that my pickle on Linux file uses Unix new lines, > > $ hexdump -C acc_rep_mat.pik | head > 00000000 28 64 70 31 0a 28 53 27 4c 27 0a 53 27 52 27 0a |(dp1.(S'L'.S'R'.| > 00000010 74 49 31 30 39 0a 73 28 53 27 49 27 0a 53 27 49 |tI109.s(S'I'.S'I| > 00000020 27 0a 74 49 31 34 35 0a 73 28 53 27 51 27 0a 53 |'.tI145.s(S'Q'.S| > 00000030 27 51 27 0a 74 49 34 32 0a 73 28 53 27 53 27 0a |'Q'.tI42.s(S'S'.| > 00000040 53 27 54 27 0a 74 49 31 37 32 0a 73 28 53 27 48 |S'T'.tI172.s(S'H| > 00000050 27 0a 53 27 54 27 0a 74 49 36 39 0a 73 28 53 27 |'.S'T'.tI69.s(S'| > 00000060 51 27 0a 53 27 59 27 0a 74 49 34 31 0a 73 28 53 |Q'.S'Y'.tI41.s(S| > 00000070 27 48 27 0a 53 27 50 27 0a 74 49 32 33 0a 73 28 |'H'.S'P'.tI23.s(| > 00000080 53 27 4e 27 0a 53 27 59 27 0a 74 49 37 35 0a 73 |S'N'.S'Y'.tI75.s| > 00000090 28 53 27 48 27 0a 53 27 4c 27 0a 74 49 37 30 0a |(S'H'.S'L'.tI70.| > > Then I converted it to DOS/Windows newlines (e.g. unix2dos > is easy if you have that, or a few lines of Python if not - see below): > > $ hexdump -C acc_rep_mat.dos.pik | head > 00000000 28 64 70 31 0d 0a 28 53 27 4c 27 0d 0a 53 27 52 |(dp1..(S'L'..S'R| > 00000010 27 0d 0a 74 49 31 30 39 0d 0a 73 28 53 27 49 27 |'..tI109..s(S'I'| > 00000020 0d 0a 53 27 49 27 0d 0a 74 49 31 34 35 0d 0a 73 |..S'I'..tI145..s| > 00000030 28 53 27 51 27 0d 0a 53 27 51 27 0d 0a 74 49 34 |(S'Q'..S'Q'..tI4| > 00000040 32 0d 0a 73 28 53 27 53 27 0d 0a 53 27 54 27 0d |2..s(S'S'..S'T'.| > 00000050 0a 74 49 31 37 32 0d 0a 73 28 53 27 48 27 0d 0a |.tI172..s(S'H'..| > 00000060 53 27 54 27 0d 0a 74 49 36 39 0d 0a 73 28 53 27 |S'T'..tI69..s(S'| > 00000070 51 27 0d 0a 53 27 59 27 0d 0a 74 49 34 31 0d 0a |Q'..S'Y'..tI41..| > 00000080 73 28 53 27 48 27 0d 0a 53 27 50 27 0d 0a 74 49 |s(S'H'..S'P'..tI| > 00000090 32 33 0d 0a 73 28 53 27 4e 27 0d 0a 53 27 59 27 |23..s(S'N'..S'Y'| > > This increases the file size from 3658 bytes to 4289 bytes. > > $ python3.4 -c "import pickle; h=open('acc_rep_mat.pik', 'rb'); > m=pickle.load(h); h.close(); print(m)" > {('E', 'M'): 33, ..., ('D', 'V'): 95} > > $ python3.4 -c "import pickle; h=open('acc_rep_mat.dos.pik', 'rb'); > m=pickle.load(h); h.close(); print(m)" > Traceback (most recent call last): > File "", line 1, in > _pickle.UnpicklingError: the STRING opcode argument must be quoted > > So I can get the exact same error under Linux now :) > > I confirmed this on Windows where my copy of git is setup to use > Unix newlines by default (I think), and the file has Unix newlines > (and is 3658 bytes). > > C:\repositories\biopython\Tests\SubsMat>c:\python34\python > Python 3.4.0 (v3.4.0:04f714765c13, Mar 16 2014, 19:24:06) [MSC v.1600 > 32 bit (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. >>>> data = open("acc_rep_mat.pik", "rb").read() >>>> with open("acc_rep_mat.dos.pik", "wb") as h: h.write(data.replace(b"\n", b"\r\n")) > ... > 4289 >>>> quit() > > C:\repositories\biopython\Tests\SubsMat>c:\python34\python -c "import > pickle; h=open('acc_rep_mat.pik', 'rb'); m=pickle.load(h); h.close(); > print(m)" > {('D', 'R'): 115, ..., ('H', 'Q'): 44} > > C:\repositories\biopython\Tests\SubsMat>c:\python34\python -c "import > pickle; h=open('acc_rep_mat.dos.pik', 'rb'); m=pickle.load(h); > h.close(); print(m)" > Traceback (most recent call last): > File "", line 1, in > _pickle.UnpicklingError: the STRING opcode argument must be quoted > > So, the upshot is that this git setting change should fix it: > https://github.com/biopython/biopython/commit/b7cc2fe199d22f794612d68e5554361413468372 > > Could you update your copy of the Biopython source code via git, > and see if that solves this pickle? > > Thank you, > > Peter From p.j.a.cock at googlemail.com Tue Apr 22 19:14:25 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 20:14:25 +0100 Subject: [Biopython-dev] Pickle problem on 64 bit Windows with Python 3.4 In-Reply-To: References: Message-ID: On Tue, Apr 22, 2014 at 8:09 PM, Manlio Calvi wrote: > Uhm no, I get the same error as before... > Seems this machine don't like conserved vegetables. > > Manlio It may just be git didn't update the file with the Unix newlines. Can you check the file (e.g. size in bytes, or some Windows hexdump equivalent like opening it for reading in Python in raw/binary mode)? Perhaps deleting the file on disk and then getting get to restore it might do the trick? Peter From zruan1991 at gmail.com Tue Apr 22 21:18:58 2014 From: zruan1991 at gmail.com (Zheng Ruan) Date: Tue, 22 Apr 2014 17:18:58 -0400 Subject: [Biopython-dev] MSAprobs test fails Message-ID: Hi, The MASprobs wrapper fails when I run the test using the latest code. The problem seems to be the err.returncode is -11 rather than 139, at least in my machine (Linux 3.12 x86_64) python2.7 and 3.3 will both generate the error. Here is scratch of the err message: FAIL: test_single_sequence (test_MSAProbs_tool.MSAProbsTestErrorConditions) Test an input file containing a single sequence. ---------------------------------------------------------------------- Traceback (most recent call last): File "./test_MSAProbs_tool.py", line 86, in test_single_sequence stdout, stderr = cline() Bio.Application.ApplicationError: Command 'msaprobs Fasta/f001' returned non-zero exit status -11, '-------------------------------------' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "./test_MSAProbs_tool.py", line 88, in test_single_sequence self.assertEqual(err.returncode, 139) AssertionError: -11 != 139 Best, Zheng Ruan From manlio.calvi at gmail.com Tue Apr 22 21:31:53 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Tue, 22 Apr 2014 23:31:53 +0200 Subject: [Biopython-dev] Pickle problem on 64 bit Windows with Python 3.4 In-Reply-To: References: Message-ID: OK this one tastes good! This problems was solved but another two emerged (or I never take notice before), time for another post... Manlio On Tue, Apr 22, 2014 at 9:14 PM, Peter Cock wrote: > On Tue, Apr 22, 2014 at 8:09 PM, Manlio Calvi wrote: >> Uhm no, I get the same error as before... >> Seems this machine don't like conserved vegetables. >> >> Manlio > > It may just be git didn't update the file with the > Unix newlines. > > Can you check the file (e.g. size in bytes, or some > Windows hexdump equivalent like opening it for > reading in Python in raw/binary mode)? > > Perhaps deleting the file on disk and then getting > get to restore it might do the trick? > > Peter From manlio.calvi at gmail.com Tue Apr 22 21:36:27 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Tue, 22 Apr 2014 23:36:27 +0200 Subject: [Biopython-dev] Win7 x64 - python 3.4 - Phylo and searchIO Message-ID: I got those two new errors: ====================================================================== ERROR: test_unicode_exception (test_Phylo.IOTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Users\Public\BioPython3\biopython\Tests\test_Phylo.py", line 57, in test_unicode_exception tree = Phylo.read(EX_NEWICK_BOM, 'newick') File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", line 65, in read tree = next(tree_gen) File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", line 53, in parse for tree in getattr(supported_formats[format], 'parse')(fp, **kwargs): File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\NewickIO.py", line 107, in parse for line in self.handle: File "C:\Python34\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 15: character maps to ====================================================================== FAIL: test_xml_2226_blastn_006 (test_SearchIO_blast_xml.BlastXmlSpecialCases) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Users\Public\BioPython3\biopython\Tests\test_SearchIO_blast_xml.py", line 3127, in test_xml_2226_blastn_006 "Expected one BiopythonParserWarning, got %r" % w) AssertionError: 2 != 1 : Expected one BiopythonParserWarning, got [, ] Manlio From p.j.a.cock at googlemail.com Tue Apr 22 21:38:57 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 22:38:57 +0100 Subject: [Biopython-dev] Pickle problem on 64 bit Windows with Python 3.4 In-Reply-To: References: Message-ID: On Tue, Apr 22, 2014 at 10:31 PM, Manlio Calvi wrote: > OK this one tastes good! This problems was solved but another two > emerged (or I never take notice before), time for another post... > > Manlio So good news (this problem is solved) but bad news (you have noticed some other issues). I'll await your next email. Thanks, Peter From p.j.a.cock at googlemail.com Tue Apr 22 21:48:28 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 22:48:28 +0100 Subject: [Biopython-dev] MSAprobs test fails In-Reply-To: References: Message-ID: Hi Zheng, Negative error codes (from Python subprocess) are really Unix signals, https://docs.python.org/2/library/subprocess.html i.e. Linux killed the job with signal 11, probably SIGSEGV - Invalid memory reference. Can you run the tool by hand? Does it seg fault or something like that? Peter On Tue, Apr 22, 2014 at 10:18 PM, Zheng Ruan wrote: > Hi, > > The MASprobs wrapper fails when I run the test using the latest code. The > problem seems to be the err.returncode is -11 rather than 139, at least in > my machine (Linux 3.12 x86_64) python2.7 and 3.3 will both generate the > error. > > Here is scratch of the err message: > > FAIL: test_single_sequence (test_MSAProbs_tool.MSAProbsTestErrorConditions) > Test an input file containing a single sequence. > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "./test_MSAProbs_tool.py", line 86, in test_single_sequence > stdout, stderr = cline() > Bio.Application.ApplicationError: Command 'msaprobs Fasta/f001' returned > non-zero exit status -11, '-------------------------------------' > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "./test_MSAProbs_tool.py", line 88, in test_single_sequence > self.assertEqual(err.returncode, 139) > AssertionError: -11 != 139 > > > Best, > Zheng Ruan > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev From p.j.a.cock at googlemail.com Tue Apr 22 21:58:59 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Tue, 22 Apr 2014 22:58:59 +0100 Subject: [Biopython-dev] Win7 x64 - python 3.4 - Phylo and searchIO In-Reply-To: References: Message-ID: On Tue, Apr 22, 2014 at 10:36 PM, Manlio Calvi wrote: > I got those two new errors: > > ====================================================================== > ERROR: test_unicode_exception (test_Phylo.IOTests) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Users\Public\BioPython3\biopython\Tests\test_Phylo.py", > line 57, in test_unicode_exception > tree = Phylo.read(EX_NEWICK_BOM, 'newick') > File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", > line 65, in read > tree = next(tree_gen) > File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", > line 53, in parse > for tree in getattr(supported_formats[format], 'parse')(fp, **kwargs): > File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\NewickIO.py", > line 107, in parse > for line in self.handle: > File "C:\Python34\lib\encodings\cp1252.py", line 23, in decode > return codecs.charmap_decode(input,self.errors,decoding_table)[0] > UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position > 15: character maps to > Yep, see https://github.com/biopython/biopython/issues/311 > ====================================================================== > FAIL: test_xml_2226_blastn_006 (test_SearchIO_blast_xml.BlastXmlSpecialCases) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Users\Public\BioPython3\biopython\Tests\test_SearchIO_blast_xml.py", > line 3127, in test_xml_2226_blastn_006 > "Expected one BiopythonParserWarning, got %r" % w) > AssertionError: 2 != 1 : Expected one BiopythonParserWarning, got > [, > ] > I suspect I have seen that before... must double check that. @Bow, I think this check needs to be relaxed slightly (e.g. at least one warning?) as it can potentially interact with global warning settings (even though the test framework tries to restore the filters to a known state before each test). Could you take a look? Thanks, Peter From zruan1991 at gmail.com Tue Apr 22 22:32:54 2014 From: zruan1991 at gmail.com (Zheng Ruan) Date: Tue, 22 Apr 2014 18:32:54 -0400 Subject: [Biopython-dev] MSAprobs test fails In-Reply-To: References: Message-ID: I see. I was just trying different alignment program. In this case, it might not be good to test the single sequence by the error code. Zheng On Tue, Apr 22, 2014 at 5:48 PM, Peter Cock wrote: > Hi Zheng, > > Negative error codes (from Python subprocess) are really Unix signals, > https://docs.python.org/2/library/subprocess.html > > i.e. Linux killed the job with signal 11, probably SIGSEGV - Invalid > memory reference. > > > Can you run the tool by hand? Does it seg fault or something like that? > > Peter > > On Tue, Apr 22, 2014 at 10:18 PM, Zheng Ruan wrote: > > Hi, > > > > The MASprobs wrapper fails when I run the test using the latest code. The > > problem seems to be the err.returncode is -11 rather than 139, at least > in > > my machine (Linux 3.12 x86_64) python2.7 and 3.3 will both generate the > > error. > > > > Here is scratch of the err message: > > > > FAIL: test_single_sequence > (test_MSAProbs_tool.MSAProbsTestErrorConditions) > > Test an input file containing a single sequence. > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File "./test_MSAProbs_tool.py", line 86, in test_single_sequence > > stdout, stderr = cline() > > Bio.Application.ApplicationError: Command 'msaprobs Fasta/f001' returned > > non-zero exit status -11, '-------------------------------------' > > > > During handling of the above exception, another exception occurred: > > > > Traceback (most recent call last): > > File "./test_MSAProbs_tool.py", line 88, in test_single_sequence > > self.assertEqual(err.returncode, 139) > > AssertionError: -11 != 139 > > > > > > Best, > > Zheng Ruan > > _______________________________________________ > > Biopython-dev mailing list > > Biopython-dev at lists.open-bio.org > > http://lists.open-bio.org/mailman/listinfo/biopython-dev > From manlio.calvi at gmail.com Wed Apr 23 04:51:31 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Wed, 23 Apr 2014 06:51:31 +0200 Subject: [Biopython-dev] Win7 x64 - python 3.4 - Phylo and searchIO In-Reply-To: References: Message-ID: Ops a bit of overzealousness here, yesterday night was a bit tired and I preferred to be sure than skip an issue. However the second one I raised on the first round of my tests but don't get addressed Manlio On Tue, Apr 22, 2014 at 11:58 PM, Peter Cock wrote: > On Tue, Apr 22, 2014 at 10:36 PM, Manlio Calvi wrote: >> I got those two new errors: >> >> ====================================================================== >> ERROR: test_unicode_exception (test_Phylo.IOTests) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "C:\Users\Public\BioPython3\biopython\Tests\test_Phylo.py", >> line 57, in test_unicode_exception >> tree = Phylo.read(EX_NEWICK_BOM, 'newick') >> File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", >> line 65, in read >> tree = next(tree_gen) >> File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", >> line 53, in parse >> for tree in getattr(supported_formats[format], 'parse')(fp, **kwargs): >> File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\NewickIO.py", >> line 107, in parse >> for line in self.handle: >> File "C:\Python34\lib\encodings\cp1252.py", line 23, in decode >> return codecs.charmap_decode(input,self.errors,decoding_table)[0] >> UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position >> 15: character maps to >> > > Yep, see https://github.com/biopython/biopython/issues/311 > >> ====================================================================== >> FAIL: test_xml_2226_blastn_006 (test_SearchIO_blast_xml.BlastXmlSpecialCases) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "C:\Users\Public\BioPython3\biopython\Tests\test_SearchIO_blast_xml.py", >> line 3127, in test_xml_2226_blastn_006 >> "Expected one BiopythonParserWarning, got %r" % w) >> AssertionError: 2 != 1 : Expected one BiopythonParserWarning, got >> [, >> ] >> > > I suspect I have seen that before... must double check that. > > @Bow, I think this check needs to be relaxed slightly (e.g. at > least one warning?) as it can potentially interact with global > warning settings (even though the test framework tries to > restore the filters to a known state before each test). Could > you take a look? > > Thanks, > > Peter From alexeymorozov1991 at gmail.com Wed Apr 23 08:09:40 2014 From: alexeymorozov1991 at gmail.com (Alexey Morozov) Date: Wed, 23 Apr 2014 17:09:40 +0900 Subject: [Biopython-dev] Codon usage module? Message-ID: Biopython currently supports only a limited set of operations for codon usage analysis, so I made my own module for stuff like counting codon usage in genes, clustering them based on codon usage, making some genome- or gene-wide statistics, and so on. If I want to get it to main release eventually, what are the rules? In particular, is NumPy dependency OK? Should all my classes inherit from something? Are there any limits on how data is to be internally stored and returned? -- Alexey Morozov, LIN SB RAS, bioinformatics group. Irkutsk, Russia. From zruan1991 at gmail.com Wed Apr 23 18:03:04 2014 From: zruan1991 at gmail.com (Zheng Ruan) Date: Wed, 23 Apr 2014 14:03:04 -0400 Subject: [Biopython-dev] [biopython] Code for Codon Alignment GSoC (#259) In-Reply-To: References: Message-ID: Hi, I integrate my document into the Tutorial ( https://github.com/zruan/biopython/tree/codonalignment/Doc) as chapter 18. Meanwhile, I split each chapter into the `chapter` directory and include them to the Tutorial.tex. The test_Tutorial.py is also updated accordingly, which is able to report per-chapter line numbers if the doctest fails. Best, Zheng On Mon, Apr 21, 2014 at 1:06 PM, Peter Cock wrote: > On Mon, Apr 21, 2014 at 6:02 PM, Zheng Ruan wrote: > > > > Hi Peter and Eric, > > > > I just made some changes to my repository. The old codonalignment branch > has been > > renamed to gsoc2013 for backup. I will write new code to the > codonalignment branch. > > OK. You could write a final GSoC blog post about this too ;) > > > Eric: > > I think the whole module should be marked as experimental like the > Bio.SearchIO > > module. I anticipate to enhance the codon alignment construction method > as well > > as add more methods for dn/ds ratio estimation and Bayes Empirical Bayes > this > > summer. > > > > I place a warning in the `__init__.py`. ( > https://github.com/zruan/biopython/commit/1c9e20c8a15febb7fd9219c632afb96127a006f1 > ) > > That sounds wise. > > > Peter: > > The NEWS file is updated ( > https://github.com/zruan/biopython/commit/0dcb39bcfd48cc009edc290ceaaa3b2c2e8f806f > ). > > I will update the wiki shortly and make the announcement. > > > > Do I need to make a separate pull request to make the above change to > biopython > > master? Or you may help me do that? > > I think Eric or I can do that from those links, Eric - do you want to do > this? > > > I am also planning to include my latex document to Tutorial.tex. I > typically use > > the Latex-Suite plugin to write latex in vim. However, it took a long > time to open the > > Tutorial.tex, which made it difficult to manage. Why not put the latex > code of each > > chapter into different tex file and use \include{} to import them into > the main tex file? > > > > Thank you! > > Zheng Ruan > > We could split the Tutorial.tex file like that - although it would mean a > bit of extra work on test_Tutorial.py (but with the benefit of getting > per-chapter line numbers). > > Thank you, > > Peter > From bow at bow.web.id Wed Apr 23 21:09:27 2014 From: bow at bow.web.id (Wibowo Arindrarto) Date: Wed, 23 Apr 2014 23:09:27 +0200 Subject: [Biopython-dev] Win7 x64 - python 3.4 - Phylo and searchIO In-Reply-To: References: Message-ID: Hi Peter, Manlio, Thanks for looking deeper into the tests :). For the SearchIO issue, I'm not too fond of relaxing the filter, as that could mean the test passes even though warnings for this particular case is never raised. After checking more, I actually found that this isn't due to the global warning filter settings. Apparently, the 'U' file read mode is deprecated in Python 3.4[1] and SearchIO is using the 'rU' mode to open its files. So the number of warnings in Python 3.4 is indeed 2: the rU DeprecationWarning and the BiopythonParserWarning we are checking. Since it looks like we can't remove 'rU' altogether from our codebase, the more appropriate fix for this case would be to check for len(w) == 2 when Python version is >= 3.4. Best, Bow [1] https://docs.python.org/3.5/whatsnew/3.4.html#deprecated-3-4 On Wed, Apr 23, 2014 at 6:51 AM, Manlio Calvi wrote: > Ops a bit of overzealousness here, yesterday night was a bit tired and > I preferred to be sure than skip an issue. > However the second one I raised on the first round of my tests but > don't get addressed > > Manlio > > On Tue, Apr 22, 2014 at 11:58 PM, Peter Cock wrote: >> On Tue, Apr 22, 2014 at 10:36 PM, Manlio Calvi wrote: >>> I got those two new errors: >>> >>> ====================================================================== >>> ERROR: test_unicode_exception (test_Phylo.IOTests) >>> ---------------------------------------------------------------------- >>> Traceback (most recent call last): >>> File "C:\Users\Public\BioPython3\biopython\Tests\test_Phylo.py", >>> line 57, in test_unicode_exception >>> tree = Phylo.read(EX_NEWICK_BOM, 'newick') >>> File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", >>> line 65, in read >>> tree = next(tree_gen) >>> File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", >>> line 53, in parse >>> for tree in getattr(supported_formats[format], 'parse')(fp, **kwargs): >>> File "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\NewickIO.py", >>> line 107, in parse >>> for line in self.handle: >>> File "C:\Python34\lib\encodings\cp1252.py", line 23, in decode >>> return codecs.charmap_decode(input,self.errors,decoding_table)[0] >>> UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position >>> 15: character maps to >>> >> >> Yep, see https://github.com/biopython/biopython/issues/311 >> >>> ====================================================================== >>> FAIL: test_xml_2226_blastn_006 (test_SearchIO_blast_xml.BlastXmlSpecialCases) >>> ---------------------------------------------------------------------- >>> Traceback (most recent call last): >>> File "C:\Users\Public\BioPython3\biopython\Tests\test_SearchIO_blast_xml.py", >>> line 3127, in test_xml_2226_blastn_006 >>> "Expected one BiopythonParserWarning, got %r" % w) >>> AssertionError: 2 != 1 : Expected one BiopythonParserWarning, got >>> [, >>> ] >>> >> >> I suspect I have seen that before... must double check that. >> >> @Bow, I think this check needs to be relaxed slightly (e.g. at >> least one warning?) as it can potentially interact with global >> warning settings (even though the test framework tries to >> restore the filters to a known state before each test). Could >> you take a look? >> >> Thanks, >> >> Peter From p.j.a.cock at googlemail.com Wed Apr 23 22:02:55 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 23 Apr 2014 23:02:55 +0100 Subject: [Biopython-dev] Win7 x64 - python 3.4 - Phylo and searchIO In-Reply-To: References: Message-ID: Good work Bow - that sounds like a better plan :) Peter On Wednesday, April 23, 2014, Wibowo Arindrarto wrote: > Hi Peter, Manlio, > > Thanks for looking deeper into the tests :). For the SearchIO issue, > I'm not too fond of relaxing the filter, as that could mean the test > passes even though warnings for this particular case is never raised. > > After checking more, I actually found that this isn't due to the > global warning filter settings. Apparently, the 'U' file read mode is > deprecated in Python 3.4[1] and SearchIO is using the 'rU' mode to > open its files. So the number of warnings in Python 3.4 is indeed 2: > the rU DeprecationWarning and the BiopythonParserWarning we are > checking. > > Since it looks like we can't remove 'rU' altogether from our codebase, > the more appropriate fix for this case would be to check for len(w) == 2 > when > Python version is >= 3.4. > > Best, > Bow > > [1] https://docs.python.org/3.5/whatsnew/3.4.html#deprecated-3-4 > > On Wed, Apr 23, 2014 at 6:51 AM, Manlio Calvi > > wrote: > > Ops a bit of overzealousness here, yesterday night was a bit tired and > > I preferred to be sure than skip an issue. > > However the second one I raised on the first round of my tests but > > don't get addressed > > > > Manlio > > > > On Tue, Apr 22, 2014 at 11:58 PM, Peter Cock > > wrote: > >> On Tue, Apr 22, 2014 at 10:36 PM, Manlio Calvi > > wrote: > >>> I got those two new errors: > >>> > >>> ====================================================================== > >>> ERROR: test_unicode_exception (test_Phylo.IOTests) > >>> ---------------------------------------------------------------------- > >>> Traceback (most recent call last): > >>> File "C:\Users\Public\BioPython3\biopython\Tests\test_Phylo.py", > >>> line 57, in test_unicode_exception > >>> tree = Phylo.read(EX_NEWICK_BOM, 'newick') > >>> File > "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", > >>> line 65, in read > >>> tree = next(tree_gen) > >>> File > "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", > >>> line 53, in parse > >>> for tree in getattr(supported_formats[format], 'parse')(fp, > **kwargs): > >>> File > "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\NewickIO.py", > >>> line 107, in parse > >>> for line in self.handle: > >>> File "C:\Python34\lib\encodings\cp1252.py", line 23, in decode > >>> return codecs.charmap_decode(input,self.errors,decoding_table)[0] > >>> UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position > >>> 15: character maps to > >>> > >> > >> Yep, see https://github.com/biopython/biopython/issues/311 > >> > >>> ====================================================================== > >>> FAIL: test_xml_2226_blastn_006 > (test_SearchIO_blast_xml.BlastXmlSpecialCases) > >>> ---------------------------------------------------------------------- > >>> Traceback (most recent call last): > >>> File > "C:\Users\Public\BioPython3\biopython\Tests\test_SearchIO_blast_xml.py", > >>> line 3127, in test_xml_2226_blastn_006 > >>> "Expected one BiopythonParserWarning, got %r" % w) > >>> AssertionError: 2 != 1 : Expected one BiopythonParserWarning, got > >>> [, > >>> ] > >>> > >> > >> I suspect I have seen that before... must double check that. > >> > >> @Bow, I think this check needs to be relaxed slightly (e.g. at > >> least one warning?) as it can potentially interact with global > >> warning settings (even though the test framework tries to > >> restore the filters to a known state before each test). Could > >> you take a look? > >> > >> Thanks, > >> > >> Peter > From bow at bow.web.id Wed Apr 23 23:06:47 2014 From: bow at bow.web.id (Wibowo Arindrarto) Date: Thu, 24 Apr 2014 01:06:47 +0200 Subject: [Biopython-dev] Win7 x64 - python 3.4 - Phylo and searchIO In-Reply-To: References: Message-ID: Thanks, Peter. The change is here: https://github.com/biopython/biopython/commit/82c23cb6afe48864f3dcdb3099cac2116842e9b2. All test pass on my box (except for the trie bug). We can't see the results in Travis yet (Python 3.4 support is coming soon they say[1]), but hopefully it will say the same thing then :). Cheers, Bow [1] https://github.com/travis-ci/travis-ci/issues/1989 On Thu, Apr 24, 2014 at 12:02 AM, Peter Cock wrote: > Good work Bow - that sounds like a better plan :) > > Peter > > On Wednesday, April 23, 2014, Wibowo Arindrarto wrote: >> >> Hi Peter, Manlio, >> >> Thanks for looking deeper into the tests :). For the SearchIO issue, >> I'm not too fond of relaxing the filter, as that could mean the test >> passes even though warnings for this particular case is never raised. >> >> After checking more, I actually found that this isn't due to the >> global warning filter settings. Apparently, the 'U' file read mode is >> deprecated in Python 3.4[1] and SearchIO is using the 'rU' mode to >> open its files. So the number of warnings in Python 3.4 is indeed 2: >> the rU DeprecationWarning and the BiopythonParserWarning we are >> checking. >> >> Since it looks like we can't remove 'rU' altogether from our codebase, >> the more appropriate fix for this case would be to check for len(w) == 2 >> when >> Python version is >= 3.4. >> >> Best, >> Bow >> >> [1] https://docs.python.org/3.5/whatsnew/3.4.html#deprecated-3-4 >> >> On Wed, Apr 23, 2014 at 6:51 AM, Manlio Calvi >> wrote: >> > Ops a bit of overzealousness here, yesterday night was a bit tired and >> > I preferred to be sure than skip an issue. >> > However the second one I raised on the first round of my tests but >> > don't get addressed >> > >> > Manlio >> > >> > On Tue, Apr 22, 2014 at 11:58 PM, Peter Cock >> > wrote: >> >> On Tue, Apr 22, 2014 at 10:36 PM, Manlio Calvi >> >> wrote: >> >>> I got those two new errors: >> >>> >> >>> ====================================================================== >> >>> ERROR: test_unicode_exception (test_Phylo.IOTests) >> >>> ---------------------------------------------------------------------- >> >>> Traceback (most recent call last): >> >>> File "C:\Users\Public\BioPython3\biopython\Tests\test_Phylo.py", >> >>> line 57, in test_unicode_exception >> >>> tree = Phylo.read(EX_NEWICK_BOM, 'newick') >> >>> File >> >>> "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", >> >>> line 65, in read >> >>> tree = next(tree_gen) >> >>> File >> >>> "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\_io.py", >> >>> line 53, in parse >> >>> for tree in getattr(supported_formats[format], 'parse')(fp, >> >>> **kwargs): >> >>> File >> >>> "C:\Users\Public\BioPython3\biopython\build\lib.win-amd64-3.4\Bio\Phylo\NewickIO.py", >> >>> line 107, in parse >> >>> for line in self.handle: >> >>> File "C:\Python34\lib\encodings\cp1252.py", line 23, in decode >> >>> return codecs.charmap_decode(input,self.errors,decoding_table)[0] >> >>> UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position >> >>> 15: character maps to >> >>> >> >> >> >> Yep, see https://github.com/biopython/biopython/issues/311 >> >> >> >>> ====================================================================== >> >>> FAIL: test_xml_2226_blastn_006 >> >>> (test_SearchIO_blast_xml.BlastXmlSpecialCases) >> >>> ---------------------------------------------------------------------- >> >>> Traceback (most recent call last): >> >>> File >> >>> "C:\Users\Public\BioPython3\biopython\Tests\test_SearchIO_blast_xml.py", >> >>> line 3127, in test_xml_2226_blastn_006 >> >>> "Expected one BiopythonParserWarning, got %r" % w) >> >>> AssertionError: 2 != 1 : Expected one BiopythonParserWarning, got >> >>> [, >> >>> ] >> >>> >> >> >> >> I suspect I have seen that before... must double check that. >> >> >> >> @Bow, I think this check needs to be relaxed slightly (e.g. at >> >> least one warning?) as it can potentially interact with global >> >> warning settings (even though the test framework tries to >> >> restore the filters to a known state before each test). Could >> >> you take a look? >> >> >> >> Thanks, >> >> >> >> Peter From p.j.a.cock at googlemail.com Thu Apr 24 08:26:34 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Thu, 24 Apr 2014 09:26:34 +0100 Subject: [Biopython-dev] Win7 x64 - python 3.4 - Phylo and searchIO In-Reply-To: References: Message-ID: Great - Tiago and I need to add Python 3.4 to the buildbot, and as you say, fingers crossed TravisCI finish rolling out Python 3.4 support soon... Peter On Thursday, April 24, 2014, Wibowo Arindrarto wrote: > Thanks, Peter. The change is here: > > https://github.com/biopython/biopython/commit/82c23cb6afe48864f3dcdb3099cac2116842e9b2 > . > All test pass on my box (except for the trie bug). We can't see the > results in Travis yet > (Python 3.4 support is coming soon they say[1]), but > hopefully it will say the same thing then :). > > Cheers, > Bow > > [1] https://github.com/travis-ci/travis-ci/issues/1989 > > > From tra at popgen.net Thu Apr 24 08:47:33 2014 From: tra at popgen.net (Tiago Antao) Date: Thu, 24 Apr 2014 09:47:33 +0100 Subject: [Biopython-dev] Win7 x64 - python 3.4 - Phylo and searchIO In-Reply-To: References: Message-ID: <20140424094733.16dafbb1@grandao> On Thu, 24 Apr 2014 09:26:34 +0100 Peter Cock wrote: > Great - Tiago and I need to add Python 3.4 to the buildbot, > and as you say, fingers crossed TravisCI finish rolling out > Python 3.4 support soon... I have been silent on this because I was waiting for a docker image based on ubuntu trusty (which has 3.4 as the default Python). The image is now available but has a bug (one of the apt-sources is still saucy). My expectation is that this will be corrected very soon. I have all the machinery ready to test on 3.4 as soon as the docker image is out and fully working. Maybe even today (as the bug is trivial to sort out - lets just hope that they do it fast). Tiago From manlio.calvi at gmail.com Thu Apr 24 09:11:47 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Thu, 24 Apr 2014 11:11:47 +0200 Subject: [Biopython-dev] Win7 x64 - python 3.4 - Phylo and searchIO In-Reply-To: <20140424094733.16dafbb1@grandao> References: <20140424094733.16dafbb1@grandao> Message-ID: In the afternoon I'll count to rebuild/retest all on 2.7, 3.3 and the 3.4 on my Win7 x64 machine On Thu, Apr 24, 2014 at 10:47 AM, Tiago Antao wrote: > > On Thu, 24 Apr 2014 09:26:34 +0100 > Peter Cock wrote: > >> Great - Tiago and I need to add Python 3.4 to the buildbot, >> and as you say, fingers crossed TravisCI finish rolling out >> Python 3.4 support soon... > > I have been silent on this because I was waiting for a docker image > based on ubuntu trusty (which has 3.4 as the default Python). The image > is now available but has a bug (one of the apt-sources is still saucy). > My expectation is that this will be corrected very soon. > > I have all the machinery ready to test on 3.4 as soon as the docker > image is out and fully working. Maybe even today (as the bug is > trivial to sort out - lets just hope that they do it fast). > > Tiago > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev From tra at popgen.net Thu Apr 24 10:36:23 2014 From: tra at popgen.net (Tiago Antao) Date: Thu, 24 Apr 2014 11:36:23 +0100 Subject: [Biopython-dev] Next biopython release (was Re: Python 3.4) In-Reply-To: References: <20140422102316.5b049105@lnx> Message-ID: <20140424113623.2bdd2d9d@lnx> On Tue, 22 Apr 2014 10:42:33 +0100 Peter Cock wrote: > I am hoping we'll have everything ready on Python 3.4 > for the next release of Biopython - aim for early May? It would be good if we rotated release managers. But, if no one else volunteers, I can put my head for the chop. Tiago From manlio.calvi at gmail.com Thu Apr 24 16:34:20 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Thu, 24 Apr 2014 18:34:20 +0200 Subject: [Biopython-dev] Testings on Win 7 x64 Message-ID: OK I launched the tests on all my serpent's nest :) As some changes are made on .git folders and files and seeing how badly git on windows manage those I erased the biopython folders and started from scratch, that's the results: Python 3.4 x64 NO error (yay!): The only note here is the fact that bio.trieand KDtree get skipped because the C module refuse to compile. Python 3.3 (WinPython 3.3.3.3 x64 2 errors (one may be temporary): CDAO test fail with (an issue we already seen): ====================================================================== ERROR: test_write_0 (test_Phylo_CDAO.WriterTests) Write and re-parse the phylogenies in test.cdao. ---------------------------------------------------------------------- Traceback (most recent call last): File ".\test_Phylo_CDAO.py", line 62, in test_write CDAOIO.write([t1], outfile) File "C:\Users\xxx\3.3 - Biopython test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\build\lib.win-amd64-3.3\Bio\Phylo\CDAOIO.py", line 74, in write return Writer(trees).write(handle, plain=plain, **kwargs) File "C:\Users\xxx\3.3 - Biopython test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\build\lib.win-amd64-3.3\Bio\Phylo\CDAOIO.py", line 268, in write handle.write('@prefix %s: <%s> .\n' % (k, v)) TypeError: 'str' does not support the buffer interface --------------------------------------- Also during the test this line popped over and over file://C:\Users\xxx\3.3 - Biopython test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Tests\CDAO\test.cdao/tree0000001#edge_annotation0000071 does not look like a valid URI, trying to serialize this will break. --------------------------------? The other isssue is related with NCBI qblast but being NCBI probably was I hit a server down. ###################################### Python 2.7 x64 NO issues! Adding to that I hit a "strangeness" I don't understand because the mentioned scipy module IS installed, the test pass but not with "flying colours": test_CodonAlign ... test_CodonAlign.py:156: UserWarning: Importing scipy.linalg.expm failed. Skip testing ML method for dN/dS estimation warnings.warn('Importing scipy.linalg.expm failed. Skip testing ML method for dN/dS estimation') test_CodonAlign.py:164: UserWarning: Importing scipy.optimize.minimize failed. Skip testing ML method for dN/dS estimation warnings.warn('Importing scipy.optimize.minimize failed. Skip testing ML method for dN/dS estimation') ok Second a more general signalation, I've seen some test depends on PIL but PIL library get discontinued and substituted by one called Pillow that is said compatible with it predecessor (but I'm not sure how much it is). This is a more general issue because require an scan of the entire codebase. Looking around seems to me the NLTK package is in a situation like this, too. Manlio From p.j.a.cock at googlemail.com Thu Apr 24 16:49:38 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Thu, 24 Apr 2014 17:49:38 +0100 Subject: [Biopython-dev] Testings on Win 7 x64 In-Reply-To: References: Message-ID: On Thu, Apr 24, 2014 at 5:34 PM, Manlio Calvi wrote: > OK I launched the tests on all my serpent's nest :) > > As some changes are made on .git folders and files and seeing how > badly git on windows manage those I erased the biopython folders and > started from scratch, that's the results: > > Python 3.4 x64 NO error (yay!): > The only note here is the fact that bio.trieand KDtree get skipped > because the C module refuse to compile. It seems none of the compiled C code is getting tested :( http://lists.open-bio.org/pipermail/biopython-dev/2014-April/011270.html > Python 3.3 (WinPython 3.3.3.3 x64 2 errors (one may be temporary): > CDAO test fail with (an issue we already seen): > > ====================================================================== > ERROR: test_write_0 (test_Phylo_CDAO.WriterTests) > Write and re-parse the phylogenies in test.cdao. > ---------------------------------------------------------------------- > Traceback (most recent call last): > File ".\test_Phylo_CDAO.py", line 62, in test_write > CDAOIO.write([t1], outfile) > File "C:\Users\xxx\3.3 - Biopython > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\build\lib.win-amd64-3.3\Bio\Phylo\CDAOIO.py", > line 74, in write > return Writer(trees).write(handle, plain=plain, **kwargs) > File "C:\Users\xxx\3.3 - Biopython > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\build\lib.win-amd64-3.3\Bio\Phylo\CDAOIO.py", > line 268, in write > handle.write('@prefix %s: <%s> .\n' % (k, v)) > TypeError: 'str' does not support the buffer interface > --------------------------------------- > > Also during the test this line popped over and over > > file://C:\Users\xxx\3.3 - Biopython > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Tests\CDAO\test.cdao/tree0000001#edge_annotation0000071 > does not look like a valid URI, trying to serialize this will break. > > -------------------------------- See this thread - no replies yet so maybe we'd better note this as a GitHub issue, http://lists.open-bio.org/pipermail/biopython-dev/2014-April/011294.html > > The other isssue is related with NCBI qblast but being NCBI probably > was I hit a server down. I've noticed the NCBI blast website is currently very slow :( > ###################################### > > Python 2.7 x64 NO issues! > Excellent - we are making progress (slowly). > > Adding to that I hit a "strangeness" I don't understand because the > mentioned scipy module IS installed, the test pass but not with > "flying colours": > test_CodonAlign ... test_CodonAlign.py:156: UserWarning: Importing > scipy.linalg.expm failed. Skip testing ML method for dN/dS estimation > warnings.warn('Importing scipy.linalg.expm failed. Skip testing ML > method for dN/dS estimation') > test_CodonAlign.py:164: UserWarning: Importing scipy.optimize.minimize > failed. Skip testing ML method for dN/dS estimation > warnings.warn('Importing scipy.optimize.minimize failed. Skip > testing ML method for dN/dS estimation') > ok The CodonAlign stuff is new, this might need looking at: https://github.com/biopython/biopython/pull/259 http://lists.open-bio.org/pipermail/biopython-dev/2014-April/011278.html > Second a more general signalation, I've seen some test depends on PIL > but PIL library get discontinued and substituted by one called Pillow > that is said compatible with it predecessor (but I'm not sure how much > it is). This is a more general issue because require an scan of the > entire codebase. Looking around seems to me the NLTK package is in a > situation like this, too. That probably refers to test_GraphicsBitmaps.py which is trying to check if ReportLab will be able to make PNG files. If ReportLab can now use Pillow or PIL, we should allow for that... https://github.com/python-imaging/Pillow (As an aside ReportLab's Python 3 support is still quite new - but I don't recall seeing any problems from Biopython's usage of it.) Peter From zruan1991 at gmail.com Thu Apr 24 16:53:53 2014 From: zruan1991 at gmail.com (Zheng Ruan) Date: Thu, 24 Apr 2014 12:53:53 -0400 Subject: [Biopython-dev] Testings on Win 7 x64 In-Reply-To: References: Message-ID: Hi Manlio, As for the test_CodonAlign warning, what's the scipy version you installed? Are you able to manually import `scipy.linalg.expm` and ` scipy.optimize.minimize` manually? scipy.optimize.minimize is added to scipy since v0.11 (http://docs.scipy.org/doc/scipy-0.11.0/reference/tutorial/optimize.html). I'm not sure why `scipy.linalg.expm` is not able to be import, since it's available in the earliest version of scipy ( http://docs.scipy.org/doc/scipy-0.7.x/reference/linalg.html). Thank you! Zheng On Thu, Apr 24, 2014 at 12:34 PM, Manlio Calvi wrote: > OK I launched the tests on all my serpent's nest :) > > As some changes are made on .git folders and files and seeing how > badly git on windows manage those I erased the biopython folders and > started from scratch, that's the results: > > Python 3.4 x64 NO error (yay!): > The only note here is the fact that bio.trieand KDtree get skipped > because the C module refuse to compile. > > Python 3.3 (WinPython 3.3.3.3 x64 2 errors (one may be temporary): > CDAO test fail with (an issue we already seen): > > ====================================================================== > ERROR: test_write_0 (test_Phylo_CDAO.WriterTests) > Write and re-parse the phylogenies in test.cdao. > ---------------------------------------------------------------------- > Traceback (most recent call last): > File ".\test_Phylo_CDAO.py", line 62, in test_write > CDAOIO.write([t1], outfile) > File "C:\Users\xxx\3.3 - Biopython > > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\build\lib.win-amd64-3.3\Bio\Phylo\CDAOIO.py", > line 74, in write > return Writer(trees).write(handle, plain=plain, **kwargs) > File "C:\Users\xxx\3.3 - Biopython > > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\build\lib.win-amd64-3.3\Bio\Phylo\CDAOIO.py", > line 268, in write > handle.write('@prefix %s: <%s> .\n' % (k, v)) > TypeError: 'str' does not support the buffer interface > --------------------------------------- > > Also during the test this line popped over and over > > file://C:\Users\xxx\3.3 - Biopython > > test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Tests\CDAO\test.cdao/tree0000001#edge_annotation0000071 > does not look like a valid URI, trying to serialize this will break. > > --------------------------------? > > The other isssue is related with NCBI qblast but being NCBI probably > was I hit a server down. > > ###################################### > > Python 2.7 x64 NO issues! > > > Adding to that I hit a "strangeness" I don't understand because the > mentioned scipy module IS installed, the test pass but not with > "flying colours": > test_CodonAlign ... test_CodonAlign.py:156: UserWarning: Importing > scipy.linalg.expm failed. Skip testing ML method for dN/dS estimation > warnings.warn('Importing scipy.linalg.expm failed. Skip testing ML > method for dN/dS estimation') > test_CodonAlign.py:164: UserWarning: Importing scipy.optimize.minimize > failed. Skip testing ML method for dN/dS estimation > warnings.warn('Importing scipy.optimize.minimize failed. Skip > testing ML method for dN/dS estimation') > ok > > Second a more general signalation, I've seen some test depends on PIL > but PIL library get discontinued and substituted by one called Pillow > that is said compatible with it predecessor (but I'm not sure how much > it is). This is a more general issue because require an scan of the > entire codebase. Looking around seems to me the NLTK package is in a > situation like this, too. > > Manlio > > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev > From p.j.a.cock at googlemail.com Thu Apr 24 18:34:16 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Thu, 24 Apr 2014 19:34:16 +0100 Subject: [Biopython-dev] Win7 x64 - python 3.4 - Phylo and searchIO In-Reply-To: References: Message-ID: On Wed, Apr 23, 2014 at 11:02 PM, Peter Cock wrote: > Good work Bow - that sounds like a better plan :) > > Peter Sadly it failed on one of the Linux machines, http://testing.open-bio.org/biopython/builders/OS%20X%20-%20Python%203.4/builds/2 I've fixed this by removing the mode U under Python 3 instead: https://github.com/biopython/biopython/commit/d308194cfc9f9535dd98b2a004e4becb52096090 And tweaked your test to got back to expecting one warning only: https://github.com/biopython/biopython/commit/3d07450abf345c62c2377e18755d96b7ae7b9263 There are other uses of mode U which we may want to change if the warnings become too visible. Peter From p.j.a.cock at googlemail.com Thu Apr 24 19:53:34 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Thu, 24 Apr 2014 20:53:34 +0100 Subject: [Biopython-dev] Versions of PyPy to support? Message-ID: Hello all, PyPy 2.3 is due out shortly, which prompts me to ask which versions of PyPy are people using Biopython with? PyPy is an alternative implementation of Python, which can often be much faster - see http://pypy.org/ We're currently testing with PyPy 1.8, 1.9, 2.0, 2.1 and 2.2 but I would like to suggest we drop at least PyPy 1.8 and 1.9. Is that OK? Thanks! Peter From manlio.calvi at gmail.com Thu Apr 24 20:13:07 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Thu, 24 Apr 2014 22:13:07 +0200 Subject: [Biopython-dev] Testings on Win 7 x64 In-Reply-To: References: Message-ID: Hi Zheng, A failed manual import of scipy make me control what happened to my distro, the scipy was there (infact for some other math computation script it works) but for some reason wasn't used by biopython, however I reinstalled the last version available, rebuilt and retest and the problem disappeared. Peter, the C modules instead I have no idea to how make them works, they're compiled and installed in place but seems that my snake's nest don't want to load it, I don't know why. Apparently they don't looks like an importable python module. Manlio On Thu, Apr 24, 2014 at 6:53 PM, Zheng Ruan wrote: > Hi Manlio, > > As for the test_CodonAlign warning, what's the scipy version you installed? > Are > you able to manually import `scipy.linalg.expm` and > `scipy.optimize.minimize` > manually? scipy.optimize.minimize is added to scipy since v0.11 > (http://docs.scipy.org/doc/scipy-0.11.0/reference/tutorial/optimize.html). > I'm not sure why `scipy.linalg.expm` is not able to be import, since it's > available > in the earliest version of scipy > (http://docs.scipy.org/doc/scipy-0.7.x/reference/linalg.html). > > Thank you! > Zheng > > > On Thu, Apr 24, 2014 at 12:34 PM, Manlio Calvi > wrote: >> >> OK I launched the tests on all my serpent's nest :) >> >> As some changes are made on .git folders and files and seeing how >> badly git on windows manage those I erased the biopython folders and >> started from scratch, that's the results: >> >> Python 3.4 x64 NO error (yay!): >> The only note here is the fact that bio.trieand KDtree get skipped >> because the C module refuse to compile. >> >> Python 3.3 (WinPython 3.3.3.3 x64 2 errors (one may be temporary): >> CDAO test fail with (an issue we already seen): >> >> ====================================================================== >> ERROR: test_write_0 (test_Phylo_CDAO.WriterTests) >> Write and re-parse the phylogenies in test.cdao. >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File ".\test_Phylo_CDAO.py", line 62, in test_write >> CDAOIO.write([t1], outfile) >> File "C:\Users\xxx\3.3 - Biopython >> >> test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\build\lib.win-amd64-3.3\Bio\Phylo\CDAOIO.py", >> line 74, in write >> return Writer(trees).write(handle, plain=plain, **kwargs) >> File "C:\Users\xxx\3.3 - Biopython >> >> test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\build\lib.win-amd64-3.3\Bio\Phylo\CDAOIO.py", >> line 268, in write >> handle.write('@prefix %s: <%s> .\n' % (k, v)) >> TypeError: 'str' does not support the buffer interface >> --------------------------------------- >> >> Also during the test this line popped over and over >> >> file://C:\Users\xxx\3.3 - Biopython >> >> test\WinPython-64bit-3.3.3.3\python-3.3.3.amd64\biopython\Tests\CDAO\test.cdao/tree0000001#edge_annotation0000071 >> does not look like a valid URI, trying to serialize this will break. >> >> --------------------------------? >> >> The other isssue is related with NCBI qblast but being NCBI probably >> was I hit a server down. >> >> ###################################### >> >> Python 2.7 x64 NO issues! >> >> >> Adding to that I hit a "strangeness" I don't understand because the >> mentioned scipy module IS installed, the test pass but not with >> "flying colours": >> test_CodonAlign ... test_CodonAlign.py:156: UserWarning: Importing >> scipy.linalg.expm failed. Skip testing ML method for dN/dS estimation >> warnings.warn('Importing scipy.linalg.expm failed. Skip testing ML >> method for dN/dS estimation') >> test_CodonAlign.py:164: UserWarning: Importing scipy.optimize.minimize >> failed. Skip testing ML method for dN/dS estimation >> warnings.warn('Importing scipy.optimize.minimize failed. Skip >> testing ML method for dN/dS estimation') >> ok >> >> Second a more general signalation, I've seen some test depends on PIL >> but PIL library get discontinued and substituted by one called Pillow >> that is said compatible with it predecessor (but I'm not sure how much >> it is). This is a more general issue because require an scan of the >> entire codebase. Looking around seems to me the NLTK package is in a >> situation like this, too. >> >> Manlio >> >> _______________________________________________ >> Biopython-dev mailing list >> Biopython-dev at lists.open-bio.org >> http://lists.open-bio.org/mailman/listinfo/biopython-dev > > From manlio.calvi at gmail.com Thu Apr 24 21:04:47 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Thu, 24 Apr 2014 23:04:47 +0200 Subject: [Biopython-dev] Versions of PyPy to support? In-Reply-To: References: Message-ID: I'm thinking about run a PyPy test on my machine, something specific to watch out? I've seen they have a beta on pypy for 3, probably a bit experimental at the moment but they saying it mostly works (as they say of PyPy in general) Manlio On Thu, Apr 24, 2014 at 9:53 PM, Peter Cock wrote: > Hello all, > > PyPy 2.3 is due out shortly, which prompts me to ask which > versions of PyPy are people using Biopython with? > > PyPy is an alternative implementation of Python, which > can often be much faster - see http://pypy.org/ > > We're currently testing with PyPy 1.8, 1.9, 2.0, 2.1 and 2.2 > but I would like to suggest we drop at least PyPy 1.8 and 1.9. > > Is that OK? > > Thanks! > > Peter > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev From p.j.a.cock at googlemail.com Thu Apr 24 21:07:54 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Thu, 24 Apr 2014 22:07:54 +0100 Subject: [Biopython-dev] Versions of PyPy to support? In-Reply-To: References: Message-ID: Try PyPy 2.2 first, and if that works you can try their experimental Python 3 support? Peter On Thursday, April 24, 2014, Manlio Calvi wrote: > I'm thinking about run a PyPy test on my machine, something specific > to watch out? > > I've seen they have a beta on pypy for 3, probably a bit experimental > at the moment but they saying it mostly works (as they say of PyPy in > general) > > Manlio > > On Thu, Apr 24, 2014 at 9:53 PM, Peter Cock > > wrote: > > Hello all, > > > > PyPy 2.3 is due out shortly, which prompts me to ask which > > versions of PyPy are people using Biopython with? > > > > PyPy is an alternative implementation of Python, which > > can often be much faster - see http://pypy.org/ > > > > We're currently testing with PyPy 1.8, 1.9, 2.0, 2.1 and 2.2 > > but I would like to suggest we drop at least PyPy 1.8 and 1.9. > > > > Is that OK? > > > > Thanks! > > > > Peter > > _______________________________________________ > > Biopython-dev mailing list > > Biopython-dev at lists.open-bio.org > > http://lists.open-bio.org/mailman/listinfo/biopython-dev > From p.j.a.cock at googlemail.com Thu Apr 24 21:31:27 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Thu, 24 Apr 2014 22:31:27 +0100 Subject: [Biopython-dev] Testings on Win 7 x64 In-Reply-To: References: Message-ID: On Thu, Apr 24, 2014 at 9:13 PM, Manlio Calvi wrote: > Hi Zheng, > > A failed manual import of scipy make me control what happened to my > distro, the scipy was there (infact for some other math computation > script it works) but for some reason wasn't used by biopython, however > I reinstalled the last version available, rebuilt and retest and the > problem disappeared. Good :) > Peter, the C modules instead I have no idea to how make them works, > they're compiled and installed in place but seems that my snake's nest > don't want to load it, I don't know why. Apparently they don't looks > like an importable python module. So Python 2.7 is working, but to confirm the C code is not working for either Python 3.3 or Python 3.4 on your 64 bit Windows 7 machine? Strange... I think we will need to wait for someone else to have a go, perhaps Tiago - or I might be able to borrow a machine... Still, I would be pleased if we can have at least Python 2.7 running on your desktop as a buildslave? Thanks, Peter From manlio.calvi at gmail.com Thu Apr 24 21:42:25 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Thu, 24 Apr 2014 23:42:25 +0200 Subject: [Biopython-dev] Testings on Win 7 x64 In-Reply-To: References: Message-ID: Confirmed, Python 2.7 miss only the test who needs external code now. I have to understand why that's don't works...Could you pass those windows compiled .pyd files to me in a separate mail? I would like to understand if it my compiled that produced malformated code(but then why only that file?) or if I've look to something else in my building environments Manlio On Thu, Apr 24, 2014 at 11:31 PM, Peter Cock wrote: > On Thu, Apr 24, 2014 at 9:13 PM, Manlio Calvi wrote: >> Hi Zheng, >> >> A failed manual import of scipy make me control what happened to my >> distro, the scipy was there (infact for some other math computation >> script it works) but for some reason wasn't used by biopython, however >> I reinstalled the last version available, rebuilt and retest and the >> problem disappeared. > > Good :) > >> Peter, the C modules instead I have no idea to how make them works, >> they're compiled and installed in place but seems that my snake's nest >> don't want to load it, I don't know why. Apparently they don't looks >> like an importable python module. > > So Python 2.7 is working, but to confirm the C code is not > working for either Python 3.3 or Python 3.4 on your 64 bit > Windows 7 machine? Strange... I think we will need to wait > for someone else to have a go, perhaps Tiago - or I might > be able to borrow a machine... > > Still, I would be pleased if we can have at least Python 2.7 > running on your desktop as a buildslave? > > Thanks, > > Peter From manlio.calvi at gmail.com Thu Apr 24 21:57:56 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Thu, 24 Apr 2014 23:57:56 +0200 Subject: [Biopython-dev] Win7 x64 - python 3.4 - Phylo and searchIO In-Reply-To: References: Message-ID: OK recompiled 3.4 and nothing to report there And I can confirm an all clear in 2.7 too apart the some warning and that compilation problem. Python 3.3 reported the same error on CDAO as before. Manlio On Thu, Apr 24, 2014 at 8:34 PM, Peter Cock wrote: > On Wed, Apr 23, 2014 at 11:02 PM, Peter Cock wrote: >> Good work Bow - that sounds like a better plan :) >> >> Peter > > Sadly it failed on one of the Linux machines, > http://testing.open-bio.org/biopython/builders/OS%20X%20-%20Python%203.4/builds/2 > > I've fixed this by removing the mode U under Python 3 instead: > https://github.com/biopython/biopython/commit/d308194cfc9f9535dd98b2a004e4becb52096090 > > And tweaked your test to got back to expecting one warning only: > https://github.com/biopython/biopython/commit/3d07450abf345c62c2377e18755d96b7ae7b9263 > > There are other uses of mode U which we may want to change > if the warnings become too visible. > > Peter > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev From anaryin at gmail.com Thu Apr 24 23:27:29 2014 From: anaryin at gmail.com (=?UTF-8?Q?Jo=C3=A3o_Rodrigues?=) Date: Fri, 25 Apr 2014 01:27:29 +0200 Subject: [Biopython-dev] SearchIO.parse HMMER3 issue Message-ID: Hi all, I downloaded HMMER3.1b today and tried to use the SearchIO module to parse the results. I ran into a weird error, something to do with a regex. The problem seems to be that after the '//' line, there is another line with '[ok]' that breaks the parser. I'm running HMMER with pretty much default options so I guess something might have changed in the latest version or something weird is happening here with mine? Cheers, Jo?o Traceback (most recent call last): File "do_hmmer.py", line 99, in results = SearchIO.read(ali_fname, 'hmmer3-text') File "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", line 363, in read second = next(generator) File "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", line 316, in parse yield qresult File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__ self.gen.throw(type, value, traceback) File "/home/software/python-libs/lib64/python2.6/site-packages/Bio/File.py", line 77, in as_handle yield fp File "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", line 315, in parse for qresult in generator: File "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/HmmerIO/hmmer3_text.py", line 47, in __iter__ for qresult in self._parse_qresult(): File "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/HmmerIO/hmmer3_text.py", line 110, in _parse_qresult qid = regx.group(1).strip() AttributeError: 'NoneType' object has no attribute 'group' From w.arindrarto at gmail.com Thu Apr 24 23:36:07 2014 From: w.arindrarto at gmail.com (Wibowo Arindrarto) Date: Fri, 25 Apr 2014 01:36:07 +0200 Subject: [Biopython-dev] SearchIO.parse HMMER3 issue In-Reply-To: References: Message-ID: Hi Joao, Yes, this is a known issue with version 3.1b1. And indeed, the cause is the '[ok]' line (just one last line in the file). I haven't finished testing some cases (and I intended to add proper support for 3.1 ~ a.ka. nhmmer + nhmmscan ~ with the update), so my updates for this has not been up (probably sometime next week, not guaranteeing anything though..). But if you need a quick solution, removing the '[ok]' line fixes the problems in my case. Which reminds me, I think I need to properly track this issue in our issue tracker :). There is an earlier post reporting this issue (http://permalink.gmane.org/gmane.comp.python.bio.general/8027), but I think we ended up emailing privately (where I mentioned this issue). Anyway, thanks for the report. Cheers, Bow On Fri, Apr 25, 2014 at 1:27 AM, Jo?o Rodrigues wrote: > Hi all, > > I downloaded HMMER3.1b today and tried to use the SearchIO module to parse > the results. I ran into a weird error, something to do with a regex. > > The problem seems to be that after the '//' line, there is another line > with '[ok]' that breaks the parser. I'm running HMMER with pretty much > default options so I guess something might have changed in the latest > version or something weird is happening here with mine? > > Cheers, > > Jo?o > > Traceback (most recent call last): > File "do_hmmer.py", line 99, in > results = SearchIO.read(ali_fname, 'hmmer3-text') > File > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", > line 363, in read > second = next(generator) > File > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", > line 316, in parse > yield qresult > File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__ > self.gen.throw(type, value, traceback) > File > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/File.py", > line 77, in as_handle > yield fp > File > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", > line 315, in parse > for qresult in generator: > File > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/HmmerIO/hmmer3_text.py", > line 47, in __iter__ > for qresult in self._parse_qresult(): > File > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/HmmerIO/hmmer3_text.py", > line 110, in _parse_qresult > qid = regx.group(1).strip() > AttributeError: 'NoneType' object has no attribute 'group' > > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev From anaryin at gmail.com Thu Apr 24 23:38:54 2014 From: anaryin at gmail.com (=?UTF-8?Q?Jo=C3=A3o_Rodrigues?=) Date: Fri, 25 Apr 2014 01:38:54 +0200 Subject: [Biopython-dev] SearchIO.parse HMMER3 issue In-Reply-To: References: Message-ID: Hey Bow, Thanks, didn't know if it was known or not and I usually don't use these sequence parsers so.. :) Removing it did the trick indeed, seemed the most practical option. Cheers, Jo?o 2014-04-25 1:36 GMT+02:00 Wibowo Arindrarto : > Hi Joao, > > Yes, this is a known issue with version 3.1b1. And indeed, the cause > is the '[ok]' line (just one last line in the file). I haven't > finished testing some cases (and I intended to add proper support for > 3.1 ~ a.ka. nhmmer + nhmmscan ~ with the update), so my updates for > this has not been up (probably sometime next week, not guaranteeing > anything though..). But if you need a quick solution, removing the > '[ok]' line fixes the problems in my case. > > Which reminds me, I think I need to properly track this issue in our > issue tracker :). There is an earlier post reporting this issue > (http://permalink.gmane.org/gmane.comp.python.bio.general/8027), but I > think we ended up emailing privately (where I mentioned this issue). > > Anyway, thanks for the report. > > Cheers, > Bow > > On Fri, Apr 25, 2014 at 1:27 AM, Jo?o Rodrigues wrote: > > Hi all, > > > > I downloaded HMMER3.1b today and tried to use the SearchIO module to > parse > > the results. I ran into a weird error, something to do with a regex. > > > > The problem seems to be that after the '//' line, there is another line > > with '[ok]' that breaks the parser. I'm running HMMER with pretty much > > default options so I guess something might have changed in the latest > > version or something weird is happening here with mine? > > > > Cheers, > > > > Jo?o > > > > Traceback (most recent call last): > > File "do_hmmer.py", line 99, in > > results = SearchIO.read(ali_fname, 'hmmer3-text') > > File > > > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", > > line 363, in read > > second = next(generator) > > File > > > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", > > line 316, in parse > > yield qresult > > File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__ > > self.gen.throw(type, value, traceback) > > File > > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/File.py", > > line 77, in as_handle > > yield fp > > File > > > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", > > line 315, in parse > > for qresult in generator: > > File > > > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/HmmerIO/hmmer3_text.py", > > line 47, in __iter__ > > for qresult in self._parse_qresult(): > > File > > > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/HmmerIO/hmmer3_text.py", > > line 110, in _parse_qresult > > qid = regx.group(1).strip() > > AttributeError: 'NoneType' object has no attribute 'group' > > > > _______________________________________________ > > Biopython-dev mailing list > > Biopython-dev at lists.open-bio.org > > http://lists.open-bio.org/mailman/listinfo/biopython-dev > From w.arindrarto at gmail.com Thu Apr 24 23:45:12 2014 From: w.arindrarto at gmail.com (Wibowo Arindrarto) Date: Fri, 25 Apr 2014 01:45:12 +0200 Subject: [Biopython-dev] SearchIO.parse HMMER3 issue In-Reply-To: References: Message-ID: Hey Joao, No problem. The discussion with the user went to private exchanges anyway, so it was not indeed publicly searchable. In any case, I posted this in our issue tracker (also to remind myself about this :) ): https://github.com/biopython/biopython/issues/313. If you find anything else breaking or something you'd like to have, feel free to post there directly. Cheers, Bow On Fri, Apr 25, 2014 at 1:38 AM, Jo?o Rodrigues wrote: > Hey Bow, > > Thanks, didn't know if it was known or not and I usually don't use these > sequence parsers so.. :) Removing it did the trick indeed, seemed the most > practical option. > > Cheers, > > Jo?o > > > 2014-04-25 1:36 GMT+02:00 Wibowo Arindrarto : > >> Hi Joao, >> >> Yes, this is a known issue with version 3.1b1. And indeed, the cause >> is the '[ok]' line (just one last line in the file). I haven't >> finished testing some cases (and I intended to add proper support for >> 3.1 ~ a.ka. nhmmer + nhmmscan ~ with the update), so my updates for >> this has not been up (probably sometime next week, not guaranteeing >> anything though..). But if you need a quick solution, removing the >> '[ok]' line fixes the problems in my case. >> >> Which reminds me, I think I need to properly track this issue in our >> issue tracker :). There is an earlier post reporting this issue >> (http://permalink.gmane.org/gmane.comp.python.bio.general/8027), but I >> think we ended up emailing privately (where I mentioned this issue). >> >> Anyway, thanks for the report. >> >> Cheers, >> Bow >> >> On Fri, Apr 25, 2014 at 1:27 AM, Jo?o Rodrigues wrote: >> > Hi all, >> > >> > I downloaded HMMER3.1b today and tried to use the SearchIO module to >> > parse >> > the results. I ran into a weird error, something to do with a regex. >> > >> > The problem seems to be that after the '//' line, there is another line >> > with '[ok]' that breaks the parser. I'm running HMMER with pretty much >> > default options so I guess something might have changed in the latest >> > version or something weird is happening here with mine? >> > >> > Cheers, >> > >> > Jo?o >> > >> > Traceback (most recent call last): >> > File "do_hmmer.py", line 99, in >> > results = SearchIO.read(ali_fname, 'hmmer3-text') >> > File >> > >> > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", >> > line 363, in read >> > second = next(generator) >> > File >> > >> > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", >> > line 316, in parse >> > yield qresult >> > File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__ >> > self.gen.throw(type, value, traceback) >> > File >> > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/File.py", >> > line 77, in as_handle >> > yield fp >> > File >> > >> > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/__init__.py", >> > line 315, in parse >> > for qresult in generator: >> > File >> > >> > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/HmmerIO/hmmer3_text.py", >> > line 47, in __iter__ >> > for qresult in self._parse_qresult(): >> > File >> > >> > "/home/software/python-libs/lib64/python2.6/site-packages/Bio/SearchIO/HmmerIO/hmmer3_text.py", >> > line 110, in _parse_qresult >> > qid = regx.group(1).strip() >> > AttributeError: 'NoneType' object has no attribute 'group' >> > >> > _______________________________________________ >> > Biopython-dev mailing list >> > Biopython-dev at lists.open-bio.org >> > http://lists.open-bio.org/mailman/listinfo/biopython-dev > > From w.arindrarto at gmail.com Fri Apr 25 00:05:51 2014 From: w.arindrarto at gmail.com (Wibowo Arindrarto) Date: Fri, 25 Apr 2014 02:05:51 +0200 Subject: [Biopython-dev] Test files leftovers Message-ID: Hello everyone, I was running full tests when I noticed that some of our test cases seem to create files for testing but not delete them afterwards. I have these from my tests: BWA/1.sai BWA/1.sam BWA/2.sai BWA/human_g1k_v37_truncated.fasta.amb BWA/human_g1k_v37_truncated.fasta.ann BWA/human_g1k_v37_truncated.fasta.bwt BWA/human_g1k_v37_truncated.fasta.pac BWA/human_g1k_v37_truncated.fasta.sa BWA/out.sam SamBam/bam1.bam.bai TreeConstruction/bootstrap_consensus.tre TreeConstruction/pars1.tre TreeConstruction/pars2.tre bam1.bam Now, I could just add these files to .gitignore, but I think it's more proper if we at least try to remove them after every test. In some cases, I saw that the creation of these files is actually the test. Which means having them before the test (i.e. from a previous run) may not be ideal. But I could be wrong, so I'm sending this first to ask whether these files should have been deleted or not :). Cheers, Bow From saketkc at gmail.com Fri Apr 25 00:11:26 2014 From: saketkc at gmail.com (Saket Choudhary) Date: Fri, 25 Apr 2014 05:41:26 +0530 Subject: [Biopython-dev] Test files leftovers In-Reply-To: References: Message-ID: Hi Bow, These are generated by the test functions written for BWA wrapper which I wrote. I could send a PR which does a 'cleanup' in tearDown()? Saket On 25 April 2014 05:35, Wibowo Arindrarto wrote: > Hello everyone, > > I was running full tests when I noticed that some of our test cases > seem to create files for testing but not delete them afterwards. > > I have these from my tests: > > BWA/1.sai > BWA/1.sam > BWA/2.sai > BWA/human_g1k_v37_truncated.fasta.amb > BWA/human_g1k_v37_truncated.fasta.ann > BWA/human_g1k_v37_truncated.fasta.bwt > BWA/human_g1k_v37_truncated.fasta.pac > BWA/human_g1k_v37_truncated.fasta.sa > BWA/out.sam > SamBam/bam1.bam.bai > TreeConstruction/bootstrap_consensus.tre > TreeConstruction/pars1.tre > TreeConstruction/pars2.tre > bam1.bam > > Now, I could just add these files to .gitignore, but I think it's more > proper if we at least try to remove them after every test. In some > cases, I saw that the creation of these files is actually the test. > Which means having them before the test (i.e. from a previous run) may > not be ideal. > > But I could be wrong, so I'm sending this first to ask whether these > files should have been deleted or not :). > > Cheers, > Bow > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev From saketkc at gmail.com Fri Apr 25 01:27:10 2014 From: saketkc at gmail.com (Saket Choudhary) Date: Fri, 25 Apr 2014 06:57:10 +0530 Subject: [Biopython-dev] Test files leftovers In-Reply-To: References: Message-ID: Maybe, I need some help here: A simple tearDown() approach won't work as I need the indexed files to be present for 'test_samse' and 'test_sampe' to be run. What I need is a tearDownClass() approach but it is not supported on python2.6. Any suggestions? Saket On 25 April 2014 05:41, Saket Choudhary wrote: > Hi Bow, > > These are generated by the test functions written for BWA wrapper > which I wrote. I could send a PR which does a 'cleanup' in > tearDown()? > > Saket > > On 25 April 2014 05:35, Wibowo Arindrarto wrote: >> Hello everyone, >> >> I was running full tests when I noticed that some of our test cases >> seem to create files for testing but not delete them afterwards. >> >> I have these from my tests: >> >> BWA/1.sai >> BWA/1.sam >> BWA/2.sai >> BWA/human_g1k_v37_truncated.fasta.amb >> BWA/human_g1k_v37_truncated.fasta.ann >> BWA/human_g1k_v37_truncated.fasta.bwt >> BWA/human_g1k_v37_truncated.fasta.pac >> BWA/human_g1k_v37_truncated.fasta.sa >> BWA/out.sam >> SamBam/bam1.bam.bai >> TreeConstruction/bootstrap_consensus.tre >> TreeConstruction/pars1.tre >> TreeConstruction/pars2.tre >> bam1.bam >> >> Now, I could just add these files to .gitignore, but I think it's more >> proper if we at least try to remove them after every test. In some >> cases, I saw that the creation of these files is actually the test. >> Which means having them before the test (i.e. from a previous run) may >> not be ideal. >> >> But I could be wrong, so I'm sending this first to ask whether these >> files should have been deleted or not :). >> >> Cheers, >> Bow >> _______________________________________________ >> Biopython-dev mailing list >> Biopython-dev at lists.open-bio.org >> http://lists.open-bio.org/mailman/listinfo/biopython-dev From saketkc at gmail.com Fri Apr 25 04:43:42 2014 From: saketkc at gmail.com (Saket Choudhary) Date: Fri, 25 Apr 2014 10:13:42 +0530 Subject: [Biopython-dev] Test files leftovers In-Reply-To: References: Message-ID: Apparently I have polluted the master branch of my biopython clone: https://github.com/saketkc/biopython PR on a fake clone: https://github.com/saketkc/biopython-2/pull/1/files On 25 April 2014 06:57, Saket Choudhary wrote: > Maybe, I need some help here: > > A simple tearDown() approach won't work as I need the indexed files to > be present for 'test_samse' and 'test_sampe' to be run. > > What I need is a tearDownClass() approach but it is not supported on python2.6. > > Any suggestions? > > Saket > > On 25 April 2014 05:41, Saket Choudhary wrote: >> Hi Bow, >> >> These are generated by the test functions written for BWA wrapper >> which I wrote. I could send a PR which does a 'cleanup' in >> tearDown()? >> >> Saket >> >> On 25 April 2014 05:35, Wibowo Arindrarto wrote: >>> Hello everyone, >>> >>> I was running full tests when I noticed that some of our test cases >>> seem to create files for testing but not delete them afterwards. >>> >>> I have these from my tests: >>> >>> BWA/1.sai >>> BWA/1.sam >>> BWA/2.sai >>> BWA/human_g1k_v37_truncated.fasta.amb >>> BWA/human_g1k_v37_truncated.fasta.ann >>> BWA/human_g1k_v37_truncated.fasta.bwt >>> BWA/human_g1k_v37_truncated.fasta.pac >>> BWA/human_g1k_v37_truncated.fasta.sa >>> BWA/out.sam >>> SamBam/bam1.bam.bai >>> TreeConstruction/bootstrap_consensus.tre >>> TreeConstruction/pars1.tre >>> TreeConstruction/pars2.tre >>> bam1.bam >>> >>> Now, I could just add these files to .gitignore, but I think it's more >>> proper if we at least try to remove them after every test. In some >>> cases, I saw that the creation of these files is actually the test. >>> Which means having them before the test (i.e. from a previous run) may >>> not be ideal. >>> >>> But I could be wrong, so I'm sending this first to ask whether these >>> files should have been deleted or not :). >>> >>> Cheers, >>> Bow >>> _______________________________________________ >>> Biopython-dev mailing list >>> Biopython-dev at lists.open-bio.org >>> http://lists.open-bio.org/mailman/listinfo/biopython-dev From manlio.calvi at gmail.com Fri Apr 25 06:27:31 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Fri, 25 Apr 2014 08:27:31 +0200 Subject: [Biopython-dev] Test files leftovers In-Reply-To: References: Message-ID: If the problem is delete the files at the end of the tests maybe the last test in BWA test suite search for all those files with one or more regex and call os.remove() method to delete it, you don't need the files after all tests on them runs out. Another cleaner way, although need more work, is create a folder test inside the BWA test suite and put there the files created, run there the tests and then remove the folder at the end, this mean rewrite the path to that test folder. Manlio On Fri, Apr 25, 2014 at 6:43 AM, Saket Choudhary wrote: > Apparently I have polluted the master branch of my biopython clone: > https://github.com/saketkc/biopython > > PR on a fake clone: https://github.com/saketkc/biopython-2/pull/1/files > > On 25 April 2014 06:57, Saket Choudhary wrote: >> Maybe, I need some help here: >> >> A simple tearDown() approach won't work as I need the indexed files to >> be present for 'test_samse' and 'test_sampe' to be run. >> >> What I need is a tearDownClass() approach but it is not supported on python2.6. >> >> Any suggestions? >> >> Saket >> >> On 25 April 2014 05:41, Saket Choudhary wrote: >>> Hi Bow, >>> >>> These are generated by the test functions written for BWA wrapper >>> which I wrote. I could send a PR which does a 'cleanup' in >>> tearDown()? >>> >>> Saket >>> >>> On 25 April 2014 05:35, Wibowo Arindrarto wrote: >>>> Hello everyone, >>>> >>>> I was running full tests when I noticed that some of our test cases >>>> seem to create files for testing but not delete them afterwards. >>>> >>>> I have these from my tests: >>>> >>>> BWA/1.sai >>>> BWA/1.sam >>>> BWA/2.sai >>>> BWA/human_g1k_v37_truncated.fasta.amb >>>> BWA/human_g1k_v37_truncated.fasta.ann >>>> BWA/human_g1k_v37_truncated.fasta.bwt >>>> BWA/human_g1k_v37_truncated.fasta.pac >>>> BWA/human_g1k_v37_truncated.fasta.sa >>>> BWA/out.sam >>>> SamBam/bam1.bam.bai >>>> TreeConstruction/bootstrap_consensus.tre >>>> TreeConstruction/pars1.tre >>>> TreeConstruction/pars2.tre >>>> bam1.bam >>>> >>>> Now, I could just add these files to .gitignore, but I think it's more >>>> proper if we at least try to remove them after every test. In some >>>> cases, I saw that the creation of these files is actually the test. >>>> Which means having them before the test (i.e. from a previous run) may >>>> not be ideal. >>>> >>>> But I could be wrong, so I'm sending this first to ask whether these >>>> files should have been deleted or not :). >>>> >>>> Cheers, >>>> Bow >>>> _______________________________________________ >>>> Biopython-dev mailing list >>>> Biopython-dev at lists.open-bio.org >>>> http://lists.open-bio.org/mailman/listinfo/biopython-dev > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev From p.j.a.cock at googlemail.com Fri Apr 25 07:08:49 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Fri, 25 Apr 2014 08:08:49 +0100 Subject: [Biopython-dev] Test files leftovers In-Reply-To: References: Message-ID: On Fri, Apr 25, 2014 at 2:27 AM, Saket Choudhary wrote: > Maybe, I need some help here: > > A simple tearDown() approach won't work as I need > the indexed files to be present for 'test_samse' and > 'test_sampe' to be run. Suggestion 1: Could you fix these test methods to be self contained? i.e. include building the index > What I need is a tearDownClass() approach but it > is not supported on python2.6. Suggestion 2: Not ideal, but use that and comment that it doesn't work on Python 2.6? > Any suggestions? > > Saket Peter From p.j.a.cock at googlemail.com Fri Apr 25 07:15:10 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Fri, 25 Apr 2014 08:15:10 +0100 Subject: [Biopython-dev] Test files leftovers In-Reply-To: References: Message-ID: On Fri, Apr 25, 2014 at 7:27 AM, Manlio Calvi wrote: > If the problem is delete the files at the end of the tests maybe the > last test in BWA test suite search for all those files with one or > more regex and call os.remove() method to delete it, you don't need > the files after all tests on them runs out. > Another cleaner way, although need more work, is create a folder test > inside the BWA test suite and put there the files created, run there > the tests and then remove the folder at the end, this mean rewrite the > path to that test folder. > > Manlio Same problem - ensuring that the folder/files are removed after all the tests. This is what the Python 2.7+ method tearDownClass() is for. Peter From p.j.a.cock at googlemail.com Fri Apr 25 07:21:27 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Fri, 25 Apr 2014 08:21:27 +0100 Subject: [Biopython-dev] Python 3.4 In-Reply-To: References: <20140422102316.5b049105@lnx> Message-ID: On Tue, Apr 22, 2014 at 10:42 AM, Peter Cock wrote: > On Tue, Apr 22, 2014 at 10:23 AM, Tiago Antao wrote: >> Dear all (especially Manlio and Peter ;) ), >> >> With Ubuntu trusty officially out, I am planning on doing a (Linux) >> docker file running Python 3.4 and attach it to buildbot. Hopefully >> today. This would allow us to have a stable testing environment of 3.4 >> against Linux... Which might give some clues to what is happening with >> 3.4 on Windows... >> >> Tiago > > Good plan :) > > I've already got Python 3.4 on three build slaves ready to > be connected (Windows XP 32 bit, Linux 64 bit, and Mac > OS X 64 bit), so we'll have the main platforms covered. I've asked the buildbot master to run tests for Python 3.4 on the Mac now, and got another 64bit Linux machine ready to run the tests. Oddly the buildbot server has become quite sluggish, perhaps we have too much old data and/or too many build targets? Removing the older PyPy versions from the matrix should help: http://lists.open-bio.org/pipermail/biopython/2014-April/009110.html > Right now the only Python 3.4 specific issue I have seen is > in Bio.trie https://github.com/biopython/biopython/issues/305 Provisional fix checked in. Peter From tra at popgen.net Fri Apr 25 09:05:03 2014 From: tra at popgen.net (Tiago Antao) Date: Fri, 25 Apr 2014 10:05:03 +0100 Subject: [Biopython-dev] Python 3.4 In-Reply-To: References: <20140422102316.5b049105@lnx> Message-ID: <20140425100503.4939bd34@lnx> On Fri, 25 Apr 2014 08:21:27 +0100 Peter Cock wrote: > I've asked the buildbot master to run tests for Python 3.4 > on the Mac now, and got another 64bit Linux machine > ready to run the tests. I hope to lunch mine this morning (or early afternoon - meetings...). The docker work is finished. > Oddly the buildbot server has become quite sluggish, > perhaps we have too much old data and/or too many > build targets? Removing the older PyPy versions > from the matrix should help: > http://lists.open-bio.org/pipermail/biopython/2014-April/009110.html I think we need to purge the old data. I do not think we have anything resembling too much build targets. I will try to purge the old data myself today. From tra at popgen.net Fri Apr 25 12:57:36 2014 From: tra at popgen.net (Tiago Antao) Date: Fri, 25 Apr 2014 13:57:36 +0100 Subject: [Biopython-dev] Python 3.4 and buildbot Message-ID: <20140425135736.7bfb4a8b@grandao> Dear all, Just a summary feedback from my part regarding buildbot and Python 3.4 Buildbot: I have cleanup the history to accelarate things. I might have cleaned too much and we lost all the past information. I guess it is not important, but it was an historical record of sorts... Python 3.4: I have set-up a couple of docker images: Biopython out of the box: https://github.com/tiagoantao/my-containers/blob/master/biopython/Biopython3 For buildbot: https://github.com/tiagoantao/my-containers/blob/master/biopython/Biopython3-Test The buildbot results are now available: http://testing.open-bio.org/biopython/builders/Linux%2064%20-%20Python%203.4/builds/1/steps/shell/logs/stdio [There seem to be a few problems] I would like to notice that it seems that there is no mysqldb connector for python3 (??!!) and thus we cannot test that. Tiago From p.j.a.cock at googlemail.com Sat Apr 26 17:19:05 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Sat, 26 Apr 2014 18:19:05 +0100 Subject: [Biopython-dev] TogoWS test failure, was: Compiling Biopython on 64 bit Windows In-Reply-To: References: Message-ID: On Sat, Apr 19, 2014 at 1:23 PM, Peter Cock wrote: > The TogoWS team confirmed this is a problem with the > server and expect it to be fixed shortly. > > Peter For the record, this is working now :) Peter From p.j.a.cock at googlemail.com Sun Apr 27 12:43:23 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Sun, 27 Apr 2014 13:43:23 +0100 Subject: [Biopython-dev] BWA command line change? Message-ID: Hi Saket, Could you have a look at this BWA problem Andreas found while testing the Debian package? I have just reordered the error message in Bio.Application to put the return code earlier in case the exception text is truncated (as here): https://github.com/biopython/biopython/commit/9860973c05aeb102f82e217ff3a88e241a3ba620 I've also switched to relative paths which will also shorten the error message: https://github.com/biopython/biopython/commit/a03d23760783c157ff7c30f483ace55beda31c09 I just tested with BWA 0.6.1 (in the Debian repository on my old desktop), and test_BWA_tool.py was fine. Thanks, Peter ---------- Forwarded message ---------- From: Andreas Tille Date: Tue, Mar 4, 2014 at 8:18 PM Subject: Re: Adding python3-biopython To: Peter Cock Cc: Debian Med Project List , Philipp Benner Hi Peter, I can confirm that I'm working on the python3 packages. I'm not finished yet but I included more verbose testing into the packaging and when I did so I realised some incompatibilities with bwa: ERROR: test_sampe (test_BWA_tool.BwaTestCase) Test for generating samfile by paired end sequencing ---------------------------------------------------------------------- Traceback (most recent call last): File "test_BWA_tool.py", line 113, in test_sampe self.do_aln(self.infile1, self.saifile1) File "test_BWA_tool.py", line 90, in do_aln stdout, stderr = cmdline(stdout=out_file) File "/tmp/buildd/python-biopython-1.63/Bio/Application/__init__.py", line 511, in __call__ stdout_str, stderr_str) ApplicationError: Command 'bwa aln /tmp/buildd/python-biopython-1.63/Tests/BWA/human_g1k_v37_truncated.fasta /tmp/buildd/python-biopython-1.63/Tests/BWA/HNSCC1_1_truncated.fastq' returne ====================================================================== ERROR: test_samse (test_BWA_tool.BwaTestCase) Test for single end sequencing ---------------------------------------------------------------------- Traceback (most recent call last): File "test_BWA_tool.py", line 102, in test_samse stdout, stderr = cmdline(stdout=self.samfile1) File "/tmp/buildd/python-biopython-1.63/Bio/Application/__init__.py", line 511, in __call__ stdout_str, stderr_str) ApplicationError: Command 'bwa samse /tmp/buildd/python-biopython-1.63/Tests/BWA/human_g1k_v37_truncated.fasta /tmp/buildd/python-biopython-1.63/Tests/BWA/1.sai /tmp/buildd/python-biopyt The used bwa says: $ bwa Program: bwa (alignment via Burrows-Wheeler transformation) Version: 0.7.6a-r433 Contact: Heng Li Usage: bwa [options] Command: index index sequences in the FASTA format mem BWA-MEM algorithm fastmap identify super-maximal exact matches fa2pac convert FASTA to PAC format pac2bwt generate BWT from PAC pac2bwtgen alternative algorithm for generating BWT bwtupdate update .bwt to the new format bwt2sa generate SA from BWT and Occ Note: To use BWA, you need to first index the genome with `bwa index'. There are three alignment algorithms in BWA: `mem', `bwasw', and `aln/samse/sampe'. If you are not sure which to use, try `bwa mem' first. Please `man ./bwa.1' for the manual. I think you should adjust your test somehow to make sure that also the latest bwa version will succeed. Kind regards Andreas. On Mon, Mar 03, 2014 at 02:08:52PM +0000, Peter Cock wrote: > Hello Philipp & DebianMed, > > The last few releases of Biopython supported Python 3 > via 2to3, but as of Biopython 1.63 (released December > 2013) we have a single codebase which works on both > Python 2.6/2.7 and Python 3.3+. ... -- http://fam-tille.de From manlio.calvi at gmail.com Mon Apr 28 08:09:54 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 28 Apr 2014 10:09:54 +0200 Subject: [Biopython-dev] Pypy2 on Win7 x64, numpy problem Message-ID: Hi all, Yesterday, after a short break on compiling, I tried to build my pypy 2 stack to run the biopython tests on it, the problem is on numpy, for every thing I tried to compile their numpy version I failed, out of curiosity I tried to compile numpy on pypy3 but get even worse. In the end both don't compile, at this point I stopped after looking my pypy envs are sane. Manlio From p.j.a.cock at googlemail.com Mon Apr 28 12:51:10 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 28 Apr 2014 13:51:10 +0100 Subject: [Biopython-dev] Pypy2 on Win7 x64, numpy problem In-Reply-To: References: Message-ID: On Mon, Apr 28, 2014 at 9:09 AM, Manlio Calvi wrote: > Hi all, > > Yesterday, after a short break on compiling, I tried to build my pypy > 2 stack to run the biopython tests on it, the problem is on numpy, for > every thing I tried to compile their numpy version I failed, out of > curiosity I tried to compile numpy on pypy3 but get even worse. In the > end both don't compile, at this point I stopped after looking my pypy > envs are sane. > > Manlio NumPy has a ton of C code, and will not work on PyPy. Attempting to compile/install NumPy under PyPy is a waste of time. However, the PyPy team are reimplementing (some of) NumPy themselves as the NumPyPy project - see e.g. http://buildbot.pypy.org/numpy-status/latest.html Currently Biopython ignores NumPy (and NumPyPy) under PyPy - last time I did a proper test there were still too many things missing to make it worth the effort. However, this is something we should review - I would hope by now that at least the PDB parsing code might work - even if things like SVD and numpy.linalg are not available on PyPy. Similarly, NumPy won't work under Jython. Peter From manlio.calvi at gmail.com Mon Apr 28 13:02:34 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 28 Apr 2014 15:02:34 +0200 Subject: [Biopython-dev] Pypy2 on Win7 x64, numpy problem In-Reply-To: References: Message-ID: I tried to install their Numpy (the Numpypy project) because I read about their difficulties about porting Cpython Numpy but even with that I get errors everywhere and don't compile it. OK giving the tests ignore Numpy(py) I'll try to build biopython against it. On Mon, Apr 28, 2014 at 2:51 PM, Peter Cock wrote: > On Mon, Apr 28, 2014 at 9:09 AM, Manlio Calvi wrote: >> Hi all, >> >> Yesterday, after a short break on compiling, I tried to build my pypy >> 2 stack to run the biopython tests on it, the problem is on numpy, for >> every thing I tried to compile their numpy version I failed, out of >> curiosity I tried to compile numpy on pypy3 but get even worse. In the >> end both don't compile, at this point I stopped after looking my pypy >> envs are sane. >> >> Manlio > > NumPy has a ton of C code, and will not work on PyPy. > Attempting to compile/install NumPy under PyPy is a waste > of time. > > However, the PyPy team are reimplementing (some of) > NumPy themselves as the NumPyPy project - see e.g. > http://buildbot.pypy.org/numpy-status/latest.html > > Currently Biopython ignores NumPy (and NumPyPy) under > PyPy - last time I did a proper test there were still too many > things missing to make it worth the effort. > > However, this is something we should review - I would hope > by now that at least the PDB parsing code might work - even > if things like SVD and numpy.linalg are not available on PyPy. > > Similarly, NumPy won't work under Jython. > > Peter From manlio.calvi at gmail.com Mon Apr 28 18:53:43 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 28 Apr 2014 20:53:43 +0200 Subject: [Biopython-dev] Pypy x64 tests Was: Pypy2 on Win7 x64, numpy problem Message-ID: OK the Tests on Pypy 2 passed but a lot was skipped for one reason to another. After this mail I'll send another with builds and tests log for archive and lookups Manlio On Mon, Apr 28, 2014 at 3:02 PM, Manlio Calvi wrote: > I tried to install their Numpy (the Numpypy project) because I read > about their difficulties about porting Cpython Numpy but even with > that I get errors everywhere and don't compile it. > OK giving the tests ignore Numpy(py) I'll try to build biopython against it. > > > > On Mon, Apr 28, 2014 at 2:51 PM, Peter Cock wrote: >> On Mon, Apr 28, 2014 at 9:09 AM, Manlio Calvi wrote: >>> Hi all, >>> >>> Yesterday, after a short break on compiling, I tried to build my pypy >>> 2 stack to run the biopython tests on it, the problem is on numpy, for >>> every thing I tried to compile their numpy version I failed, out of >>> curiosity I tried to compile numpy on pypy3 but get even worse. In the >>> end both don't compile, at this point I stopped after looking my pypy >>> envs are sane. >>> >>> Manlio >> >> NumPy has a ton of C code, and will not work on PyPy. >> Attempting to compile/install NumPy under PyPy is a waste >> of time. >> >> However, the PyPy team are reimplementing (some of) >> NumPy themselves as the NumPyPy project - see e.g. >> http://buildbot.pypy.org/numpy-status/latest.html >> >> Currently Biopython ignores NumPy (and NumPyPy) under >> PyPy - last time I did a proper test there were still too many >> things missing to make it worth the effort. >> >> However, this is something we should review - I would hope >> by now that at least the PDB parsing code might work - even >> if things like SVD and numpy.linalg are not available on PyPy. >> >> Similarly, NumPy won't work under Jython. >> >> Peter From manlio.calvi at gmail.com Mon Apr 28 19:07:56 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 28 Apr 2014 21:07:56 +0200 Subject: [Biopython-dev] Tests on x64 Win 7 Message-ID: Hi people, I rebuild and retest the last biopython commit, no test failed so I here signal some recurrent warning in python3.4, nothing too bad but there is a lot of file left not closed. This problem could be related to the one with the remaing files after the tests? Seems to me was the same files. If that the case it could be a class who open and don't the files after the read. PS after this mail I'll send another mail with the logs. Please Peter, acknowledge that mail (and the one before that on Pypy subject). Manlio From manlio.calvi at gmail.com Mon Apr 28 18:55:33 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 28 Apr 2014 20:55:33 +0200 Subject: [Biopython-dev] Pypy x64 tests Was: Pypy2 on Win7 x64, numpy problem In-Reply-To: References: Message-ID: Here's the logs, enjoy! Manlio On Mon, Apr 28, 2014 at 8:53 PM, Manlio Calvi wrote: > OK the Tests on Pypy 2 passed but a lot was skipped for one reason to another. > After this mail I'll send another with builds and tests log for > archive and lookups > > Manlio > > > On Mon, Apr 28, 2014 at 3:02 PM, Manlio Calvi wrote: >> I tried to install their Numpy (the Numpypy project) because I read >> about their difficulties about porting Cpython Numpy but even with >> that I get errors everywhere and don't compile it. >> OK giving the tests ignore Numpy(py) I'll try to build biopython against it. >> >> >> >> On Mon, Apr 28, 2014 at 2:51 PM, Peter Cock wrote: >>> On Mon, Apr 28, 2014 at 9:09 AM, Manlio Calvi wrote: >>>> Hi all, >>>> >>>> Yesterday, after a short break on compiling, I tried to build my pypy >>>> 2 stack to run the biopython tests on it, the problem is on numpy, for >>>> every thing I tried to compile their numpy version I failed, out of >>>> curiosity I tried to compile numpy on pypy3 but get even worse. In the >>>> end both don't compile, at this point I stopped after looking my pypy >>>> envs are sane. >>>> >>>> Manlio >>> >>> NumPy has a ton of C code, and will not work on PyPy. >>> Attempting to compile/install NumPy under PyPy is a waste >>> of time. >>> >>> However, the PyPy team are reimplementing (some of) >>> NumPy themselves as the NumPyPy project - see e.g. >>> http://buildbot.pypy.org/numpy-status/latest.html >>> >>> Currently Biopython ignores NumPy (and NumPyPy) under >>> PyPy - last time I did a proper test there were still too many >>> things missing to make it worth the effort. >>> >>> However, this is something we should review - I would hope >>> by now that at least the PDB parsing code might work - even >>> if things like SVD and numpy.linalg are not available on PyPy. >>> >>> Similarly, NumPy won't work under Jython. >>> >>> Peter -------------- next part -------------- A non-text attachment was scrubbed... Name: buildpypy.log Type: application/octet-stream Size: 41621 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: testpypy.log Type: application/octet-stream Size: 14772 bytes Desc: not available URL: From manlio.calvi at gmail.com Mon Apr 28 19:09:43 2014 From: manlio.calvi at gmail.com (Manlio Calvi) Date: Mon, 28 Apr 2014 21:09:43 +0200 Subject: [Biopython-dev] Tests on x64 Win 7 In-Reply-To: References: Message-ID: Here's the files Manlio On Mon, Apr 28, 2014 at 9:07 PM, Manlio Calvi wrote: > Hi people, > > I rebuild and retest the last biopython commit, no test failed so I > here signal some recurrent warning in python3.4, nothing too bad but > there is a lot of file left not closed. This problem could be related > to the one with the remaing files after the tests? Seems to me was the > same files. > If that the case it could be a class who open and don't the files > after the read. > > PS after this mail I'll send another mail with the logs. Please Peter, > acknowledge that mail (and the one before that on Pypy subject). > > Manlio -------------- next part -------------- A non-text attachment was scrubbed... Name: build27.log Type: application/octet-stream Size: 63700 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test27.log Type: application/octet-stream Size: 10789 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: build34.log Type: application/octet-stream Size: 63796 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test34.log Type: application/octet-stream Size: 24973 bytes Desc: not available URL: From p.j.a.cock at googlemail.com Mon Apr 28 19:49:16 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 28 Apr 2014 20:49:16 +0100 Subject: [Biopython-dev] Tests on x64 Win 7 In-Reply-To: References: Message-ID: On Mon, Apr 28, 2014 at 8:07 PM, Manlio Calvi wrote: > Hi people, > > I rebuild and retest the last biopython commit, no test failed so I > here signal some recurrent warning in python3.4, nothing too bad but > there is a lot of file left not closed. This problem could be related > to the one with the remaing files after the tests? Seems to me was the > same files. > If that the case it could be a class who open and don't the files > after the read. The problem is lots of (old) Python code tends to do something like this: handle = open(filename) handle.read() etc #handle.close() missing This relies on C Python doing garbage collection when the handle variable goes out of scope, which will close the handle. This is not so predictable on alternative Python implementations like Jython and PyPy so the handle may not get closed for some time. That can cause problems, for example the OS imposes limits on the number of open files, and Windows will refuse to delete a file if there are handles open to it. The solution is to either close the handle explicitly, or use a context manager (with statement). Now in some senses the warning is harmless, see: http://emptysqua.re/blog/against-resourcewarnings-in-python-3/ By default the ResourceWarning is silent, but it seems something sometimes goes wrong with our test framework and this starts to show up (due to a possible design flaw, the Python warnings filters are global): http://lists.open-bio.org/pipermail/biopython-dev/2013-April/010521.html How we test for warnings and/or silence them is something in need of an overhaul - perhaps the warnings.catch_warnings context manager would help (new in Python 2.6, so we can use that)? > PS after this mail I'll send another mail with the logs. Please Peter, > acknowledge that mail (and the one before that on Pypy subject). I approved those emails with attachments which were waiting for moderation. Peter From saketkc at gmail.com Mon Apr 28 20:07:43 2014 From: saketkc at gmail.com (Saket Choudhary) Date: Tue, 29 Apr 2014 01:37:43 +0530 Subject: [Biopython-dev] BWA command line change? In-Reply-To: References: Message-ID: This is turning out to be an issue with this particular version of bwa[0.7.6a-r433];l $ bwa7.6a aln [main] unrecognized command 'aln' However the later two versions do support it, and the tests do pass: $ bwa7.8 aln Usage: bwa aln [options] $ bwa7.7 aln Usage: bwa aln [options] I have bwa7.5[ 0.7.5a-r405] installed which I used for testing and it seems to be working as well. Unfortunately, I was not able to locate this error being reported on the bwa mailing list[The reason probably being more people using 'mem' now] Andrew, can you check this with the latest version of bwa?[0.7.8] Thanks, Saket On 27 April 2014 18:13, Peter Cock wrote: > Hi Saket, > > Could you have a look at this BWA problem Andreas found > while testing the Debian package? > > I have just reordered the error message in Bio.Application > to put the return code earlier in case the exception text is > truncated (as here): > https://github.com/biopython/biopython/commit/9860973c05aeb102f82e217ff3a88e241a3ba620 > > I've also switched to relative paths which will also shorten > the error message: > https://github.com/biopython/biopython/commit/a03d23760783c157ff7c30f483ace55beda31c09 > > I just tested with BWA 0.6.1 (in the Debian repository on > my old desktop), and test_BWA_tool.py was fine. > > Thanks, > > Peter > > > ---------- Forwarded message ---------- > From: Andreas Tille > Date: Tue, Mar 4, 2014 at 8:18 PM > Subject: Re: Adding python3-biopython > To: Peter Cock > Cc: Debian Med Project List , Philipp > Benner > > > Hi Peter, > > I can confirm that I'm working on the python3 packages. I'm not > finished yet but I included more verbose testing into the packaging and > when I did so I realised some incompatibilities with bwa: > > > ERROR: test_sampe (test_BWA_tool.BwaTestCase) > Test for generating samfile by paired end sequencing > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "test_BWA_tool.py", line 113, in test_sampe > self.do_aln(self.infile1, self.saifile1) > File "test_BWA_tool.py", line 90, in do_aln > stdout, stderr = cmdline(stdout=out_file) > File "/tmp/buildd/python-biopython-1.63/Bio/Application/__init__.py", > line 511, in __call__ > stdout_str, stderr_str) > ApplicationError: Command 'bwa aln > /tmp/buildd/python-biopython-1.63/Tests/BWA/human_g1k_v37_truncated.fasta > /tmp/buildd/python-biopython-1.63/Tests/BWA/HNSCC1_1_truncated.fastq' > returne > > ====================================================================== > ERROR: test_samse (test_BWA_tool.BwaTestCase) > Test for single end sequencing > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "test_BWA_tool.py", line 102, in test_samse > stdout, stderr = cmdline(stdout=self.samfile1) > File "/tmp/buildd/python-biopython-1.63/Bio/Application/__init__.py", > line 511, in __call__ > stdout_str, stderr_str) > ApplicationError: Command 'bwa samse > /tmp/buildd/python-biopython-1.63/Tests/BWA/human_g1k_v37_truncated.fasta > /tmp/buildd/python-biopython-1.63/Tests/BWA/1.sai > /tmp/buildd/python-biopyt > > > > The used bwa says: > > $ bwa > > Program: bwa (alignment via Burrows-Wheeler transformation) > Version: 0.7.6a-r433 > Contact: Heng Li > > Usage: bwa [options] > > Command: index index sequences in the FASTA format > mem BWA-MEM algorithm > fastmap identify super-maximal exact matches > > fa2pac convert FASTA to PAC format > pac2bwt generate BWT from PAC > pac2bwtgen alternative algorithm for generating BWT > bwtupdate update .bwt to the new format > bwt2sa generate SA from BWT and Occ > > Note: To use BWA, you need to first index the genome with `bwa index'. > There are three alignment algorithms in BWA: `mem', `bwasw', and > `aln/samse/sampe'. If you are not sure which to use, try `bwa mem' > first. Please `man ./bwa.1' for the manual. > > > I think you should adjust your test somehow to make sure that > also the latest bwa version will succeed. > > Kind regards > > Andreas. > > > On Mon, Mar 03, 2014 at 02:08:52PM +0000, Peter Cock wrote: >> Hello Philipp & DebianMed, >> >> The last few releases of Biopython supported Python 3 >> via 2to3, but as of Biopython 1.63 (released December >> 2013) we have a single codebase which works on both >> Python 2.6/2.7 and Python 3.3+. > ... > > -- > http://fam-tille.de From p.j.a.cock at googlemail.com Mon Apr 28 20:23:51 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Mon, 28 Apr 2014 21:23:51 +0100 Subject: [Biopython-dev] BWA command line change? In-Reply-To: References: Message-ID: On Mon, Apr 28, 2014 at 9:07 PM, Saket Choudhary wrote: > This is turning out to be an issue with this particular version of > bwa[0.7.6a-r433];l > $ bwa7.6a aln > [main] unrecognized command 'aln' > > However the later two versions do support it, and the tests do pass: The simplest solution may be to check for that error during the header where we see if bwa is installed, and if it is missing the aln command skip the tests? Peter From saketkc at gmail.com Mon Apr 28 21:17:35 2014 From: saketkc at gmail.com (Saket Choudhary) Date: Tue, 29 Apr 2014 02:47:35 +0530 Subject: [Biopython-dev] BWA command line change? In-Reply-To: References: Message-ID: Commited a fix here: https://github.com/saketkc/biopython/commit/d884ee6e12a798b974b3a3996e22b5292ab79be6 If it looks good, I can send a PR? Unfortunately my commit history is a bit muddled up on master. On 29 April 2014 01:53, Peter Cock wrote: > On Mon, Apr 28, 2014 at 9:07 PM, Saket Choudhary wrote: >> This is turning out to be an issue with this particular version of >> bwa[0.7.6a-r433];l >> $ bwa7.6a aln >> [main] unrecognized command 'aln' >> >> However the later two versions do support it, and the tests do pass: > > The simplest solution may be to check for that error during > the header where we see if bwa is installed, and if it is > missing the aln command skip the tests? > > Peter From saketkc at gmail.com Tue Apr 29 12:45:24 2014 From: saketkc at gmail.com (Saket Choudhary) Date: Tue, 29 Apr 2014 18:15:24 +0530 Subject: [Biopython-dev] BioPython Debian packages uploaded (Was: BWA command line change?) In-Reply-To: <20140429091359.GC20235@an3as.eu> References: <20140429091359.GC20235@an3as.eu> Message-ID: Thanks Andreas, Just for the record, I have now submitted this as a pull request: https://github.com/biopython/biopython/pull/315 Saket On 29 April 2014 14:43, Andreas Tille wrote: > Hi, > > I have included the patch below in the just uploaded BioPython package. > The package will need to passe the new queue due to the new packages for > Python 3.3 and 3.4 so it might need some days to become available in > Debian unstable. > > Besides the Python 3 support the other major change is that we try to > run as many tests of the test suite as possible at package build time > and there is also an autopkgtest added which means that in a regular > basis the package is installed automatically on a clean system and the > test suite is runned on the installation. > > As an additional hint for you as developers: Here you can find all > patches applied to the package: > > http://anonscm.debian.org/viewvc/debian-med/trunk/packages/python-biopython/trunk/debian/patches/ > > I think the only patch which I did not reported is > biopython-from-pil-import-image.patch which I'd recommend to apply > upstream as well (description is included). > > Kind regards and thanks for providing BioPython as free software > > Andreas. > > On Tue, Apr 29, 2014 at 02:47:35AM +0530, Saket Choudhary wrote: >> Commited a fix here: >> https://github.com/saketkc/biopython/commit/d884ee6e12a798b974b3a3996e22b5292ab79be6 >> >> If it looks good, I can send a PR? >> Unfortunately my commit history is a bit muddled up on master. >> >> On 29 April 2014 01:53, Peter Cock wrote: >> > On Mon, Apr 28, 2014 at 9:07 PM, Saket Choudhary wrote: >> >> This is turning out to be an issue with this particular version of >> >> bwa[0.7.6a-r433];l >> >> $ bwa7.6a aln >> >> [main] unrecognized command 'aln' >> >> >> >> However the later two versions do support it, and the tests do pass: >> > >> > The simplest solution may be to check for that error during >> > the header where we see if bwa is installed, and if it is >> > missing the aln command skip the tests? >> > >> > Peter >> > > -- > http://fam-tille.de From p.j.a.cock at googlemail.com Wed Apr 30 10:02:52 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 30 Apr 2014 11:02:52 +0100 Subject: [Biopython-dev] Python 3.4 - UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 Message-ID: Hi all, One of the 64 bit Linux buildslaves is showing a unicode problem on Python 3.4 in test_SeqIO_SeqXML.py and test_Phylo.py - perhaps locale related? e.g. http://testing.open-bio.org/biopython/builders/Linux%2064%20-%20Python%203.4/builds/9/steps/shell/logs/stdio UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 797: ordinal not in range(128) Tiago, this is on your new docker3 buildslave - can you reproduce this error 'by hand' and look at the locale settings please? These seem to be the "problem" files: $ hexdump -C PhyloXML/phyloxml_examples.xml | grep " c3 " 00003320 20 5a c3 bc 72 69 63 68 3c 2f 64 65 73 63 3e 0d | Z..rich.| $ hexdump -C SeqXML/rna_example.xml | grep " c3 " 00000310 3c 64 65 73 63 72 69 70 74 69 6f 6e 3e c3 a5 c3 |...| 00000320 85 c3 bc c3 b6 c3 96 c3 9f c3 b8 c3 a4 c2 a2 c2 |................| These are deliberate tests of accented characters (and other non-ASCII text) in a unicode description. Both XML files says they are using UTF-8 as the encoding. Thanks, Peter From p.j.a.cock at googlemail.com Wed Apr 30 10:06:22 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 30 Apr 2014 11:06:22 +0100 Subject: [Biopython-dev] Python 3.4 In-Reply-To: <20140425100503.4939bd34@lnx> References: <20140422102316.5b049105@lnx> <20140425100503.4939bd34@lnx> Message-ID: On Fri, Apr 25, 2014 at 10:05 AM, Tiago Antao wrote: > On Fri, 25 Apr 2014 08:21:27 +0100 Peter Cock wrote: > >> I've asked the buildbot master to run tests for Python 3.4 >> on the Mac now, and got another 64bit Linux machine >> ready to run the tests. > > I hope to lunch mine this morning (or early afternoon - meetings...). > The docker work is finished. I can see it as a buildslave now - well done :) >> Oddly the buildbot server has become quite sluggish, >> perhaps we have too much old data and/or too many >> build targets? Removing the older PyPy versions >> from the matrix should help: >> http://lists.open-bio.org/pipermail/biopython/2014-April/009110.html > > > I think we need to purge the old data. I do not think we have anything > resembling too much build targets. I will try to purge the old data > myself today. Thanks Tiago - its a shame to lose the old builds with error logs, but it was of limited value now. Thanks for doing that - the buildbot webpages are very responsive now :) Now some good news for Python 3.4 - this is now supported by TravisCI, and our tests pass: https://github.com/biopython/biopython/commit/e61afb73fdc6cf5721e16299414e6b4b161a5099 Bad news - another unicode error under Python 3.4: http://lists.open-bio.org/pipermail/biopython-dev/2014-April/011355.html Regards, Peter From p.j.a.cock at googlemail.com Wed Apr 30 10:25:41 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 30 Apr 2014 11:25:41 +0100 Subject: [Biopython-dev] Python 3.4 In-Reply-To: References: <20140422102316.5b049105@lnx> <20140425100503.4939bd34@lnx> Message-ID: For the README file we currently say: We currently recommend using Python 2.7 from http://www.python.org which is the final version of Python 2. Early adopters are encouraged to try Biopython under Python 3.3 as well. How about this?: Most of the third party Python libraries Biopython can use are now supported under Python 3 (e.g. NumPy and ReportLab), but not all (e.g. MySQLdb). Overall we therefore still recommend using Python 2.7 from http://www.python.org which is the final version of Python 2. Peter From tra at popgen.net Wed Apr 30 10:33:11 2014 From: tra at popgen.net (Tiago Antao) Date: Wed, 30 Apr 2014 11:33:11 +0100 Subject: [Biopython-dev] Python 3.4 In-Reply-To: References: <20140422102316.5b049105@lnx> <20140425100503.4939bd34@lnx> Message-ID: <20140430113311.34cff5e1@lnx> I am wondering if it would make sense to replace mysqldb with the pure python mysql connector? That would cut a non-python dependency out whilst making everything python 3 compliant. I would be willing to volunteer time (after the next release) for this. On Wed, 30 Apr 2014 11:25:41 +0100 Peter Cock wrote: > For the README file we currently say: > > We currently recommend using Python 2.7 from http://www.python.org > which is the final version of Python 2. Early adopters are encouraged > to try Biopython under Python 3.3 as well. > > How about this?: > > Most of the third party Python libraries Biopython can use are now > supported under Python 3 (e.g. NumPy and ReportLab), but not all > (e.g. MySQLdb). Overall we therefore still recommend using Python 2.7 > from http://www.python.org which is the final version of Python 2. > > Peter > _______________________________________________ > Biopython-dev mailing list > Biopython-dev at lists.open-bio.org > http://lists.open-bio.org/mailman/listinfo/biopython-dev From p.j.a.cock at googlemail.com Wed Apr 30 10:38:29 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 30 Apr 2014 11:38:29 +0100 Subject: [Biopython-dev] Python 3.4 In-Reply-To: <20140430113311.34cff5e1@lnx> References: <20140422102316.5b049105@lnx> <20140425100503.4939bd34@lnx> <20140430113311.34cff5e1@lnx> Message-ID: Since mysqldb isn't being actively maintained, that seems like a good plan - thanks :) Note several very old bits of Biopython use MySQL (not just BioSQL), but I suspect they are not in wide use, and could be targets for deprecation. Peter On Wed, Apr 30, 2014 at 11:33 AM, Tiago Antao wrote: > I am wondering if it would make sense to replace mysqldb with the pure > python mysql connector? That would cut a non-python dependency out > whilst making everything python 3 compliant. > > I would be willing to volunteer time (after the next release) for > this. > > On Wed, 30 Apr 2014 11:25:41 +0100 > Peter Cock wrote: > >> For the README file we currently say: >> >> We currently recommend using Python 2.7 from http://www.python.org >> which is the final version of Python 2. Early adopters are encouraged >> to try Biopython under Python 3.3 as well. >> >> How about this?: >> >> Most of the third party Python libraries Biopython can use are now >> supported under Python 3 (e.g. NumPy and ReportLab), but not all >> (e.g. MySQLdb). Overall we therefore still recommend using Python 2.7 >> from http://www.python.org which is the final version of Python 2. >> >> Peter >> _______________________________________________ >> Biopython-dev mailing list >> Biopython-dev at lists.open-bio.org >> http://lists.open-bio.org/mailman/listinfo/biopython-dev > From p.j.a.cock at googlemail.com Wed Apr 30 13:24:42 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 30 Apr 2014 14:24:42 +0100 Subject: [Biopython-dev] Bug#746484: file not distributable In-Reply-To: <20140430131703.GD758@an3as.eu> References: <20140430131703.GD758@an3as.eu> Message-ID: Hi Andreas, Are you specifically asking about Biopython 1.63 here? I think you can reasonable exclude this DTD file (and any others under the Bio/Entrez/DTD file). Biopython 1.63 will warn if they are missing but attempt to download them automatically. We're looking at dropping all the NCBI Entrez related DTD files, since the next Biopython release (v1.64) will automatically download AND cache them - see recent discussion, e.g. http://lists.open-bio.org/pipermail/biopython-dev/2014-March/011205.html We haven't actually removed the files on GitHub yet, but this might be an incentive to do so. Thanks, Peter On Wed, Apr 30, 2014 at 2:17 PM, Andreas Tille wrote: > Hello, > > our ftpmaster has detected an issue with one of the DTDs which are > distributed with BioPython source. > > I can confirm that after applying the following patch > > > --- a/Bio/Entrez/DTDs/modules.ent > +++ b/Bio/Entrez/DTDs/modules.ent > @@ -350,13 +350,6 @@ Version Reason/Occasion > "mathmlsetup.ent" > > > > - > - - PUBLIC > -"-//W3C//ENTITIES MathML 2.0 Qualified Names 1.0//EN" > -"mathml/mathml2-qname-1.mod" > > - > - > > "-//W3C//DTD MathML 2.0//EN" > > > the file in question can be removed from the archive without breaking > the build (including the test suite). I would like to suggest to drop > the file in question from your distribution tarball in case my analysis > that it is not needed is correct. > > Kind regards > > Andreas. > > On Wed, Apr 30, 2014 at 02:25:09PM +0200, Thorsten Alteholz wrote: >> Package: python-biopython >> Version: 1.63-2 >> Severity: serious >> User: alteholz at debian.org >> Usertags: ftp >> X-Debbugs-CC: ftpmaster at ftp-master.debian.org >> thanks >> >> Dear Maintainer, >> >> according to: >> http://www.w3.org/Consortium/Legal/2002/copyright-documents-20021231 >> the file >> biopython-1.63\Bio\Entrez\DTDs\mathml2-qname-1.mod >> may not be modified and such this file is not distributable in main. >> >> Thorsten >> >> _______________________________________________ >> Debian-med-packaging mailing list >> Debian-med-packaging at lists.alioth.debian.org >> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/debian-med-packaging >> > > -- > http://fam-tille.de From p.j.a.cock at googlemail.com Wed Apr 30 13:38:11 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 30 Apr 2014 14:38:11 +0100 Subject: [Biopython-dev] Bug#746484: file not distributable In-Reply-To: <20140430133016.GE758@an3as.eu> References: <20140430131703.GD758@an3as.eu> <20140430133016.GE758@an3as.eu> Message-ID: On Wed, Apr 30, 2014 at 2:30 PM, Andreas Tille wrote: > Hi Peter, > > thanks for your super-fast response. > > On Wed, Apr 30, 2014 at 02:24:42PM +0100, Peter Cock wrote: >> Are you specifically asking about Biopython 1.63 here? > > Yes. Since I have added python3 binary packages Biopython 1.63 went > through manual inspection by ftpmaster and this issue was noticed. Very through of them - thanks! Also thank you for doing the Debian Python3 packaging of Biopython :) >> I think you >> can reasonable exclude this DTD file (and any others under the >> Bio/Entrez/DTD file). Biopython 1.63 will warn if they are missing >> but attempt to download them automatically. > > OK. > >> We're looking at dropping all the NCBI Entrez related DTD files, >> since the next Biopython release (v1.64) will automatically download >> AND cache them - see recent discussion, e.g. >> >> http://lists.open-bio.org/pipermail/biopython-dev/2014-March/011205.html > > That's fine. > >> We haven't actually removed the files on GitHub yet, but this >> might be an incentive to do so. > > OK, meanwhile (as long as 1.64 is not yet released) I will remove the > file from the Debian archive. > > Thanks for the clarification > > Andreas. Great, Peter P.S. I'm skimming over the Debian patches to see what we can fix: http://anonscm.debian.org/viewvc/debian-med/trunk/packages/python-biopython/trunk/debian/patches/ e.g. https://github.com/biopython/biopython/commit/2f098ac5311e0eec3d6737f4fff60e18c50b9481 From p.j.a.cock at googlemail.com Wed Apr 30 14:05:51 2014 From: p.j.a.cock at googlemail.com (Peter Cock) Date: Wed, 30 Apr 2014 15:05:51 +0100 Subject: [Biopython-dev] PIL/Pillow import, was: Testings on Win 7 x64 Message-ID: On Thu, Apr 24, 2014 at 5:49 PM, Peter Cock wrote: > On Thu, Apr 24, 2014 at 5:34 PM, Manlio Calvi wrote: >> >> Second a more general signalation, I've seen some test depends on PIL >> but PIL library get discontinued and substituted by one called Pillow >> that is said compatible with it predecessor (but I'm not sure how much >> it is). This is a more general issue because require an scan of the >> entire codebase. Looking around seems to me the NLTK package is in a >> situation like this, too. > > That probably refers to test_GraphicsBitmaps.py which is trying to > check if ReportLab will be able to make PNG files. If ReportLab > can now use Pillow or PIL, we should allow for that... > https://github.com/python-imaging/Pillow > > (As an aside ReportLab's Python 3 support is still quite new - > but I don't recall seeing any problems from Biopython's usage > of it.) During Debian packaging Andreas Tille had to tweak Biopython 1.63 to work nicely with Pillow (which does seem to have effectively taken over from the older PIL project): http://anonscm.debian.org/viewvc/debian-med/trunk/packages/python-biopython/trunk/debian/patches/biopython-from-pil-import-image.patch?revision=16855&view=markup I've made the following (hopefully) backwards compatible change: https://github.com/biopython/biopython/commit/3c2a2f683d766d43964e7271e90c28c16be3b06e Peter