[Biopython-dev] Bug#751277: python-biopython: FTBFS on mips* powerpc s390x

[Peter Cock]

On Fri, Jun 13, 2014 at 11:05 PM, Andreas Tille <andreas at an3as.eu> wrote:
> Hi Peter,
>
> thanks for your quick response.
>
> On Fri, Jun 13, 2014 at 02:57:37PM +0100, Peter Cock wrote:
>> > I have updated the Debian package to version 1.64 (BTW, it is fine to
>> > ping debian-med at lists.debian.org about new upstream versions - we might
>> > become more quick in packaging new versions).
>>
>> Good point. I used to email the Phillipe & Charles when there was
>> a change I anticipated might cause trouble.
>>
>> Should we just email debian-med at lists.debian.org after each new
>> Biopython release?
>
> Yes, that's perfectly OK.  It might happen that certain persons might be
> busy with other things.  I also see no point to waste the chance to make
> new versions of BioPython even more popular - so let readers of the
> Debian Med list know about what you are doing.
>
>> If so we can add that to our build process:
>> http://biopython.org/wiki/Building_a_release
>
> +1

Done. We can add other Linux etc packaging contacts too...

>> > Jakub Wilk was kind enough to point to the real problems of the tests
>> > which can be read below (since the end of the build log does not say
>> > a lot).
>> >
>> > It would be great if you give some advise how to deal with these
>> > problems.
>>
>> Most of these are tests where we call an external command line
>> tool (i.e. muscle, dnal, dialign2-2). The purpose of the tests is
>> in part to check our command line wrappers are current (and
>> catch any API changes), but also in many cases to check we
>> can parse the current output (to catch any format changes).
>>
>> I would *guess* that some of these platforms have problems
>> in these underlying tools - or a very different version is being
>> tested? i.e. Older than the mainstream platforms.
>
> Usually all these tools are in sync.  However, not all of these are
> tested since most are missing unit tests - so BioPython is a great
> test for these tools.

Indirectly yes :)

>> --
>>
>> The final category of failures was from test_Cluster.py under
>> powerpc and s390x, under Python 3.4, which suggests it
>> could be something in the C code for Bio.Cluster - probably
>> Python 3 specific.
>>
>> From line 138-141,
>>
>>      clusterid, error, nfound = kcluster(data, nclusters=nclusters,
>>                                             mask=mask, weight=weight,
>>                                             transpose=0, npass=100,
>>                                             method='a', dist='e')
>>
>> Line 210-212,
>>
>>     distance = clusterdistance(data, mask=mask, weight=weight,
>>                                    index1=c1, index2=c2, dist='e',
>>                                    method='a', transpose=0)
>>
>> Line 289-290,
>>
>>         tree = treecluster(data=data1, mask=mask1, weight=weight1,
>>                            transpose=0, method='a', dist='e')
>>
>> Line 555-557,
>>
>>         clusterid, celldata = somcluster(data=data, mask=mask, weight=weight,
>>                                          transpose=0, nxgrid=10, nygrid=10,
>>                                          inittau=0.02, niter=100, dist='e')
>>
>> This all give the following error via C function distance_converter
>> in Bio/Cluster/clustermodule.c
>>
>> ValueError: distance should be a single character
>>
>> Yet in all those examples, dist='e' which is a single character...
>>
>> The good news is I can reproduce a related problem on Mac OS X
>> under Python 3.3 and 3.4 where this error is not raised:
>>
>> Test branch:
>> https://github.com/peterjc/biopython/tree/cluster_single_char
>>
>> This commit makes the error messages more explicit:
>> https://github.com/peterjc/biopython/commit/fa597040cfb7e5f18d55257367397e88274563b8
>>
>> This commit verifies the errors are thrown (and they are not
>> under Python 3 on the Mac):
>> https://github.com/peterjc/biopython/commit/5b99854a82f08321ad78feaf0b362002d2d1fd2b
>>
>> I'm going to have to pass this one to Michiel to look at... but it
>> looks like a glitch in the bytes vs unicode handling, which for
>> some reason mostly works - but breaks under some unusual
>> platforms?
>
> I'll try to ask porters to check this patch on the different
> architectures.

I think Jakub is correct that the original problem is a big-endian
issue, but it appears to have helped me spot a Python 3 specific
problem in our error handling as well (the work on that branch).

Thanks,

Peter

P.S. There is a problem with our mail server, which is being
looked, at but some of these messages may have been lost
and never appear on the Biopython archives :(