[Biopython-dev] Using NumPy/SciPy/scikit-learn approach to building wheels

Mon Jul 17 17:03:36 UTC 2017

Dear Biopythoneers,

Another update - after some hackery to change the TravisCI
Linux system's locale and white list the known test failures
(see the issues mentioned in my previous email), the wheel
tests are all passing. Woo!

https://github.com/biopython/biopython-wheels

Logs:

https://travis-ci.org/biopython/biopython-wheels/builds
https://ci.appveyor.com/project/biopython/biopython-wheels/history

Matthew Brett has kindly setup the sklearn RackSpace
credentials, which means the latest wheels are available
here as a staging area prior to pushing them to PyPI:

http://a365fff413fe338398b6-1c8a9b3114517dc5fe17b7c3f8c63a43.r19.cf2.rackcdn.com/

At this point I would like to encourage people on the dev
list to download and test these work. i.e. Please download
the appropriate wheel file, then:

$ pip install biopython-1.70-XXX.whl

(You may wish to first uninstall any prior Biopython on
your system)

If you get the wrong wheel for your system or version of
Python, it should abort immediately saying:

$ pip3 install biopython-1.70-ZZZ.whl
biopython-1.70-ZZZ.whl is not a supported wheel on this platform.

Of course, rather than XXX or ZZZ there is actually a
complicated platform specific name here - described here:

https://www.python.org/dev/peps/pep-0427/#file-name-convention

Please also if possible report the checksum of any file you've
tried (since if an extra build is triggered on biopython-wheels,
I think it will over-write the existing wheels):

$ md5sum biopython-1.70-*.whl
57ff2ce4a6bead102ab41556d8540d29
biopython-1.70-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
e7d7f128c3137f8f412a581fad436be4  biopython-1.70-cp27-cp27m-manylinux1_i686.whl
27b3a8588b042ae01c16c39cfa356e43
biopython-1.70-cp27-cp27m-manylinux1_x86_64.whl
95488ebb6dd318a247ecb8d9ad415fdf  biopython-1.70-cp27-cp27mu-manylinux1_i686.whl
e74c4418a5a0b24abd538ec13ad31be4
biopython-1.70-cp27-cp27mu-manylinux1_x86_64.whl
c3930614c24b851f40cbe1c72b0a214f  biopython-1.70-cp27-cp27m-win32.whl
55252b72706404a64ed814ce03e19132  biopython-1.70-cp27-cp27m-win_amd64.whl
0728881bdf2d83c07cba1f1ae4253276
biopython-1.70-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
4b16ae733d781aa41a33452bd03b5a1b  biopython-1.70-cp34-cp34m-manylinux1_i686.whl
72b088cbd5d701ce25cdf375fdce0a1f
biopython-1.70-cp34-cp34m-manylinux1_x86_64.whl
5f4b2d89b9ac8d77b7680ffd7b4f1bd4  biopython-1.70-cp34-cp34m-win32.whl
5a79fc6c22347ce7a1605da8af0ee128  biopython-1.70-cp34-cp34m-win_amd64.whl
6f5ed87b4332e141f3f9a5b2127d3849
biopython-1.70-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
c2f7c8ca471b42497a255270dc2f2022  biopython-1.70-cp35-cp35m-manylinux1_i686.whl
f3ccdbf8156a32e74a322cb648c39ffd
biopython-1.70-cp35-cp35m-manylinux1_x86_64.whl
6d14ac2b0aa7c186d96b031018c31039  biopython-1.70-cp35-cp35m-win32.whl
106304d4173eed93e580a0482e8af3cc  biopython-1.70-cp35-cp35m-win_amd64.whl
df515e7feff2635f8ee15b3836b4277b
biopython-1.70-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
1b9d6ffc27f70dadc85e04a5a7adef81  biopython-1.70-cp36-cp36m-manylinux1_i686.whl
205ddda12e73f48aec88c5898a0bb8ff
biopython-1.70-cp36-cp36m-manylinux1_x86_64.whl
670f62814cbd658ec9e6b326b986444a  biopython-1.70-cp36-cp36m-win32.whl
90cc0ff2e3b17209078e25127278b6f4  biopython-1.70-cp36-cp36m-win_amd64.whl

$ shasum -a 256 biopython-1.70-*.whl
a57a3306fe230ba17ad2bbd770ff931907f0c6cd9d58a4b6181ab3299f6c2716
biopython-1.70-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
92062175cf9496c086402534c8862ed6427b7928301c6830443c8b1a1f686037
biopython-1.70-cp27-cp27m-manylinux1_i686.whl
5ae16c07ddc0993b60d6eeee8af1740a7fc5dad5b91aff790c0bb982c511a6fd
biopython-1.70-cp27-cp27m-manylinux1_x86_64.whl
f669b0ce6031cd0c853c94659667623b7e5a03c4e10ddd210bd8692239237c46
biopython-1.70-cp27-cp27mu-manylinux1_i686.whl
30bfcf145759ad99e53ed1413cf73b3eeb49f73dcbb053d88803d50a36e3e3d4
biopython-1.70-cp27-cp27mu-manylinux1_x86_64.whl
73f7aed83d0d4460291724b4c108bb6269b31476b74466b8b827ca1716302a3c
biopython-1.70-cp27-cp27m-win32.whl
3cd9ad7c1492988488d57ef3e1e065e46e54ce6a9ada8c77ad0bf338d99d3dc6
biopython-1.70-cp27-cp27m-win_amd64.whl
198df927da67b2e45980c75635005cf0c61b11c68a4131465b3ecf3573961948
biopython-1.70-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
704b0b850ad4140c375d1fbb470c31e6fe18584e1a7253b927b656395ff58b88
biopython-1.70-cp34-cp34m-manylinux1_i686.whl
d1c5088b16295d2eaa874fe341e17f2300acb82c7d96438bb9fc3d28d610ff3a
biopython-1.70-cp34-cp34m-manylinux1_x86_64.whl
057f4a1a33e996189a9edcf9d6863f2a8148b1bbe813d009c81f00ae3b6d5de8
biopython-1.70-cp34-cp34m-win32.whl
e4c28ff28eefc5c2b3bda13194007981cac2aced1068702d5e26336b2b17e641
biopython-1.70-cp34-cp34m-win_amd64.whl
a679e18af4f52e614e4efef5193800d8eda1a79211e35767a603729dadca1cae
biopython-1.70-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
76189584603f9c2ae67059e59490a48bf9c0a7f1a03ee6f019047e5cea7ad206
biopython-1.70-cp35-cp35m-manylinux1_i686.whl
bff538d0ccb5c3d7da7459ca33de8d35e14b5eeec16a9414c79d3a6e5303328f
biopython-1.70-cp35-cp35m-manylinux1_x86_64.whl
1fd1cb42e7bb5042f65732dbd76f9e378f57f4ed6b02176de7ae6d818493ca5b
biopython-1.70-cp35-cp35m-win32.whl
ca91d1aafb1273659f3d581017dbd5981ad8246adbb31c074ee2c7e3f34e944d
biopython-1.70-cp35-cp35m-win_amd64.whl
2a4e9d351073249143ab1dae5041420e6c35c69699e0b2570b9b2950f73dceff
biopython-1.70-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
66863e72932a698a845e0b75a89d73708e843279e9fff9b1fa988b98645d7c46
biopython-1.70-cp36-cp36m-manylinux1_i686.whl
2f5c8dce8f6118aeaea58b3ca89e90bb852a84934efd616865ecb3ab88de0826
biopython-1.70-cp36-cp36m-manylinux1_x86_64.whl
15f42ef90b6743708751ef5e7a2688cc39f0e379e733c77faaccd7fbc7a25c78
biopython-1.70-cp36-cp36m-win32.whl
a5994b4351487ec320edfeadea9ff951e3767c1e567e3c309d7ae6937a19a30a
biopython-1.70-cp36-cp36m-win_amd64.whl

Right now I'd particularly like to hear test reports from Windows
users (32 and 64bit, Python 2 and 3), since that is currently
most difficult for me to test.

Once we have some positive replies, I'll upload these to PyPI,
and work on revising our release building instructions :)

Thanks,

Peter

On Thu, Jul 13, 2017 at 5:00 PM, Peter Cock <p.j.a.cock at googlemail.com> wrote:
> A brief update - the multibuild system works in that we can
> build wheels for Linux, Mac, and Windows (32-bit and 64-bit).
>
> https://github.com/biopython/biopython-wheels
>
> Logs:
>
> https://travis-ci.org/biopython/biopython-wheels/builds
> https://ci.appveyor.com/project/biopython/biopython-wheels/history
>
> I've not yet asked for the RackSpace credentials to actually
> save the wheels prior to uploading them to PyPI.
>
> Unfortunately, getting all the tests to pass is proving frustrating -
> the multibuild platform has highlighted some locale issues:
>
> https://github.com/biopython/biopython/issues/855
> https://github.com/biopython/biopython/issues/1319
> https://github.com/biopython/biopython/issues/1320
> https://github.com/biopython/biopython/issues/1321
>
> And a strange floating point nan issue:
>
> https://github.com/biopython/biopython/issues/1316
>
> Plus a minor int/long doctest which I fixed (5 vs 5L), fixed:
>
> https://github.com/biopython/biopython/commit/a1d14584f3fa8002b42ebcf9b79ff26fe82235d7
>
> On the bright side, so far there is nothing which makes me
> want to rush out another release.
>
> I'm happy now with the Windows tests (skipping two problem
> tests), so perhaps we should use these wheels as is?
>
> https://ci.appveyor.com/project/biopython/biopython-wheels/build/1.0.23
>
> Peter
>
> On Fri, Jul 7, 2017 at 4:31 PM, Peter Cock <p.j.a.cock at googlemail.com> wrote:
>> Matthew Brett is happy with this plan :)
>>
>> I'll be creating a new repository for this (probably next week):
>>
>> https://github.com/biopython/biopython-wheels
>>
>> Once that is up and running, the goal would be to make a
>> trivial commit there to update the version tag, and then TravisCI
>> and AppVeyor would be triggered to fetch that tagged commit
>> from the main repository, and build all the compiled wheels for us.
>>
>> e.g. An example from AstroPy:
>>
>> https://github.com/MacPython/astropy-wheels/commit/cf5f4e80d3a6644f7aadc6375662209be85ca2c1
>>
>> For now it would be a manual step (after final QC) to upload
>> the wheels to PyPI for public use.
>>
>> Regards,
>>
>> Peter
>>
>> On Wed, Jul 5, 2017 at 11:48 AM, Peter Cock <p.j.a.cock at googlemail.com> wrote:
>>> Dear Biopythoneers,
>>>
>>> RE: https://github.com/biopython/biopython/issues/1296
>>>
>>> There has been an impressive amount of work in the wider
>>> NumPy ecosystem to automate building compiled wheels for
>>> Linux, Mac and Windows:
>>>
>>> Here is how NumPy etc does their wheels:
>>>
>>> https://github.com/MacPython/numpy-wheels
>>>
>>> A more relevant example of another project using NumPy:
>>>
>>> https://github.com/MacPython/astropy-wheels
>>>
>>> Background to the setup:
>>>
>>> https://github.com/MacPython/wiki/wiki
>>>
>>> https://github.com/matthew-brett/multibuild
>>>
>>> Unless anyone has a better idea, I plan to ask Matthew Brett
>>> if Biopython can also use the Rackspace donated resources
>>> originally for the scikit-learn team for hosting the wheels prior
>>> to uploading to PyPI:
>>>
>>> https://github.com/MacPython/wiki/wiki/Managing-Rackspace
>>>
>>> Regards,
>>>
>>> Peter