[Biopython-dev] Python 3 and subprocess .communicate() bug

[Peter Cock]

Hi Tiago,

We'd been talking briefly off-list about the recent buildbot
failures under Python 3 where the recent change to using
subprocess in the PopGen module was causing failures.

Sadly while it seems to work on Python 3.1 and 3.2 my
suggestion to try using bytes with the communicate call
fails on Python 3.3 and under Windows:
https://github.com/biopython/biopython/commit/912692ee2b57e8c075ba38bdf814c9dbe4f5cdb9

e.g. After the change to use bytes,
http://testing.open-bio.org/biopython/builders/Linux%2064%20-%20Python%203.3/builds/202
http://testing.open-bio.org/biopython/builders/Windows%20XP%20-%20Python%203.1/builds/816
http://testing.open-bio.org/biopython/builders/Windows%20XP%20-%20Python%203.2/builds/680
http://testing.open-bio.org/biopython/builders/Windows%20XP%20-%20Python%203.3/builds/206

This appears to be a known bug in the subprocess
module, http://bugs.python.org/issue16903 which
should be fixed in Python 3.2.4 and Python 3.3.
It appears not to have been fixed on Python 3.1.

I see two options,

Option One, revert that commit (i.e. send unicode strings
as before, not bytes). This will work on Python 3.2.4+
onwards including Windows. It will fail on Python 3.1
and out of date Python 3.2 through 3.2.3 releases.

Option Two, don't use universal_newlines=True which
then requires us to use byte strings for all the stdin,
stdout and stderr processing. More work, but it should
in principle work on old and new Python 3 releases.

Note that while we're not seeing any problems yet, I
suspect this issue would affect our Bio.Application
wrappers __call__ function as well when used to send
data to stdin. Here again we could switch to using
bytes and universal_newlines=False and do any
bytes/unicode handling within the __call_ function,
on just insist on a fixed version of Python.

If we decide to recommend at least Python 3.2.4
(when using Python 3), then we could add a warning
to the relevant modules to catch this issue?

What do people think?

Regards,

Peter