[Biopython-dev] Output of Biopython tests

[Michiel De Hoon]

Hi everybody,

With the help of several Biopython developers, especially Peter, the problems
with Martel and the new mxTextTools release have now been solved (in the
sense that all unit tests now succeed). So we're a lot closer to a new
Biopython release. Thanks everybody!

When I was running the Biopython tests, one thing bothered me though. All
Biopython tests now have a corresponding output file that contains the output
the test should generate if it runs correctly. For some tests, this makes
perfect sense, particularly if the output is large. For others, on the other
hand, having the test output explicitly in a file doesn't actually add much
information. For example, the output for test_psw is

test_psw
test_AlignmentColumn_assertions (test_psw.TestPSW) ... ok
test_AlignmentColumn_full (test_psw.TestPSW) ... ok
test_AlignmentColumn_kinds (test_psw.TestPSW) ... ok
test_AlignmentColumn_repr (test_psw.TestPSW) ... ok
test_Alignment_assertions (test_psw.TestPSW) ... ok
test_Alignment_normal (test_psw.TestPSW) ... ok
test_ColumnUnit (test_psw.TestPSW) ... ok
Doctest: Bio.Wise.psw.parse_line ... ok

----------------------------------------------------------------------
Ran 8 tests in 0.002s

OK

For comparison, this is the test output if test_psw.py fails:

test_AlignmentColumn_assertions (__main__.TestPSW) ... ok
test_AlignmentColumn_full (__main__.TestPSW) ... ok
test_AlignmentColumn_kinds (__main__.TestPSW) ... FAIL
test_AlignmentColumn_repr (__main__.TestPSW) ... ok
test_Alignment_assertions (__main__.TestPSW) ... ok
test_Alignment_normal (__main__.TestPSW) ... ok
test_ColumnUnit (__main__.TestPSW) ... ok
Doctest: Bio.Wise.psw.parse_line ... ok

======================================================================
FAIL: test_AlignmentColumn_kinds (__main__.TestPSW)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_psw.py", line 47, in test_AlignmentColumn_kinds
    self.assertEqual(ac.kind,
"some_funny_output_I_made_up_instead_of_INSERT")
AssertionError: 'INSERT' != 'some_funny_output_I_made_up_instead_of_INSERT'

----------------------------------------------------------------------
Ran 8 tests in 0.000s


The point is that for this test, having the output explicitly is not needed
in order to identify the problem.

Now, for some tests having the output explicitly actually causes a problem.
I'm thinking about those unit tests that only run if some particular software
is installed on the system (for example, SQL). In those cases, we need to
distinguish failure due to missing software from a true failure (the former
may not bother the user much if he's not interested in that particular part
of Biopython). If a test cannot be run because of missing prerequisites,
currently a unit test generates an ImportError, which is then caught inside
run_tests. Hence, we get the following output when running the Biopython
tests:

test_BioSQL ... Skipping test because of import error: Skipping BioSQL tests
--
enable tests in Tests/test_BioSQL.py
ok

When you look inside test_BioSQL.py, you'll see that the actual error is not
an ImportError. In addition, if a true ImportError occurs during the test,
the test will inadvertently be treated as skipped.

My solution would be to skip tests inside test_BioSQL if the prerequisites
are not met. However, in that case the test output no longer agrees with the
expected test output, generating a failure message.

I'd therefore like to suggest the following:
1) Keep the test output, but let each test_* script (instead of run_tests.py)
be responsible of comparing the test output with the expected output.
2) If the expected output is trivial, simply use the assert statements to
verify the test output instead of storing them in a file and reading them
from there.

Any objections?

--Michiel.

Michiel de Hoon
Center for Computational Biology and Bioinformatics
Columbia University
1150 St Nicholas Avenue
New York, NY 10032