[Biopython-dev] Rethinking Biopython's testing framework

[Brad Chapman]

Hi Michiel;
Good thoughts on this; my comments are below.

> Biopython's testing framework is built on top of Python's unit testing
> framewerk. Python's unit testing framework makes use of assertion
> statements to compare the result of a command to the expected result.
> Biopython uses test scripts that print output to stdout, together with
> an output file that contains the correct output. After running each
> test script, it compares the generated output with the correct output
> to see if the test was successful.

Agreed with the distinction between the unit tests and the "dump
lots of text and compare" approach. I've written both and do think
the unit testing/assertion model is more robust since you can go
back and actually get some insight into what someone was thinking
when they wrote an assertion.

> However, more than half of Biopython's tests do not actually make use of this testing framework:
[...]
> These tests have trivial output, for example test_Cluster:
> 
> test_Cluster
> test_clusterdistance (test_Cluster.TestCluster) ... ok
> test_distancematrix_kmedoids (test_Cluster.TestCluster) ... ok
> test_kcluster (test_Cluster.TestCluster) ... ok
> test_matrix_parse (test_Cluster.TestCluster) ... ok
> test_median_mean (test_Cluster.TestCluster) ... ok
> test_somcluster (test_Cluster.TestCluster) ... ok
> test_treecluster (test_Cluster.TestCluster) ... ok

They really do make use of the framework, but at a higher level. I
agree that if you run a single test it makes little difference
whether you use 'run_tests.py test_Cluster' or just run
'test_Cluster.py' directly. However, when you are running all the
tests as is regular done in development or before pushing releases,
this comparison is important. It will will pick out if you get a
line like:

test_clusterdistance (test_Cluster.TestCluster) ... ERROR

instead of the expected ok and report this in the summary for all of
the tests. Otherwise this is likely to get lost in all of the
results.

> Personally, I find Python's unit testing framework easier to
> understand than Biopython's testing framework. It doesn't need a
> separate output file, and it is easier to match each line of code with
> the correct behavior.
>
> I would therefore like to suggest to move from Biopython's testing
> framework to Python's testing framework. This also relieves us of the
> task of explaining Biopython's testing framework to contributors,
> and allows us to make better use of what Python already provides.
> Comparing output line-by-line, as Biopython's testing framework
> currently does, can still be used by test scripts that need this
> functionality.

Is the testing framework you are proposing different from the unit
tests used the individual tests? How does your proposed manage the
higher level functionality of checking if all sub-tests within one
of the test suites passes?

Brad