[Biopython-dev] Rethinking Biopython's testing framework

[Michiel de Hoon]

> > However, more than half of Biopython's tests do
> > not actually make use of this testing framework:
> > [...]
> > These tests have trivial output, for example
> test_Cluster:
> > 
> > test_Cluster
> > test_clusterdistance (test_Cluster.TestCluster) ... ok
> > test_distancematrix_kmedoids
> > (test_Cluster.TestCluster) ... ok
> > test_kcluster (test_Cluster.TestCluster) ... ok
> > test_matrix_parse (test_Cluster.TestCluster) ... ok
> > test_median_mean (test_Cluster.TestCluster) ... ok
> > test_somcluster (test_Cluster.TestCluster) ... ok
> > test_treecluster (test_Cluster.TestCluster) ... ok
> 
> They really do make use of the framework, but at a higher
> level. I agree that if you run a single test it makes little
> difference whether you use 'run_tests.py test_Cluster' or just
> run 'test_Cluster.py' directly. However, when you are
> running all the tests as is regular done in development
> or before pushing releases, this comparison is important. It
> will pick out if you get a line like:
> 
> test_clusterdistance (test_Cluster.TestCluster) ... ERROR
> 
> instead of the expected ok and report this in the summary
> for all of the tests. Otherwise this is likely to get lost
> in all of the results.

Actually, I never use the summary produced by run_tests.py. I just check which tests failed, and then fix them one by one by running the individual test scripts.

> > I would therefore like to suggest to move from
> > Biopython's testing framework to Python's testing
> > framework. This also relieves us of the
> > task of explaining Biopython's testing framework
> > to contributors, and allows us to make better use
> > of what Python already provides.
...
> Is the testing framework you are proposing different from
> the unit tests used the individual tests?

I am proposing to use the regular Python unit testing framework as it is. This means that most Biopython tests do not change at all (or only trivially). The run_tests.py script will need to be modified though to remove the requirement of having an output file for each individual test.

> How does your proposed
> manage the higher level functionality of checking if all sub-tests
> within one of the test suites passes?

If one of the sub-tests fails, Python's unit testing framework will tell us so, though (perhaps) not exactly which sub-test fails. However, that is easy to figure out just by running the individual test script by itself.

--Michiel