[BioPython] Re: UnitTests

[Jeffrey Chang]

Hi Cayte and Andrew,

Thanks for sharing this discussion!  We should definitely hash out the
existing issues and come to some agreement on which code base to start
building upon.

The thing to keep in mind, though, is that in addition to having a robust
framework, it will need to be well-documented and easy to use.  In order
to be successful, tests will have to be written.  Unfortunately, it's
usually low on people's priority lists (I'm guilty, too!), so there's
going to have to be some simple API, or people will not take the time to
learn how.

Jeff



>  > Okay, now for the program as a whole.  You create an instance of a
>  > TestCase, then invoke with the module name (BTW, you can pass modules
>  > around as an object instead of going through
>  > sys.modules[module_name]).
>  >
>  > Once created, the methods are scanned to make a list of
>  > UnitTestCallers.  When the test suite is run, the TestCase is
>  > initialized using "setup", then the individual test callers are
>  > executed.  They return with either None (to indicate success), or an
>  > object (to indicate an error) or via an exception (to indicate a
>  > failure).
>  >
>  > After the test is run, the test case is "tear_down"ed (potentially to
>  > allow the tests to be run again?).
>  >
>  > The comparison of test results with the gold standard is done through
>  > methods named "assert_*".
>  >
>  >
>  > If that's how it is supposed to work, here are some questions:
>  >
>  >   1) is the order of method execution guaranteed?  I have test cases
>  >   which build on previous results.  It appears either those cases will
>  >   have to all be in a single method, or I need a guarantee of order so
>  >   I can store results in the instance.
>  >
>    No.  Actually, one of the things I like about the Xtreme approach is that
>  each test is isolated and you don't have to hunt for hidden dependencies.
>  IMHO, tests with dependencies should be grouped together.
> >
> >   2) how do I test exceptions?  Is it
>  >    try:
>  >      obj.meth()
>  >    except TypeError, obj:
>  >      self.assert_equals(obj.__class__, TypeError)
>  >
>  >   or simply
>  >
>  >    try:
>  >      obj.meth()
>  >    except TypeError, obj:
>  >      pass
>  >
>  >   and let the caller catch any other exceptions (as a failure rather
>  >   than an error).  (Should there be the difference between those two
>  >   types of errors?)
>  >
>     Yes, I think so.
> 
>  >   3) this is a big one ...  What advantage does this framework give me
>  >   over the, perhaps, naive version I've been working with?
>  >
>    1.  Each test is isolated, so I only have to look for dependencies in
>  setup and the local test function.
> 
>  2.  This isolation also makes it easier for someone reading the code.  You
>  may remember the sequence but s/he doesn't.
> 
>  3. I can create suites of just a few test cases.  If only three tests fail,
>  I don't have to rerun everything.
> 
>  4.  I can separate the design of test cases from the mechanics of
>  implementing.  Because, I can use a rote mechanical method to implement
> test
>  cases, almost like filling out a form, at least for me, it frees me to
> think
>  more about how to test.
>  .
> 
>  5. If there's a failure, I know exactly where to look in the code.
> 
>  > Have you taken a look at the one I used?  It drives a set of test_*.py
>  > scripts, captures text written to stdout, and compares the text to a
>  > golden copy.  Errors are signified either by a difference in the test,
>  > or finishing with an exception.
>  >
> 
>  I've looked superficially, but I plan to look closer this weekend.
> 
>  > This corresponds roughly to each test_*.py file is a TestCase, where
>  > the setup() does nothing and the tear_down() deletes all variables.
>  > The assert_* function is an assert_equals between the generated output
>  > and the expected output.
>  >
>  > The main difference I see is my code fails a test at the first time
>  > the two outputs differ, while yours can keep on going.  However,
>  > that's not quite true if attributes are stored in a TestCase for
>  > other, future test cases.  In fact, if things aren't stored, that's
>  > equivalent to spliting up a test_*.py file into several different
>  > ones.
>  >
>  > Mine has the advantage that if something changes, then once you've
>  > verified the new regression output, it can be made into the new gold
>  > standard.  With your code, you have to modify scatters assert_equals
>  > and assert_condition lines.
>  >
>     I use the diff function when it comes to comparing output files, but I'm
>  not sure it's appropriate for every situation.  I think a list of passed
> and
>  failed test cases provides a useful summary and if mnemonic names are used,
>  give you an idea of what was covered.