[Open-bio-l] Automated testing server; was [BioRuby] tutorial

[Chris Fields]

On Mar 4, 2011, at 5:45 AM, Peter Cock wrote:

> Hi all,
> 
> I've retitled this thread to reflect the current topic, and CC'd the open-bio-l
> mailing list as Raoul suggested. Can we continue the discussion there?
> http://lists.open-bio.org/mailman/listinfo/open-bio-l

Agreed.  I'm replying only on open-bio-l to promote that.

> On Thu, Mar 3, 2011 at 8:52 PM, Chris Fields <cjfields at illinois.edu> wrote:
>> On Mar 3, 2011, at 2:29 PM, Raoul Bonnal wrote:
>> 
>>> Dear All,
>>> a) this discussion is a good starting point for a workgroup during Codefest
>>> 2011 and/or BOSC (can't post to bopen-bio list cause I don't have the addr
>>> here, sorry) a.idea) why not create a common "machine" on the cloud for
>>> testing our projects ?! It would be fun to see all the bio* with stats about
>>> testing etc...
>> 
>> I think the buildbot instance covers some of that, but they are mainly
>> automated builds:
>> 
>> http://testing.open-bio.org/
>> 
>> biopython is the only one set up currently.
> 
> Note that this is a small Amazon cloud machine, and doesn't actually run
> the tests itself (that would require a more expensive machine due to the
> higher load). Instead we need client machines (covering a range of OS,
> Python version etc) whose run-time is donated (currently by Biopython
> developers or their employers).
> 
> When setting up http://testing.open-bio.org/ the intention was to have
> it host other Bio* projects too - and they don't have to be using the
> buildbot software (but they could if they wanted to).

I think it would be nice to have a smoke server there as well (I don't think buildbot can do that, AFAIK).  Will ask Chris D about it.

I like the fact that we now are diversifying beyond perl-based solutions with buildbot and redmine (I would include mediawiki, but PHP gives me the hives :).  Any easily-installable Java-based ones?

>>> b) testing is very important and new contributes should be accepted
>>> only with tests, which implies a problem how to check fake tests? :-)
>> 
>> Code review and checking the test results.  Otherwise, you have to trust
>> your developers.
> 
> This is something we're pushing much more with Biopython in recent
> years (a depressing amount of legacy code had no tests when I first
> got involved, things are a lot better now). For any new code contributions
> we do ask for tests and minimal documentation (and try and work
> with the contributor to help this happen).

Same here.  This is a common cycle with OS projects.

>>> c) usually ruby's community relies a lot on doc==tests that's
>>> completely wrong, because the newbies can't read fluently the
>>> code, also because there are different tools and sometimes a lot
>>> of mocking.
> 
> With Biopython we're starting to make more use of testable docs
> (Python's doctest system) which is really good I think. But you are
> right, you need proper unit tests AS WELL. With our doctests,
> the primary aim is to be useful documentation (which can double
> as a basic functional test).

The problem with bioperl is that much of the testing framework for in-line or testable docs wasn't present during the initial development process.  We also have a LOT of cruft modules that lack true author support. 

What we are planning on for future releases is to modularize bioperl installations, with an installer to put together the dists one wants.  The current release has well over 1000 classes and modules; modularizing will help remove some of the maintenance headaches we had been facing during development.  We are also pushing new code to go into separate CPAN installations, as these can be added to an installer as needed (as someone else long ago put it, we're essentially creating a mini-BioPerlPAN within CPAN).

>>> c.a) recently i had an idea, but I have no time to do that: establish
>>> a relationship between a mock and a real object and validate the
>>> mock or enable it only if the real object has been tested before.
>>> That would make mocking more sens to me, but i think this is
>>> another story.
>> 
>> Yes, I've thought about that as well re: Moose-based work, haven't
>> wrapped my head around it either.
> 
> I'm not quite following - it may be a terminology difference.

Perl thing; Moose makes tests using mocked objects a bit easier.  I have a side project that re-implements bioperl classes using Moose, has had some pretty promising results.

chris

>>> d) I agree with Toshiaki, after this very useful brain storming
>>> start working (Impressed to see all this traffic :-) )
>>> 
>>> Cheers.
>> 
>> +1.  Will be interesting to see how things progress.
>> 
>> chris
> 
> Same here :)
> 
> Peter