[BioRuby] Nightly build testing, was: Broken link on wiki installation page

[Peter Cock]

Hello all,

Note I've retitled the thread since the topic has change.

Apologies for the long email - I was hoping to talk
though most of this first with Raoul, Goto-san and
Toshiaki-san while in Japan, but there was plenty
of other things going on. Sorry.

On Wed, Sep 21, 2011 at 9:07 PM, Pjotr Prins <pjotr.public14 at thebird.nl> wrote:
> Normally it is just running 'rake test' in the base dir. I think
> gems can test also at install time.

Actually I may have been over enthusiastic regarding
testing the BioRuby gems:
http://lists.open-bio.org/pipermail/bioruby/2011-September/001896.html

One of the key points of the buildbot software (which
we are using for the Biopython nightly testing) is it is
linked to a repository, and you can see the test results
versus the revisions. This includes git support which
is good. You can browse this here:
http://testing.open-bio.org/biopython/

Personally I like the transposed grid on my widescreen
monitor at work, where each row is a revision:
http://testing.open-bio.org/biopython/tgrid

When the row turns red, you have a major cross
platform regression. When a subgroup turns red,
you have a platform specific failure (e.g. just
Windows, or a specific version of Python).

We also have failure emails sent out, and this can
be followed by an RSS feed too.

There is also a time based view, the "waterfall":
http://testing.open-bio.org/biopython/waterfall

If all the tests are triggered automatically on a
nightly basis, the chronological order of the tests
will match the changes made to the repository.
However, via the web interface (with a password)
you can request a buildslave run the tests using
a particular branch/revision. This is handy if a
buildslave was offline for some reason and
missed a nightly test.

Now, in the typical setup you have one software
suite with one repository and one test suite.
That's what we have for Biopython, and what
I was picturing initially for BioRuby.

In the case of the BioRuby gems, we could set
this up against the core BioRuby repository, and
have the core BioRuby tests run (against the
known revision from that repository) plus the
BioGem tests run with the latest Gem code.
The downside is that wouldn't track changes
to the Gems themselves. Is that likely to be
a problem?

I've not looked into the details of possible
multi-repository setups - perhaps Tiago
knows more (he setup the buildbot for
Biopython, CC'd).

Thinking out loud, one solution would be a
meta repository - I think you can have a git
repository with sub projects... maybe that
could be done for the BioRuby core plus
(important) Gems?

Alternatively, perhaps a different test server
infrastructure might support a composite
project with multiple sub projects, each
with their own repository? You don't have
to use buildbot - Chris Fields was talking
about something called smoke test for
BioPerl (CC'd).

> Anyway, I am sure some of us are
> interested in helping out, including myself.
>

Excellent. You (BioRuby) will need several volunteer
machines as buildslaves which are are either on all
the time, or at least online regularly. I'm willing to offer
one.

You (BioRuby) will also need to decide which platforms
you'll be testing - by which I mean what combinations
of operating system and version of Ruby. You will need
at least one buildslave for each platform. I plan to
offer at least one of my machines (it will make testing
an initial setup much easier).

For Biopython we try to cover Mac, Windows, Linux,
both 32 bit and 64 bit, and a range of versions of the
standard Python implementation (in C) plus also
Jython (Python on the Java Virtual Machine), and
in the future perhaps PyPy and IronPython.

For each platform we will need the (short) steps
to build/compile BioRuby from a fresh git checkout,
and run the tests.

e.g. For Biopython under Windows on Python 2.5,

C:\Python25\python.exe setup.py build
C:\Python25\python.exe setup.py test

While on Linux on Python 2.7 it would be:

python2.7 setup.py build
python2.7 setup,py test

We try and have each buildslave cover a couple of different
versions of Python - thus we use explicit and slightly different
commands in each case (rather than just using the default
installation of Python).

We can go though a lot of this kind of detail off the
mailing list if you prefer.

I hope I haven't bored most of the mailing list
subscribers. There isn't a separate list for BioRuby
development is there?

Regards,

Peter