[BioRuby] Nightly build testing, was: Broken link on wiki installation page

[Pjotr Prins]

Hi Peter,

It is for reason of limited installs we have meta-packages.
Meta-packages provide a sense of stability, and is what most users
will install. 

I suggest testing for:

  - bio     
  - bio-core, which contains core pure ruby biogems
  - bio-core-ext, which contains non-pure ruby biogems
    (extensions)

anything worth testing will be in there. We can help create the test
scripts for those packages.

Also, it looks like the testing environment should be hosted in a
ready made VM image. I can create a CloudBioLinux flavour for that.
Once we know what to install. I will also want tests for biolib_hpc,
when that gets production ready (btw). It is a long standing wish on
my end to get this sorted.

Pj.

On Wed, Sep 21, 2011 at 10:17:46PM +0100, Peter Cock wrote:
> Hello all,
> 
> Note I've retitled the thread since the topic has change.
> 
> Apologies for the long email - I was hoping to talk
> though most of this first with Raoul, Goto-san and
> Toshiaki-san while in Japan, but there was plenty
> of other things going on. Sorry.
> 
> On Wed, Sep 21, 2011 at 9:07 PM, Pjotr Prins <pjotr.public14 at thebird.nl> wrote:
> > Normally it is just running 'rake test' in the base dir. I think
> > gems can test also at install time.
> 
> Actually I may have been over enthusiastic regarding
> testing the BioRuby gems:
> http://lists.open-bio.org/pipermail/bioruby/2011-September/001896.html
> 
> One of the key points of the buildbot software (which
> we are using for the Biopython nightly testing) is it is
> linked to a repository, and you can see the test results
> versus the revisions. This includes git support which
> is good. You can browse this here:
> http://testing.open-bio.org/biopython/
> 
> Personally I like the transposed grid on my widescreen
> monitor at work, where each row is a revision:
> http://testing.open-bio.org/biopython/tgrid
> 
> When the row turns red, you have a major cross
> platform regression. When a subgroup turns red,
> you have a platform specific failure (e.g. just
> Windows, or a specific version of Python).
> 
> We also have failure emails sent out, and this can
> be followed by an RSS feed too.
> 
> There is also a time based view, the "waterfall":
> http://testing.open-bio.org/biopython/waterfall
> 
> If all the tests are triggered automatically on a
> nightly basis, the chronological order of the tests
> will match the changes made to the repository.
> However, via the web interface (with a password)
> you can request a buildslave run the tests using
> a particular branch/revision. This is handy if a
> buildslave was offline for some reason and
> missed a nightly test.
> 
> Now, in the typical setup you have one software
> suite with one repository and one test suite.
> That's what we have for Biopython, and what
> I was picturing initially for BioRuby.
> 
> In the case of the BioRuby gems, we could set
> this up against the core BioRuby repository, and
> have the core BioRuby tests run (against the
> known revision from that repository) plus the
> BioGem tests run with the latest Gem code.
> The downside is that wouldn't track changes
> to the Gems themselves. Is that likely to be
> a problem?
> 
> I've not looked into the details of possible
> multi-repository setups - perhaps Tiago
> knows more (he setup the buildbot for
> Biopython, CC'd).
> 
> Thinking out loud, one solution would be a
> meta repository - I think you can have a git
> repository with sub projects... maybe that
> could be done for the BioRuby core plus
> (important) Gems?
> 
> Alternatively, perhaps a different test server
> infrastructure might support a composite
> project with multiple sub projects, each
> with their own repository? You don't have
> to use buildbot - Chris Fields was talking
> about something called smoke test for
> BioPerl (CC'd).
> 
> > Anyway, I am sure some of us are
> > interested in helping out, including myself.
> >
> 
> Excellent. You (BioRuby) will need several volunteer
> machines as buildslaves which are are either on all
> the time, or at least online regularly. I'm willing to offer
> one.
> 
> You (BioRuby) will also need to decide which platforms
> you'll be testing - by which I mean what combinations
> of operating system and version of Ruby. You will need
> at least one buildslave for each platform. I plan to
> offer at least one of my machines (it will make testing
> an initial setup much easier).
> 
> For Biopython we try to cover Mac, Windows, Linux,
> both 32 bit and 64 bit, and a range of versions of the
> standard Python implementation (in C) plus also
> Jython (Python on the Java Virtual Machine), and
> in the future perhaps PyPy and IronPython.
> 
> For each platform we will need the (short) steps
> to build/compile BioRuby from a fresh git checkout,
> and run the tests.
> 
> e.g. For Biopython under Windows on Python 2.5,
> 
> C:\Python25\python.exe setup.py build
> C:\Python25\python.exe setup.py test
> 
> While on Linux on Python 2.7 it would be:
> 
> python2.7 setup.py build
> python2.7 setup,py test
> 
> We try and have each buildslave cover a couple of different
> versions of Python - thus we use explicit and slightly different
> commands in each case (rather than just using the default
> installation of Python).
> 
> We can go though a lot of this kind of detail off the
> mailing list if you prefer.
> 
> I hope I haven't bored most of the mailing list
> subscribers. There isn't a separate list for BioRuby
> development is there?
> 
> Regards,
> 
> Peter
>