[Biopython-dev] Continuous integration server

[Peter]

2010/10/30 Tiago Antão <tiagoantao at gmail.com>:
> Hi all,
>
> I've been hacking with buildbot, an integration server. This is to
> allow continuous testing of Biopython. So that we are alerted of any
> problems as soon as somebody does a dreadful commit (I have the top 5
> of most dreadful commits, so it was fair that I should try to do
> something about it).
>
> Things are still incomplete, but I think it is time to inform the list
> of this effort...
> To know more about buildbot you can either go to the buildbot site
> http://buildbot.net/ or see the draft doc that I have been preparing
> http://biopython.org/wiki/Continuous_integration
> There is a draft server here:
> http://events.open-bio.org:8010/
> The cool thing about buildbot is that actual testing is done by
> volunteer computers. Want to test on OS y, Python version z? You can
> offer the idle time of your laptop for that...
>

It is looking impressive Tiago - excellent work :)

>
> Obvious things missing:
>
> 0. First and foremost, see if people like this?

Looks very promising.

> 1. Changing the biopython test code to avoid stressing the network
> (i.e., having a run_tests option that will not test network tests).
> This to avoid imposing continuous traffic on genbank and friends. This
> is a show stopper.

Certainly we can't scale this up to many machines running regular
testing without limiting the network access somewhat.

> 2. Maybe warn the mailing list when some fundamental build stops
> working (e.g. send an email when a python 2.x build stops working)
> 3. Have test servers with all the applications installed (do you want
> to volunteer? This is more to do with volunteers)

I would expect "core" developers to have machines with most of
the command line applications used in Biopython's tests already
installed - but yes, we do want to make sure each optional
command line tool or library is installed on at least one build slave.

> 4. Maybe change run_tests to require all tests to be done. If we are
> doing integration testing, we want all tests to be done (missing
> applications or libraries should be an error). As an example, none of
> my tests are complete

This is about how it currently skips tests missing external
dependencies (like PopGen command line tools in your case).
I think that is OK, otherwise we'll get false positives (see below,
we can't satisfy all dependencies on all platforms).

> 5. Support mac (my access to Mr Job's fashion machines is limited).
> Again this is more a volunteer issue.

My main work machine is a Mac, so this shouldn't be an issue.

> 6. Discuss policies: One test a day? Full tests or updates? Full
> network tests (probably sporadically)? Send emails?

Right now triggering tests after each commit isn't easy to do
is it (due to limited git support in builtbot)? That might be nice
but in the short term running the tests once a day is a big step
forward.

I'd suggest we do network tests once a week (or fortnight?).

> 7. Find volunteers to cover several OSes and several Python
> versions. Assure that people do full tests (i.e. with all applications
> and libraries)

That isn't possible - some applications are not available on Windows,
and some libraries are not available on Jython or Python 3 (yet).

> 8. While I have volunteer Windows testing myself, I will not be able
> to maintain it regularly.

I have access to a Windows machine (which I use to build the
Biopython installers) but currently it is only online intermittently.
I'd have to reorganise machines due to limited network ports in
the office, but it could in principle be used as a builtbot slave.

>
> Opinions are most welcome
>

What is wrong with your Linux Python 3.1 slave? It seems that
2to3 is failing on the doctest conversion.

Peter