[Biopython-dev] Continuous integration

[Peter]

2010/10/11 Tiago Antão <tiagoantao at gmail.com>:
> Hi Peter,
>
> 2010/10/11 Peter <biopython at maubp.freeserve.co.uk>:
>> (1) Several of our tests go online to the NCBI or UniProt etc.
>> These tests can and do fail sometimes due to network issues.
>> Also, having some/many buildbot slaves running on a regular
>> basis (once a week? once a day?) would add up and this
>> load may be unwelcome. Perhaps we need to add an -offline
>> flag to run_tests.py which can skip any online tests?
>
> That might be a good idea (to have an --offline flag, I mean). A very
> good idea, indeed.
>
> I would like to put the infrastructure in place (if people are
> interested in going ahead with this...), but after that we need to
> stabilize a test policy and that will mean answering questions like
> that.
>
> As far as I see we will have many builders (tests under different
> conditions). Say 5 different Python versions (Jython included), at
> least 3 OSes. This is already 15 builders. This can easily creep up.
> Though the numbers are high, it is quite easy to maintain all this
> stuff: 3 volunteer machines (one for each OS) are enough. The cool
> thing about buildbot is that it is designed for volunteer machines to
> be added, so you can start your buildbot slave on your laptop when you
> are idle. It does not need an array of servers on demand to produce
> the tests.
>
> NCBI and Uniprot might not like to see 30 daily connections for tests
> :( . So we might need to have, say, one weekly test for each OS doing
> the network stuff (just a single Python version per OS, maybe) and
> dailies not doing network loads.

Exactly.

>> (2) You mention buildbot doesn't have built in support for
>> spotting changes in a git repository - but can it do this for
>> SVN? Since github.com also allow access to the git repo
>> via svn that might be a more elegant workaround.
>
> There are 2 different things to consider:
> 1. Spotting the git repository. There is no builtin support, but this
> is TRIVIAL nonetheless with the general adaptor of buildbot. It works
> like this:
>     a. a developer does a push
>     b. github has a hook system which allows for reporting a change
> to the repository to a certain URL/CGI. Fully automated, transparent
> to the developer.
>     c. We supply a CGI that receives the event and informs buildbot.
> There are CGIs for github. We just have to stuff one in a webserver.

Do you even need a post-commit hook? Unless you want
to automatically run the tests after every commit (which might
be useful) wouldn't it be enough to do a daily checkout?

> 2. The slaves/builders have to download github code. In this case,
> buildbot HAS NATIVE SUPPORT.

Understood.

>> (3) Does the buildbot master require the buildbot slaves
>> be online most/all of the time? Would a desktop machine
>> which is typically only on during office hours on week
>> days still be useful? I could probably answer this myself
>> with a bit more background reading ;)
>
> That is one of the wonders of buildbot. Just the server needs to be online.
> You can indeed have a desktop machine: Whenever it suits you better
> you start your buildbot slave, it connects to the server to see if
> there is work to do and the server supplies work to be done. The
> server can be instructed to only allow the slave to do a single task
> at a time (to avoid overloading the slave).

Excellent. I guess the specifics of starting the buildbot slave
will be OS specific, thus it would be up to the machine owner
if this should happen automatically at login or not.

> I am now at a stage were I really need a server to test (with a public
> address). I would volunteer to do the installation myself, but I would
> need shell access to a machine where I could run a server process. No
> root access is needed, but a web server is not enough as buildbot is
> twisted based. Maybe we can convince the OBF to help. Again, I can
> volunteer to do the installation. No root access is needed, just the
> ability to run a server process and a couple of open ports.

I think we should have a work with the OBF as running this on
one of their servers does seem the best plan. I'll email you.

Peter