[BioRuby] BioRuby Digest, Vol 110, Issue 1

[Francesco Strozzi]

This is indeed very interesting, thanks for sharing the link!

Cheers
Francesco

--
Francesco Strozzi



On Thu, 5 Mar 2015 at 22:14 Michael Barton <mail at michaelbarton.me.uk> wrote:

> If you're considering docker support you might be interested in this
> project I've been working on recently - github.com/bioboxes/rfc. We're
> aiming to standardise many of the common bioinformatics tools inside docker
> containers so they can be used interchangeably.
>
> On 4 March 2015 at 02:38, Yannick Wurm <y.wurm at qmul.ac.uk> wrote:
>
> > Hey Francesco,
> >
> > that's very cool. I like the fact that it abstracts away all the
> > complication of the queuing system. Can you use pipengine without a
> queuing
> > system/scheduler? (i.e. on a single 48-core fat node)?
> >
> > Is there an easily searchable bioinfo-core mailing list archive? I am a
> > member but cannot easily find the discussion you mention.
> >
> > I agree that its challenging to find/create one-size-fits-all solutions.
> > However I do think that there is a need for a "pipelining" solution that
> is
> > sufficiently biologist-friendly to get them to immediately see the value
> > (saving them time AND improving
> > agility/reproducibitliy/maintainabiltiy/sharability). Ad-hoc solutions
> > produced by biologists tend to do everything badly...
> >
> > Cheers,
> > Yannick
> >
> > p.s.: Sorry about the Gsoc & thanks for your efforts in putting it
> > together...
> > p.p.s.: docker is amazeballs :)
> >         Have a look at (WIP) https://github.com/yeban/oswitch
> >         We're facilitating transparent switching (files/paths/ids
> > conserved)
> >         back and forth between different OS.
> >
> >
> >
> > > On 3 Mar 2015, at 12:00, bioruby-request at mailman.open-bio.org wrote:
> > >
> > > Hi Yannick,
> > > that's an interesting topic.
> > > I have been working for a while on a Ruby package to handle pipelines
> and
> > > distributed analyses in our Bioinformatics core: the code is here
> > > https://github.com/fstrozzi/bioruby-pipengine .
> > >
> > > With this solution we have decided to stick to a simple approach, i.e.
> > > pipelines templates written in YAML where you can put raw command lines
> > > with simple placeholders that get substituted at run time according to
> > your
> > > project and samples. So the DSL is reduced to a minimum and the tool
> then
> > > creates runnable scripts that can be send through a queuing system.
> There
> > > is also a simple error control for jobs and also checkpoints to skip
> > > already completed steps for a given pipeline.
> > > This is *very* Illumina-centric and so far it works only through a
> > > Torque/PBS scheduler (this is what we have in-house). It is a bit rough
> > but
> > > we are using it since >2 years now and we are quite happy. I know it
> has
> > > been used also in other places. I've recently started a Scala
> > > implementation of this code (https://github.com/fstrozzi/PipEngine),
> to
> > > make it more portable and also to introduce a number of improvements.
> > It's
> > > still very work in progress, but among other things we want to add the
> > > support for multiple queuing systems, step dependencies and Docker
> > support.
> > >
> > > Anyway, the point with these solutions, in my opinion, is that I do not
> > > think there could be a perfect tool that can fit every purpose or
> > scenario
> > > or environment. There was a similar discussion also on the biocore
> > mailing
> > > list some time ago and it turned out that many centres either use their
> > own
> > > systems or take existing solutions, such as for instance Bpipe, and
> > modify
> > > them to fit their needs. Nextflow is also a very nice tool.
> > >
> > > In the end we have done the same and developed a solution that, even if
> > > with its own limitations, fits our needs and our way of structuring and
> > > organising the data analyses.
> > >
> > > Cheers
> > > Francesco
> >
> >
> >
> > -------------------------------------------------------
> > Yannick Wurm - http://wurmlab.github.io
> > Ants, Genomes & Evolution ⋅ y.wurm at qmul.ac.uk ⋅ skype:yannickwurm ⋅ +44
> > 207 882 3049
> > 5.03A Fogg ⋅ School of Biological & Chemical Sciences ⋅ Queen Mary,
> > University of London ⋅ Mile End Road ⋅ E1 4NS London ⋅ UK
> >
> >
> > _______________________________________________
> > BioRuby Project - http://www.bioruby.org/
> > BioRuby mailing list
> > BioRuby at mailman.open-bio.org
> > http://mailman.open-bio.org/mailman/listinfo/bioruby
> _______________________________________________
> BioRuby Project - http://www.bioruby.org/
> BioRuby mailing list
> BioRuby at mailman.open-bio.org
> http://mailman.open-bio.org/mailman/listinfo/bioruby