[Bioperl-pipeline] Re: BioPipe
Juguang Xiao
juguang at tll.org.sg
Mon Jun 30 14:04:26 EDT 2003
>
>> Currently, one manager is in charge of one running pipe. if you want
>> to run more, just start up more managers, why do not need one manager
>> handlers many pipes?
>
> Because it is extremely counter-intuitive to have to run multiple
> applications to run many pipelines. Just like the whole cluster is
> managed with one application, so should pipelines. This is not only a
> user interface issue. This is also a change to the schema, to handle
> multiple pipelines in one database, rather than many databases, and
> have pipeline ids on the jobs, on the files,etc. etc.
>
One thing I need to remind is about Perl's thread. As I gave up the
Bio::DB::BioSQL::MultiDB using thread, thread-version perl raise more
problems than its benefit. If you want to implement the above
mentioned, you bet the Perl thread.
If the thread problem cannot be solved in perl, you cannot make
pipeline manager server program access multiple connection as well.
My suggest is to let pipeline managers running as process and each has
one database. On the top of it, there builds a server program to
connect with client and simply submit the process on the cluster. Due
to the perl thread problem, if it is true, we should use Java. We do
not need a lot intercommunication between the Java marshall and the
perl's pipeline managers, I think.
my $.02
Juguang
More information about the bioperl-pipeline
mailing list