[Bioperl-pipeline] web, restart

[aaron at tll.org.sg]

Dear Shawn,

Hey, hope your network problems have cleared up, I can't detect anything
wrong from this end (even ssh connections from home).. was it a particular
machine or leo?

As for the web pipeline... it certainly sounds like an interesting option.
Was working on web-based monitoring software earlier that went thru three
development revamps over 2 years. 1st version perl.cgi. 2nd pure java (app)
thru proprietary tcp-based protocol, and we had to finally settle on
jsp/tomcat to avoid firewall/port problems, etc.

Our solution was based on a tiny server stub that maintains central control
of  the processes called a monitor. Then each web connection in will query
this monitor for current process state, etc. Back then we used shared memory
to decouple the actual processes from state monitoring. This can be done
through Perl IPC V5 calls too, though our implementation was C.

Let me read thru the biopipe architecture to see how the processes are
handled (forked/threaded), managed by PID/lockfiles, and I'll get back to
the list on how best to begin the design of the web monitor/launcher.

Cheers,
aaron

>
> On Thursday, June 26, 2003, at 07:17  PM, jeremyp at sgx3.bmb.uga.edu
> wrote:
>
>> Hi,
>>
>> I remember there was some talk about including software to allow for
>> running pipelines through a web interface. Has this been done?
>>
>
> Uhm, I personally have not been working on this but I think some folks
> at TLL might
> be working on this. Aaron you wanna chime in? Should we start some
> discussion on what some issues that need
> to be addressed to get this developing seriously?
>
>> Also, is there a way to restart the PipelineManager? That is, if the
>> PipelineManager were killed but I wanted the pipeline it was running
>> when
>> killed to continue running at a later date, is there a way to do this?
>>
>
> Oh definitely, we do it all the time. When you kill the
> PipelineManager, all the job states are
> stored in the database. Normally what happens in this scenario is that
> the pipeline user
>
> 1) first kills the PipelineManager script
> 2) does a bkill 0 (for lsf) for his jobs
> 3) Do whatever fixing one needs and make sure that the inputs and
> output databases are cleaned appropriately
> 3) The state of the jobs in the job table will have a mix of jobs that
> have status Failed, New, Submitted
>     You will need to set the jobs that are in Submitted state back to
> New or Failed. This is because we killed
>     the jobs with the bkill before they could write their status back
> to the table. So the PipelineManager upon
>    restart will think they are still running (not so clever as to check
>
> with LSF yet) and only fetch the New|Failed jobs.
>
>   So execute : update job set status="NEW" where status="SUBMITTED" in
>   your pipeline database
> 4) Remove the Pipeline lock file and run PipelineManager again OR
>     run PipelineManager with the -f option and it should remove it for
> you.
>
> hope that is clear
>
> cheers,
>
> shawn
>
>
>
>
>> Thanks,
>> Jeremy
>> _______________________________________________
>> bioperl-pipeline mailing list
>> bioperl-pipeline at bioperl.org
>> http://bioperl.org/mailman/listinfo/bioperl-pipeline