[Bioperl-l] Problems with Bioperl-ext package on WinVista?

Chris Fields cjfields at illinois.edu
Sun Aug 16 12:53:29 UTC 2009 

That worked!  Thanks Yee Man!

chris

ps - let me know how you want to deal with a release.

On Aug 16, 2009, at 4:36 AM, Yee Man Chan wrote:

> Hi Chris
>
>   Thanks for your suggestions. I think it is indeed better to check  
> sum to 1.0 using sprintf. I fixed this in the newly committed HMM.pm
>
>   I also fixed codes that will lead to warnings with use warnings.
>
>   So now the only problem left is that "monotonic increasing" error.  
> For that part of the code, I was trying to perform an expectation  
> maximization step. Theoretically, the expectation should  
> monotonically increase in every step. But I suppose this is not  
> necessarily true when double precision floating point numbers are  
> involved. I don't know why I used a 1e-100 tolerance for this.  
> Therefore I "fixed" it by using the same tolerance to terminate the  
> maximization step (ie .000001). I suppose this "fix" will make it  
> much more unlikely to throw exception with your 5.10.0 perl.
>
>   Can you give that a try again and see if it works now.
>
> Thank you
> Yee Man
>
>
>
> --- On Sat, 8/15/09, Chris Fields <cjfields at illinois.edu> wrote:
>
>> From: Chris Fields <cjfields at illinois.edu>
>> Subject: Re: [Bioperl-l] Problems with Bioperl-ext package on  
>> WinVista?
>> To: "Yee Man Chan" <ymc at yahoo.com>
>> Cc: "Robert Buels" <rmb32 at cornell.edu>, "BioPerl List" <bioperl-l at lists.open-bio.org 
>> >
>> Date: Saturday, August 15, 2009, 10:38 PM
>> Yee,
>>
>> I took the liberty of making a few simple changes to
>> Bio::Tools::HMM in svn to point out the problem and possible
>> solutions.  Feel free to revert these as needed.
>>
>> I'm seeing two errors, which appear randomly when running
>> 'make test'.  The first is easily fixable, the second,
>> I'm not so sure.  I'll let you make the decisions on
>> both.
>>
>> 1)  There is an assumption in the module that, when
>> adding floating points, you will always get 1.0.  You
>> may run into problems: see 'perldoc -q long decimals'.
>> Lines like this (two places in the module):
>>   ...
>>   if ($sum != 1.0) {
>>      $self->throw("Sum of
>> probabilities for each state must be 1.0; got $sum\n");
>>   }
>>   ...
>>
>> won't work as expected (note I added a simple diagnostic,
>> just print out the 'bad' sum).  With perl 5.8.8, this
>> appears to work fine, but this is what I get with perl 5.10
>> (64-bit):
>>
>> pyrimidine1:HMM cjfields$ make test
>> PERL_DL_NONLAZY=1 /opt/perl/bin/perl "-Iblib/lib"
>> "-Iblib/arch" test.pl
>> Baum-Welch Training
>> ===================
>> Initial Probability Array:
>> 0.499978    0.500022
>> Transition Probability Matrix:
>> 0.499978    0.500022
>> 0.499978    0.500022
>> Emission Probability Matrix:
>> 0.133333    0.143333
>> 0.163333    0.123333
>> 0.143333    0.293333
>> 0.133333    0.143333
>> 0.163333    0.123333
>> 0.143333    0.293333
>>
>> Log Probability of sequence 1: -521.808
>> Log Probability of sequence 2: -426.057
>>
>> Statistical Training
>> ====================
>> Initial Probability Array:
>> 1    0
>> Transition Probability Matrix:
>>
>> ------------- EXCEPTION -------------
>> MSG: Sum of probabilities for each from-state must be 1.0;
>> got 0.999999999999999976
>>
>> STACK Bio::Tools::HMM::transition_prob
>> /Users/cjfields/bioperl/bioperl-live/Bio/Tools/HMM.pm:499
>> STACK toplevel test.pl:82
>> -------------------------------------
>>
>> make: *** [test_dynamic] Error 255
>>
>> I'm assuming this needs to simply be rounded up to
>> 1.0.  That could be accomplished with something like
>> 'if (sprintf("%.2f", $sum) != 1.0) {...}'
>>
>> 2) The second error is a little stranger.  I have been
>> randomly getting this:
>>
>> pyrimidine1:HMM cjfields$ make test
>> PERL_DL_NONLAZY=1 /opt/perl/bin/perl "-Iblib/lib"
>> "-Iblib/arch" test.pl
>> Baum-Welch Training
>> ===================
>> S should be monotonic increasing!
>> make: *** [test_dynamic] Error 255
>>
>> When I add strict and warnings pragmas to Bio::Tools::HMM
>> (with a little additional cleanup to get things running), I
>> get an additional warning (arrow):
>>
>> pyrimidine1:HMM cjfields$ make test
>> PERL_DL_NONLAZY=1 /opt/perl/bin/perl "-Iblib/lib"
>> "-Iblib/arch" test.pl
>> Argument "FL" isn't numeric in numeric lt (<) at
>> /Users/cjfields/bioperl/bioperl-live/Bio/Tools/HMM.pm line
>> 188. <----
>> Baum-Welch Training
>> ===================
>> S should be monotonic increasing!
>> make: *** [test_dynamic] Error 255
>>
>> So something is not being converted as expected.
>>
>> chris
>>
>> On Aug 15, 2009, at 11:32 PM, Yee Man Chan wrote:
>>
>>> When are you going to release 1.6? Maybe let me work
>> on it before it releases. If it doesn't resolve the problem,
>> then we can think about other alternatives.
>>>
>>> Also, please show me the latest errors you have for
>> 5.10.0.
>>>
>>> Thanks
>>> Yee Man
>>>
>>> --- On Sat, 8/15/09, Chris Fields <cjfields at illinois.edu>
>> wrote:
>>>
>>>> From: Chris Fields <cjfields at illinois.edu>
>>>> Subject: Re: [Bioperl-l] Problems with Bioperl-ext
>> package on WinVista?
>>>> To: "Yee Man Chan" <ymc at yahoo.com>
>>>> Cc: "Robert Buels" <rmb32 at cornell.edu>,
>> "BioPerl List" <bioperl-l at lists.open-bio.org>
>>>> Date: Saturday, August 15, 2009, 7:05 PM
>>>> I'm still seeing the same errors on
>>>> Mac OS X for 64-bit perl 5.10.0.  Mac OS X,
>> native perl
>>>> (v5.8.8) passes fine now (as well as perl 5.8.8
>> on
>>>> dev.open-bio.org).
>>>>
>>>> I'm wondering if this is a problem with my local
>> perl
>>>> build.  I'm very tempted to push the
>> HMM-related code
>>>> into a separate distribution (bioperl-hmm) and
>> make a CPAN
>>>> release out of it so it gets wider testing via
>> CPAN testers;
>>>> it would just require a minimum bioperl 1.6
>> installation for
>>>> Bio::Tools::HMM and any related modules.
>> Yee, would
>>>> that be okay with you?
>>>>
>>>> chris
>>>>
>>>> On Aug 15, 2009, at 8:23 PM, Yee Man Chan wrote:
>>>>
>>>>>
>>>>> I just committed HMM.xs and typemap to SVN.
>> Can you
>>>> test it to confirm it works in 64-bit machines?
>>>>>
>>>>> Thanks
>>>>> Yee Man
>>>>>
>>>>> --- On Sat, 8/15/09, Chris Fields <cjfields at illinois.edu>
>>>> wrote:
>>>>>
>>>>>> From: Chris Fields <cjfields at illinois.edu>
>>>>>> Subject: Re: [Bioperl-l] Problems with
>> Bioperl-ext
>>>> package on WinVista?
>>>>>> To: "Robert Buels" <rmb32 at cornell.edu>
>>>>>> Cc: "Yee Man Chan" <ymc at yahoo.com>,
>>>> "BioPerl List" <bioperl-l at lists.open-bio.org>
>>>>>> Date: Saturday, August 15, 2009, 12:11 PM
>>>>>> I'm not sure, but it makes more sense
>>>>>> to commit these changes directly.
>> Yee, need
>>>> us to set
>>>>>> you up with a commit bit?  If so,
>> fill out
>>>> the
>>>>>> information on this page:
>>>>>>
>>>>>> http://www.bioperl.org/wiki/SVN_Account_Request
>>>>>>
>>>>>> and forward it to support at open-bio.org.
>>>>>> I'll sponsor you.
>>>>>>
>>>>>> chris
>>>>>>
>>>>>> On Aug 15, 2009, at 11:44 AM, Robert Buels
>> wrote:
>>>>>>
>>>>>>> The usual procedure for developing
>> code is to
>>>> exchange
>>>>>> code via commits to a version control
>>>> system.  Yee, do
>>>>>> you know how to use Subversion? Does Yee
>> need a
>>>> commit bit?
>>>>>>>
>>>>>>> Rob
>>>>>>>
>>>>>>> Yee Man Chan wrote:
>>>>>>>> Hi Chris
>>>>>>>>      I find
>> that there is a
>>>> memory
>>>>>> access bug in my code. Attached is the
>> fixed
>>>> HMM.xs. This
>>>>>> file together with the simpler typemap
>> should fix
>>>> all
>>>>>> problems. (I hope..)
>>>>>>>>      Please let
>> me know if it
>>>> works
>>>>>> for you.
>>>>>>>> Sorry for the bug...
>>>>>>>> Yee Man
>>>>>>>> --- On Fri, 8/14/09, Chris Fields
>> <cjfields at illinois.edu>
>>>>>> wrote:
>>>>>>>>> From: Chris Fields <cjfields at illinois.edu>
>>>>>>>>> Subject: Re: [Bioperl-l]
>> Problems
>>>> with
>>>>>> Bioperl-ext package on WinVista?
>>>>>>>>> To: "Yee Man Chan" <ymc at yahoo.com>
>>>>>>>>> Cc: "Robert Buels" <rmb32 at cornell.edu>,
>>>>>> "Jonny Dalzell" <jdalzell03 at qub.ac.uk>,
>>>>>> "BioPerl List" <Bioperl-l at lists.open-bio.org>
>>>>>>>>> Date: Friday, August 14, 2009,
>> 8:31
>>>> AM
>>>>>>>>> Yee Man,
>>>>>>>>>
>>>>>>>>> I tested this out locally
>> (perl 5.8.8
>>>> 32-bit,
>>>>>> perl 5.10.0
>>>>>>>>> 64-bit) and on
>> dev.open-bio.org (which
>>>> is perl
>>>>>> 5.8.8,
>>>>>>>>> appears to be 32-bit).
>> The patch
>>>> results
>>>>>> in cleaning
>>>>>>>>> up warnings for 5.10.0 but
>> results in
>>>> similar
>>>>>> warnings for
>>>>>>>>> 5.8.8 (linux or OS X).
>>>>>>>>>
>>>>>>>>> On OS X perl 5.8.8, this
>> sometimes
>>>> passes
>>>>>> (note the first
>>>>>>>>> attempt fails, the second
>> succeeds),
>>>> so it's
>>>>>> not entirely a
>>>>>>>>> 32-bit issue:
>>>>>>>>>
>>>>>>>>> http://gist.github.com/167860
>>>>>>>>>
>>>>>>>>> OS X and perl 5.10.0, this
>> always
>>>> fails as the
>>>>>> previous
>>>>>>>>> gist shows, but demonstrates
>> similar
>>>> behavior
>>>>>> (multiple
>>>>>>>>> attempts to test get
>> different
>>>> responses):
>>>>>>>>>
>>>>>>>>> http://gist.github.com/167542
>>>>>>>>>
>>>>>>>>> On linux, everything passes
>> with or
>>>> w/o the
>>>>>> patched files
>>>>>>>>> (patched files have warnings
>> as
>>>> indicated
>>>>>> above):
>>>>>>>>>
>>>>>>>>> Specs for all three perl
>> executables
>>>> (they
>>>>>> vary a bit):
>>>>>>>>>
>>>>>>>>> http://gist.github.com/167883
>>>>>>>>>
>>>>>>>>> chris
>>>>>>>>>
>>>>>>>>> On Aug 14, 2009, at 3:27 AM,
>> Yee Man
>>>> Chan
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Ah.. I find that the
>> typemap can
>>>> become as
>>>>>> simple as
>>>>>>>>> this
>>>>>>>>>> =====================
>>>>>>>>>> TYPEMAP
>>>>>>>>>> HMM *
>> T_PTROBJ
>>>>>>>>>> =====================
>>>>>>>>>>
>>>>>>>>>> Then the generated HMM.c
>> will have
>>>> a
>>>>>> function called
>>>>>>>>> INT2PTR to do the pointer
>> conversion.
>>>> I
>>>>>> believe this should
>>>>>>>>> solve the warnings.
>>>>>>>>>> Attached are the updated
>> HMM.xs
>>>> and
>>>>>> typemap. Can
>>>>>>>>> someone with a 64-bit machine
>> give it
>>>> a try?
>>>>>>>>>> Thank you
>>>>>>>>>> Yee Man
>>>>>>>>>> --- On Thu, 8/13/09, Chris
>> Fields
>>>> <cjfields at illinois.edu>
>>>>>>>>> wrote:
>>>>>>>>>>> From: Chris Fields
>> <cjfields at illinois.edu>
>>>>>>>>>>> Subject: Re:
>> [Bioperl-l]
>>>> Problems with
>>>>>> Bioperl-ext
>>>>>>>>> package on WinVista?
>>>>>>>>>>> To: "Yee Man Chan"
>> <ymc at yahoo.com>
>>>>>>>>>>> Cc: "Robert Buels"
>> <rmb32 at cornell.edu>,
>>>>>>>>> "Jonny Dalzell" <jdalzell03 at qub.ac.uk>,
>>>>>>>>> "BioPerl List" <Bioperl-l at lists.open-bio.org>
>>>>>>>>>>> Date: Thursday, August
>> 13,
>>>> 2009, 5:31
>>>>>> PM
>>>>>>>>>>> (just to point out to
>>>> everyone, Yee
>>>>>>>>>>> Man's contact
>> information was
>>>> in the
>>>>>> POD)
>>>>>>>>>>>
>>>>>>>>>>> Yee Man,
>>>>>>>>>>>
>>>>>>>>>>> I have the output in
>> the below
>>>> link:
>>>>>>>>>>>
>>>>>>>>>>> http://gist.github.com/167542
>>>>>>>>>>>
>>>>>>>>>>> There are similar
>> problems
>>>> popping up
>>>>>> on 32- and
>>>>>>>>> 64-bit
>>>>>>>>>>> perl 5.10.0, Mac OS X
>> 10.5.
>>>>>> Haven't had time
>>>>>>>>> to debug
>>>>>>>>>>> it unfortunately.
>>>>>>>>>>>
>>>>>>>>>>> I think we should
>> seriously
>>>> consider
>>>>>> spinning this
>>>>>>>>> code off
>>>>>>>>>>> into it's own
>> distribution
>>>> for
>>>>>> CPAN.  It's
>>>>>>>>>>> unfortunately
>> bit-rotting away
>>>> in
>>>>>>>>> bioperl-ext.  If you
>>>>>>>>>>> want to continue
>> supporting it
>>>> I can
>>>>>> help set that
>>>>>>>>> up.
>>>>>>>>>>> chris
>>>>>>>>>>>
>>>>>>>>>>> On Aug 13, 2009, at
>> 6:58 PM,
>>>> Yee Man
>>>>>> Chan wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi
>>>>>>>>>>>>
>>>>>>>>>>>>
>>    So is
>>>> this
>>>>>> an HMM only
>>>>>>>>> problem? Or does
>>>>>>>>>>> it apply to other
>> bioperl-ext
>>>>>> modules?
>>>>>>>>>>>>
>>    What
>>>>>> exactly are the
>>>>>>>>> compilation errors
>>>>>>>>>>> for HMM? I believe my
>>>> implementation
>>>>>> is just a
>>>>>>>>> simple one
>>>>>>>>>>> based on Rabiner's
>> paper.
>>>>>>>>>>>> http://www.google.com/url?sa=t&source=web&ct=res&cd=1&url=http%3A%2F%2Fwww.cs.ubc.ca%2F 
>>>>>>>>>>>> ~murphyk%2FBayes 
>>>>>>>>>>>> %2Frabiner.pdf&ei=QqiESvClHNCfkQXbh8mWBw&rct=j&q=rabiner 
>>>>>>>>>>>> +hmm&usg=AFQjCNHeXLhTHmuKUXKKCHYSs58TxVGfZg
>>>>>>>>>>>>
>>>>>>>>>>>>
>>    I
>>>> don't
>>>>>> think I did
>>>>>>>>> anything fancy that
>>>>>>>>>>> makes it machine
>> dependent or
>>>> non-ANSI
>>>>>> C.
>>>>>>>>>>>> Yee Man
>>>>>>>>>>>>
>>>>>>>>>>>> --- On Thu,
>> 8/13/09, Chris
>>>> Fields
>>>>>> <cjfields at illinois.edu>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>> From: Chris
>> Fields
>>>> <cjfields at illinois.edu>
>>>>>>>>>>>>> Subject: Re:
>>>> [Bioperl-l]
>>>>>> Problems with
>>>>>>>>> Bioperl-ext
>>>>>>>>>>> package on WinVista?
>>>>>>>>>>>>> To: "Robert
>> Buels"
>>>> <rmb32 at cornell.edu>
>>>>>>>>>>>>> Cc: "Jonny
>> Dalzell"
>>>> <jdalzell03 at qub.ac.uk>,
>>>>>>>>>>> "BioPerl List" <Bioperl-l at lists.open-bio.org>,
>>>>>>>>>>> "Yee Man Chan" <ymc at yahoo.com>
>>>>>>>>>>>>> Date:
>> Thursday, August
>>>> 13,
>>>>>> 2009, 3:18 PM
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Aug 13,
>> 2009, at
>>>> 4:37 PM,
>>>>>> Robert Buels
>>>>>>>>> wrote:
>>>>>>>>>>>>>> Jonny
>> Dalzell
>>>> wrote:
>>>>>>>>>>>>>>> Is it
>>>> ridiculous of me
>>>>>> to expect
>>>>>>>>> ubuntu to
>>>>>>>>>>> take
>>>>>>>>>>>>> care of this
>> for
>>>> me?  How
>>>>>> do
>>>>>>>>>>>>>>> I go
>> about
>>>> compiling
>>>>>> the HMM?
>>>>>>>>>>>>>> Yes.
>> This is
>>>> a very
>>>>>> specialized
>>>>>>>>> thing
>>>>>>>>>>> that
>>>>>>>>>>>>> you're doing,
>> and
>>>> Ubuntu does
>>>>>> not have
>>>>>>>>> the
>>>>>>>>>>> resources to
>>>>>>>>>>>>> package every
>> single
>>>> thing.
>>>>>>>>>>>>>>
>> Unfortunately, it
>>>> looks
>>>>>> like
>>>>>>>>> bioperl-ext
>>>>>>>>>>> package is
>>>>>>>>>>>>> not
>> installable under
>>>> Ubuntu
>>>>>> 9.04 anyway,
>>>>>>>>> which is
>>>>>>>>>>> what I'm
>>>>>>>>>>>>> running.
>> For
>>>> others on
>>>>>> this list,
>>>>>>>>> if
>>>>>>>>>>> somebody is
>>>>>>>>>>>>> interested in
>> doing
>>>>>> maintaining it, I'd be
>>>>>>>>> happy
>>>>>>>>>>> to help out
>>>>>>>>>>>>> by testing on
>>>> Debian-based
>>>>>> Linux
>>>>>>>>> platforms.
>>>>>>>>>>> We need to
>>>>>>>>>>>>> clarify this
>>>> package's
>>>>>> maintenance status:
>>>>>>>>> if
>>>>>>>>>>> there is
>>>>>>>>>>>>> nobody
>> interested in
>>>>>> maintaining it, I
>>>>>>>>> would
>>>>>>>>>>> recommend that
>>>>>>>>>>>>> bioperl-ext be
>> removed
>>>> from
>>>>>> distribution.
>>>>>>>>>>> It's not in
>>>>>>>>>>>>> anybody's
>> interest to
>>>> have
>>>>>> unmaintained
>>>>>>>>> software
>>>>>>>>>>> out there
>>>>>>>>>>>>> causing
>> confusion.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have cc'd
>> Yee Man
>>>> Chan for
>>>>>> this.
>>>>>>>>> If there
>>>>>>>>>>> isn't a
>>>>>>>>>>>>> response or
>> the
>>>> message
>>>>>> bounces, we do one
>>>>>>>>> of two
>>>>>>>>>>> things:
>>>>>>>>>>>>> 1) consider
>> it
>>>> deprecated
>>>>>> (probably
>>>>>>>>> safest).
>>>>>>>>>>>>> 2) spin it out
>> into a
>>>> separate
>>>>>> module.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Just tried to
>> comile
>>>> it myself
>>>>>> and am
>>>>>>>>> getting
>>>>>>>>>>> errors (using
>>>>>>>>>>>>> 64bit perl
>> 5.10), so I
>>>> think,
>>>>>> unless
>>>>>>>>> someone wants
>>>>>>>>>>> to take
>>>>>>>>>>>>> this on,
>> option #1 is
>>>> best.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> So Jonny,
>> in
>>>> short, I
>>>>>> would say "do
>>>>>>>>> not use
>>>>>>>>>>>>> bioperl-ext".
>>>>>>>>>>>>>
>>>>>>>>>>>>> In general,
>> that's a
>>>> safe
>>>>>> bet.  We're
>>>>>>>>> moving
>>>>>>>>>>> most of
>>>>>>>>>>>>> our C/C++
>> bindings to
>>>> BioLib.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Step
>> back.
>>>> What are
>>>>>> you trying
>>>>>>>>> to
>>>>>>>>>>>>> accomplish?
>>>> Chris
>>>>>> already
>>>>>>>>> recommended some
>>>>>>>>>>> alternative
>>>>>>>>>>>>> methods in his
>> email
>>>> of 8/11
>>>>>> on this
>>>>>>>>>>> subject.
>> Perhaps
>>>>>>>>>>>>> we can guide
>> you to
>>>> some
>>>>>> software that is
>>>>>>>>>>> actively
>>>>>>>>>>>>> maintained and
>> will
>>>> meet your
>>>>>> needs.
>>>>>>>>>>>>>> Rob
>>>>>>>>>>>>> Exactly.
>> Lots of
>>>> other
>>>>>> (better
>>>>>>>>> supported!)
>>>>>>>>>>> options
>>>>>>>>>>>>> out there.
>>>> HMMER, SeqAn,
>>>>>> and
>>>>>>>>> others.
>>>>>>>>>>>>> chris
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>>>
>> __________________________________________________
>>>>>>>>>> Do You Yahoo!?
>>>>>>>>>> Tired of spam?
>> Yahoo! Mail
>>>> has the
>>>>>> best spam
>>>>>>>>> protection around
>>>>>>>>>> http://mail.yahoo.com
>>>>>>>>>
>>>>>>
>>>>
>> <HMM.xs><typemap>_______________________________________________
>>>>>>>>>> Bioperl-l mailing list
>>>>>>>>>> Bioperl-l at lists.open-bio.org
>>>>>>>>>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --Robert Buels
>>>>>>> Bioinformatics Analyst, Sol Genomics
>> Network
>>>>>>> Boyce Thompson Institute for Plant
>> Research
>>>>>>> Tower Rd
>>>>>>> Ithaca, NY  14853
>>>>>>> Tel: 503-889-8539
>>>>>>> rmb32 at cornell.edu
>>>>>>> http://www.sgn.cornell.edu
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>
>
>

More information about the Bioperl-l mailing list