From gbottu at ben.vub.ac.be Fri Mar 2 06:25:48 2007 From: gbottu at ben.vub.ac.be (Guy Bottu) Date: Fri, 2 Mar 2007 12:25:48 +0100 Subject: [emboss-dev] Upper limit to "floats" - Checked by AntiVir DEMO version - Message-ID: <20070302112548.GA5122@bigben.ulb.ac.be> Dear colleagues, While testing a problem related to the difference between BLAST outputs at the NCBI and at BEN I tried to run blast2seq (our "wrapper" for bl2seq) with parameter -effdbsize=19973047279.0 and got a : Uncaught exception: Assertion failed, raised at ajmem.c:134 The crash occurs when the program reads the "float". I tried to add a "maximum: 1000000000000" to the ACD, but this does not help. Can you confirm that there is an upper limit to the size of floating point numbers EMBOSS can handle ? And do you think it is a good idea to increase this value, since e.g. the total size of the NCBI "nr" databank is higher and so we might need our programs to handle such values ? Regards, Guy Bottu, BEN From pmr at ebi.ac.uk Fri Mar 2 07:20:08 2007 From: pmr at ebi.ac.uk (Peter Rice) Date: Fri, 02 Mar 2007 12:20:08 +0000 Subject: [emboss-dev] Upper limit to "floats" - Checked by AntiVir DEMO version - In-Reply-To: <20070302112548.GA5122@bigben.ulb.ac.be> References: <20070302112548.GA5122@bigben.ulb.ac.be> Message-ID: <45E81678.8000105@ebi.ac.uk> Guy Bottu wrote: > While testing a problem related to the difference between BLAST outputs > at the NCBI and at BEN I tried to run blast2seq (our "wrapper" for > bl2seq) with parameter -effdbsize=19973047279.0 and got a : > Uncaught exception: Assertion failed, raised at ajmem.c:134 Curious. This error suggests you have called ajMemCalloc with a request for an array of zero (or negative) length. I suspect you have exceeded the size of an integer. It does not look like a floating point problem. Macros that call ajMemCalloc include: AJALLOC(nbytes) AJALLOC0(nbytes) AJCALLOC(count, nbytes) AJCALLOC0(count, nbytes) In each case, "nbytes" is the value that was not a positive integer. In EMBOSS memory handling nbytes is of type ajlong. It may be you need to store the database size in a larger integer datatype, or perhaps some automatic conversion to int is happening. Let me know if you need more clues. regards, Peter From pmr at ebi.ac.uk Fri Mar 2 08:40:26 2007 From: pmr at ebi.ac.uk (Peter Rice) Date: Fri, 02 Mar 2007 13:40:26 +0000 Subject: [emboss-dev] Upper limit to "floats" - Checked by AntiVir DEMO vers In-Reply-To: <20070302132046.GA6813@bigben.ulb.ac.be> References: <20070302112548.GA5122@bigben.ulb.ac.be> <45E81678.8000105@ebi.ac.uk> <20070302132046.GA6813@bigben.ulb.ac.be> Message-ID: <45E8294A.3080705@ebi.ac.uk> Hi Guy, Guy Bottu wrote: > On Fri, Mar 02, 2007 at 12:20:08PM +0000, Peter Rice wrote: >> Curious. This error suggests you have called ajMemCalloc with a request >> for an array of zero (or negative) length. >> I suspect you have exceeded the size of an integer. It does not look like >> a floating point problem. > > Actually, the ACD contains : > > float: effdbsize [ > information: "Effective databank size for statistical > calculations" > minimum: "0.0" > ] > > and the program : > > float CLfloat; /* real number on command line */ > ... > embInit ("blast2seq", argc, argv); > (here it crashes) Aha ... and it crashes in valgrind too. Excellent timing, just as we are thinking about 4.1.0. Is this a problem with the current CVS code? We made changes to the way this works, so 4.0.0 seems to have no problem. The problem is in function ajStrFromFloat where converting the float to a string goes through extracting the absolute value as an integer to work out how long the string needs to be. We do this because all ACD values are stored as strings internally. To fix it, in ajmath.c look at functions ajNumLengthFloat and ajNumLengthDouble and replace int with long, and replace UINT_MAX wth ULONG_MAX Fixed in 4.1.0 and I will commit ajmath.c once the QAtests complete (they take rather longer than they used to now - we added a lot more memory tests :-) From gbottu at ben.vub.ac.be Fri Mar 2 11:25:48 2007 From: gbottu at ben.vub.ac.be (Guy Bottu) Date: Fri, 2 Mar 2007 12:25:48 +0100 Subject: [emboss-dev] Upper limit to "floats" - Checked by AntiVir DEMO version - Message-ID: <20070302112548.GA5122@bigben.ulb.ac.be> Dear colleagues, While testing a problem related to the difference between BLAST outputs at the NCBI and at BEN I tried to run blast2seq (our "wrapper" for bl2seq) with parameter -effdbsize=19973047279.0 and got a : Uncaught exception: Assertion failed, raised at ajmem.c:134 The crash occurs when the program reads the "float". I tried to add a "maximum: 1000000000000" to the ACD, but this does not help. Can you confirm that there is an upper limit to the size of floating point numbers EMBOSS can handle ? And do you think it is a good idea to increase this value, since e.g. the total size of the NCBI "nr" databank is higher and so we might need our programs to handle such values ? Regards, Guy Bottu, BEN From pmr at ebi.ac.uk Fri Mar 2 12:20:08 2007 From: pmr at ebi.ac.uk (Peter Rice) Date: Fri, 02 Mar 2007 12:20:08 +0000 Subject: [emboss-dev] Upper limit to "floats" - Checked by AntiVir DEMO version - In-Reply-To: <20070302112548.GA5122@bigben.ulb.ac.be> References: <20070302112548.GA5122@bigben.ulb.ac.be> Message-ID: <45E81678.8000105@ebi.ac.uk> Guy Bottu wrote: > While testing a problem related to the difference between BLAST outputs > at the NCBI and at BEN I tried to run blast2seq (our "wrapper" for > bl2seq) with parameter -effdbsize=19973047279.0 and got a : > Uncaught exception: Assertion failed, raised at ajmem.c:134 Curious. This error suggests you have called ajMemCalloc with a request for an array of zero (or negative) length. I suspect you have exceeded the size of an integer. It does not look like a floating point problem. Macros that call ajMemCalloc include: AJALLOC(nbytes) AJALLOC0(nbytes) AJCALLOC(count, nbytes) AJCALLOC0(count, nbytes) In each case, "nbytes" is the value that was not a positive integer. In EMBOSS memory handling nbytes is of type ajlong. It may be you need to store the database size in a larger integer datatype, or perhaps some automatic conversion to int is happening. Let me know if you need more clues. regards, Peter From pmr at ebi.ac.uk Fri Mar 2 13:40:26 2007 From: pmr at ebi.ac.uk (Peter Rice) Date: Fri, 02 Mar 2007 13:40:26 +0000 Subject: [emboss-dev] Upper limit to "floats" - Checked by AntiVir DEMO vers In-Reply-To: <20070302132046.GA6813@bigben.ulb.ac.be> References: <20070302112548.GA5122@bigben.ulb.ac.be> <45E81678.8000105@ebi.ac.uk> <20070302132046.GA6813@bigben.ulb.ac.be> Message-ID: <45E8294A.3080705@ebi.ac.uk> Hi Guy, Guy Bottu wrote: > On Fri, Mar 02, 2007 at 12:20:08PM +0000, Peter Rice wrote: >> Curious. This error suggests you have called ajMemCalloc with a request >> for an array of zero (or negative) length. >> I suspect you have exceeded the size of an integer. It does not look like >> a floating point problem. > > Actually, the ACD contains : > > float: effdbsize [ > information: "Effective databank size for statistical > calculations" > minimum: "0.0" > ] > > and the program : > > float CLfloat; /* real number on command line */ > ... > embInit ("blast2seq", argc, argv); > (here it crashes) Aha ... and it crashes in valgrind too. Excellent timing, just as we are thinking about 4.1.0. Is this a problem with the current CVS code? We made changes to the way this works, so 4.0.0 seems to have no problem. The problem is in function ajStrFromFloat where converting the float to a string goes through extracting the absolute value as an integer to work out how long the string needs to be. We do this because all ACD values are stored as strings internally. To fix it, in ajmath.c look at functions ajNumLengthFloat and ajNumLengthDouble and replace int with long, and replace UINT_MAX wth ULONG_MAX Fixed in 4.1.0 and I will commit ajmath.c once the QAtests complete (they take rather longer than they used to now - we added a lot more memory tests :-)