[Biopython-dev] Re: Work towards getting KDTree compiling

[Michiel Jan Laurens de Hoon]

Brad Chapman wrote:
>>o) On Windows, the compilation of KDTree fails:
>>
>>building 'Bio.KDTree._CKDTree' extension
>>creating build\temp.win32-2.3\Release\bio\kdtree
>>C:\cygwin\bin\cc.exe -Ic:\Python23\include -Ic:\Python23\PC -c 
>>Bio/KDTree/KDTree
>>.swig.cpp -o build\temp.win32-2.3\Release\bio\kdtree\kdtree.swig.o
>>error: command 'cc' failed: Invalid argument
>>
>>I am compiling with python setup.py build --compiler=mingw32. The C 
>>extensions in Biopython (e.g. Bio.Cluster) compile without problems.
>>This may be a bug in distutils for mingw32 when dealing with C++.
> 
> 
> Yeah, I don't really understand the error (Invalid argument seems to
> imply there is something wrong with the commandline, and I'm not
> sure what that is), but digging through distutils code it looks like
> C++ in mingw32 is not really supported.
> 
> As a complete guess, I tried applying the same changes I used for
> Python 2.2.x to mingw32 compilation. Will you give the new setup.py 
> a try and see if it improves anything at all?

Sorry, doesn't work. Using Python 2.3.3:

building 'Bio.KDTree._CKDTree' extension
creating build\temp.win32-2.3\Release\bio\kdtree
Traceback (most recent call last):
   File "setup.py", line 505, in ?
     data_files=DATA_FILES,
   File "c:\Python23\lib\distutils\core.py", line 149, in setup
     dist.run_commands()
   File "c:\Python23\lib\distutils\dist.py", line 907, in run_commands
     self.run_command(cmd)
   File "c:\Python23\lib\distutils\dist.py", line 927, in run_command
     cmd_obj.run()
   File "c:\Python23\lib\distutils\command\build.py", line 107, in run
     self.run_command(cmd_name)
   File "c:\Python23\lib\distutils\cmd.py", line 333, in run_command
     self.distribution.run_command(command)
   File "c:\Python23\lib\distutils\dist.py", line 927, in run_command
     cmd_obj.run()
   File "setup.py", line 166, in run
     build_ext.run(self)
   File "c:\Python23\lib\distutils\command\build_ext.py", line 269, in run
     self.build_extensions()
   File "setup.py", line 171, in build_extensions
     build_ext.build_extensions(self)
   File "c:\Python23\lib\distutils\command\build_ext.py", line 395, in build_exte
nsions
     self.build_extension(ext)
   File "setup.py", line 197, in build_extension
     build_ext.build_extension(self, ext)
   File "c:\Python23\lib\distutils\command\build_ext.py", line 460, in build_exte
nsion
     depends=ext.depends)
   File "c:\Python23\lib\distutils\ccompiler.py", line 695, in compile
     self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
   File "c:\Python23\lib\distutils\cygwinccompiler.py", line 137, in _compile
     self.spawn(self.compiler_so + cc_args + [src, '-o', obj] +
   File "c:\Python23\lib\distutils\ccompiler.py", line 1036, in spawn
     spawn (cmd, dry_run=self.dry_run)
   File "c:\Python23\lib\distutils\spawn.py", line 39, in spawn
     _spawn_nt(cmd, search_path, dry_run=dry_run)
   File "c:\Python23\lib\distutils\spawn.py", line 72, in _spawn_nt
     cmd = _nt_quote_args(cmd)
   File "c:\Python23\lib\distutils\spawn.py", line 62, in _nt_quote_args
     if string.find(args[i], ' ') != -1:
   File "C:\Python23\lib\string.py", line 178, in find
     return s.find(*args)
AttributeError: 'NoneType' object has no attribute 'find'


>>o) On SunOS 5.8, the compilation runs fine, using the native cc compiler 
>>for the C extensions and the g++ compiler for the C++ extensions. Note that 
>>mixing the cc compiler and the g++ compiler may lead to crashes. But at 
>>least on SunOS 5.8, no such problems occur when importing KDTree or Affy.
> 
> 
> Does Tests/test_KDTree.py work for you as well? If there are no
> problems then I say we are all in the clear. Since distutils is
> really just pulling up the C++ compiler information from the
> environment, there is really nothing we can do about the cc/g++
> conflict -- honestly this is probably a issue where a solaris user
> has to set CXX to point to the native c++ compiler.

It works! Allright! Using Python 2.3.3 on SunOS 5.8:

anago{mdehoon}8: python -i test_KDTree.py
Passed.
Passed.
Passed.
Passed.
Passed.
Passed.
Passed.
Passed.
Passed.
Passed.
Passed.
Passed.
Passed.
Passed.
Passed.
Passed.
Passed.
Passed.
Passed.
Passed.
 >>>


>>o) On Mac OS X, the compilation seems to run fine, but python crashes if 
>>one of the C++ modules is imported. The C modules do not cause a crash. I 
>>am not sure what causes this crash; the C and C++ compilers are consistent 
>>with each other.
> 
> 
> Hmmm. How do they crash? Just a core dump? Can you attach gdb and
> see anything? Also, is this with gcc? What versions? For the record,
> I'm compiling with gcc version 2.95.4 and 3.3.3 without any
> problems.
> 

I'm not sure if this will help much but this is what I get from gdb:

 >>> from Bio.KDTree import *
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Reading symbols for shared libraries bfd_mach_o_scan_read_symtab_symbol: symbol 
name out of range (399792 >= 5512)
. done

Program received signal EXC_BAD_ACCESS, Could not access memory.
0x8fe0cc18 in __dyld_is_symbol_coalesced ()
(gdb)

> Can any other Mac OS people confirm this problem? Any solutions from
> anyone?

The celmodule.cc code in Affy is very short and uses few C++ specific routines. 
Converting this code to standard C is almost trivial and would more robust than 
modifying setup.py to handle C++. Converting KDTree would require some more work 
though.

--Michiel.

-- 
Michiel de Hoon, Assistant Professor
University of Tokyo, Institute of Medical Science
Human Genome Center
4-6-1 Shirokane-dai, Minato-ku
Tokyo 108-8639
Japan
http://bonsai.ims.u-tokyo.ac.jp/~mdehoon