[Biopython-dev] GenePop tests - No such file or directory: 'big.gen.IN2'
Peter Cock
p.j.a.cock at googlemail.com
Sat Oct 5 10:15:33 UTC 2013
Hi Tiago,
The buildbot has often been failing, particularly on Jython 2.7 on
Windows XP, but also one Linux from time to time. The failure is
often stochastic - a rerun can fix it. e.g.
http://testing.open-bio.org/biopython/builders/Linux%2064%20-%20Jython%202.7/builds/267/steps/shell/logs/stdio
======================================================================
ERROR: test_get_heterozygosity_info (test_PopGen_GenePop_EasyController.AppTest)
Test heterozygosity info.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home_local/buildslave/BuildBot_Biopython/jython27lin64/build/Tests/test_PopGen_GenePop_EasyController.py",
line 53, in test_get_heterozygosity_info
hz_info = self.ctrl.get_heterozygosity_info(0, "Locus2")
File "/home_local/buildslave/BuildBot_Biopython/jython27lin64/build/Bio/PopGen/GenePop/EasyController.py",
line 76, in get_heterozygosity_info
geno_freqs = self._controller.calc_allele_genotype_freqs(self._fname)
File "/home_local/buildslave/BuildBot_Biopython/jython27lin64/build/Bio/PopGen/GenePop/Controller.py",
line 679, in calc_allele_genotype_freqs
locf = open(fname + ".IN2")
IOError: [Errno 2] No such file or directory: 'big.gen.IN2'
----------------------------------------------------------------------
The error appears to be in Bio/PopGen/GenePop/Controller.py
where sometimes copying a file and then opening it immediately
can fail:
popf = open(fname + ".INF")
shutil.copyfile(fname + ".INF", fname + ".IN2")
locf = open(fname + ".IN2")
pop_iter = _FileIterator(pop_parser, popf, fname + ".INF")
locus_iter = _FileIterator(locus_parser, locf, fname + ".IN2")
It seems the _FileIterator class is a wrapper to loop over
a file and then delete the file - and since you need to
parse the file in two ways (pop_parser and locus_parser)
you've made a copy of the file.
I presume these files are too big to load into memory
and then delete immediately?
In terms of fixing the symptoms, we could try something
crude like this (untested):
popf = open(fname + ".INF")
shutil.copyfile(fname + ".INF", fname + ".IN2")
while not os.path.isfile(fname + ".IN2"):
sleep(0.5)
locf = open(fname + ".IN2")
pop_iter = _FileIterator(pop_parser, popf, fname + ".INF")
locus_iter = _FileIterator(locus_parser, locf, fname + ".IN2")
I'm not familiar with the output files so can't immediately
see a more satisfactory solution.
Peter
P.S. As an aside, I would refactor this so that _FileIterator
opens the handle itself from the filename given, which
seems cleaner (opening and closing the handle in one
place) and makes the calling code shorter too:
shutil.copyfile(fname + ".INF", fname + ".IN2")
pop_iter = _FileIterator(pop_parser, fname + ".INF")
locus_iter = _FileIterator(locus_parser, fname + ".IN2")
(If you have no objections, I'm happy to make that change)
More information about the Biopython-dev
mailing list