[Biopython-dev] GenePop tests - No such file or directory: 'big.gen.IN2'

[Peter Cock]

Hi Tiago,

The buildbot has often been failing, particularly on Jython 2.7 on
Windows XP, but also one Linux from time to time. The failure is
often stochastic - a rerun can fix it. e.g.

http://testing.open-bio.org/biopython/builders/Linux%2064%20-%20Jython%202.7/builds/267/steps/shell/logs/stdio

======================================================================
ERROR: test_get_heterozygosity_info (test_PopGen_GenePop_EasyController.AppTest)
Test heterozygosity info.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home_local/buildslave/BuildBot_Biopython/jython27lin64/build/Tests/test_PopGen_GenePop_EasyController.py",
line 53, in test_get_heterozygosity_info
    hz_info = self.ctrl.get_heterozygosity_info(0, "Locus2")
  File "/home_local/buildslave/BuildBot_Biopython/jython27lin64/build/Bio/PopGen/GenePop/EasyController.py",
line 76, in get_heterozygosity_info
    geno_freqs = self._controller.calc_allele_genotype_freqs(self._fname)
  File "/home_local/buildslave/BuildBot_Biopython/jython27lin64/build/Bio/PopGen/GenePop/Controller.py",
line 679, in calc_allele_genotype_freqs
    locf = open(fname + ".IN2")
IOError: [Errno 2] No such file or directory: 'big.gen.IN2'

----------------------------------------------------------------------

The error appears to be in Bio/PopGen/GenePop/Controller.py
where sometimes copying a file and then opening it immediately
can fail:

        popf = open(fname + ".INF")
        shutil.copyfile(fname + ".INF", fname + ".IN2")
        locf = open(fname + ".IN2")
        pop_iter = _FileIterator(pop_parser, popf, fname + ".INF")
        locus_iter = _FileIterator(locus_parser, locf, fname + ".IN2")

It seems the _FileIterator class is a wrapper to loop over
a file and then delete the file - and since you need to
parse the file in two ways (pop_parser and locus_parser)
you've made a copy of the file.

I presume these files are too big to load into memory
and then delete immediately?

In terms of fixing the symptoms, we could try something
crude like this (untested):

        popf = open(fname + ".INF")
        shutil.copyfile(fname + ".INF", fname + ".IN2")
        while not os.path.isfile(fname + ".IN2"):
            sleep(0.5)
        locf = open(fname + ".IN2")
        pop_iter = _FileIterator(pop_parser, popf, fname + ".INF")
        locus_iter = _FileIterator(locus_parser, locf, fname + ".IN2")

I'm not familiar with the output files so can't immediately
see a more satisfactory solution.

Peter

P.S. As an aside, I would refactor this so that _FileIterator
opens the handle itself from the filename given, which
seems cleaner (opening and closing the handle in one
place) and makes the calling code shorter too:

        shutil.copyfile(fname + ".INF", fname + ".IN2")
        pop_iter = _FileIterator(pop_parser, fname + ".INF")
        locus_iter = _FileIterator(locus_parser, fname + ".IN2")

(If you have no objections, I'm happy to make that change)