[Biopython-dev] Need guidance.

Peter Cock p.j.a.cock at googlemail.com
Thu Jul 6 21:16:56 UTC 2017 

That is useful Adam - deterministic tests are a
problem, and easily missed with the continuous
integration tests during a pull request.

Adil spent some time already on dealing with
random test sequences, so hopefully he can see
why this is happening...

Thank you!

Peter

On Thu, Jul 6, 2017 at 9:25 PM, Adam Kurkiewicz <adam at kurkiewicz.pl> wrote:
> I've understand now why running the test directly from Scripts didn't
> work.
>
> I can get the test cases to pass OK 90% of the time inside Tests by
> running
>
> `python test_testseq.py`
>
> However, these test cases are still non-deterministic, in particular
> test_codon_tables fails about 10% of the time. Is this intended
> behaviour?
>
> That's the stack trace:
>
> picrin at lamport:~/programming/biopython/adil/Tests$ python
> test_testseq.py
> test_alphabet (__main__.TestTestseq)
> Testing 'alphabet' argument... ... ok
> test_codon_tables (__main__.TestTestseq)
> Testing codon sets... ... FAIL
> test_gc_target (__main__.TestTestseq)
> Testing 'gc_target' argument... ... ok
> test_messenger (__main__.TestTestseq)
> Testing 'messenger' argument... ... ok
> test_seeding (__main__.TestTestseq)
> Testing 'rand_seed' argument... ... ok
> test_size (__main__.TestTestseq)
> Testing 'size' and 'truncate' arguments... ... ok
>
> ======================================================================
> FAIL: test_codon_tables (__main__.TestTestseq)
> Testing codon sets...
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "test_testseq.py", line 87, in test_codon_tables
>     self.assertFalse(seq[0] == "M")
> AssertionError: True is not false
>
> ----------------------------------------------------------------------
> Ran 6 tests in 0.008s
>
> FAILED (failures=1)
>
> Adam
>
> On Thu, Jul 6, 2017, at 09:04 PM, Adam Kurkiewicz wrote:
>> Another issue, perhaps more likely to be a real issue is that the test
>> in line 127 fails about 50% of the time, and I've seen occasional
>> failure in line 107.
>>
>> Are the results of the tests non-deterministic? If so, would it be
>> advisable to make them fail much less often, e.g. once per billion runs
>> (this can usually be achieved by doing a probability calculation)?
>>
>> That's the stack trace of the failing tests:
>>
>> **********************************************************************
>> File "testseq.py", line 107, in __main__.testseq
>> Failed example:
>>     my_seq2.alphabet
>> Expected:
>>     IUPACProtein()
>> Got:
>>     HasStopCodon(IUPACProtein(), '*')
>> **********************************************************************
>> File "testseq.py", line 127, in __main__.testseq
>> Failed example:
>>     -5 < error < 5
>> Expected:
>>     False
>> Got:
>>     True
>> **********************************************************************
>>
>>
>> Adam
>>
>> On Thu, Jul 6, 2017, at 08:46 PM, Adam Kurkiewicz wrote:
>> > Hi,
>> >
>> > I've been just having a look at the pull request.
>> >
>> > Please correct me if I'm wrong.
>> >
>> > 1. At the moment one of the ways to run tests locally is to execute
>> > `python Tests/run_tests.py` from the project's main directory.
>> > 2. Adil's code will not be tested in this way.
>> > 3. In order to test Adil's code one has to `run python
>> > Scripts/testseq.py` from the main directory.
>> >
>> > If that's the case, then it doesn't work on my machine (but I'm most
>> > likely missing something):
>> >
>> > picrin at lamport:~/programming/biopython/adil$ which python
>> > /home/picrin/anaconda3/bin/python
>> > picrin at lamport:~/programming/biopython/adil$ python --version
>> > Python 3.6.0 :: Anaconda custom (64-bit)
>> > picrin at lamport:~/programming/biopython/adil$ python Scripts/testseq.py
>> > Running doctests...
>> > **********************************************************************
>> > File "Scripts/testseq.py", line 65, in __main__.testseq
>> > Failed example:
>> >     from Scripts.testseq import testseq
>> > Exception raised:
>> >     Traceback (most recent call last):
>> >       File "/home/picrin/anaconda3/lib/python3.6/doctest.py", line 1330,
>> >       in __run
>> >         compileflags, 1), test.globs)
>> >       File "<doctest __main__.testseq[0]>", line 1, in <module>
>> >         from Scripts.testseq import testseq
>> >     ModuleNotFoundError: No module named 'Scripts.testseq'
>> > **********************************************************************
>> > 1 items had failures:
>> >    1 of  41 in __main__.testseq
>> > ***Test Failed*** 1 failures.
>> > Done
>> >
>> >
>> > Adam
>> >
>> > On Tue, Jul 4, 2017, at 04:41 PM, Peter Cock wrote:
>> > > Could someone else please take a look at Adil's pull request,
>> > > particularly his doctest tweaks and import magic in order to
>> > > test the code under Scripts/ rather than under Bio/ as usual:
>> > >
>> > > https://github.com/biopython/biopython/pull/1306
>> > >
>> > > (I'd just like the reassurance of a second opinion here, as
>> > > the import stuff has a lot of subtleties.)
>> > >
>> > > Many thanks,
>> > >
>> > > Peter
>> > >
>> > >
>> > > On Thu, Jun 15, 2017 at 10:33 AM, Peter Cock <p.j.a.cock at googlemail.com>
>> > > wrote:
>> > > > Hi Adil,
>> > > >
>> > > > You've certainly learnt lots about imports and doctests - and probably
>> > > > know more than me now.
>> > > >
>> > > > I think you've made a sensible choice in restricting the special cases
>> > > > to your new test_testseq.py while leaving run_tests.py and the
>> > > > existing doctests as they are.
>> > > >
>> > > > Biopython is likely to keep supporting Python 2.7 until the year 2020, see:
>> > > >
>> > > > http://mailman.open-bio.org/pipermail/biopython-dev/2016-December/021613.html
>> > > >
>> > > > Peter
>> > > >
>> > > > On Thu, Jun 15, 2017 at 5:43 AM, Adil Iqbal <aiqbal85 at gmail.com> wrote:
>> > > >> Hey, though I managed to solve the importing issue, in my conversation with
>> > > >> Peter, I realized I made a rather large decision without consulting the
>> > > >> community.
>> > > >>
>> > > >> In my current build, I am omitting all modules that are not part of the Bio
>> > > >> package from executing doctests in version 2.7.
>> > > >>
>> > > >> My reason for doing that is copy-pasted below this message. Please read it
>> > > >> for context.
>> > > >>
>> > > >> I realize that 'testseq' is the only module this change effects but is it
>> > > >> something that we should extend to future modules?
>> > > >>
>> > > >> Best,
>> > > >> Adil
>> > > >>
>> > > >> That method will certainly succeed in importing the ['testseq'] module. The
>> > > >> issue is that once the importing is complete, the doctests are run. And the
>> > > >> doctest also contain an import statement. The doctests cause an error in 2.7
>> > > >> because they are subject to the same importing limitations as
>> > > >> "run_tests.py". I will now describe these limitations below:
>> > > >>
>> > > >> Prior to 3.3, hierarchical importing (with "dot" notation) required a
>> > > >> directory to be initialized with a "__init__.py" file. With the introduction
>> > > >> of 3.3, hierarchical importing was generalized to all directories,
>> > > >> regardless of initialization. This is why my commits were passing all tests
>> > > >> except 2.7.
>> > > >>
>> > > >> Since we can't initialize folders carelessly, its best to remove the non-Bio
>> > > >> modules from the "run_tests.py" file just prior to version 3.3. All versions
>> > > >> of python after-and-including 3.3 don't require an extra "__init__.py" file,
>> > > >> so the doctests will run fine on 3.3, 3.4, 3.5, and 3.6. That's why the most
>> > > >> recent build passed the Travis CI test.
>> > > >>
>> > > >> I agree that my import methods were becoming overly-complicated. In the most
>> > > >> recent build, I've eliminated the complicated code in favor of just deleting
>> > > >> non-Bio files.The only complicated code left is limited to just the unit
>> > > >> test ("test_testseq.py").
>> > > >>
>> > > >> Thankfully, the unit test for 'testseq' runs all of the same tests that are
>> > > >> in the doctests, so we can be confident that 'testseq' is compatible with
>> > > >> all versions of python. In the future, if "biopython" decides to stop
>> > > >> supporting python 2.7, I would be happy to remove the last bit of
>> > > >> complicated code.
>> > > >>
>> > > >> TL:DR - In 2.7 only: Modules outside 'Bio' package can be imported from
>> > > >> "run_tests.py", however, their doctests will likely fail because they must
>> > > >> also contain import statements. It's best to remove them from doctesting in
>> > > >> earlier versions of python and allow the unit tests to confirm
>> > > >> compatibility. (Only for non-Bio files in only 2.7).
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >> _______________________________________________
>> > > >> Biopython-dev mailing list
>> > > >> Biopython-dev at mailman.open-bio.org
>> > > >> http://mailman.open-bio.org/mailman/listinfo/biopython-dev
>> > > _______________________________________________
>> > > Biopython-dev mailing list
>> > > Biopython-dev at mailman.open-bio.org
>> > > http://mailman.open-bio.org/mailman/listinfo/biopython-dev

More information about the Biopython-dev mailing list