Tests too slow, need much faster feedback

[nirs]

Even after #13, tests are too slow.

$ tox -e 2.7 -- py.test -v --durations=10
2.7 develop-inst-nodeps: /home/nsoffer/src/python-manhole
2.7 runtests: PYTHONHASHSEED='4080427460'
2.7 runtests: commands[0] | py.test -v --durations=10
============================================================================= test session starts ==============================================================================
platform linux2 -- Python 2.7.5 -- py-1.4.23 -- pytest-2.6.1 -- /home/nsoffer/src/python-manhole/.tox/2.7/bin/python2.7
plugins: cov, capturelog
collected 14 items

tests/test_manhole.py::test_simple PASSED
tests/test_manhole.py::test_fork_exec PASSED
tests/test_manhole.py::test_socket_path PASSED
tests/test_manhole.py::test_socket_path_with_fork PASSED
tests/test_manhole.py::test_exit_with_grace PASSED
tests/test_manhole.py::test_with_fork PASSED
tests/test_manhole.py::test_with_forkpty PASSED
tests/test_manhole.py::test_auth_fail PASSED
tests/test_manhole.py::test_activate_on_usr2 PASSED
tests/test_manhole.py::test_activate_on_with_oneshot_on PASSED
tests/test_manhole.py::test_oneshot_on_usr2 PASSED
tests/test_manhole.py::test_fail_to_cry PASSED
tests/test_manhole.py::test_oneshot_on_usr2_error PASSED
tests/test_manhole.py::test_interrupt_on_accept PASSED

========================================================================== slowest 10 test durations ===========================================================================
11.11s call tests/test_manhole.py::test_oneshot_on_usr2_error
10.81s call tests/test_manhole.py::test_oneshot_on_usr2
10.70s call tests/test_manhole.py::test_activate_on_usr2
4.39s call tests/test_manhole.py::test_simple
2.62s call tests/test_manhole.py::test_with_fork
2.57s call tests/test_manhole.py::test_with_forkpty
2.56s call tests/test_manhole.py::test_socket_path_with_fork
2.12s call tests/test_manhole.py::test_interrupt_on_accept
1.16s call tests/test_manhole.py::test_fork_exec
0.61s call tests/test_manhole.py::test_exit_with_grace
==================================================================== 14 passed, 2 warnings in 49.98 seconds ====================================================================

[nirs]

Plan:

• Separate functional tests from stress tests
  • functional tests: reconnect, re-install after fork
  • stress tests: reconnect 20 times in a raw, re-install after fork 100 times
• Move stress tests to separate module, or mark tests to they can be skipped by default

[ionelmc]

Ideally we'd just leave the current suite as is (they are integration tests after all) and write a completely separate "unit" suite with mocks and stuff that runs fast. What do you think ?

[ionelmc]

I can help adding a testing skeleton and regrouping tests for this to work and maybe some start tests with mocks/stubs. Are you interested in working on the bulk of the 'fast' suite - further reorganisation etc?

For the separation we could have a "FAST/SLOW" switch via environment variable (so you can still run tox for both). Or we could have 2 sets of tox environments - the default list of envs will be the fast ones.

[nirs]

I don't think we need unit tests and specially not complicated mocking and stubs. This is very simple code with almost no logic, reusing ready components (like the interactive console). I think we need simple integration tests that run quickly and provide fast feedback when you change stuff, and stress tests to catch those rare issues that happen only under load.

I think the simplest way to have both kind of tests would be to mark stress tests using a decorator that skip these tests unless an environment variable is set:

For example:

def stresstest(f):
    def wrapper(*a, **kw):
       if not os.getenv('STRESSTEST'):
           skip('stress tests are disabled')
       return f(*a, **kw)
   return wrapper

@stresstest
def test_reconnect_stress():
    ...

Running stress tests:

STRESSTEST=1 tox -e 2.7

Similar system works nicely for vdsm:
https://github.com/oVirt/vdsm/blob/master/tests/testValidation.py
https://github.com/oVirt/vdsm/blob/master/tests/utilsTests.py#L505

In the vdsm, it need much more code, because we use nose plugins
and the defaults are wrong, you have to disable the stress tests instead
of enable them.

The simple code above is the only code needed to support this, and
it will work with any test framework (assuming test skipping support).

The nice thing about this system is that it is very easy to add tests, and
understand existing tests, since all the tests for particular code are
in the same place.

[Alessandro-Barbieri]

This one tests/test_manhole.py::test_stderr_doesnt_deadlock is super slow