add docs/tests.rst documenting how to run tests and how to interpret the output #1439

New Issue

tahoe-lafs · 2011-07-23T00:30:38Z

davidsarah commented

2011-07-23 00:30:38 +00:00

In /tahoe-lafs/trac-2024-07-25/issues/5207#comment:26, Zooko wrote about various ways to run the Tahoe-LAFS test suite, some of which work and some of which do not. He suggested adding a docs/tests.rst to document this.

tests.rst should also document cases in which the tests are known to fail (such as when we're not testing the right code), how to interpret skips, todos, and other sources of noise, and how to get Twisted log output from test cases.

In [/tahoe-lafs/trac-2024-07-25/issues/5207](/tahoe-lafs/trac-2024-07-25/issues/5207)#comment:26, Zooko wrote about various ways to run the Tahoe-LAFS test suite, some of which work and some of which do not. He suggested adding a `docs/tests.rst` to document this. tests.rst should also document cases in which the tests are known to fail (such as when we're not testing the right code), how to interpret skips, todos, and other sources of noise, and how to get Twisted log output from test cases.

tahoe-lafs added the

labels 2011-07-23 00:30:38 +00:00

tahoe-lafs added this to the eventually milestone 2011-07-23 00:30:38 +00:00

zooko commented

2014-09-23 22:20:17 +00:00

See also [//pipermail/tahoe-dev/2014-August/009152.html Nathan's comments about "How To Run Tests" during a Nuts & Bolts meeting].

warner commented

2014-09-23 23:07:34 +00:00

I'm picky about making tests run quickly, so I've identified three ways to run them, with various levels of nuisance and performance:

1: python setup.py test [-s allmydata.test.test_FOO]
2: ./bin/tahoe debug trial allmydata.test.test_FOO
3: PYTHONPATH=support/lib/pythonX.Y/site-packages trial allmydata.test.test_FOO

The last is the fastest, but you have to have twisted/trial already installed (or add more to $PATH/$PYTHONPATH), tests which involve searching $PATH won't work (or add more to $PATH), and you have to know what minor version of python you're running (to get the support/ directory right).

2 is easier, and only slightly slower: the bin/tahoe wrapper performs a fork+exec to re-run the entry point with the right $PATH/$PYTHONPATH

1 is how the test automation does it, because it's not particularly tahoe-specific. It's even slower because it runs a second subprocess.call to invoke the second form. More importantly, it actually runs setup.py build first (because of the aliases in setup.cfg), which can spend several seconds looking at your support/ directory and deciding whether to install dependencies or not.

And if you use python setup.py test --coverage, then you get one subprocess.call to invoke bin/tahoe @coverage, a second one inside the bin/tahoe wrapper to run the real bin/tahoe, and third one in bin/tahoe to run coverage. But for large tests, that's usually small compared to the overhead induced by the coverage tracing function.

I'm hoping that our #2255/#2077 overhauls can remove these extra calls.

I'm picky about making tests run quickly, so I've identified three ways to run them, with various levels of nuisance and performance: * 1: `python setup.py test [-s allmydata.test.test_FOO]` * 2: `./bin/tahoe debug trial allmydata.test.test_FOO` * 3: `PYTHONPATH=support/lib/pythonX.Y/site-packages trial allmydata.test.test_FOO` The last is the fastest, but you have to have twisted/trial already installed (or add more to $PATH/$PYTHONPATH), tests which involve searching $PATH won't work (or add more to $PATH), and you have to know what minor version of python you're running (to get the support/ directory right). 2 is easier, and only slightly slower: the `bin/tahoe` wrapper performs a fork+exec to re-run the entry point with the right $PATH/$PYTHONPATH 1 is how the test automation does it, because it's not particularly tahoe-specific. It's even slower because it runs a second subprocess.call to invoke the second form. More importantly, it actually runs `setup.py build` first (because of the aliases in `setup.cfg`), which can spend several seconds looking at your support/ directory and deciding whether to install dependencies or not. And if you use `python setup.py test --coverage`, then you get one subprocess.call to invoke `bin/tahoe @coverage`, a second one inside the `bin/tahoe` wrapper to run the real bin/tahoe, and third one in bin/tahoe to run `coverage`. But for large tests, that's usually small compared to the overhead induced by the coverage tracing function. I'm hoping that our #2255/#2077 overhauls can remove these extra calls.

Sign in to join this conversation.