Run tests in parallel #2845

New Issue

tahoe-lafs · 2016-11-29T14:10:29Z

rvs commented

2016-11-29 14:10:29 +00:00

Currently tox takes quite a lot of time, even on beefy machine with SSD & many CPUs.

Looks like xdist plugin can run tests in paralel which could speed things up quite a bit.

See e.g. https://qxf2.com/blog/xdist-run-tests-parallel-pytest/

Currently `tox` takes quite a lot of time, even on beefy machine with SSD & many CPUs. Looks like xdist plugin can run tests in paralel which could speed things up quite a bit. See e.g. <https://qxf2.com/blog/xdist-run-tests-parallel-pytest/>

tahoe-lafs added the

labels 2016-11-29 14:10:29 +00:00

tahoe-lafs added this to the soon milestone 2016-11-29 14:10:29 +00:00

tahoe-lafs added

code

and removed

unknown

labels 2016-12-23 13:46:30 +00:00

tahoe-lafs changed title from ~~Run tests in parrallel~~ to Run tests in parallel

2017-01-10 17:38:13 +00:00

rvs commented

2017-01-12 23:43:08 +00:00

(https://github.com/tahoe-lafs/tahoe-lafs/pull/395)

warner commented

2017-01-13 00:04:06 +00:00

Huh.. I had no idea that trial could parallelize like that. Let's see how well travis handles the PR, but then yeah, let's land that.

Any idea if we could use this on tox -e coverage too? I don't know how the .coverage files are written, or if the multiple workers would clobber each other's data.

Huh.. I had no idea that trial could parallelize like that. Let's see how well travis handles the PR, but then yeah, let's land that. Any idea if we could use this on `tox -e coverage` too? I don't know how the `.coverage` files are written, or if the multiple workers would clobber each other's data.

warner commented

2017-01-13 02:46:41 +00:00

The github PR is showing failures on the integration tests: I think they're more sensitive to concurrency, so we probably can't enable it for those.

Also it looks like trial -j causes _trial_temp/test.log to be split into several pieces (one per worker, in files like _trial_temp/0/test.log and _trial_temp/1/test.log). I believe our buildbot reads from test.log to count various things, so we may need to rewrite some tooling to handle that. But the speedup is so impressive, I think it's worth the effort.

The github PR is showing failures on the integration tests: I think they're more sensitive to concurrency, so we probably can't enable it for those. Also it looks like `trial -j` causes `_trial_temp/test.log` to be split into several pieces (one per worker, in files like `_trial_temp/0/test.log` and `_trial_temp/1/test.log`). I believe our buildbot reads from `test.log` to count various things, so we may need to rewrite some tooling to handle that. But the speedup is so impressive, I think it's worth the effort.

exarkun commented

2017-08-07 18:29:16 +00:00

Sorry, https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2894 was a duplicate of this.

Sorry, <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2894> was a duplicate of this.

exarkun added the

fixed

label 2017-08-07 18:29:16 +00:00

exarkun closed this issue

2017-08-07 18:29:16 +00:00

Sign in to join this conversation.