test_mutable.Update takes too long to run #1500

New Issue

warner · 2011-08-23T21:37:09Z

warner commented

2011-08-23 21:37:09 +00:00

The new MDMF tests in test_mutable increase the runtime (on my 2010 MacBookPro laptop) from 24s (in 1.8.2) to 7.5 minutes. Almost all of this extra time is spent in test_mutable.Update. I think a lot of the slowdown is because each test creates four new mutable files (two SDMF, two MDMF), which takes about 26s each, even if they don't end up using the file for anything.

First step is to figure out why we're taking 7 seconds to create a new file: we ought to be using 522-bit keys, but maybe something overrode that. Second step is to trim out the unnecessary file creations.

This isn't necessarily a blocker for 1.9, but it slows down development considerably, so I want to see it fixed soon.

The new MDMF tests in test_mutable increase the runtime (on my 2010 MacBookPro laptop) from 24s (in 1.8.2) to 7.5 minutes. Almost all of this extra time is spent in `test_mutable.Update`. I think a lot of the slowdown is because each test creates four new mutable files (two SDMF, two MDMF), which takes about 26s each, even if they don't end up using the file for anything. First step is to figure out why we're taking 7 seconds to create a new file: we ought to be using 522-bit keys, but maybe something overrode that. Second step is to trim out the unnecessary file creations. This isn't necessarily a blocker for 1.9, but it slows down development considerably, so I want to see it fixed soon.

warner added the

labels 2011-08-23 21:37:09 +00:00

warner added this to the 1.9.0 milestone 2011-08-23 21:37:09 +00:00

kevan commented

2011-08-26 18:50:00 +00:00

Two of the four mutable files are created with 255 shares; this was meant to test the boundary calculations for MDMF caps (though the SDMF file isn't necessary for that). If I comment out lines 3290 and 3291 of test_mutable.py, the runtime on my machine drops from 121 seconds to 21 seconds, suggesting that MDMF or SDMF files take a lot longer to process when created with a large number of shares. I'm not sure if that's a regression relative to 1.8.2; I'm not aware of any existing tests that exercise the code on files with a large number of shares.

Using 255 shares isn't a reliable testing technique unless full-sized keys are also used, and full-sized keys won't be used by default in these tests (which is why they passed when the boundary calculations were wrong). I like the idea of testing files with 255 shares, but I don't think it is worthwhile to add minutes of runtime to the test suite for unreliable tests. Maybe we should add tests for this case in an area of the test suite that runs with larger keys and remove them from test_mutable.Update.

Two of the four mutable files are created with 255 shares; this was meant to test the boundary calculations for MDMF caps (though the SDMF file isn't necessary for that). If I comment out lines 3290 and 3291 of test_mutable.py, the runtime on my machine drops from 121 seconds to 21 seconds, suggesting that MDMF or SDMF files take a lot longer to process when created with a large number of shares. I'm not sure if that's a regression relative to 1.8.2; I'm not aware of any existing tests that exercise the code on files with a large number of shares. Using 255 shares isn't a reliable testing technique unless full-sized keys are also used, and full-sized keys won't be used by default in these tests (which is why they passed when the boundary calculations were wrong). I like the idea of testing files with 255 shares, but I don't think it is worthwhile to add minutes of runtime to the test suite for unreliable tests. Maybe we should add tests for this case in an area of the test suite that runs with larger keys and remove them from `test_mutable.Update`.

Brian Warner <warner@lothar.com> commented

2011-08-29 07:27:51 +00:00

In changeset:980eb778c1f3d40e:

test_mutable.Update: only upload the files needed for each test. refs #1500

This first step shaves 15% off the runtime: from 139s to 119s on my laptop.
It also fixes a couple of places where a Deferred was being dropped, which
would cause two tests to run in parallel and also confuse error reporting.

In changeset:980eb778c1f3d40e: ``` test_mutable.Update: only upload the files needed for each test. refs #1500 This first step shaves 15% off the runtime: from 139s to 119s on my laptop. It also fixes a couple of places where a Deferred was being dropped, which would cause two tests to run in parallel and also confuse error reporting. ```

warner commented

2011-09-05 08:08:43 +00:00

incidentally, test_mutable.MultipleEncodings uses a hard-coded one second delay to get the shares to be returned in the right order, and this probably gets invoked 3 or 4 times during the test, so changing that to use a manual deliver-all-pending-reads trigger (probably invoked after download_best_version() was started but before expecting its Deferred to fire) could reduce the runtime by 3 or 4 seconds.

incidentally, `test_mutable.MultipleEncodings` uses a hard-coded one second delay to get the shares to be returned in the right order, and this probably gets invoked 3 or 4 times during the test, so changing that to use a manual deliver-all-pending-reads trigger (probably invoked after download_best_version() was started but before expecting its Deferred to fire) could reduce the runtime by 3 or 4 seconds.

warner commented

2011-10-13 17:06:14 +00:00

not happening in 1.9

warner modified the milestone from 1.9.0 to 1.10.0

2011-10-13 17:06:14 +00:00

Sign in to join this conversation.