Use CircleCI's "parallelism" feature to speed up runs of the integration test suite #4054

Open
opened 2023-08-01 14:02:05 +00:00 by exarkun · 0 comments

CircleCI has a feature where you can tell it to run several executors concurrently for a single job. It tells each executor how many concurrent executors it is running and what its unique index in that sequence is. Then it is up to the job to pick a slice of work which, when combined with the choice made on all the other jobs, combine to represent the full test suite run.

For a test suite (as opposed to something like the compilation of a large C program), the "slice of work" can be some subset of the tests in the test suite.

There are a few possible approaches for convincing pytest to run such a subset. If we combine one of these with the CircleCI feature then we can leverage more CPU power to (possibly) complete the integration tests more quickly.

CircleCI has a feature where you can tell it to run several executors concurrently for a single job. It tells each executor how many concurrent executors it is running and what its unique index in that sequence is. Then it is up to the job to pick a slice of work which, when combined with the choice made on all the other jobs, combine to represent the full test suite run. For a test suite (as opposed to something like the compilation of a large C program), the "slice of work" can be some subset of the tests in the test suite. There are a few possible approaches for convincing pytest to run such a subset. If we combine one of these with the CircleCI feature then we can leverage more CPU power to (possibly) complete the integration tests more quickly.
exarkun added the
unknown
normal
defect
n/a
labels 2023-08-01 14:02:05 +00:00
exarkun added this to the undecided milestone 2023-08-01 14:02:05 +00:00
Sign in to join this conversation.
No Milestone
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Reference: tahoe-lafs/trac-2024-07-25#4054
No description provided.