Ratchet based on test module, not individual test #3372

New Issue

chadwhitacre · 2020-08-07T12:38:23Z

chadwhitacre commented

2020-08-07 12:38:23 +00:00

The unit of work for the Python 3 port is the module. Currently we ratchet up successful tests at the individual test level, which means that randomly successful tests in unrelated modules that we don't care about yet distract us from the module currently in focus. At the very least, they are noise during dev and code review. If they start failing again, the developer either has to manually remove them from the "binary-ish" ratchet-passing, which is an annoyance, or fix them, which is a rabbit hole.

In #3369 there is a suggestion to not fail ratchet in CI if unexpected successes are discovered. But, since devs are expected to add expected successes to ratchet before pushing, this still places a burden on devs to then manually edit ratchet-passing and distinguish expected and unexpected successes.

Acceptance

tox -e py36 only runs the tests explicitly listed in allmydata.utils._python3.PORTED_TEST_MODULES.
There's no automated "ratcheting" per se. CI semantics are the same as for any other env, just limited in scope to the specified set of test modules, which is manually ratcheted (as it were).

The unit of work for the Python 3 [port](https://tahoe-lafs.org/trac/tahoe-lafs/wiki/Python3) is the module. Currently we ratchet up successful tests at the individual test level, which means that randomly successful tests in unrelated modules that we don't care about yet distract us from the module currently in focus. At the very least, they are noise during dev and code review. If they start failing again, the developer either has to manually remove them from the "binary-ish" `ratchet-passing`, which is an annoyance, or fix them, which is a rabbit hole. In #3369 there is a suggestion to not fail ratchet in CI if unexpected successes are discovered. But, since devs are expected to add expected successes to ratchet before pushing, this still places a burden on devs to then manually edit `ratchet-passing` and distinguish expected and unexpected successes. ### Acceptance 1. `tox -e py36` only runs the tests explicitly listed in `allmydata.utils._python3.PORTED_TEST_MODULES`. 1. There's no automated "ratcheting" per se. CI semantics are the same as for any other env, just limited in scope to the specified set of test modules, which is manually ratcheted (as it were).

chadwhitacre added the

labels 2020-08-07 12:38:23 +00:00

chadwhitacre added this to the Support Python 3 milestone 2020-08-07 12:38:23 +00:00

GitHub <noreply@github.com> commented

2020-08-07 15:13:01 +00:00

In fc901e96/trunk:

Merge pull request #765 from tahoe-lafs/3372.alphabetize-PORTED_MODULES

Sort the Python 3 ported modules list (as we said we would)

Fixes: ticket:3372

In [fc901e96/trunk](/tahoe-lafs/trac-2024-07-25/commit/fc901e9614029e1a1713993ce543ea139ea7fb53): ``` Merge pull request #765 from tahoe-lafs/3372.alphabetize-PORTED_MODULES Sort the Python 3 ported modules list (as we said we would) Fixes: ticket:3372 ```

tahoe-lafs added the

fixed

label 2020-08-07 15:13:01 +00:00

GitHub <noreply@github.com> closed this issue

2020-08-07 15:13:01 +00:00

chadwhitacre commented

2020-08-11 11:53:08 +00:00

I ended up going a slightly different route. tox -e py36 by itself will fail. I added shtuff to the Travis invocation to constrain the set of tests we exercise under py36 in CI. In local dev one may want to use tox to run tests that haven't been ported yet.

PR: https://github.com/tahoe-lafs/tahoe-lafs/pull/770

I ended up going a slightly different route. `tox -e py36` by itself will fail. I added shtuff to the Travis invocation to constrain the set of tests we exercise under py36 in CI. In local dev one may want to use `tox` to run tests that haven't been ported yet. PR: <https://github.com/tahoe-lafs/tahoe-lafs/pull/770>

exarkun commented

2020-08-13 13:01:00 +00:00

Re-opening. Some unrelated PR marked this as fixed. :/ I can't find the ticket that the sorted-ported-modules PR was supposed to fix ...

exarkun removed the

fixed

label 2020-08-13 13:01:00 +00:00

exarkun reopened this issue

2020-08-13 13:01:00 +00:00

GitHub <noreply@github.com> commented

2020-08-14 15:55:35 +00:00

In bc78797/trunk:

Merge pull request #777 from tahoe-lafs/3372.ratchet-by-module.python3.6

Ratchet by module, not by individual test (in python3.6)

Fixes: ticket:3372

In [bc78797/trunk](/tahoe-lafs/trac-2024-07-25/commit/bc787975da1e5ac7bfe2453720589d57bac1c768): ``` Merge pull request #777 from tahoe-lafs/3372.ratchet-by-module.python3.6 Ratchet by module, not by individual test (in python3.6) Fixes: ticket:3372 ```

tahoe-lafs added the

fixed

label 2020-08-14 15:55:35 +00:00

GitHub <noreply@github.com> closed this issue

2020-08-14 15:55:35 +00:00

Sign in to join this conversation.