tahoe-lafs
/
trac-2024-07-25
2
0
Issues 1.1k Wiki

Decide where "packaging tests" should live. #2049

New Issue
Open
opened 2013-08-07 19:37:33 +00:00 by nejucomo · 30 comments
nejucomo commented 2013-08-07 19:37:33 +00:00
Copy Link

Task: Define a policy about where packaging tests live, where buildbot configuration lives, then specify that policy on the wiki.

Details:

We'd benefit from "packaging tests" which are different from unittests. The latter target a prepared installation or repository checkout, whereas packaging tests have these distinctions:

Inputs: Packaging tests may target as input:

  • build tests - a fresh repository checkout in order to test creation of distribution formats -- e.g. sdist, linux distro packages, sumo tarballs, etc...
  • installation tests - a locally prepared distribution format -- e.g. sdist, .deb, etc...
  • distribution tests - the network -- e.g. a PyPI package name and http(s) connections to python packaging mirrors; a debian package name and connections to debian repositories; etc...

Staging or Production: We might like to run the same tests against unreleased versions and distinctly against "the live internet".

  • Staging tests might rely on a local PyPI mirror to ensure that if we were to distribute sdists to the real PyPI, then users will probably successfully be able to install them.
  • Production tests rely on the actual distribution that live users use. Failures of these tests let us know when our released packages are unavailable to users, and can help identify why.

Brittleness: sensitivity to transient conditions:

  • Packaging tests start from either a repository clone or an sdist and should behave fairly deterministically without networking.
  • Installation tests start from local packaging files, and should behave deterministically, barring system state issues.
  • Distributions tests necessarily use the internet and will give false positives for transient errors. (Note: Errors due to transient outages are important information! Suppose tahoe-lafs install relies on some.third-party.package.host.example.com and that site goes down. We want to know that people are now unable to install tahoe-lafs through specific channels.)

We may want different packaging tests to live in different places depending on these axes. Some of these tests may not require buildbot, and/or may not be developed or maintained by us at all! For example, tests which verify live PyPI packages may be run by PyPI itself. We should still account for this in an "official policy".

We probably want test infrastructure operators to have some flexibility to only run some categories of tests. For example, a debian test box would not want to run windows installation tests, or some operators may prefer to limit networking and only run non-distribution tests.

**Task:** Define a policy about where packaging tests live, where buildbot configuration lives, then specify that policy on the wiki. **Details:** We'd benefit from "packaging tests" which are different from unittests. The latter target a prepared installation or repository checkout, whereas packaging tests have these distinctions: **Inputs:** Packaging tests may target as input: * *build tests* - a fresh repository checkout in order to test creation of distribution formats -- *e.g.* `sdist`, `linux distro packages`, sumo tarballs, etc... * *installation tests* - a locally prepared distribution format -- *e.g.* `sdist`, `.deb`, etc... * *distribution tests* - the network -- *e.g.* a PyPI package name and `http(s)` connections to python packaging mirrors; a debian package name and connections to debian repositories; etc... **Staging or Production**: We might like to run the same tests against unreleased versions and distinctly against "the live internet". * *Staging tests* might rely on a local PyPI mirror to ensure that *if we were* to distribute sdists to the real PyPI, *then* users will probably successfully be able to install them. * *Production tests* rely on the actual distribution that live users use. Failures of these tests let us know when our released packages are unavailable to users, and can help identify why. **Brittleness**: sensitivity to transient conditions: * Packaging tests start from either a repository clone or an sdist and should behave fairly deterministically without networking. * Installation tests start from local packaging files, and should behave deterministically, barring system state issues. * Distributions tests necessarily use the internet and will give false positives for transient errors. (*Note:* Errors due to transient outages are important information! Suppose `tahoe-lafs` install relies on `some.third-party.package.host.example.com` and that site goes down. We want to know that people are now unable to install `tahoe-lafs` through specific channels.) We may want different packaging tests to live in different places depending on these axes. Some of these tests may not require buildbot, and/or may not be developed or maintained by us at all! For example, tests which verify live PyPI packages may be run by PyPI itself. We should still account for this in an "official policy". We probably want test infrastructure operators to have some flexibility to only run some categories of tests. For example, a debian test box would not want to run windows installation tests, or some operators may prefer to limit networking and only run non-distribution tests.
nejucomo added the
unknown
normal
task
1.10.0
 labels 2013-08-07 19:37:33 +00:00
nejucomo added this to the undecided milestone 2013-08-07 19:37:33 +00:00
nejucomo commented 2013-08-07 19:41:29 +00:00
Author
Copy Link

Note: We might have some tests that are abstracted over the target package. For example a distribution test which takes as input a PyPI package named $PKG:

  1. Create a new pristine virtualenv; activate it.
  2. $ pip install $pkg
  3. Fail if the installation fails.
  4. Fail if http connections were attempted without ssl.
  5. Run the installed packages unit tests; fail if they fail.
  6. Otherwise, pass.

If this were adopted as a packaging goal/specification, we could apply it to allmydata-tahoe, zfec, pyutil, pycryptopp, etc...

Note: We might have some tests that are abstracted over the target package. For example a distribution test which takes as input a PyPI package named $PKG: 1. Create a new pristine virtualenv; activate it. 1. $ pip install $pkg 1. Fail if the installation fails. 1. Fail if `http` connections were attempted without `ssl`. 1. Run the installed packages unit tests; fail if they fail. 1. Otherwise, pass. If this were adopted as a packaging goal/specification, we could apply it to `allmydata-tahoe`, `zfec`, `pyutil`, `pycryptopp`, etc...
nejucomo commented 2013-08-07 19:43:22 +00:00
Author
Copy Link

Zooko points out that if the tests live in the main tahoe-lafs repository, then when they are invoked they may incorrectly import the local unpackaged code, or make other dependency mistakes.

Zooko points out that if the tests live in the main `tahoe-lafs` repository, then when they are invoked they may incorrectly import the local unpackaged code, or make other dependency mistakes.
daira commented 2013-08-08 00:22:30 +00:00
Owner
Copy Link

I'm strongly in favour of having these tests in a separate repo on github (and moving any existing such tests into that repo).

I'm strongly in favour of having these tests in a separate repo on github (and moving any existing such tests into that repo).
daira commented 2013-08-18 20:55:02 +00:00
Owner
Copy Link

(https://github.com/tahoe-lafs/packaging-tests) ?

(https://github.com/tahoe-lafs/packaging-tests) ?
tahoe-lafs added
dev-infrastructure
 and removed
unknown
 labels 2013-08-18 20:56:39 +00:00
tahoe-lafs modified the milestone from undecided to soon (release n/a) 2013-08-18 20:56:39 +00:00
zooko commented 2013-08-21 03:11:58 +00:00
Copy Link

Replying to daira:

https://github.com/tahoe-lafs/packaging-tests ?

Yes, please!

As far as I know, only Brian has the authority to create new repos under https://github.com/tahoe-lafs.

(Well, only Brian, all the people who have root at github.com, and DigiCert, and all of the root CAs, and people who have owned root CAs, and the makers of web browsers. I actually hope to fix this before colonizing outer space.)

Replying to [daira](/tahoe-lafs/trac-2024-07-25/issues/2049#issuecomment-93005): > <https://github.com/tahoe-lafs/packaging-tests> ? Yes, please! As far as I know, only Brian has the authority to create new repos under <https://github.com/tahoe-lafs>. (Well, only Brian, all the people who have root at github.com, and DigiCert, and all of the root CAs, and people who have owned root CAs, and the makers of web browsers. I actually hope to fix this *before* colonizing outer space.)
zooko commented 2013-08-21 03:22:29 +00:00
Copy Link

Replying to daira:

I'm strongly in favour of having these tests in a separate repo on github (and moving any existing such tests into that repo).

Any existing such tests:

Replying to [daira](/tahoe-lafs/trac-2024-07-25/issues/2049#issuecomment-93004): > I'm strongly in favour of having these tests in a separate repo on github (and moving any existing such tests into that repo). Any existing such tests: * [TestOldDep](https://github.com/zooko/buildbot-config-tahoe/blame/027e2fe1646863df46c15977cb3ed9badde100bf/bbsupport.py#l123) * [TestAlreadyHaveDep](https://github.com/zooko/buildbot-config-tahoe/blame/027e2fe1646863df46c15977cb3ed9badde100bf/bbsupport.py#l136) * [InstallToEgg](https://github.com/zooko/buildbot-config-tahoe/blame/027e2fe1646863df46c15977cb3ed9badde100bf/bbsupport.py#l440) * [InstallToPrefixDir](https://github.com/zooko/buildbot-config-tahoe/blame/027e2fe1646863df46c15977cb3ed9badde100bf/bbsupport.py#l461) * [TestFromEggTrial](https://github.com/zooko/buildbot-config-tahoe/blame/027e2fe1646863df46c15977cb3ed9badde100bf/bbsupport.py#l477) * [TestFromEgg](https://github.com/zooko/buildbot-config-tahoe/blame/027e2fe1646863df46c15977cb3ed9badde100bf/bbsupport.py#l518) * [TestFromPrefixDirPyunit](https://github.com/zooko/buildbot-config-tahoe/blame/027e2fe1646863df46c15977cb3ed9badde100bf/bbsupport.py#l561) * [TestFromPrefixDirRunTrial](https://github.com/zooko/buildbot-config-tahoe/blame/027e2fe1646863df46c15977cb3ed9badde100bf/bbsupport.py#l598)
zooko commented 2013-08-27 12:02:08 +00:00
Copy Link

Brian: would you please create a repo named https://github.com/tahoe-lafs/packaging-tests ?

Brian: would you please create a repo named <https://github.com/tahoe-lafs/packaging-tests> ?
daira commented 2013-08-27 12:05:34 +00:00
Owner
Copy Link

Replying to [zooko]comment:6:

[...]
(Well, only Brian, all the people who have root at github.com, and DigiCert, and all of the root CAs, and people who have owned root CAs, and the makers of web browsers. I actually hope to fix this before colonizing outer space.)

Hmm, I'm not sure that should be a blocker for our interstellar exploration plans ;-)

Replying to [zooko]comment:6: > [...] > (Well, only Brian, all the people who have root at github.com, and DigiCert, and all of the root CAs, and people who have owned root CAs, and the makers of web browsers. I actually hope to fix this *before* colonizing outer space.) Hmm, I'm not sure that should be a blocker for our interstellar exploration plans ;-)
zooko commented 2013-08-27 12:13:41 +00:00
Copy Link

Oh look there was an old duplicate of this: #1248, which has some relevant detail in it. Please check it out. I'm closing #1248 as a duplicate of this.

Oh look there was an old duplicate of this: #1248, which has some relevant detail in it. Please check it out. I'm closing #1248 as a duplicate of this.
warner commented 2013-08-27 17:39:59 +00:00
Copy Link

I'm -0. I'll create that repo if you want, but first I'd like to push back on the notion that these tests ought to live outside of the main source tree.

Zooko points out that if the tests live in the main tahoe-lafs repository, then when they are invoked they may incorrectly import the local unpackaged code, or make other dependency mistakes.

I think it's easy to avoid that. Our bin/tahoe looks in its parent directory for the marker file that indicates it should include ../src and ../support, etc. But if we unpack the sdist into a subdirectory, and run the new bin/tahoe, it won't look in the original tree for anything.

(One counter-example would be if something in the build process wants to know if it's in a git checkout, for which git walks all the way to the filesystem root looking for a .git directory. I don't think that's a big deal.)

More repositories means more places for contributors to look, more divergence between code and tests. Also it raises the buildbot question of which repo's changes should trigger the tests being run (presumably you'd want to trigger on changes in either source-repo or tests-repo, and that's not trivial with buildbot).

My weak suggestion would be to put all of these "packaging tests" into a subdirectory of the main tree, and have the buildbot (or whatever) check out trunk, then run one of them. The packaging tests should not feel obligated to use the rest of the source code for anything, but if the test happens to need a trunk checkout, one is nearby. I could be talked out of this, though.

I'm -0. I'll create that repo if you want, but first I'd like to push back on the notion that these tests ought to live outside of the main source tree. > Zooko points out that if the tests live in the main tahoe-lafs repository, then when they are invoked they may incorrectly import the local unpackaged code, or make other dependency mistakes. I think it's easy to avoid that. Our `bin/tahoe` looks in its parent directory for the marker file that indicates it should include `../src` and `../support`, etc. But if we unpack the sdist into a subdirectory, and run the new `bin/tahoe`, it won't look in the original tree for anything. (One counter-example would be if something in the build process wants to know if it's in a git checkout, for which `git` walks all the way to the filesystem root looking for a `.git` directory. I don't think that's a big deal.) More repositories means more places for contributors to look, more divergence between code and tests. Also it raises the buildbot question of which repo's changes should trigger the tests being run (presumably you'd want to trigger on changes in either source-repo or tests-repo, and that's not trivial with buildbot). My weak suggestion would be to put all of these "packaging tests" into a subdirectory of the main tree, and have the buildbot (or whatever) check out trunk, then run one of them. The packaging tests should not feel obligated to use the rest of the source code for anything, but if the test happens to need a trunk checkout, one is nearby. I could be talked out of this, though.
zooko commented 2013-08-27 18:05:31 +00:00
Copy Link

Well, I still feel like it is better to maintain the packaging tests and the Tahoe-LAFS code separately. I don't foresee it being a problem that there is "divergence" between the packaging tests and the Tahoe-LAFS code. Those two things probably oughtn't be changed in tight synchronization with each other anyway!

I've had problems in the past with packaging code accidentally importing the wrong version of a package, and it was very confusing, so I feel nervous about the packaging tests coming with a copy of the code-under-test in a sibling directory. I think it could cause trouble.

I think it's easy to avoid that. Our bin/tahoe looks in its parent directory for the marker file that indicates it should include ../src and ../support, etc. But if we unpack the sdist into a subdirectory, and run the new bin/tahoe, it won't look in the original tree for anything.

I don't think this is easy! I find this stuff very confusing. You can't rely on setting PYTHONPATH, by the time you can edit sys.path it is sometimes too late, you can't rely on editing sys.modules (although I occasionally have to do that). Having your Python code crawl around in your file system and look for magic files in sibling directories, like we do in bin/tahoe is icky and non-standard, even though it has worked so far bin/tahoe's purposes.

In short, if I want some code (the packaging tests) to have control over what specific copy of some other code (the code-under-test) it starts with, then my preferred way of achieving that is:

  1. The test code starts with no extant copies of any version of the other code present on its filesystem.

  2. The test code fetches a specifically chosen version, by tarball-download of a specific URL, or by git fetch of a specific branch, tag, or commit-hash.

Well, I still feel like it is better to maintain the packaging tests and the Tahoe-LAFS code separately. I don't foresee it being a problem that there is "divergence" between the packaging tests and the Tahoe-LAFS code. Those two things probably oughtn't be changed in tight synchronization with each other anyway! I've had problems in the past with packaging code accidentally importing the wrong version of a package, and it was very confusing, so I feel nervous about the packaging tests coming with a copy of the code-under-test in a sibling directory. I think it could cause trouble. > I think it's easy to avoid that. Our bin/tahoe looks in its parent directory for the marker file that indicates it should include ../src and ../support, etc. But if we unpack the sdist into a subdirectory, and run the new bin/tahoe, it won't look in the original tree for anything. I don't think this is easy! I find this stuff very confusing. You can't rely on setting `PYTHONPATH`, by the time you can edit `sys.path` it is sometimes too late, you can't rely on editing `sys.modules` (although I occasionally have to do that). Having your Python code crawl around in your file system and look for magic files in sibling directories, like we do in `bin/tahoe` is icky and non-standard, even though it has worked so far `bin/tahoe`'s purposes. In short, if I want some code (the packaging tests) to have control over what specific copy of some other code (the code-under-test) it starts with, then my preferred way of achieving that is: 1. The test code starts with no extant copies of any version of the other code present on its filesystem. 2. The test code fetches a specifically chosen version, by tarball-download of a specific URL, or by git fetch of a specific branch, tag, or commit-hash.
warner commented 2013-08-27 18:41:55 +00:00
Copy Link

Replying to zooko:

I'm closing #1248 as a duplicate of this.

Ok, but then let's please have "move test code out of buildbot's master.cfg and into some repo" be part of this (#2049) ticket. That was the point of #1248, and it's not clear thaht it was included in the remit for #2049.

Replying to [zooko](/tahoe-lafs/trac-2024-07-25/issues/2049#issuecomment-93013): > I'm closing #1248 as a duplicate of this. Ok, but then let's please have "move test code out of buildbot's master.cfg and into some repo" be part of this (#2049) ticket. That was the point of #1248, and it's not clear thaht it was included in the remit for #2049.
warner commented 2013-08-27 18:46:32 +00:00
Copy Link

Eh, ok. Repo created at https://github.com/tahoe-lafs/packaging-tests . I+zooko+daira have commit.

Eh, ok. Repo created at <https://github.com/tahoe-lafs/packaging-tests> . I+zooko+daira have commit.
nejucomo commented 2013-10-23 16:09:12 +00:00
Author
Copy Link

We chatted about next steps in a weekly dev chat just now. I propose the next steps are:

  1. ☑ Review and merge https://github.com/tahoe-lafs/buildbot-config-tahoe/pulls
  2. ☐ Create a local master/slave setup to test changes.
  3. ☐ Create a test runner framework in https://github.com/tahoe-lafs/packaging-tests
  4. ☐ Ensure that the new test framework executes the same tests with the same output reported as the existing production buildbot deployment.
  5. ☐ Swap in this new configuration into the existing build-bot network (-and carefully consider contingency downgrade)
  6. ☐ Verify that the new buildbot operation is functionally equivalent to the current operation
  7. ☐ Close this ticket as resolved

After that, we can iterate on packaging-tests to try to nail down installation issues, using separate tickets.

Edit: Added checkboxes for the tasks and marked the first as complete.

Edit 2014-09-25: Changed a step from "
Move the Tahoe ./misc/build_helpers tests into packaging-tests" to be about non-regression criterion.

Edit 2014-09-25: Inserted a new step: "Create a local master/slave setup to test changes." See comment 25#comment:-1.

We chatted about next steps in a weekly dev chat just now. I propose the next steps are: 1. ☑ Review and merge <https://github.com/tahoe-lafs/buildbot-config-tahoe/pulls> 1. ☐ Create a local master/slave setup to test changes. 1. ☐ Create a test runner framework in <https://github.com/tahoe-lafs/packaging-tests> 1. ☐ Ensure that the new test framework executes the same tests with the same output reported as the existing production buildbot deployment. 1. ☐ Swap in this new configuration into the existing build-bot network (-and carefully consider contingency downgrade) 1. ☐ Verify that the new buildbot operation is functionally equivalent to the current operation 1. ☐ Close this ticket as resolved After that, we can iterate on ``packaging-tests`` to try to nail down installation issues, using separate tickets. **Edit:** Added checkboxes for the tasks and marked the first as complete. **Edit 2014-09-25:** Changed a step from " Move the Tahoe ``./misc/build_helpers`` tests into ``packaging-tests``" to be about non-regression criterion. **Edit 2014-09-25:** Inserted a new step: "Create a local master/slave setup to test changes." See [comment 25](/tahoe-lafs/trac-2024-07-25/issues/7079)#[comment:-1](/tahoe-lafs/trac-2024-07-25/issues/2049#issuecomment--1).
zooko commented 2014-07-08 17:13:46 +00:00
Copy Link

We've done step 1 from comment:93018.

We've done step 1 from [comment:93018](/tahoe-lafs/trac-2024-07-25/issues/2049#issuecomment-93018).
warner was unassigned by zooko 2014-07-08 17:13:46 +00:00
nejucomo was assigned by zooko 2014-07-08 17:13:46 +00:00
nejucomo commented 2014-09-25 16:24:19 +00:00
Author
Copy Link

Note: I removed the checkboxes in the quotes to avoid state confusion. (I intended to update comment 15 to reflect new changes in addition to adding comments.)

Replying to nejucomo:

We chatted about next steps in a weekly dev chat just now. I propose the next steps are:

  1. Review and merge https://github.com/tahoe-lafs/buildbot-config-tahoe/pulls
  2. Create a test runner framework in https://github.com/tahoe-lafs/packaging-tests

I propose we use twisted trial as "the framework" and run tests with trial $OPTIONS packaging_tests.

  1. Move the Tahoe ./misc/build_helpers tests into packaging-tests

Oh, I should look at these before selecting the framework. Hopefully these are easy to hook into trial tests.

  1. Swap in this new configuration into the existing build-bot network (-and carefully consider contingency downgrade)
  2. Verify that the new buildbot operation is functionally equivalent to the current operation
  3. Close this ticket as resolved

After that, we can iterate on packaging-tests to try to nail down installation issues, using separate tickets.

Edit: Added checkboxes for the tasks and marked the first as complete.

So right now I'm going to examine ./misc/build_helpers and determine if they'd fit into a trial design.

**Note:** I removed the checkboxes in the quotes to avoid state confusion. (I intended to update comment 15 to reflect new changes in addition to adding comments.) Replying to [nejucomo](/tahoe-lafs/trac-2024-07-25/issues/2049#issuecomment-93018): > We chatted about next steps in a weekly dev chat just now. I propose the next steps are: > > 1. Review and merge <https://github.com/tahoe-lafs/buildbot-config-tahoe/pulls> > 2. Create a test runner framework in <https://github.com/tahoe-lafs/packaging-tests> I propose we use twisted trial as "the framework" and run tests with ``trial $OPTIONS packaging_tests``. > 3. Move the Tahoe ``./misc/build_helpers`` tests into ``packaging-tests`` Oh, I should look at these before selecting the framework. Hopefully these are easy to hook into trial tests. > 4. Swap in this new configuration into the existing build-bot network (-and carefully consider contingency downgrade) > 5. Verify that the new buildbot operation is functionally equivalent to the current operation > 6. Close this ticket as resolved > > After that, we can iterate on ``packaging-tests`` to try to nail down installation issues, using separate tickets. > > **Edit:** Added checkboxes for the tasks and marked the first as complete. So right now I'm going to examine ``./misc/build_helpers`` and determine if they'd fit into a trial design.
zancas commented 2014-09-25 17:58:41 +00:00
Copy Link

Should nejucomo be given commit access to:

https://github.com/tahoe-lafs/packaging-tests

and/or:

https://github.com/tahoe-lafs/buildbot-config-tahoe

?

Should nejucomo be given commit access to: <https://github.com/tahoe-lafs/packaging-tests> and/or: <https://github.com/tahoe-lafs/buildbot-config-tahoe> ?
daira commented 2014-09-25 18:12:05 +00:00
Owner
Copy Link

Probably. I don't have admin access to those either.

Probably. I don't have admin access to those either.
zancas commented 2014-09-25 18:13:45 +00:00
Copy Link

Replying to nejucomo:

Task: Define a policy about where packaging tests live, where buildbot configuration lives, then specify that policy on the wiki.

Details:

We'd benefit from "packaging tests" which are different from unittests. The latter target a prepared installation or repository checkout, whereas packaging tests have these distinctions:

Inputs: Packaging tests may target as input:

  • build tests - a fresh repository checkout in order to test creation of distribution formats -- e.g. sdist, linux distro packages, sumo tarballs, etc...
  • installation tests - a locally prepared distribution format -- e.g. sdist, .deb, etc...
  • distribution tests - the network -- e.g. a PyPI package name and http(s) connections to python packaging mirrors; a debian package name and connections to debian repositories; etc...

Staging or Production: We might like to run the same tests against unreleased versions and distinctly against "the live internet".

  • Staging tests might rely on a local PyPI mirror to ensure that if we were to distribute sdists to the real PyPI, then users will probably successfully be able to install them.
  • Production tests rely on the actual distribution that live users use. Failures of these tests let us know when our released packages are unavailable to users, and can help identify why.

Brittleness: sensitivity to transient conditions:

  • Packaging tests start from either a repository clone or an sdist and should behave fairly deterministically without networking.
  • Installation tests start from local packaging files, and should behave deterministically, barring system state issues.
  • Distributions tests necessarily use the internet and will give false positives for transient errors. (Note: Errors due to transient outages are important information! Suppose tahoe-lafs install relies on some.third-party.package.host.example.com and that site goes down. We want to know that people are now unable to install tahoe-lafs through specific channels.)

We may want different packaging tests to live in different places depending on these axes. Some of these tests may not require buildbot, and/or may not be developed or maintained by us at all! For example, tests which verify live PyPI packages may be run by PyPI itself. We should still account for this in an "official policy".

We probably want test infrastructure operators to have some flexibility to only run some categories of tests. For example, a debian test box would not want to run windows installation tests, or some operators may prefer to limit networking and only run non-distribution tests.

Here're some tweaks to the test type taxonomy outlined above.

  1. I think the name "network distribution tests" is easier to understand, perhaps because it's more specific, than "distribution tests". I believe this because when I added the "network " before instances of "distribution test" I found the text easier to understand.

  2. If I Understand Correctly (IIUC) the word "Packaging" which is the first word after the first bullet point in the "Brittleness" section is a substitution error, and should be the word "Build".

  3. It seems to me that "Brittleness" is a criteria, like "Inputs", by which one can categorize test "packaging test" types, whereas "Staging or Production" is a discriminator that applies more agnostically to all tests. Therefore I propose that the description will have a more intuitive flow if the order of the sections: "Brittleness" and "Staging or Production" are swapped. Also it might be nice to provide some other indicator that the "Staging or Production" section is not quite the same kind of section as the other two, e.g. an interposed sentence, or making both "Inputs" and "Brittleness" be subsections of a "Test Type" section.

  4. I propose that we replace the term "Brittleness" with the word "Robustness".

Replying to [nejucomo](/tahoe-lafs/trac-2024-07-25/issues/7079): > **Task:** Define a policy about where packaging tests live, where buildbot configuration lives, then specify that policy on the wiki. > > **Details:** > > We'd benefit from "packaging tests" which are different from unittests. The latter target a prepared installation or repository checkout, whereas packaging tests have these distinctions: > > **Inputs:** Packaging tests may target as input: > > * *build tests* - a fresh repository checkout in order to test creation of distribution formats -- *e.g.* `sdist`, `linux distro packages`, sumo tarballs, etc... > * *installation tests* - a locally prepared distribution format -- *e.g.* `sdist`, `.deb`, etc... > * *distribution tests* - the network -- *e.g.* a PyPI package name and `http(s)` connections to python packaging mirrors; a debian package name and connections to debian repositories; etc... > > **Staging or Production**: We might like to run the same tests against unreleased versions and distinctly against "the live internet". > > * *Staging tests* might rely on a local PyPI mirror to ensure that *if we were* to distribute sdists to the real PyPI, *then* users will probably successfully be able to install them. > * *Production tests* rely on the actual distribution that live users use. Failures of these tests let us know when our released packages are unavailable to users, and can help identify why. > > **Brittleness**: sensitivity to transient conditions: > > * Packaging tests start from either a repository clone or an sdist and should behave fairly deterministically without networking. > * Installation tests start from local packaging files, and should behave deterministically, barring system state issues. > * Distributions tests necessarily use the internet and will give false positives for transient errors. (*Note:* Errors due to transient outages are important information! Suppose `tahoe-lafs` install relies on `some.third-party.package.host.example.com` and that site goes down. We want to know that people are now unable to install `tahoe-lafs` through specific channels.) > > > We may want different packaging tests to live in different places depending on these axes. Some of these tests may not require buildbot, and/or may not be developed or maintained by us at all! For example, tests which verify live PyPI packages may be run by PyPI itself. We should still account for this in an "official policy". > > We probably want test infrastructure operators to have some flexibility to only run some categories of tests. For example, a debian test box would not want to run windows installation tests, or some operators may prefer to limit networking and only run non-distribution tests. Here're some tweaks to the test type taxonomy outlined above. 1. I think the name "network distribution tests" is easier to understand, perhaps because it's more specific, than "distribution tests". I believe this because when I added the "network " before instances of "distribution test" I found the text easier to understand. 2. If I Understand Correctly (IIUC) the word "Packaging" which is the first word after the first bullet point in the "Brittleness" section is a substitution error, and should be the word "Build". 3. It seems to me that "Brittleness" is a criteria, like "Inputs", by which one can categorize test "packaging test" types, whereas "Staging or Production" is a discriminator that applies more agnostically to all tests. Therefore I propose that the description will have a more intuitive flow if the order of the sections: "Brittleness" and "Staging or Production" are swapped. Also it might be nice to provide some other indicator that the "Staging or Production" section is not quite the same kind of section as the other two, e.g. an interposed sentence, or making both "Inputs" and "Brittleness" be subsections of a "Test Type" section. 4. I propose that we replace the term "Brittleness" with the word "Robustness".
nejucomo commented 2014-09-25 18:14:19 +00:00
Author
Copy Link

Replying to Zancas:

Should nejucomo be given commit access to:

https://github.com/tahoe-lafs/packaging-tests

and/or:

https://github.com/tahoe-lafs/buildbot-config-tahoe

?

I'll just do the simple non-blocking github flow: fork and make a pull request. I think I prefer that to having commit access anyway. The main reason I would want commit access is if we frequently block on the existing set of people with access.

Replying to [Zancas](/tahoe-lafs/trac-2024-07-25/issues/2049#issuecomment-93023): > Should nejucomo be given commit access to: > > <https://github.com/tahoe-lafs/packaging-tests> > > and/or: > > <https://github.com/tahoe-lafs/buildbot-config-tahoe> > > ? I'll just do the simple non-blocking github flow: fork and make a pull request. I think I prefer that to having commit access anyway. The main reason I would want commit access is if we frequently block on the existing set of people with access.
nejucomo commented 2014-09-25 18:32:25 +00:00
Author
Copy Link

Replying to [Zancas]comment:21:

...

Zancas and I are chatting in realtime and we both agree that having clear terminology could help avoid confusion.

As an example, in the Inputs section, first bullet, I say "build", but what I means is not what python ./setup.py build does.

In fact, all of this description kind of ignores what python ./setup.py build does, which is kind of a "local compilation and install without creating any package".

Other sources of confusion:

"distribution" - does this mean the process of copying bytes from some "package repository" onto a local system and then installing it? -or- does it mean "linux distro" or does it mean the format of the bytes in a package or what?

"packaging" - Is this the process of converting source code into a package file (tarball, .whl, .deb, .dmg, etc...)? Is it the process of publishing such files to "package repositories"? Is it the process of installing?

I propose we focus on the checklist in /tahoe-lafs/trac-2024-07-25/issues/7079#comment:93018 and ignore the description for now. Later we can categorize the different tests we create and create a canonical lexicon.

Replying to [Zancas]comment:21: > ... Zancas and I are chatting in realtime and we both agree that having clear terminology could help avoid confusion. As an example, in the **Inputs** section, first bullet, I say "build", but what I means is *not what* `python ./setup.py build` does. In fact, all of this description kind of ignores what `python ./setup.py build` does, which is kind of a "local compilation and install without creating any package". Other sources of confusion: "distribution" - does this mean the process of copying bytes from some "package repository" onto a local system and then installing it? *-or-* does it mean "linux distro" or does it mean the format of the bytes in a package or what? "packaging" - Is this the process of converting source code into a package file (tarball, ``.whl``, ``.deb``, ``.dmg``, etc...)? Is it the process of publishing such files to "package repositories"? Is it the process of installing? I propose we focus on the checklist in [/tahoe-lafs/trac-2024-07-25/issues/7079](/tahoe-lafs/trac-2024-07-25/issues/7079)#[comment:93018](/tahoe-lafs/trac-2024-07-25/issues/2049#issuecomment-93018) and ignore the description *for now*. Later we can categorize the different tests we create and create a canonical lexicon.
nejucomo commented 2014-09-25 19:51:06 +00:00
Author
Copy Link

Zancas and I are reading over buildbot-config-tahoe/tahoe/git/master.cfg in order to learn what should appear in packaging-tests.

In my opinion, the master.cfg should be minimal, and I'd like to move some of the operations here into packaging-tests, and here are some examples of why:

  • In master.cfg line 92 there's a conditional on a per-buildslave option do_test_already_have_dep which determines whether or not to run a test. I would rather have two explicitly named tests such as test_tahoe_build_with_preexisting_pycryptopp versus test_tahoe_build_without_preexisting_pyrcryptopp and then configure buildslaves by specifying which tests they run. (In fact, a single buildslave may be able to run both tests which would be handy especially when we have few buildslaves for a given architecture.)

  • In master.cfg line 97 we run ./setup.py build, which precludes tests such as "run pip install . on a git checkout inside of a virtualenv and verify this works".

-etc...

I'm a bit confused though about how buildslave's are or should be configured though. For the short term (for the purposes of this ticket), my goal is to change master.cfg minimally.

edit: I wrote this comment with multiple submissions because I'm paranoid of killing my browser tab when I really mean "delete previous word", ie: ctrl-w. :-/

Zancas and I are reading over [buildbot-config-tahoe/tahoe/git/master.cfg](https://github.com/tahoe-lafs/buildbot-config-tahoe/blob/master/tahoe/git/master.cfg) in order to learn what should appear in [packaging-tests](https://github.com/tahoe-lafs/packaging-tests). In my opinion, the `master.cfg` should be minimal, and I'd like to move some of the operations here into `packaging-tests`, and here are some examples of why: * In [master.cfg line 92](https://github.com/tahoe-lafs/buildbot-config-tahoe/blob/master/tahoe/git/master.cfg#L92) there's a conditional on a per-buildslave option `do_test_already_have_dep` which determines whether or not to run a test. I would rather have two explicitly named tests such as `test_tahoe_build_with_preexisting_pycryptopp` versus `test_tahoe_build_without_preexisting_pyrcryptopp` and then configure buildslaves by specifying which tests they run. (In fact, a single buildslave may be able to run both tests which would be handy especially when we have few buildslaves for a given architecture.) * In [master.cfg line 97](https://github.com/tahoe-lafs/buildbot-config-tahoe/blob/master/tahoe/git/master.cfg#L97) we run `./setup.py build`, which precludes tests such as "run `pip install .` on a git checkout inside of a virtualenv and verify this works". -etc... I'm a bit confused though about how buildslave's are or should be configured though. For the short term (for the purposes of this ticket), my goal is to change `master.cfg` minimally. **edit:** I wrote this comment with multiple submissions because I'm paranoid of killing my browser tab when I really mean "delete previous word", ie: ctrl-w. :-/
nejucomo commented 2014-09-25 20:45:12 +00:00
Author
Copy Link

Ok, my next goal is to set up a local test buildmaster / buildslave using this config so that I can compare its results to stuff inside https://tahoe-lafs.org/buildbot-tahoe-lafs/ before changing anything.

Note: I edited the checklist in comment 15#comment:93018 to reflect this goal.

Ok, my next goal is to set up a local test buildmaster / buildslave using this config so that I can compare its results to stuff inside <https://tahoe-lafs.org/buildbot-tahoe-lafs/> before changing anything. **Note:** I edited the checklist in [comment 15](/tahoe-lafs/trac-2024-07-25/issues/7079)#[comment:93018](/tahoe-lafs/trac-2024-07-25/issues/2049#issuecomment-93018) to reflect this goal.
daira commented 2014-09-26 23:42:44 +00:00
Owner
Copy Link

Replying to [nejucomo]comment:22:

Replying to Zancas:

Should nejucomo be given commit access to:

https://github.com/tahoe-lafs/packaging-tests

and/or:

https://github.com/tahoe-lafs/buildbot-config-tahoe

?

I'll just do the simple non-blocking github flow: fork and make a pull request. I think I prefer that to having commit access anyway. The main reason I would want commit access is if we frequently block on the existing set of people with access.

I see no reason why the permissions for those two repos shouldn't be identical to those for tahoe-lafs/tahoe-lafs.

Replying to [nejucomo]comment:22: > Replying to [Zancas](/tahoe-lafs/trac-2024-07-25/issues/2049#issuecomment-93023): > > Should nejucomo be given commit access to: > > > > <https://github.com/tahoe-lafs/packaging-tests> > > > > and/or: > > > > <https://github.com/tahoe-lafs/buildbot-config-tahoe> > > > > ? > > I'll just do the simple non-blocking github flow: fork and make a pull request. I think I prefer that to having commit access anyway. The main reason I would want commit access is if we frequently block on the existing set of people with access. I see no reason why the permissions for those two repos shouldn't be identical to those for tahoe-lafs/tahoe-lafs.
nejucomo commented 2014-09-29 16:54:24 +00:00
Author
Copy Link

Replying to [daira]comment:26:

Replying to [nejucomo]comment:22:

Replying to Zancas:

Should nejucomo be given commit access to:

https://github.com/tahoe-lafs/packaging-tests

and/or:

https://github.com/tahoe-lafs/buildbot-config-tahoe

?

I'll just do the simple non-blocking github flow: fork and make a pull request. I think I prefer that to having commit access anyway. The main reason I would want commit access is if we frequently block on the existing set of people with access.

I see no reason why the permissions for those two repos shouldn't be identical to those for tahoe-lafs/tahoe-lafs.

Agreed.

Replying to [daira]comment:26: > Replying to [nejucomo]comment:22: > > Replying to [Zancas](/tahoe-lafs/trac-2024-07-25/issues/2049#issuecomment-93023): > > > Should nejucomo be given commit access to: > > > > > > <https://github.com/tahoe-lafs/packaging-tests> > > > > > > and/or: > > > > > > <https://github.com/tahoe-lafs/buildbot-config-tahoe> > > > > > > ? > > > > I'll just do the simple non-blocking github flow: fork and make a pull request. I think I prefer that to having commit access anyway. The main reason I would want commit access is if we frequently block on the existing set of people with access. > > I see no reason why the permissions for those two repos shouldn't be identical to those for tahoe-lafs/tahoe-lafs. Agreed.
nejucomo commented 2014-09-29 17:00:57 +00:00
Author
Copy Link

Note, the new packaging tests, in addition to performing all tests as the previous buildbot configuration should additionally test the OSX packaging added in #182.

Ticket #2303 is about ensuring a buildslave will run this new test.

Note, the new packaging tests, in addition to performing all tests as the previous buildbot configuration should additionally test the OSX packaging added in #182. Ticket #2303 is about ensuring a buildslave will run this new test.
nejucomo commented 2014-09-29 17:21:27 +00:00
Author
Copy Link

The plan is to add one or more new Builders for packaging tests and not modify existing builders.

The plan is to add one or more new Builders for packaging tests and not modify existing builders.
nejucomo commented 2014-09-29 17:26:41 +00:00
Author
Copy Link

In a meeting now, we're verbally deciding to bump this out of the OpenITP packaging grant. Instead we will add new packaging tests into tahoe-lafs ./misc/ somewhere and modify the existing buildbot master.cfg for tahoe directly for those tests.

In a meeting now, we're verbally deciding to bump this out of the OpenITP packaging grant. Instead we will add new packaging tests into tahoe-lafs ``./misc/`` somewhere and modify the existing buildbot ``master.cfg`` for tahoe directly for those tests.
nejucomo commented 2014-09-29 17:27:21 +00:00
Author
Copy Link

Note, I've removed openitp-packaging keyword to reflect our decision.

Note, I've removed ``openitp-packaging`` keyword to reflect our decision.
meejah commented 2021-03-30 18:41:12 +00:00
Owner
Copy Link

Ticket retargeted after milestone closed

Ticket retargeted after milestone closed
meejah modified the milestone from soon (release n/a) to soon 2021-03-30 18:41:12 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
0.2.0

   
0.3.0

   
0.4.0

   
0.5.0

   
0.5.1

   
0.6.0

   
0.6.1

   
0.7.0

   
0.8.0

   
0.9.0

   
1.0.0

   
1.1.0

   
1.10.0

   
1.10.1

   
1.10.2

   
1.10a2

   
1.11.0

   
1.12.0

   
1.12.1

   
1.13.0

   
1.14.0

   
1.15.0

   
1.15.1

   
1.2.0

   
1.3.0

   
1.4.1

   
1.5.0

   
1.6.0

   
1.6.1

   
1.7.0

   
1.7.1

   
1.7β

   
1.8.0

   
1.8.1

   
1.8.2

   
1.8.3

   
1.8β

   
1.9.0

   
1.9.0-s3branch

   
1.9.0a1

   
1.9.0a2

   
1.9.0b1

   
1.9.1

   
1.9.2

   
1.9.2a1

   
LeastAuthority.com automation

   
blocker

   
cannot reproduce

   
cloud-branch

   
code

catch-all for anything that can be fixed by writing code but isn't classified better

  
code-dirnodes

anything involving tahoe directories

  
code-encoding

immutable-file encoding format/layout, cap format, encoding/decoding algorithms

  
code-frontend

general frontend things: new user-facing interfaces, FTP/SFTP servers, FUSE-like interfaces

  
code-frontend-cli

CLI tools

  
code-frontend-ftp-sftp

   
code-frontend-magic-folder

   
code-frontend-web

webapi interfaces, user-facing "WUI" interfaces

  
code-mutable

anything involving mutable files: encoding/decoding, share format

  
code-network

Introducer, locating storage servers, establishing/maintaining connections to them

  
code-nodeadmin

node startup/shutdown, tools to create nodes, logging tools, management/monitoring tools, manhole, disk-space-limiting tools

  
code-peerselection

server selection/sorting algorithm, rebalancing code, allocation policy, swarming-download

  
code-storage

storage-server implementation, share-storage directory layout, client-to-server share transfer protocol

  
contrib

problems with non-core things that live in the source:contrib/ directory

  
critical

   
defect

   
dev-infrastructure

things that help us build Tahoe: buildbot, automated speed/memory/code-coverage tests and results-displayers, IRC channels, mailing lists, this Trac instance

  
documentation

docs items. Also use 'docs' keyword on tickets in more specific components.

  
duplicate

   
enhancement

   
fixed

   
invalid

   
major

   
minor

   
n/a

   
normal

   
operational

things involved in running a tahoe grid, external tools to monitor/maintain a grid (like "nodeadmin" but larger-scale). Was also used for issues with the allmydata.com grid.

  
packaging

setup.py script, library dependencies, binary packages, licensing, installation instructions

  
somebody else's problem

   
supercritical

   
task

   
trivial

   
unknown

Uncategorized. All tickets default to this component until someone updates it.

  
was already fixed

   
website

things about the tahoe-lafs.org website (and sometimes this Trac instance, shared with 'dev-infrastructure')

  
wontfix

   
worksforme
No Label
0.2.0
0.3.0
0.4.0
0.5.0
0.5.1
0.6.0
0.6.1
0.7.0
0.8.0
0.9.0
1.0.0
1.1.0
1.10.0
1.10.1
1.10.2
1.10a2
1.11.0
1.12.0
1.12.1
1.13.0
1.14.0
1.15.0
1.15.1
1.2.0
1.3.0
1.4.1
1.5.0
1.6.0
1.6.1
1.7.0
1.7.1
1.7β
1.8.0
1.8.1
1.8.2
1.8.3
1.8β
1.9.0
1.9.0-s3branch
1.9.0a1
1.9.0a2
1.9.0b1
1.9.1
1.9.2
1.9.2a1
LeastAuthority.com automation
blocker
cannot reproduce
cloud-branch
code
code-dirnodes
code-encoding
code-frontend
code-frontend-cli
code-frontend-ftp-sftp
code-frontend-magic-folder
code-frontend-web
code-mutable
code-network
code-nodeadmin
code-peerselection
code-storage
contrib
critical
defect
dev-infrastructure
documentation
duplicate
enhancement
fixed
invalid
major
minor
n/a
normal
operational
packaging
somebody else's problem
supercritical
task
trivial
unknown
was already fixed
website
wontfix
worksforme
Milestone
Clear milestone
No items
No Milestone
soon
Assignees
Clear assignees
No Assignees
nejucomo
6 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Reference: tahoe-lafs/trac-2024-07-25#2049
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?