"make test" tests the installed version of allmydata, not the local sandbox version of allmydata #145
Labels
No Label
0.2.0
0.3.0
0.4.0
0.5.0
0.5.1
0.6.0
0.6.1
0.7.0
0.8.0
0.9.0
1.0.0
1.1.0
1.10.0
1.10.1
1.10.2
1.10a2
1.11.0
1.12.0
1.12.1
1.13.0
1.14.0
1.15.0
1.15.1
1.2.0
1.3.0
1.4.1
1.5.0
1.6.0
1.6.1
1.7.0
1.7.1
1.7β
1.8.0
1.8.1
1.8.2
1.8.3
1.8β
1.9.0
1.9.0-s3branch
1.9.0a1
1.9.0a2
1.9.0b1
1.9.1
1.9.2
1.9.2a1
LeastAuthority.com automation
blocker
cannot reproduce
cloud-branch
code
code-dirnodes
code-encoding
code-frontend
code-frontend-cli
code-frontend-ftp-sftp
code-frontend-magic-folder
code-frontend-web
code-mutable
code-network
code-nodeadmin
code-peerselection
code-storage
contrib
critical
defect
dev-infrastructure
documentation
duplicate
enhancement
fixed
invalid
major
minor
n/a
normal
operational
packaging
somebody else's problem
supercritical
task
trivial
unknown
was already fixed
website
wontfix
worksforme
No Milestone
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Reference: tahoe-lafs/trac-2024-07-25#145
Loading…
Reference in New Issue
Block a user
No description provided.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
It seems like .pth paths get searched earlier than PYTHONPATH, so if you install allmydata and then change something in your local sandbox and then run "make test", it will test the installed version, not the local one that you changed.
eek. I thought that .pth files had path-manipulation logic in them to make
sure that everything which gets added is really inserted into the middle of
sys.path, in the same place that the .pth file was found.
Hmm. I went to confirm this, and realized that I can't seem to find any .pth
files on my system that contain code, and I can't seem to find a reference to
'.pth' in the site.py files there.
This one bit me again, as I sleepily couldn't figure out why my new unit test wasn't failing as it should. When I deleted the installed version of allmydata, this was fixed.
One workaround to this which might not be a terrible idea in any case is to exclude test files from the installed files.
I'd like to understand the reasons first.. what was the PYTHONPATH used for that test?
I'm also inclined to leave the test sub-package installed, since that allows users who have just installed tahoe (perhaps from some binary installer) to run the unit tests on their own. FWIW, Twisted does the same thing, and following their lead I did the same for Buildbot and Foolscap.
When tahoe isn't installed, then
make test
starts with:I'll investigate more later...
Okay here is the difference between sys.path when running unit tests when tahoe is installed vs. when it is not installed:
So all of the installed .eggs from the Python site-packages dir, including the allmydata-tahoe .egg, are earlier on the path than
$PWD/src
is.Bumping to v0.7.
this is currently giving us false-positive test results on the Solaris buildslave, and possibly
others, so I'm raising the priority.
One way to lower the priority would be to modify the test process and buildbot config to emit the
version number of the code actually being tested (as extracted from the _trial_temp/twistd.log file) in the test step, so we could spot the discrepancy. At the moment, the buildbot is silently testing the wrong versions on platforms where tahoe was installed into a normal system path.
from the following mailing list articles:
http://www.eby-sarna.com/pipermail/peak/2006-june/002582.html
http://mail.python.org/pipermail/distutils-sig/2006-July/006492.html
it looks like using eggs always results in the following import order:
and for us, the tahoe "egg in site-packages" is overriding the code-under-test "non-egg on PYTHONPATH".
It appears that the recommended solution might be to use "setup.py develop", to create an .egg (really a directory, rather than a .zip file) that contains symlinks into our source tree. Then put that .egg in a local directory on PYTHONPATH.
We're focussing on an imminent v0.7.0 which hopefully has [#197 Small Distributed Mutable Files] and also a fix for [#199 bad SHA-256]. So I'm bumping less urgent tickets to v0.7.1.
I uninstalled the allmydata-tahoe package from the system on nooxie, so now the buildslave on nooxie is testing its local checkout of the trunk source.
So while this is still a bug, it is no longer the cause of any build failures on nooxie.
Brian, do I understand correctly that the fix is for "make test" to do "./setup.py develop" to produce an egg which symlinks into the directory and put that egg on the PYTHONPATH?
Nowadays
make build
runs./setup.py develop
, so this might fix this bug. I haven't tested it yet, though.Yes, this is fixed.
Hm. This is apparently not fixed. If you easy_install tahoe then
make test
tests that installed version instead of the one in ./support.No, I checked and this seems to be fixed. If anyone else can reproduce the problem --
make test
testing some other copy of the allmydata-tahoe source code instead of the source code in the current directory -- please let us know.Also, I guess, if
./setup.py trial
tests anyallmydata
package other than the one in the current directory, that too would mean this ticket should be re-opened.See #1137 for a reoccurrence of this problem, in the test-from-egg and test-from-prefixdir buildbot steps.
Why do we think it was fixed for
setup.py trial
? I see no test that would fail if we were testing the wrong code. ([test_path in test_runner.py]source:src/allmydata/test/test_runner.py@4581#L35 checks that we run the same code by invokingbin/tahoe
that we are currently testing, but ifbin/tahoe
andsetup.py trial
were running the same installed version -- which is entirely plausible because they both work by setting PYTHONPATH -- then we wouldn't detect that.)Hm, [test_path in test_runner.py]source:src/allmydata/test/test_runner.py@4581#L35 detects whether starting from the
allmydata
directory that contains theallmydata/*init*.py
file and going../../bin/tahoe
finds a workingtahoe
executable that prints out the same path. Hm, and I guess this will fail if we are running the tests of an installed version, for example on my Mac:Ah, but in this case the test is skipped:
So this test does fail in (some) cases that it is running a
test_runner.py
from an installed version instead of from a source tree, but this is not what we want! I want to be able to run the tests on an installed version and for them to be green, and I also want a different way to ensure that we are testing the right versions of the source in normal tests, test-from-egg and test-from-prefix-installdir on the buildbots. Hm.Replying to zooko:
Actually the test will always be skipped rather than failing because of this.
It is skipped in the case in #1137, for example.
I think running the tests on an installed version (as opposed to a version that is built at a prefixdir) isn't currently a supported thing to do; you have to be in a directory that contains at least a
setuptools-*.egg
, and presumably that also looks like a Tahoe-LAFS source distribution in other ways (the missingsetuptools-*.egg
is as far as I got). That it passes when you import an installed version instead of the one insrc/allmydata
is an accident; I'll open another ticket to support that case.The
test_the_right_code
test that I added on the ticket1074 branch should fail in this case, but an additional problem is that the version we're incorrectly testing might not have that test (or, we might fix a bug in that the test in future, and be confused by its difference in behaviour in the version we're actually testing).Making a change in setuptools_trial would have the same problem because we might be using an installed version of that. Making a change in
build_helpers/run_trial.py
would work, though. I'm trying that approach now.Hm, let's list the supported ways to run tests:
cd $SOURCEDIR && python setup.py test
cd $SOURCEDIR && trial allmydata.test
mkdir tempempty && cd tempempty && trial allmydata.test
# should run unit tests of the installed source code, might skip some tests -- test_runner -- that depend on the test code figuring our where the 'tahoe' executable livesmkdir -p egginstalldir && PYTHONPATH=egginstalldir:${PYTHONPATH} easy_install -d egginstalldir dist/*.egg && cd egginstalldir && PYTHONPATH=.:${PYTHONPATH} trial allmydata.test
# should run unit tests of the code in this egginstalldir including test_runner. This is what install-to-egg and test-from-egg on the buildbot is trying to do.python setup.py install --single-version-externally-managed --record=/dev/null --prefix=prefixinstalldir && cd prefixinstalldir && PYTHONPATH=lib/python2.6/site-packages:${PYTHONPATH} PATH=bin:${PATH} trial allmydata.test
# should run unit tests of the code in this prefixinstalldir including test_runner. This is what install-to-prefixdir and test-from-prefixdir on the buildbot is trying to doIs that it?
Note: I just tried each of these manually and here are my results:
/Library/Python/2.6/site-packages/allmydata_tahoe-1.7.1-py2.6.egg
instead of the current source tree, even if I set thePYTHONPATH
before running trial./Library/Python/2.6/site-packages/allmydata_tahoe-1.7.1-py2.6.egg
instead of the current source tree, even if I set thePYTHONPATH
before running trial.Note that as part of #1296, all of the supported ways to run tests (including the Makefile targets) now directly or indirectly invoke
bin/tahoe debug trial
. The ways to run tests described in comment:26 that didn't work, still don't work, but they are no longer used on the buildbots and I don't think they should be considered supported.It is still possible for
--single-version-externally-managed
installs (and other installs that copy code to site directories, such as the Twisted installers for Windows) to mess things up, although we try very hard to detect such cases. The only such cases I'm aware of are caused by #1258. Can we consider this ticket to be a duplicate of that one?Let's document the "supported and unsupported ways to run tests" from comment:26. Perhaps in a new file named source:docs/tests.rst?
Replying to zooko:
Filed as #1439. I haven't seen any other cases than #1258 where we test the wrong code, so I'm marking this as a duplicate.