build failure: mock vs zetuptoolz #2465

New Issue

warner · 2015-07-15T19:01:02Z

warner commented

2015-07-15 19:01:02 +00:00

Zooko and Jun Matsushita reported this one via email (as Trac was temporarily broken at the time). Building tahoe (either 1.10.1 or trunk) is currently broken.

The "mock" library (which we use for running unit tests, but not during runtime) recently received its first update in two years, and the new version (1.1.3) depends upon setuptools 17.1 or newer. Tahoe's forked "zsetuptoolz" doesn't meet the requirement.

I'm not entirely sure why the 17.1 requirement was added: the previous mock-1.1.2 didn't have that requirement, and Tahoe builds failed with an error like this:

  File "build/setuptools-0.6c16dev6.egg/setuptools/command/easy_install.py", line 385, in install_egg_scripts
    self.install_wrapper_scripts(dist)
  File "build/setuptools-0.6c16dev6.egg/setuptools/command/easy_install.py", line 577, in install_wrapper_scripts
    for args in get_script_args(dist, script_dir=self.script_dir):
TypeError: override_get_script_args() got an unexpected keyword argument 'script_dir'

It's possible that the dependency is actually imposed by the "pbr" library that mock now uses. The current mock-1.1.3 error looks like:

  File "build/setuptools-0.6c16dev6.egg/setuptools/command/easy_install.py", line 938, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "build/setuptools-0.6c16dev6.egg/setuptools/command/easy_install.py", line 927, in run_setup
    run_setup(setup_script, args)
  File "build/setuptools-0.6c16dev6.egg/setuptools/sandbox.py", line 62, in run_setup
    lambda: execfile(
  File "build/setuptools-0.6c16dev6.egg/setuptools/sandbox.py", line 105, in run
    return func()
  File "build/setuptools-0.6c16dev6.egg/setuptools/sandbox.py", line 64, in <lambda>
    {'__file__':setup_script, '__name__':'__main__'}
  File "setup.py", line 10, in <module>
    #
AttributeError: 'module' object has no attribute 'version'

And it looks like mock trunk made a change yesterday (which will probably be released as 1.1.4) to get the setuptools version from the right place, which will probably change that error to something like:

mock requires setuptools>=17.1. Aborting installation

Some potential fixes/improvements:

Remove "mock" from the set of things installed by "setup.py build": it's only for unit tests, and end-users don't need it. But this won't help developers, and might be hard to implement because tahoe has extensive (and overgrown, IMO) code to check the versions of all its dependencies, and that code doesn't have the notion of "tests_requires".
Stop using "mock" altogether. Zooko points out that twisted.trial has a patch-during-tests utility that's better (it understands Deferreds). We have 9 test_*.py files that import mock.
Stop forking setuptools. This is obviously the best long-term fix, and would be an opportunity to clean up a lot of our build code.
Prolong the agony: modify zetuptoolz to handle whatever the newer mock needs.

Zooko and Jun Matsushita reported this one via email (as Trac was temporarily broken at the time). Building tahoe (either 1.10.1 or trunk) is currently broken. The "mock" library (which we use for running unit tests, but not during runtime) recently received its first update in two years, and the new version (1.1.3) depends upon setuptools 17.1 or newer. Tahoe's forked "zsetuptoolz" doesn't meet the requirement. I'm not entirely sure why the 17.1 requirement was added: the previous mock-1.1.2 didn't have that requirement, and Tahoe builds failed with an error like this: ``` File "build/setuptools-0.6c16dev6.egg/setuptools/command/easy_install.py", line 385, in install_egg_scripts self.install_wrapper_scripts(dist) File "build/setuptools-0.6c16dev6.egg/setuptools/command/easy_install.py", line 577, in install_wrapper_scripts for args in get_script_args(dist, script_dir=self.script_dir): TypeError: override_get_script_args() got an unexpected keyword argument 'script_dir' ``` It's possible that the dependency is actually imposed by the "pbr" library that mock now uses. The current mock-1.1.3 error looks like: ``` File "build/setuptools-0.6c16dev6.egg/setuptools/command/easy_install.py", line 938, in build_and_install self.run_setup(setup_script, setup_base, args) File "build/setuptools-0.6c16dev6.egg/setuptools/command/easy_install.py", line 927, in run_setup run_setup(setup_script, args) File "build/setuptools-0.6c16dev6.egg/setuptools/sandbox.py", line 62, in run_setup lambda: execfile( File "build/setuptools-0.6c16dev6.egg/setuptools/sandbox.py", line 105, in run return func() File "build/setuptools-0.6c16dev6.egg/setuptools/sandbox.py", line 64, in <lambda> {'__file__':setup_script, '__name__':'__main__'} File "setup.py", line 10, in <module> # AttributeError: 'module' object has no attribute 'version' ``` And it looks like mock trunk made a change yesterday (which will probably be released as 1.1.4) to get the setuptools version from the right place, which will probably change that error to something like: ``` mock requires setuptools>=17.1. Aborting installation ``` Some potential fixes/improvements: * Remove "mock" from the set of things installed by "setup.py build": it's only for unit tests, and end-users don't need it. But this won't help developers, and might be hard to implement because tahoe has extensive (and overgrown, IMO) code to check the versions of all its dependencies, and that code doesn't have the notion of "tests_requires". * Stop using "mock" altogether. Zooko points out that twisted.trial has a patch-during-tests utility that's better (it understands Deferreds). We have 9 `test_*.py` files that import mock. * Stop forking setuptools. This is obviously the best long-term fix, and would be an opportunity to clean up a lot of our build code. * Prolong the agony: modify zetuptoolz to handle whatever the newer mock needs.

warner added the

labels 2015-07-15 19:01:02 +00:00

warner added this to the soon milestone 2015-07-15 19:01:02 +00:00

warner commented

2015-07-16 17:06:08 +00:00

Daira implemented (in baaf2a4) the 5th option which I forgot about: change our mock dependency to reject any of the newer versions. This is a short-term fix, as really we want to be compatible with the latest version of everything. So I think we still need to implement one of the other longer-term fixes.

Lowering the priority, since this is no longer breaking all builds.

Daira implemented (in baaf2a4) the 5th option which I forgot about: change our `mock` dependency to reject any of the newer versions. This is a short-term fix, as really we want to be compatible with the latest version of everything. So I think we still need to implement one of the other longer-term fixes. Lowering the priority, since this is no longer breaking all builds.

warner added

normal

and removed

critical

labels 2015-07-16 17:06:08 +00:00

daira commented

2015-07-16 18:52:36 +00:00

... but it is still breaking builds of 1.10.1 and earlier :-(

I was hoping this would be something that could be easily fixed by releasing a new version of mock, but it sounds not. We might need to release a 1.10.2 to fix this.

(Note: this doesn't affect the Debian tahoe-lafs package because they won't have packaged mock 1.1.2+ yet.)

... but it is still breaking builds of 1.10.1 and earlier :-( I was hoping this would be something that could be easily fixed by releasing a new version of mock, but it sounds not. We might need to release a 1.10.2 to fix this. (Note: this doesn't affect the Debian tahoe-lafs package because they won't have packaged mock 1.1.2+ yet.)

daira commented

2015-07-16 18:54:00 +00:00

It's really quite irritating that this happened a few weeks after releasing 1.10.1 when we'd held off that release for 2 years :-/

daira commented

2015-07-16 18:56:02 +00:00

I'm in favour of removing the mock dependency. The style of tests it results in are quite complicated to follow and I don't think it is buying us much, relative to using trial's patch method.

I'm in favour of removing the mock dependency. The style of tests it results in are quite complicated to follow and I don't think it is buying us much, relative to using trial's `patch` method.

daira commented

2015-07-16 19:29:31 +00:00

I made a start on eliminating mock in this branch: https://github.com/tahoe-lafs/tahoe-lafs/commits/2465.eliminate-mock.0. It doesn't seem too painful.

I made a start on eliminating mock in this branch: <https://github.com/tahoe-lafs/tahoe-lafs/commits/2465.eliminate-mock.0>. It doesn't seem too painful.

daira commented

2015-07-17 21:23:48 +00:00

Review needed for https://github.com/tahoe-lafs/tahoe-lafs/pull/175.

Review needed for <https://github.com/tahoe-lafs/tahoe-lafs/pull/175>.

tahoe-lafs modified the milestone from soon to 1.10.2

2015-07-17 21:28:46 +00:00

tahoe-lafs added

major

and removed

normal

labels 2015-07-17 22:00:44 +00:00

warner commented