Tahoe-lafs fails to start with python 2.7 due to missing argparse module #1714

New Issue

tahoe-lafs · 2012-04-10T13:27:15Z

vrusinov commented

2012-04-10 13:27:15 +00:00

Traceback (most recent call last):
  File "/usr/bin/tahoe", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2707, in <module>
    working_set.require(__requires__)
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 686, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 584, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: argparse>=0.8

But since python 2.7 argparse is included in standard library: http://www.python.org/dev/peps/pep-0389/, so my system does not have separate module for argparse.

``` Traceback (most recent call last): File "/usr/bin/tahoe", line 6, in <module> from pkg_resources import load_entry_point File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2707, in <module> working_set.require(__requires__) File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 686, in require needed = self.resolve(parse_requirements(requirements)) File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 584, in resolve raise DistributionNotFound(req) pkg_resources.DistributionNotFound: argparse>=0.8 ``` But since python 2.7 argparse is included in standard library: <http://www.python.org/dev/peps/pep-0389/>, so my system does not have separate module for argparse.

tahoe-lafs added the

labels 2012-04-10 13:27:15 +00:00

tahoe-lafs added this to the undecided milestone 2012-04-10 13:27:15 +00:00

warner commented

2012-04-10 18:42:39 +00:00

Out of curiosity, what operating system is this?

(I suspect that this is another setuptools-ish problem. I don't believe that the python standard library is expected to provide pkg-info metadata for its contents, which makes it hard to use pkg_resources to detect availability of such modules).

Out of curiosity, what operating system is this? (I suspect that this is another setuptools-ish problem. I don't believe that the python standard library is expected to provide pkg-info metadata for its contents, which makes it hard to use pkg_resources to detect availability of such modules).

warner added

packaging

and removed

unknown

labels 2012-04-10 18:42:58 +00:00

davidsarah commented

2012-04-11 01:57:04 +00:00

It is not setuptools' responsibility to detect the standard library version of argparse, or the responsibility of that version to provide pkg-info metadata. What is supposed to happen, I believe, is that the argparse package is always built (if not installed as a package) when building dependencies of zfec, and that the version so built should work fine on Python 2.7. It is not intended that the standard library version be used by zfec (but if it were, as a result of a strangely configured sys.path for example, it wouldn't matter because the standard library version of argparse is compatible with the package version; and besides Tahoe doesn't use the zfec command-line tools).

vrusinov: how did you install Tahoe? If you installed it from an OS package, then that package is broken because it should have declared a dependency on the argparse package regardless of whether the OS has Python 2.7.

If you installed it using 'python setup.py build', then it should have downloaded and built the argparse package, again regardless of whether the OS has Python 2.7.

It is not setuptools' responsibility to detect the standard library version of argparse, or the responsibility of that version to provide pkg-info metadata. What is supposed to happen, I believe, is that the argparse package is always built (if not installed *as a package*) when building dependencies of zfec, and that the version so built [should work fine on Python 2.7](https://code.google.com/p/argparse/source/browse/README.txt). It is not intended that the standard library version be used by zfec (but if it were, as a result of a strangely configured sys.path for example, it wouldn't matter because the standard library version of argparse is compatible with the package version; and besides Tahoe doesn't use the zfec command-line tools). vrusinov: how did you install Tahoe? If you installed it from an OS package, then that package is broken because it should have declared a dependency on the argparse package regardless of whether the OS has Python 2.7. If you installed it using 'python setup.py build', then it should have downloaded and built the argparse package, again regardless of whether the OS has Python 2.7.

zooko commented

2012-04-11 02:50:54 +00:00

We can also change zfec to declare a dependency on the argparse package only if the argparse module can't be imported. This would be analogous to the way Tahoe-LAFS [declares a dependency on the pysqlite package]source:git/src/allmydata/_auto_deps.py?annotate=blame&rev=cef646c4a70537d618459044ea0a8b590732ca90#L80 only if the sqlite3 module can't be imported.

A simpler solution to this problem is to require Python 2.7! I'm excited about Python 2.7 because the Python developers have sworn to stop improving the Python 2 line.

Realistically, though, it will probably be years before we're willing to tell people they can't run Tahoe-LAFS with the Python 2.6 that came with their operating system. See #1658 (drop support for Python < 2.6) for a ticket where we are dithering about when to drop support for Python < 2.6. To simplify the subject of this ticket we would also have to drop support for Python < 2.7.

We can also change zfec to declare a dependency on the argparse package only if the argparse module can't be imported. This would be analogous to the way Tahoe-LAFS [declares a dependency on the pysqlite package]source:git/src/allmydata/_auto_deps.py?annotate=blame&rev=cef646c4a70537d618459044ea0a8b590732ca90#L80 only if the `sqlite3` module can't be imported. A simpler solution to this problem is to require Python 2.7! I'm excited about Python 2.7 because the Python developers have sworn to stop improving the Python 2 line. Realistically, though, it will probably be years before we're willing to tell people they can't run Tahoe-LAFS with the Python 2.6 that came with their operating system. See #1658 (drop support for Python < 2.6) for a ticket where we are dithering about when to drop support for Python < 2.6. To simplify the subject of *this* ticket we would *also* have to drop support for Python < 2.7.

zooko commented

2012-04-11 03:05:14 +00:00

Hey waitaminute. I went to implement my idea from comment:88164 to have zfec declare a dependency on the argparse package only if the argparse module couldn't be imported. However, I discovered that zfec already [//trac/zfec/browser/trunk/zfec/setup.py?annotate=blame&rev=390#L156 declares a dependency on argparse only if Python < 2.7]. So how did you get a version of zfec installed into your system that says that it requires the argparse package? My guess is that you installed zfec with Python 2.6 and then upgraded your system to Python 2.7. I guess this shows that the hack of "try to import the argparse module and, if that doesn't work, declare a dependency on the argparse package" is more robust.

Assigning to vrusinov to see if he can explain how his install of zfec has this dependency on argparse even though he has Python 2.7. (Hint: where is zfec located? You can tell with $ python -c 'import zfec;print zfec'.)

Oh, another explanation would be that you have a version of zfec from before its dependency on argparse was conditionalized. That would be... let's see...

[//trac/zfec/changeset/390/trunk https://tahoe-lafs.org/trac/zfec/changeset/390/trunk]

Oh dear... I guess I haven't made a new release of zfec in the 13 months since I committed that patch.

Okay, so a good next step would be for me to edit zfec to use the "try import and then fallback to require" approach instead of the "Python < 2.7" approach, and then make a new release of zfec.

Hey waitaminute. I went to implement my idea from [comment:88164](/tahoe-lafs/trac-2024-07-25/issues/1714#issuecomment-88164) to have `zfec` declare a dependency on the `argparse` package only if the `argparse` module couldn't be imported. However, I discovered that `zfec` already [//trac/zfec/browser/trunk/zfec/setup.py?annotate=blame&rev=390#L156 declares a dependency on argparse only if Python < 2.7]. So how did you get a version of zfec installed into your system that says that it requires the `argparse` package? My guess is that you installed zfec with Python 2.6 and then upgraded your system to Python 2.7. I guess this shows that the hack of "try to import the `argparse` module and, if that doesn't work, declare a dependency on the `argparse` package" is more robust. Assigning to vrusinov to see if he can explain how his install of `zfec` has this dependency on `argparse` even though he has Python 2.7. (Hint: where is `zfec` located? You can tell with $ `python -c 'import zfec;print zfec'`.) Oh, another explanation would be that you have a version of `zfec` from before its dependency on `argparse` was conditionalized. That would be... let's see... [//trac/zfec/changeset/390/trunk <https://tahoe-lafs.org/trac/zfec/changeset/390/trunk>] Oh dear... I guess I haven't made a new release of zfec in the 13 months since I committed that patch. Okay, so a good next step would be for me to edit `zfec` to use the "try import and then fallback to require" approach instead of the "Python < 2.7" approach, and then make a new release of zfec.

davidsarah commented

2012-04-11 03:09:12 +00:00

Replying to zooko:

Okay, so a good next step would be for me to edit zfec to use the "try import and then fallback to require" approach instead of the "Python < 2.7" approach, and then make a new release of zfec.

Why is that change needed? Just making a new release of zfec as-is would solve the problem, no?

(I'm generally -1 on attempting to import modules that are normally only runtime dependencies at build time.)

Replying to [zooko](/tahoe-lafs/trac-2024-07-25/issues/1714#issuecomment-88165): > Okay, so a good next step would be for me to edit `zfec` to use the "try import and then fallback to require" approach instead of the "Python < 2.7" approach, and then make a new release of zfec. Why is that change needed? Just making a new release of zfec as-is would solve the problem, no? (I'm generally -1 on attempting to import modules that are normally only runtime dependencies at build time.)

davidsarah commented

2012-04-11 03:12:30 +00:00

BTW, making a new release of zfec will not help if vrusinov's (or anyone else's with the same problem) system is using the OS-packaged zfec. So we still need to know how vrusinov installed Tahoe.

davidsarah commented

2012-04-11 03:22:25 +00:00

Oh, and zooko: you have a habit of getting halfway through a comment and finding out something that negates the evidence for a previous hypothesis (e.g. "My guess is that you installed zfec with Python 2.6..." above), and then not editing or ~~striking out~~ the hypothesis. It's confusing!

Oh, and zooko: you have a habit of getting halfway through a comment and finding out something that negates the evidence for a previous hypothesis (e.g. "My guess is that you installed zfec with Python 2.6..." above), and then not editing or ~~striking out~~ the hypothesis. It's confusing!

zooko commented

2012-04-11 03:25:31 +00:00

Okay, I've posted zfec v1.4.23 which hopefully fixes this problem. vrusinov: please try again!

zooko commented

2012-04-11 03:25:45 +00:00

Replying to davidsarah:

Oh, and zooko: you have a habit of getting halfway through a comment and finding out something that negates the evidence for a previous hypothesis (e.g. "My guess is that you installed zfec with Python 2.6..." above), and then not editing or ~~striking out~~ the hypothesis. It's confusing!

Sorry! You're right.

Replying to [davidsarah](/tahoe-lafs/trac-2024-07-25/issues/1714#issuecomment-88168): > Oh, and zooko: you have a habit of getting halfway through a comment and finding out something that negates the evidence for a previous hypothesis (e.g. "My guess is that you installed zfec with Python 2.6..." above), and then not editing or ~~striking out~~ the hypothesis. It's confusing! Sorry! You're right.

zooko commented

2012-04-11 03:30:29 +00:00

Replying to [davidsarah]comment:6:

Why is that change needed? Just making a new release of zfec as-is would solve the problem, no?

I guess my main reason was that I wanted zfec's build system to be more like Tahoe-LAFS's. Also, I'm not sure if there could be some situation where Python 2.7 is in use, but argparse isn't importable. This happened recently, for example, when a Tahoe-LAFS user reported that their Python 2.7 didn't have the sqlite module. It wasn't entirely clear if this was the *BSD packaging of Python or some custom build of that user that had managed to omit this module from the Python standard library.

In general, I prefer the approach of "try the most specific test that is accurate" over "determine the functionality from a version number", when practical.

(I'm generally -1 on attempting to import modules that are normally only runtime dependencies at build time.)

That's a good consideration, too. In this case I currently prefer the above considerations, but I guess I could go either way.

Replying to [davidsarah]comment:6: > > Why is that change needed? Just making a new release of zfec as-is would solve the problem, no? I guess my main reason was that I wanted `zfec`'s build system to be more like `Tahoe-LAFS`'s. Also, I'm not sure if there could be some situation where Python 2.7 is in use, but argparse isn't importable. This happened recently, for example, when a Tahoe-LAFS user reported that their Python 2.7 didn't have the `sqlite` module. It wasn't entirely clear if this was the *BSD packaging of Python or some custom build of that user that had managed to omit this module from the Python standard library. In general, I prefer the approach of "try the most specific test that is accurate" over "determine the functionality from a version number", when practical. > (I'm generally -1 on attempting to import modules that are normally only runtime dependencies at build time.) That's a good consideration, too. In this case I currently prefer the above considerations, but I guess I could go either way.

vrusinov commented

2012-04-11 06:58:02 +00:00

Out of curiosity, what operating system is this?

It's Fedora 16.

vrusinov: how did you install Tahoe? If you installed it from an OS package, then that
package is broken because it should have declared a dependency on the argparse package
regardless of whether the OS has Python 2.7.

If you installed it using 'python setup.py build', then it should have downloaded and built
the argparse package, again regardless of whether the OS has Python 2.7.

I'm using my own spec to build tahoe-lafs rpm and some additional specs to build missing dependencies (including zfec). Specs does nothing more than python setup.py build and python setup.py install.

It works pefectly on RHEL 5 & 6 (python 2.4 and 2.6), but does not works on Fedora 16.

Of course, I can port python-argparse (which presents on RHEL) to Fedora 16, but it will look ugly and I'm sure it's not necessary.

A simpler solution to this problem is to require Python 2.7! I'm excited about Python 2.7
because the Python developers have sworn to stop improving the Python 2 line.

Yep, but it will break existing installs on all RHEL/CentOS/SL versions (not sure about status of 2.7 on other distros), and nobody really want to install 2.7 on such distro. Personally I think any wide-used project should support at least 2.4. I share you pain as a developer, but as a sysadmin I understand the reason why there are lots of users with older long-term support systems.

My guess is that you installed zfec with Python 2.6 and then upgraded your system to
Python 2.7.

No, I did not. I've build zfec 1.4.22 just some days ago. And, oh, Thank you! I see you've just released a new version. I will try it.

BTW, making a new release of zfec will not help if vrusinov's (or anyone else's with the same problem) system is using the OS-packaged zfec. So we still need to know how vrusinov installed Tahoe.

It will help if tahoe-lafs will declare >= zfec-1.4.23 dependency on python 2.7. At least it will be clear from error message what needs to be upgraded.

> Out of curiosity, what operating system is this? It's Fedora 16. > vrusinov: how did you install Tahoe? If you installed it from an OS package, then that > package is broken because it should have declared a dependency on the argparse package > regardless of whether the OS has Python 2.7. > > If you installed it using 'python setup.py build', then it should have downloaded and built > the argparse package, again regardless of whether the OS has Python 2.7. I'm using my own spec to build tahoe-lafs rpm and some additional specs to build missing dependencies (including zfec). Specs does nothing more than `python setup.py build` and `python setup.py install`. It works pefectly on RHEL 5 & 6 (python 2.4 and 2.6), but does not works on Fedora 16. Of course, I can port python-argparse (which presents on RHEL) to Fedora 16, but it will look ugly and I'm sure it's not necessary. > A simpler solution to this problem is to require Python 2.7! I'm excited about Python 2.7 > because the Python developers have sworn to stop improving the Python 2 line. Yep, but it will break existing installs on all RHEL/CentOS/SL versions (not sure about status of 2.7 on other distros), and nobody really want to install 2.7 on such distro. Personally I think any wide-used project should support at least 2.4. I share you pain as a developer, but as a sysadmin I understand the reason why there are lots of users with older long-term support systems. > My guess is that you installed zfec with Python 2.6 and then upgraded your system to > Python 2.7. No, I did not. I've build zfec 1.4.22 just some days ago. And, oh, Thank you! I see you've just released a new version. I will try it. > BTW, making a new release of zfec will not help if vrusinov's (or anyone else's with the same problem) system is using the OS-packaged zfec. So we still need to know how vrusinov installed Tahoe. It will help if tahoe-lafs will declare >= zfec-1.4.23 dependency on python 2.7. At least it will be clear from error message what needs to be upgraded.

vrusinov commented

2012-04-11 07:50:25 +00:00

I will try it.

Well, it not so easy: https://tahoe-lafs.org/trac/zfec/ticket/15

I've built it, but I don't want to install manually (as it will mess with system packaging) and don't want to use virtualenv (as I want tahoe & zfec installed system-wide).

> I will try it. Well, it not so easy: <https://tahoe-lafs.org/trac/zfec/ticket/15> I've built it, but I don't want to install manually (as it will mess with system packaging) and don't want to use virtualenv (as I want tahoe & zfec installed system-wide).

zooko commented

2013-06-19 18:00:44 +00:00

What's the status of this ticket? Did the fact that I made a new release of zfec solve it and now it can be closed?

Or should we implement one of the suggestions from the comments, such as "It will help if tahoe-lafs will declare >= zfec-1.4.23 dependency" (from comment:88172)?

What's the status of this ticket? Did the fact that I made a new release of zfec solve it and now it can be closed? Or should we implement one of the suggestions from the comments, such as "It will help if tahoe-lafs will declare >= zfec-1.4.23 dependency" (from [comment:88172](/tahoe-lafs/trac-2024-07-25/issues/1714#issuecomment-88172))?

daira commented

2013-06-21 00:54:00 +00:00

I still don't understand why the original DistributionNotFound: argparse>=0.8 occurred, because python setup.py build should have downloaded and built argparse, which should work absolutely fine on Python 2.7. Of course it might have failed to do so due to one of the numerous setuptools bugs (or some other network or packaging error), but we don't have enough information to tell which one.

I still don't understand why the original `DistributionNotFound: argparse>=0.8` occurred, because `python setup.py build` should have downloaded and built argparse, which should work absolutely fine on Python 2.7. Of course it might have failed to do so due to one of the numerous setuptools bugs (or some other network or packaging error), but we don't have enough information to tell which one.

daira commented

2013-06-21 00:56:50 +00:00

By the way, if the zfec requirement is bumped, it should be to >=1.4.24, not >=1.4.23 (due to https://tahoe-lafs.org/trac/zfec/ticket/15 ).

By the way, if the zfec requirement is bumped, it should be to >=1.4.24, not >=1.4.23 (due to <https://tahoe-lafs.org/trac/zfec/ticket/15> ).

zooko commented

2013-06-21 15:02:38 +00:00

Okay, it sounds like our options are:

a) close this ticket as "invalid", "wontfix", "irreproducible", or "someone else's problem"
b) drop support for any version of Python that isn't Python v2.7.x! ☺
c) bump the requirement on zfec to be ≥ 1.4.24

I vote for a).

Okay, it sounds like our options are: * a) close this ticket as "invalid", "wontfix", "irreproducible", or "someone else's problem" * b) drop support for any version of Python that isn't Python v2.7.x! ☺ * c) bump the requirement on zfec to be `≥ 1.4.24` I vote for a).

zooko commented

2013-06-21 15:04:56 +00:00

Addendum: I propose to close this ticket because, as people upgrade to zfec ≥ 1.4.24 and as they upgrade to Python ≥ 2.7, this problem will stop happening. Also, since we don't understand exactly how it happened in the first place, it could have been something very specific to vrusinov's system, that won't affect other users.

Addendum: I propose to close this ticket because, as people upgrade to zfec ≥ 1.4.24 and as they upgrade to Python ≥ 2.7, this problem will stop happening. Also, since we don't understand *exactly* how it happened in the first place, it could have been something very specific to vrusinov's system, that won't affect other users.

tahoe-lafs added the

cannot reproduce

label 2013-06-21 23:13:59 +00:00

tahoe-lafs closed this issue

2013-06-21 23:13:59 +00:00

daira commented

2013-06-21 23:15:15 +00:00

vrusinov (or anyone else), please feel free to reopen this ticket if it happens again with any recent version of Tahoe-LAFS.

Sign in to join this conversation.