depend on Twisted[windows] (or something) to get pypiwin32 installed #2392

New Issue

zooko · 2015-03-19T20:02:45Z

zooko commented

2015-03-19 20:02:45 +00:00

(https://pypi.python.org/pypi/pypiwin32)

This is a fork (actually a very small fork, a.k.a. a toothpick) of pywin32, produced by Glyph in order to fix the problem of pywin32 not being pip-installable:

http://sourceforge.net/p/pywin32/bugs/669/

Depending on this would allow us to use newer versions of Twisted, which would remove the problem with comment:93324. I think this would allow us to close #2066.

(note: as of March-2016, we do have a platform-conditional install_requires= dependency on pypiwin32, and this ticket is about finding a better way to express that dependency)

(https://pypi.python.org/pypi/pypiwin32) This is a fork (actually a very small fork, a.k.a. a toothpick) of pywin32, produced by Glyph in order to fix the problem of pywin32 not being pip-installable: <http://sourceforge.net/p/pywin32/bugs/669/> Depending on this would allow us to use newer versions of Twisted, which would remove the problem with [comment:93324](/tahoe-lafs/trac-2024-07-25/issues/2066#issuecomment-93324). I think this would allow us to close #2066. (note: as of March-2016, we *do* have a platform-conditional `install_requires=` dependency on `pypiwin32`, and this ticket is about finding a better way to express that dependency)

zooko added the

labels 2015-03-19 20:02:45 +00:00

zooko added this to the undecided milestone 2015-03-19 20:02:45 +00:00

daira commented

2015-03-19 23:36:57 +00:00

Can zetuptoolz use wheels? I think it is forked from a version of setuptools that predates wheels.

zooko commented

2015-07-21 20:10:06 +00:00

glyph: could you build a .egg for Windows of pypiwin32?

glyph commented

2015-07-21 23:08:39 +00:00

I don't think so. My initial attempts produce errors, although I need to do some diagnosis as to why (it may be a local development environment setup problem).

I'm also reluctant to upload them. Eggs require easy_install, and easy_install doesn't verify SSL certificates, so this would be facilitating the arbitrary execution of unverified code I've been trying so hard to avoid in Python-packaging-land. I take it there are still some difficulties dealing with wheels in tahoe? I will keep trying, and I'll upload them if there's really no other way to do this in Tahoe.

I don't think so. My initial attempts produce errors, although I need to do some diagnosis as to why (it may be a local development environment setup problem). I'm also reluctant to upload them. Eggs require `easy_install`, and `easy_install` doesn't verify SSL certificates, so this would be facilitating the arbitrary execution of unverified code I've been trying so hard to avoid in Python-packaging-land. I take it there are still some difficulties dealing with wheels in tahoe? I will keep trying, and I'll upload them if there's really no other way to do this in Tahoe.

glyph commented

2015-07-21 23:10:00 +00:00

Ideally, you would just add a dependency on twistedwindows_platform. But that will add a dependency on pyOpenSSL - are you still trying to avoid that? I wonder if there's some other way to break down our extras to give you the matrix Tahoe wants.

Ideally, you would just add a dependency on `twistedwindows_platform`. But that will add a dependency on pyOpenSSL - are you still trying to avoid that? I wonder if there's some other way to break down our extras to give you the matrix Tahoe wants.

warner commented

2016-03-25 16:59:21 +00:00

twistedwindows_SOMETHING is how I'd prefer to do this. We're no longer trying to avoid pyopenssl, in fact because of a setuptools bug/lacking-feature, I think we need twistedtls,windows_SOMETHING. (Foolscap requires twistedtls, and Twisted requires foolscap, but since setuptools will see Tahoe's twisted first, it will install twisted (without TLS), and then when it gets to foolscap later, it ignores the tls, and we miss out on service-identity, and things break).

At present (Twisted-16.0.0), twistedwindows_platform would also pull in the following _PLATFORM_INDEPENDENT features:

tls : that ok
conch : that's ok too
soap: um, I'd like to avoid it: 760KB of wheels, wstools, defusedxml, and docutils
serial: probably small, but completely unrelated to Tahoe

Also, because I suspect the same setuptools bug would cause problems if we Tahoe had two separate Twisted dependencies with different extras (e.g. Twistedtls, Twistedwindows_platform), the conditional logic in our setup.py/_auto_deps.py would need to be a bit grotty. So if it doesn't sound like Twisted is likely to change its windows dependency any time soon, I think we can continue to just conditionally depend on pypiwin32 when setup.py is run on windows.

BTW, we're all about wheels now. The existing pypiwin32 wheels should suit us just fine.

If you were going to re-slice the Twisted extras, maybe I'd suggest just a Twistedwindows extra, that only pulls in what's necessary to run the reactor on windows (pypiwin32). In fact I'm not entirely sure what the windows_platform and osx_platform extras are for. E.g. on OS-X it would seem appropriate for us for depend on Twistedosx, just in case there's some special magic that's necessary, but the current Twistedosx_platform includes pyobjc, which is gigantic, not available as wheels, takes something like an hour to compile, and not something we need (yet.. if/when we manage a non-web-based GUI, it may become necessary).

`twistedwindows_SOMETHING` is how I'd prefer to do this. We're no longer trying to avoid pyopenssl, in fact because of a setuptools bug/lacking-feature, I think we need `twistedtls,windows_SOMETHING`. (Foolscap requires `twistedtls`, and Twisted requires `foolscap`, but since setuptools will see Tahoe's `twisted` first, it will install `twisted` (without TLS), and then when it gets to foolscap later, it ignores the `tls`, and we miss out on `service-identity`, and things break). At present (Twisted-16.0.0), `twistedwindows_platform` would also pull in the following `_PLATFORM_INDEPENDENT` features: * `tls` : that ok * `conch` : that's ok too * `soap`: um, I'd like to avoid it: 760KB of wheels, `wstools`, `defusedxml`, and `docutils` * `serial`: probably small, but completely unrelated to Tahoe Also, because I suspect the same setuptools bug would cause problems if we Tahoe had two separate Twisted dependencies with different extras (e.g. `Twistedtls, Twistedwindows_platform`), the conditional logic in our `setup.py`/`_auto_deps.py` would need to be a bit grotty. So if it doesn't sound like Twisted is likely to change its windows dependency any time soon, I think we can continue to just conditionally depend on `pypiwin32` when `setup.py` is run on windows. BTW, we're all about wheels now. The existing `pypiwin32` wheels should suit us just fine. If you *were* going to re-slice the Twisted extras, maybe I'd suggest just a `Twistedwindows` extra, that only pulls in what's necessary to run the reactor on windows (`pypiwin32`). In fact I'm not entirely sure what the `windows_platform` and `osx_platform` extras are for. E.g. on OS-X it would seem appropriate for us for depend on `Twistedosx`, just in case there's some special magic that's necessary, but the current `Twistedosx_platform` includes `pyobjc`, which is gigantic, not available as wheels, takes something like an hour to compile, and not something we need (yet.. if/when we manage a non-web-based GUI, it may become necessary).

warner commented

2016-03-25 17:09:45 +00:00

Changing the title of this ticket, since we actually added the dependency on pypiwin32 a few weeks ago (as part of switching to pip, in #1582).

Changing the title of this ticket, since we actually added the dependency on `pypiwin32` a few weeks ago (as part of switching to pip, in #1582).

warner changed title from ~~add dependency on pypiwin32 to satisfy Twisted on Windows~~ to depend on `Twisted[windows]` (or something) to get pypiwin32 installed

2016-03-25 17:09:45 +00:00

glyph commented

2016-03-25 17:26:58 +00:00

If you were going to re-slice the Twisted extras ...

It seems like this should really be in a ticket on Twisted :).

> If you *were* going to re-slice the Twisted extras ... It seems like this should really be in a ticket on Twisted :).

glyph commented

2016-03-25 17:32:42 +00:00

the current Twistedosx_platform includes pyobjc, which is gigantic, not available as wheels, takes something like an hour to compile, and not something we need (yet.. if/when we manage a non-web-based GUI, it may become necessary).

This is actually necessary for doing anything that talks to OS X platform APIs, even those that have nothing to do with the GUI, like CoreLocation, and CoreWLAN, and SCNetworkReachability.

However, Twisted's dependency on all of pyobjc is somewhat mistaken; we could get away with just pyobjc-core and pyobjc-framework-Cocoa which should be substantially faster to install.

In the meanwhile, for developers that do a lot with tahoe, if they're hitting this problem now, one way to jump-start your local wheel cache is to make a temporary virtualenv, and then do:

pip install pyobjc-core
pip install pyobjc

as two separate commands; compiling pyobjc's frameworks is much, much faster if you already have an importable pyobjc-core in your environment. Once you've done this once, pip install pyobjc will be plenty fast since you'll have local wheels.

> the current Twistedosx_platform includes pyobjc, which is gigantic, not available as wheels, takes something like an hour to compile, and not something we need (yet.. if/when we manage a non-web-based GUI, it may become necessary). This is actually necessary for doing anything that talks to OS X platform APIs, even those that have nothing to do with the GUI, like [CoreLocation](wiki/CoreLocation), and CoreWLAN, and SCNetworkReachability. However, Twisted's dependency on *all* of pyobjc is somewhat mistaken; we could get away with just `pyobjc-core` and `pyobjc-framework-Cocoa` which should be substantially faster to install. In the meanwhile, for developers that do a lot with tahoe, if they're hitting this problem now, one way to jump-start your local wheel cache is to make a temporary virtualenv, and then do: ``` pip install pyobjc-core pip install pyobjc ``` as two separate commands; compiling pyobjc's frameworks is much, much faster if you already have an importable `pyobjc-core` in your environment. Once you've done this once, `pip install pyobjc` will be plenty fast since you'll have local wheels.

exarkun commented

2018-08-06 17:50:53 +00:00

Dropping the design-review-needed keyword as there is no discernible design here for review.

Dropping the `design-review-needed` keyword as there is no discernible design here for review.

exarkun commented

2020-01-17 16:12:55 +00:00

The limitations of setuptools, pip, "extras", etc make it unlikely anything significant is going to change on this front any time soon.

The one thing that should likely be changed is that "pywin32" now ships wheels and "pypiwin32" is no longer relevant (and also increasingly out of date, one supposes - although if it works "out of date" might be a good thing rather than a bad thing). Also, Twisted now depends on "pywin32" on Windows so Tahoe-LAFS currently depends on "pypiwin32" *and" pywin32" on Windows. Oops. That's easy enough to fix, though (at least until Twisted changes its "windows_platform" extra again; sigh, duplication).

The limitations of setuptools, pip, "extras", etc make it unlikely anything significant is going to change on this front any time soon. The one thing that should likely be changed is that "pywin32" now ships wheels and "pypiwin32" is no longer relevant (and also increasingly out of date, one supposes - although if it works "out of date" might be a good thing rather than a bad thing). Also, Twisted now depends on "pywin32" on Windows so Tahoe-LAFS currently depends on "pypiwin32" *and" pywin32" on Windows. Oops. That's easy enough to fix, though (at least until Twisted changes its "windows_platform" extra again; sigh, duplication).

exarkun commented

2020-01-17 16:13:46 +00:00

(https://github.com/tahoe-lafs/tahoe-lafs/pull/685)

GitHub <noreply@github.com> commented

2020-01-20 19:09:37 +00:00

In 5bd84895/trunk:

Merge pull request #685 from tahoe-lafs/2392.pywin32

Replace pypiwin32 with pywin32

Fixes: ticket:2392

In [5bd84895/trunk](/tahoe-lafs/trac-2024-07-25/commit/5bd84895fdc436feb0753e824044a016430896da): ``` Merge pull request #685 from tahoe-lafs/2392.pywin32 Replace pypiwin32 with pywin32 Fixes: ticket:2392 ```

tahoe-lafs added the

fixed

label 2020-01-20 19:09:37 +00:00

GitHub <noreply@github.com> closed this issue

2020-01-20 19:09:37 +00:00

glyph commented

2020-01-20 19:20:37 +00:00

🎉❗️

Sign in to join this conversation.