make Tahoe-LAFS work under PyPy #1792

New Issue

tahoe-lafs · 2012-07-10T22:58:09Z

davidsarah commented

2012-07-10 22:58:09 +00:00

If I remember correctly, the main obstacle was extension modules. Current PyPy supports extension modules compiled for CPython, but is not 100% compatible with the CPython extension API.

[Twisted apparently now builds correctly with PyPy.](https://bugs.pypy.org/issue735)

If I remember correctly, the main obstacle was extension modules. Current [PyPy](wiki/PyPy) [supports extension modules compiled for CPython](http://morepypy.blogspot.co.uk/2010/04/using-cpython-extension-modules-with.html), but is not 100% compatible with the CPython extension API. [Twisted apparently now builds correctly with [PyPy](wiki/PyPy).](https://bugs.pypy.org/issue735)

tahoe-lafs added the

labels 2012-07-10 22:58:09 +00:00

tahoe-lafs added this to the eventually milestone 2012-07-10 22:58:09 +00:00

davidsarah commented

2012-07-11 21:31:16 +00:00

fijal wrote in #pypy:

davidsarah: pyopenssl works as long as you don't have multiple threads

fijal wrote in #pypy: > davidsarah: pyopenssl works as long as you don't have multiple threads

hawkowl commented

2015-07-29 04:10:16 +00:00

From an IRC discussion about why it doesn't work (pycryptopp), I decided to look at Cryptography and pynacl, and what uses pycryptopp in tahoe.

ed25519 is supported by pynacl.
AES is supported by Cryptography.
RSA keys (in many formats) are supported by Cryptography.
SHA256 hashes are supported by Cryptography.

Zooko mentioned that Cryptography has a hard dependency on OpenSSL (which he rightfully dislikes) but my unqualified self thinks that it might not be so bad. Worth a shot, maybe? I have (some) time on Tuesday's PyCon AU sprint session to maybe take a hack on it, if the maintainers don't find the current-hard-dep on OpenSSL a dealbreaker.

From an IRC discussion about why it doesn't work (pycryptopp), I decided to look at Cryptography and pynacl, and what uses pycryptopp in tahoe. - ed25519 is supported by pynacl. - AES is supported by Cryptography. - RSA keys (in many formats) are supported by Cryptography. - SHA256 hashes are supported by Cryptography. Zooko mentioned that Cryptography has a hard dependency on OpenSSL (which he rightfully dislikes) but my unqualified self thinks that it might not be so bad. Worth a shot, maybe? I have (some) time on Tuesday's [PyCon](wiki/PyCon) AU sprint session to maybe take a hack on it, if the maintainers don't find the current-hard-dep on OpenSSL a dealbreaker.

warner commented

2015-07-29 05:42:20 +00:00

Foolscap depends upon OpenSSL too, and even though we're planning on replacing Foolscap (#510), for backwards compatibility we'll probably need it for quite a while yet. Also when we replace it, we may wind up wanting TLS-protected HTTP instead of just plain HTTP (to maintain privacy of which shares you're fetching), which might be another thing that depends upon OpenSSL. I'm not a fan of OpenSSL either, but I don't think depending upon it would be the worst direction to go. Maybe our policy could be to use pynacl for new features, and migrate old ones from pycryptopp to cryptography?

hawkowl commented

2015-07-29 06:34:58 +00:00

That policy sounds reasonable -- although pynacl doesn't expose much more than ed22519 keys and "boxes" (Curve25519-pubpriv, salsa20+Poly1305 symmetric), so, unless the new features use that, you'll have to use Cryptography.

To replace pycryptopp in Tahoe, both Cryptography and pynacl will have to be used (pynacl for the ed22519, cryptography for AES, RSA, and SHA-2), so new features could possibly rely on either. The best case is that Cryptography later comes with a non-OpenSSL backend (that being said, LibreSSL works, if I'm not mistaken), which is improbable (because it's a lot of work) but possible (because it's been written from the start to have multiple backends -- on OS X it uses CommonCrypto for instance).

I'll see what I can get from a day of hacking in the sprints.

That policy sounds reasonable -- although pynacl doesn't expose much more than ed22519 keys and "boxes" (Curve25519-pubpriv, salsa20+Poly1305 symmetric), so, unless the new features use that, you'll have to use Cryptography. To replace pycryptopp in Tahoe, both Cryptography and pynacl will have to be used (pynacl for the ed22519, cryptography for AES, RSA, and SHA-2), so new features could possibly rely on either. The best case is that Cryptography later comes with a non-OpenSSL backend (that being said, LibreSSL works, if I'm not mistaken), which is improbable (because it's a lot of work) but possible (because it's been written from the start to have multiple backends -- on OS X it uses [CommonCrypto](wiki/CommonCrypto) for instance). I'll see what I can get from a day of hacking in the sprints.

exarkun commented

2019-06-25 14:14:31 +00:00

Note that efforts are underway to switch from pycryptopp to cryptography (https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3031) as part of the Python 3 porting effort.

Note that efforts are underway to switch from pycryptopp to cryptography (<https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3031>) as part of the Python 3 porting effort.

exarkun commented

2019-12-18 15:44:52 +00:00

There is a PR that seems to get us most of the way there now - https://github.com/tahoe-lafs/tahoe-lafs/pull/669

There is a PR that seems to get us most of the way there now - <https://github.com/tahoe-lafs/tahoe-lafs/pull/669>

exarkun commented

2020-11-30 13:46:08 +00:00

Merged in https://github.com/tahoe-lafs/tahoe-lafs/commit/466369dd6948d8fb4ecc3b9fea0c98ba6aca140d (some while ago)

Merged in <https://github.com/tahoe-lafs/tahoe-lafs/commit/466369dd6948d8fb4ecc3b9fea0c98ba6aca140d> (some while ago)

exarkun added the

fixed

label 2020-11-30 13:46:08 +00:00

exarkun closed this issue