when cryptography has random-access AES, update helper to use it #266

New Issue

warner · 2008-01-08T23:49:41Z

warner commented

2008-01-08 23:49:41 +00:00

to build the offloaded-uploader (#116), we need the ability to do AES CTR
mode at arbitrary places in the input stream. I think a second (optional)
argument to aes.AES.process() that accepts a byte offset would do the
trick. I'm imagining something like:

   def process(self, data, offset=None):
       if offset is None:
           offset = self._last_offset
       block = aes_encrypt(self._key, number_as_string(offset))
       output = xor(data, block)
       self._last_offset = offset + len(data)
       return output

(with all the appropriate joy of handling block boundaries, of course)

to build the offloaded-uploader (#116), we need the ability to do AES CTR mode at arbitrary places in the input stream. I think a second (optional) argument to `aes.AES.process()` that accepts a byte offset would do the trick. I'm imagining something like: ``` def process(self, data, offset=None): if offset is None: offset = self._last_offset block = aes_encrypt(self._key, number_as_string(offset)) output = xor(data, block) self._last_offset = offset + len(data) return output ``` (with all the appropriate joy of handling block boundaries, of course)

warner added the

labels 2008-01-08 23:49:41 +00:00

warner added this to the 0.8.0 (Allmydata 3.0 Beta) milestone 2008-01-08 23:49:41 +00:00

zooko was assigned by warner

2008-01-08 23:49:41 +00:00

warner commented

2008-01-09 01:14:34 +00:00

hmm, robk astutely points out that we might want to re-encrypt the data on
the second pass anyways, to build up the hash trees. So we can build the
offloaded uploader without this. Lowering the priority.

hmm, robk astutely points out that we might want to re-encrypt the data on the second pass anyways, to build up the hash trees. So we can build the offloaded uploader without this. Lowering the priority.

warner added

minor

and removed

major

labels 2008-01-09 01:14:34 +00:00

zooko modified the milestone from 0.8.0 (Allmydata 3.0 Beta) to undecided

2008-01-23 02:22:27 +00:00

warner commented

2008-01-24 02:00:31 +00:00

Note: once pycryptopp can do this, revisit upload.py EncryptoAnUploadable.read_encrypted, to allow it to avoid encrypting all the data that we're going to throw away (when hash_only==True).

warner commented

2008-02-14 00:02:03 +00:00

it sounds like zooko just added this feature to pycryptopp-0.3.0, right?

zooko commented

2008-02-14 03:12:07 +00:00

Yes. You can pass an "iv" argument to the constructor of an AES object now.

warner commented

2008-04-25 00:00:30 +00:00

ok, changing the ticket description to reflect that this is now an action item for the helper, rather than pycryptopp

warner removed the

code

label 2008-04-25 00:00:30 +00:00

warner changed title from ~~pycryptopp: we need random-access AES encryption~~ to update helper to take advantage of random-access AES encryption

2008-04-25 00:00:30 +00:00

zooko was unassigned by warner

2008-04-25 00:00:45 +00:00

warner self-assigned this 2008-04-25 00:00:45 +00:00

warner commented

2009-04-15 19:18:31 +00:00

hrm. Tell me about this "iv" argument: is it a string or an integer? I think I'd rather have pycryptopp be responsible for the pack-int-to-string and block-alignment issues.. it is more likely to get them right than I would.

It would also be nice to be able to pass an offset into process(), so you could create one AES object and then seek around with it (rather than creating a new AES object for every segment), but if that were a hassle I suppose I could live without it.

What happens if my segment starts on a non-AES-blocksize boundary?

Incidentally, I added pycryptopp#18 to request this feature, having forgotten about this #266 ticket.

hrm. Tell me about this "iv" argument: is it a string or an integer? I think I'd rather have pycryptopp be responsible for the pack-int-to-string and block-alignment issues.. it is more likely to get them right than I would. It would also be nice to be able to pass an offset into `process()`, so you could create one AES object and then seek around with it (rather than creating a new AES object for every segment), but if that were a hassle I suppose I could live without it. What happens if my segment starts on a non-AES-blocksize boundary? Incidentally, I added [pycryptopp#18](http://allmydata.org/trac/pycryptopp/ticket/18) to request this feature, having forgotten about this #266 ticket.

warner commented

2009-08-27 06:21:17 +00:00

As I understand it, pycryptopp's iv= argument is not sufficient for this, at least not without a huge amount of work on the tahoe side. iv= is a string, which defaults to an AES-sized block of all zeros. To process data at an arbitrary location, the application must figure out the right offset, pack that value into the IV block, handle non-multiple-of-the-blocksize shifts, and then keep track of exactly how much data you pass into process() if you want to call it more than once. Then, to actually seek() to a different location, you have to throw away that AES object and make a new one, since the AES() constructor is the only place that takes an iv= argument, not process().

Updating title to reflect this. Note that pycryptopp ticket 18 is still the feature request: this ticket is to remind us (on tahoe) that this sort of feature is blocked on pycryptopp-18. Also note that better random-access downloading (i.e. anything other than starting at the beginning of the file and fetching everything from there to the point of interest) also requires random-access AES.

As I understand it, pycryptopp's `iv=` argument is *not* sufficient for this, at least not without a huge amount of work on the tahoe side. `iv=` is a string, which defaults to an AES-sized block of all zeros. To process data at an arbitrary location, the application must figure out the right offset, pack that value into the IV block, handle non-multiple-of-the-blocksize shifts, and then keep track of exactly how much data you pass into process() if you want to call it more than once. Then, to actually seek() to a different location, you have to throw away that AES object and make a new one, since the AES() constructor is the only place that takes an `iv=` argument, not process(). Updating title to reflect this. Note that pycryptopp ticket 18 is still the feature request: this ticket is to remind us (on tahoe) that this sort of feature is blocked on pycryptopp-18. Also note that better random-access downloading (i.e. anything other than starting at the beginning of the file and fetching everything from there to the point of interest) also requires random-access AES.

warner changed title from ~~update helper to take advantage of random-access AES encryption~~ to when pycryptopp has random-access AES, update helper to use it

2009-08-27 06:21:17 +00:00

tahoe-lafs added the

code-encoding

label 2009-12-04 04:54:24 +00:00

exarkun commented

2019-07-25 13:13:50 +00:00

ticket:3031 switched Tahoe-LAFS from pycryptopp to cryptography but cryptography also doesn't offer a random-access AES so I'm modifying this ticket to be about the same problem but with cryptography.

exarkun changed title from ~~when pycryptopp has random-access AES, update helper to use it~~ to when cryptography has random-access AES, update helper to use it

2019-07-25 13:13:50 +00:00

exarkun commented

2019-07-25 13:58:02 +00:00

I filed ticket:3230 for considering an alternate solution. From #cryptography-dev there was neither a clear yes or no on whether providing a simpler API in cryptography would be sensible. The use case is slightly obscure and the functionality is already available, just with a slightly inconvenient API. However, OpenStack Swift does basically the same thing as Tahoe-LAFS:
https://github.com/openstack/swift/blob/ae36030278b244148363d4a1c6ed0c7fc7cb5204/swift/common/middleware/crypto/crypto_utils.py#L76-L84

The fact that there are two applications seemed to carry some weight.

I filed ticket:3230 for considering an alternate solution. From #cryptography-dev there was neither a clear yes or no on whether providing a simpler API in `cryptography` would be sensible. The use case is slightly obscure and the functionality is already available, just with a slightly inconvenient API. However, [OpenStack](wiki/OpenStack) Swift does basically the same thing as Tahoe-LAFS: <https://github.com/openstack/swift/blob/ae36030278b244148363d4a1c6ed0c7fc7cb5204/swift/common/middleware/crypto/crypto_utils.py#L76-L84> The fact that there are two applications seemed to carry some weight.

Sign in to join this conversation.