publish-and-subscribe to change events on mutables #2555

New Issue

zooko · 2015-10-27T18:25:30Z

zooko commented

2015-10-27 18:25:30 +00:00

Currently if a client wants to keep track of changes being made by someone else to a mutable, it has to poll that mutable. In Magic Folder, for example, we poll every few seconds. Polling is terrible! It has bad latency (a few seconds) and also bad load (new requests every few seconds).

Instead we should implement a publish-and-subscribe notification system by which clients can maintain an open (but idle) connection to a server, and then get low-latency, cheap notifications from servers about events.

The storage server should provide an API which does something like this:

remote_registerWatcher(watcher, storage_index, previous_watcher=None)

Currently if a client wants to keep track of changes being made by someone else to a mutable, it has to poll that mutable. In Magic Folder, for example, we poll every few seconds. Polling is terrible! It has bad latency (a few seconds) and also bad load (new requests every few seconds). Instead we should implement a publish-and-subscribe notification system by which clients can maintain an open (but idle) connection to a server, and then get low-latency, cheap notifications from servers about events. The storage server should provide an API which does something like this: ``` remote_registerWatcher(watcher, storage_index, previous_watcher=None) ```

zooko added the

labels 2015-10-27 18:25:30 +00:00

zooko added this to the undecided milestone 2015-10-27 18:25:30 +00:00

zooko commented

2015-10-27 18:27:32 +00:00

Passing the previous_watcher to the storage server is just a way to let the storage server free up resources by ceasing to send notifications to that old one. The client does not rely on the storage server's behavior with regard to that! The client implements a purely client-side ordering guarantee in which it synchronously disables the old watcher (the old watcher is effectively a revocable forwarder) before it sends this request to the server and gives the server access to the new watcher.

Passing the `previous_watcher` to the storage server is just a way to let the storage server free up resources by ceasing to send notifications to that old one. The client does *not* rely on the storage server's behavior with regard to that! The client implements a purely client-side ordering guarantee in which it synchronously disables the old watcher (the old watcher is effectively a revocable forwarder) before it sends this request to the server and gives the server access to the new watcher.

warner commented

2015-10-27 19:24:21 +00:00

To be specific:

registerWatch would be a method of RIStorageServer, next to get_buckets() and slot_readv()
the watcher would be called with no arguments, or maybe just the storage index as an argument: in particular, it would not be called with any contents of the file. It's purely a "something has changed, you might want to come back and read it soon" notification.
Daira pointed out that calling it with some contents of the file might let us optimize out a roundtrip from the subsequent mapupdate/read operation. To avoid making the server know about how mutable shares are arranged, we'd have to specify a range or a read-vector of which parts of the share to deliver along with the "something has changed" notification
I countered that it would take some difficult code changes to take advantage of that. Mutable reads do two share reads: one for the "mapupdate" phase (which reads at least the signed roothash/seqnum, and might optimistically read and cache a little bit more), then a second that reads the main data. To remove one of these, we'd need to change the mutable downloader to accept a blob of data from outside the usual control flow.
Also, really the notification would need to deliver the entire contents of the file, since there are almost never situations where some data has changed, but then you don't end up reading the entire share
we're trying to avoid using notifyOnDisconnect here
the idea is that every 5 minutes we do two things:
establish a watcher
once established, read the share
replacing the watcher with a new one is just to avoid needing to pay attention to whether the old one is still working or not

We think it'd be a good idea to require that the mutable share exists before a caller can establish a watcher on it. Ideally we'd require that the caller proves knowledge of a readcap before allowing them to add the watcher, but our share formats don't really make that possible. Also maybe the API should let you watch multiple storage indicies at once. But maybe not.

At the next higher level up, I think we'll add a IMutableFileNode.watch(callback) object, and maybe unwatch, which will do a mapupdate of the mutable file, locate all the servers that currently have shares, then register a watcher on each one. Actually it should probably establish a loop which does:

does a mapupdate on the file
adds a watcher on each share that was found
read the file contents
wait 5 minutes
re-establish the watchers
re-read the file contents
wait, etc

Basically we want a max-5-minutes-latency fallback in case the watcher connection fails, without trying to react to Foolscap's (unreliable / racy) notifyOnDisconnect feature (which we're trying to remove).

Also, note that when we move to an HTTP-based storage server, this will probably be implemented with an EventSource or WebSocket. The IMutableFileNode API should hide the details.

To be specific: * registerWatch would be a method of `RIStorageServer`, next to `get_buckets()` and `slot_readv()` * the watcher would be called with no arguments, or maybe just the storage index as an argument: in particular, it would *not* be called with any contents of the file. It's purely a "something has changed, you might want to come back and read it soon" notification. * Daira pointed out that calling it with *some* contents of the file might let us optimize out a roundtrip from the subsequent mapupdate/read operation. To avoid making the server know about how mutable shares are arranged, we'd have to specify a range or a read-vector of which parts of the share to deliver along with the "something has changed" notification * I countered that it would take some difficult code changes to take advantage of that. Mutable reads do two share reads: one for the "mapupdate" phase (which reads at least the signed roothash/seqnum, and might optimistically read and cache a little bit more), then a second that reads the main data. To remove one of these, we'd need to change the mutable downloader to accept a blob of data from outside the usual control flow. * Also, really the notification would need to deliver the entire contents of the file, since there are almost never situations where some data has changed, but then you *don't* end up reading the entire share * we're trying to avoid using `notifyOnDisconnect` here * the idea is that every 5 minutes we do two things: * establish a watcher * once established, read the share * replacing the watcher with a new one is just to avoid needing to pay attention to whether the old one is still working or not We think it'd be a good idea to require that the mutable share exists before a caller can establish a watcher on it. Ideally we'd require that the caller proves knowledge of a readcap before allowing them to add the watcher, but our share formats don't really make that possible. Also maybe the API should let you watch multiple storage indicies at once. But maybe not. At the next higher level up, I think we'll add a `IMutableFileNode.watch(callback)` object, and maybe `unwatch`, which will do a mapupdate of the mutable file, locate all the servers that currently have shares, then register a watcher on each one. Actually it should probably establish a loop which does: * does a mapupdate on the file * adds a watcher on each share that was found * read the file contents * wait 5 minutes * re-establish the watchers * re-read the file contents * wait, etc Basically we want a max-5-minutes-latency fallback in case the watcher connection fails, without trying to react to Foolscap's (unreliable / racy) `notifyOnDisconnect` feature (which we're trying to remove). Also, note that when we move to an HTTP-based storage server, this will probably be implemented with an EventSource or WebSocket. The `IMutableFileNode` API should hide the details.

warner added

code-mutable

and removed

unknown

labels 2016-09-13 17:44:40 +00:00

exarkun commented

2020-01-17 16:22:33 +00:00

There is currently a WIP PR for a WebSocket-based design - https://github.com/tahoe-lafs/tahoe-lafs/pull/513

There is currently a WIP PR for a [WebSocket](wiki/WebSocket)-based design - <https://github.com/tahoe-lafs/tahoe-lafs/pull/513>

Sign in to join this conversation.