maybe add node metadata? (in addition to edge metadata) #307

New Issue

warner · 2008-02-08T22:12:08Z

warner commented

2008-02-08 22:12:08 +00:00

At present, our only metadata is on directory edges. Files are plain untyped
sequences of bytes, directories are dictionaries of (rwcap, rocap, metadata)
tuples.

In kicking ideas around with Rob, it occurred to me that there might be value
to being able to put metadata inside directory nodes: not on the edges that
point to its children, but attached to the node itself. One use for this that
we've discussed in the past is how to coordinate writes: if you could put a
FURL in the dirnode, you could establish a convention that says "you must try
to talk to the coordination server (at this FURL) and get a write-lock before
modifying the dirnode". The example that came up yesterday was some sort of
publish-subscribe server: "I promise to announce my changes to this dirnode
through the named pubsub server each time I change it", which would allow
clients to cache the contents for a long period of time (by subscribing to
hear about changes). It isn't clear that this particular purpose would be
useful, but a generalized per-dirnode metadata structure could make it easy
to implement things like this in the future.

Likewise, per-filenode metadata could be used to transform our untyped
sequence-of-bytes into typed files, which would make certain operations
easier (like viewing files in the web interface). Different filesystems offer
different sorts of metadata.. posix extended attributes, file mtime/ctime,
Mac resource forks. Some of these are attached to the edge, some are attached
to the node.

It might be useful to define a layer on top of flat sequence-of-bytes
immutable files to provide a place for this sort of filenode metadata. It
might be useful to combine a mutable metadata file with an immutable data
file, to alow mutable metadata. Or bind two mutable files together (one for
metadata and one for the main data) in an immutable form that would represent
the same semantic unit as a traditional unix file.

There are a lot of possibilites. Some systems put information in the metadata
because it is faster to read than the main data. Some put it in the metadata
because then other applications can read it without understanding the format
of the main file. We'd have to take considerations like this into account
before actually defining anything.

At present, our only metadata is on directory edges. Files are plain untyped sequences of bytes, directories are dictionaries of (rwcap, rocap, metadata) tuples. In kicking ideas around with Rob, it occurred to me that there might be value to being able to put metadata inside directory nodes: not on the edges that point to its children, but attached to the node itself. One use for this that we've discussed in the past is how to coordinate writes: if you could put a FURL in the dirnode, you could establish a convention that says "you must try to talk to the coordination server (at this FURL) and get a write-lock before modifying the dirnode". The example that came up yesterday was some sort of publish-subscribe server: "I promise to announce my changes to this dirnode through the named pubsub server each time I change it", which would allow clients to cache the contents for a long period of time (by subscribing to hear about changes). It isn't clear that this particular purpose would be useful, but a generalized per-dirnode metadata structure could make it easy to implement things like this in the future. Likewise, per-filenode metadata could be used to transform our untyped sequence-of-bytes into typed files, which would make certain operations easier (like viewing files in the web interface). Different filesystems offer different sorts of metadata.. posix extended attributes, file mtime/ctime, Mac resource forks. Some of these are attached to the edge, some are attached to the node. It might be useful to define a layer on top of flat sequence-of-bytes immutable files to provide a place for this sort of filenode metadata. It might be useful to combine a mutable metadata file with an immutable data file, to alow mutable metadata. Or bind two mutable files together (one for metadata and one for the main data) in an immutable form that would represent the same semantic unit as a traditional unix file. There are a lot of possibilites. Some systems put information in the metadata because it is faster to read than the main data. Some put it in the metadata because then other applications can read it without understanding the format of the main file. We'd have to take considerations like this into account before actually defining anything.