tahoe-lafs/tahoe-lafs
tahoe-lafs
/
tahoe-lafs
2
0
Fork 0
Code Issues Pull Requests 23 Packages Projects Releases 1 Wiki Activity

docs: fix tab-vs-spaces, make some CLI examples <tt>/"literal", wrap some to 

80-cols, remove spurious whitespace. Add rst2html.py rule to Makefile.
This commit is contained in:
Brian Warner 2010-10-14 23:04:18 -07:00
parent 1e76d8ac39
commit c4f8376a20
4 changed files with 337 additions and 326 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
docs/Makefile
View File

@ -14,5 +14,8 @@ images-eps: $(EPSS)
%.eps: %.svg %.eps: %.svg
inkscape --export-eps $@ $< inkscape --export-eps $@ $<
%.html: %.rst
rst2html.py $< $@
clean: clean:
rm -f *.png *.eps rm -f *.png *.eps

18
docs/backupdb.rst
View File

@ -13,10 +13,10 @@ To speed up backup operations, Tahoe maintains a small database known as the
"backupdb". This is used to avoid re-uploading files which have already been "backupdb". This is used to avoid re-uploading files which have already been
uploaded recently. uploaded recently.
This database lives in ~/.tahoe/private/backupdb.sqlite, and is a SQLite This database lives in ``~/.tahoe/private/backupdb.sqlite``, and is a SQLite
single-file database. It is used by the "tahoe backup" command. In the future, single-file database. It is used by the "tahoe backup" command. In the
it will also be used by "tahoe mirror", and by "tahoe cp" when the future, it will also be used by "tahoe mirror", and by "tahoe cp" when the
--use-backupdb option is included. ``--use-backupdb`` option is included.
The purpose of this database is twofold: to manage the file-to-cap The purpose of this database is twofold: to manage the file-to-cap
translation (the "upload" step) and the directory-to-cap translation (the translation (the "upload" step) and the directory-to-cap translation (the
@ -121,9 +121,9 @@ If ctime, mtime, or size is different, the client will upload the file, as
above. above.
If these identifiers are the same, the client will assume that the file is If these identifiers are the same, the client will assume that the file is
unchanged (unless the --ignore-timestamps option is provided, in which case unchanged (unless the ``--ignore-timestamps`` option is provided, in which
the client always re-uploads the file), and it may be allowed to skip the case the client always re-uploads the file), and it may be allowed to skip
upload. For safety, however, we require the client periodically perform a the upload. For safety, however, we require the client periodically perform a
filecheck on these probably-already-uploaded files, and re-upload anything filecheck on these probably-already-uploaded files, and re-upload anything
that doesn't look healthy. The client looks the fileid up in the that doesn't look healthy. The client looks the fileid up in the
'last_checked' table, to see how long it has been since the file was last 'last_checked' table, to see how long it has been since the file was last
@ -151,8 +151,8 @@ checked and found healthy, the 'last_upload' entry is updated.
Relying upon timestamps is a compromise between efficiency and safety: a file Relying upon timestamps is a compromise between efficiency and safety: a file
which is modified without changing the timestamp or size will be treated as which is modified without changing the timestamp or size will be treated as
unmodified, and the "tahoe backup" command will not copy the new contents unmodified, and the "tahoe backup" command will not copy the new contents
into the grid. The --no-timestamps can be used to disable this optimization, into the grid. The ``--no-timestamps`` can be used to disable this
forcing every byte of the file to be hashed and encoded. optimization, forcing every byte of the file to be hashed and encoded.
Directory Operations Directory Operations
==================== ====================

504
docs/configuration.rst
View File

@ -66,64 +66,66 @@ set the tub.location option described below.
nickname = (UTF-8 string, optional) nickname = (UTF-8 string, optional)
This value will be displayed in management tools as this node's "nickname". This value will be displayed in management tools as this node's
If not provided, the nickname will be set to "<unspecified>". This string "nickname". If not provided, the nickname will be set to "<unspecified>".
shall be a UTF-8 encoded unicode string. This string shall be a UTF-8 encoded unicode string.
web.port = (strports string, optional) web.port = (strports string, optional)
This controls where the node's webserver should listen, providing filesystem This controls where the node's webserver should listen, providing
access and node status as defined in webapi.txt . This file contains a filesystem access and node status as defined in webapi.txt . This file
Twisted "strports" specification such as "3456" or contains a Twisted "strports" specification such as "3456" or
"tcp:3456:interface=127.0.0.1". The 'tahoe create-node' or 'tahoe create-client' "tcp:3456:interface=127.0.0.1". The 'tahoe create-node' or 'tahoe
commands set the web.port to "tcp:3456:interface=127.0.0.1" by default; this create-client' commands set the web.port to
is overridable by the "--webport" option. You can make it use SSL by writing "tcp:3456:interface=127.0.0.1" by default; this is overridable by the
"--webport" option. You can make it use SSL by writing
"ssl:3456:privateKey=mykey.pem:certKey=cert.pem" instead. "ssl:3456:privateKey=mykey.pem:certKey=cert.pem" instead.
If this is not provided, the node will not run a web server. If this is not provided, the node will not run a web server.
web.static = (string, optional) web.static = (string, optional)
This controls where the /static portion of the URL space is served. The This controls where the /static portion of the URL space is served. The
value is a directory name (~username is allowed, and non-absolute names are value is a directory name (~username is allowed, and non-absolute names
interpreted relative to the node's basedir) which can contain HTML and other are interpreted relative to the node's basedir) which can contain HTML
files. This can be used to serve a javascript-based frontend to the Tahoe and other files. This can be used to serve a javascript-based frontend to
node, or other services. the Tahoe node, or other services.
The default value is "public_html", which will serve $BASEDIR/public_html . The default value is "public_html", which will serve $BASEDIR/public_html .
With the default settings, http://127.0.0.1:3456/static/foo.html will serve With the default settings, http://127.0.0.1:3456/static/foo.html will
the contents of $BASEDIR/public_html/foo.html . serve the contents of $BASEDIR/public_html/foo.html .
tub.port = (integer, optional) tub.port = (integer, optional)
This controls which port the node uses to accept Foolscap connections from This controls which port the node uses to accept Foolscap connections
other nodes. If not provided, the node will ask the kernel for any available from other nodes. If not provided, the node will ask the kernel for any
port. The port will be written to a separate file (named client.port or available port. The port will be written to a separate file (named
introducer.port), so that subsequent runs will re-use the same port. client.port or introducer.port), so that subsequent runs will re-use the
same port.
tub.location = (string, optional) tub.location = (string, optional)
In addition to running as a client, each Tahoe node also runs as a server, In addition to running as a client, each Tahoe node also runs as a
listening for connections from other Tahoe clients. The node announces its server, listening for connections from other Tahoe clients. The node
location by publishing a "FURL" (a string with some connection hints) to the announces its location by publishing a "FURL" (a string with some
Introducer. The string it publishes can be found in connection hints) to the Introducer. The string it publishes can be found
$BASEDIR/private/storage.furl . The "tub.location" configuration controls in $BASEDIR/private/storage.furl . The "tub.location" configuration
what location is published in this announcement. controls what location is published in this announcement.
If you don't provide tub.location, the node will try to figure out a useful If you don't provide tub.location, the node will try to figure out a
one by itself, by using tools like 'ifconfig' to determine the set of IP useful one by itself, by using tools like 'ifconfig' to determine the set
addresses on which it can be reached from nodes both near and far. It will of IP addresses on which it can be reached from nodes both near and far.
also include the TCP port number on which it is listening (either the one It will also include the TCP port number on which it is listening (either
specified by tub.port, or whichever port was assigned by the kernel when the one specified by tub.port, or whichever port was assigned by the
tub.port is left unspecified). kernel when tub.port is left unspecified).
You might want to override this value if your node lives behind a firewall You might want to override this value if your node lives behind a
that is doing inbound port forwarding, or if you are using other proxies firewall that is doing inbound port forwarding, or if you are using other
such that the local IP address or port number is not the same one that proxies such that the local IP address or port number is not the same one
remote clients should use to connect. You might also want to control this that remote clients should use to connect. You might also want to control
when using a Tor proxy to avoid revealing your actual IP address through the this when using a Tor proxy to avoid revealing your actual IP address
Introducer announcement. through the Introducer announcement.
The value is a comma-separated string of host:port location hints, like The value is a comma-separated string of host:port location hints, like
this: this:
@ -131,100 +133,103 @@ set the tub.location option described below.
A few examples: A few examples:
Emulate default behavior, assuming your host has IP address 123.45.67.89 Emulate default behavior, assuming your host has IP address
and the kernel-allocated port number was 8098: 123.45.67.89 and the kernel-allocated port number was 8098:
tub.port = 8098 tub.port = 8098
tub.location = 123.45.67.89:8098,127.0.0.1:8098 tub.location = 123.45.67.89:8098,127.0.0.1:8098
Use a DNS name so you can change the IP address more easily: Use a DNS name so you can change the IP address more easily:
tub.port = 8098 tub.port = 8098
tub.location = tahoe.example.com:8098 tub.location = tahoe.example.com:8098
Run a node behind a firewall (which has an external IP address) that has Run a node behind a firewall (which has an external IP address) that
been configured to forward port 7912 to our internal node's port 8098: has been configured to forward port 7912 to our internal node's port
8098:
tub.port = 8098 tub.port = 8098
tub.location = external-firewall.example.com:7912 tub.location = external-firewall.example.com:7912
Run a node behind a Tor proxy (perhaps via torsocks), in client-only mode Run a node behind a Tor proxy (perhaps via torsocks), in client-only
(i.e. we can make outbound connections, but other nodes will not be able to mode (i.e. we can make outbound connections, but other nodes will not
connect to us). The literal 'unreachable.example.org' will not resolve, but be able to connect to us). The literal 'unreachable.example.org' will
will serve as a reminder to human observers that this node cannot be not resolve, but will serve as a reminder to human observers that this
reached. "Don't call us.. we'll call you": node cannot be reached. "Don't call us.. we'll call you":
tub.port = 8098 tub.port = 8098
tub.location = unreachable.example.org:0 tub.location = unreachable.example.org:0
Run a node behind a Tor proxy, and make the server available as a Tor Run a node behind a Tor proxy, and make the server available as a Tor
"hidden service". (this assumes that other clients are running their node "hidden service". (this assumes that other clients are running their
with torsocks, such that they are prepared to connect to a .onion address). node with torsocks, such that they are prepared to connect to a .onion
The hidden service must first be configured in Tor, by giving it a local address). The hidden service must first be configured in Tor, by giving
port number and then obtaining a .onion name, using something in the torrc it a local port number and then obtaining a .onion name, using
file like: something in the torrc file like:
HiddenServiceDir /var/lib/tor/hidden_services/tahoe HiddenServiceDir /var/lib/tor/hidden_services/tahoe
HiddenServicePort 29212 127.0.0.1:8098 HiddenServicePort 29212 127.0.0.1:8098
once Tor is restarted, the .onion hostname will be in once Tor is restarted, the .onion hostname will be in
/var/lib/tor/hidden_services/tahoe/hostname . Then set up your tahoe.cfg /var/lib/tor/hidden_services/tahoe/hostname . Then set up your
like: tahoe.cfg like:
tub.port = 8098 tub.port = 8098
tub.location = ualhejtq2p7ohfbb.onion:29212 tub.location = ualhejtq2p7ohfbb.onion:29212
Most users will not need to set tub.location . Most users will not need to set tub.location .
Note that the old 'advertised_ip_addresses' file from earlier releases is no Note that the old 'advertised_ip_addresses' file from earlier releases is
longer supported. Tahoe 1.3.0 and later will ignore this file. no longer supported. Tahoe 1.3.0 and later will ignore this file.
log_gatherer.furl = (FURL, optional) log_gatherer.furl = (FURL, optional)
If provided, this contains a single FURL string which is used to contact a If provided, this contains a single FURL string which is used to contact
'log gatherer', which will be granted access to the logport. This can be a 'log gatherer', which will be granted access to the logport. This can
used by centralized storage meshes to gather operational logs in a single be used by centralized storage meshes to gather operational logs in a
place. Note that when an old-style BASEDIR/log_gatherer.furl file exists single place. Note that when an old-style BASEDIR/log_gatherer.furl file
(see 'Backwards Compatibility Files', below), both are used. (for most other exists (see 'Backwards Compatibility Files', below), both are used. (for
items, the separate config file overrides the entry in tahoe.cfg) most other items, the separate config file overrides the entry in
tahoe.cfg)
timeout.keepalive = (integer in seconds, optional) timeout.keepalive = (integer in seconds, optional)
timeout.disconnect = (integer in seconds, optional) timeout.disconnect = (integer in seconds, optional)
If timeout.keepalive is provided, it is treated as an integral number of If timeout.keepalive is provided, it is treated as an integral number of
seconds, and sets the Foolscap "keepalive timer" to that value. For each seconds, and sets the Foolscap "keepalive timer" to that value. For each
connection to another node, if nothing has been heard for a while, we will connection to another node, if nothing has been heard for a while, we
attempt to provoke the other end into saying something. The duration of will attempt to provoke the other end into saying something. The duration
silence that passes before sending the PING will be between KT and 2*KT. of silence that passes before sending the PING will be between KT and
This is mainly intended to keep NAT boxes from expiring idle TCP sessions, 2*KT. This is mainly intended to keep NAT boxes from expiring idle TCP
but also gives TCP's long-duration keepalive/disconnect timers some traffic sessions, but also gives TCP's long-duration keepalive/disconnect timers
to work with. The default value is 240 (i.e. 4 minutes). some traffic to work with. The default value is 240 (i.e. 4 minutes).
If timeout.disconnect is provided, this is treated as an integral number of If timeout.disconnect is provided, this is treated as an integral number
seconds, and sets the Foolscap "disconnect timer" to that value. For each of seconds, and sets the Foolscap "disconnect timer" to that value. For
connection to another node, if nothing has been heard for a while, we will each connection to another node, if nothing has been heard for a while,
drop the connection. The duration of silence that passes before dropping the we will drop the connection. The duration of silence that passes before
connection will be between DT-2*KT and 2*DT+2*KT (please see ticket #521 for dropping the connection will be between DT-2*KT and 2*DT+2*KT (please see
more details). If we are sending a large amount of data to the other end ticket #521 for more details). If we are sending a large amount of data
(which takes more than DT-2*KT to deliver), we might incorrectly drop the to the other end (which takes more than DT-2*KT to deliver), we might
connection. The default behavior (when this value is not provided) is to incorrectly drop the connection. The default behavior (when this value is
disable the disconnect timer. not provided) is to disable the disconnect timer.
See ticket #521 for a discussion of how to pick these timeout values. Using See ticket #521 for a discussion of how to pick these timeout values.
30 minutes means we'll disconnect after 22 to 68 minutes of inactivity. Using 30 minutes means we'll disconnect after 22 to 68 minutes of
Receiving data will reset this timeout, however if we have more than 22min inactivity. Receiving data will reset this timeout, however if we have
of data in the outbound queue (such as 800kB in two pipelined segments of 10 more than 22min of data in the outbound queue (such as 800kB in two
shares each) and the far end has no need to contact us, our ping might be pipelined segments of 10 shares each) and the far end has no need to
delayed, so we may disconnect them by accident. contact us, our ping might be delayed, so we may disconnect them by
accident.
ssh.port = (strports string, optional) ssh.port = (strports string, optional)
ssh.authorized_keys_file = (filename, optional) ssh.authorized_keys_file = (filename, optional)
This enables an SSH-based interactive Python shell, which can be used to This enables an SSH-based interactive Python shell, which can be used to
inspect the internal state of the node, for debugging. To cause the node to inspect the internal state of the node, for debugging. To cause the node
accept SSH connections on port 8022 from the same keys as the rest of your to accept SSH connections on port 8022 from the same keys as the rest of
account, use: your account, use:
[tub] [tub]
ssh.port = 8022 ssh.port = 8022
ssh.authorized_keys_file = ~/.ssh/authorized_keys ssh.authorized_keys_file = ~/.ssh/authorized_keys
@ -233,13 +238,13 @@ set the tub.location option described below.
This specifies a temporary directory for the webapi server to use, for This specifies a temporary directory for the webapi server to use, for
holding large files while they are being uploaded. If a webapi client holding large files while they are being uploaded. If a webapi client
attempts to upload a 10GB file, this tempdir will need to have at least 10GB attempts to upload a 10GB file, this tempdir will need to have at least
available for the upload to complete. 10GB available for the upload to complete.
The default value is the "tmp" directory in the node's base directory (i.e. The default value is the "tmp" directory in the node's base directory
$NODEDIR/tmp), but it can be placed elsewhere. This directory is used for (i.e. $NODEDIR/tmp), but it can be placed elsewhere. This directory is
files that usually (on a unix system) go into /tmp . The string will be used for files that usually (on a unix system) go into /tmp . The string
interpreted relative to the node's base directory. will be interpreted relative to the node's base directory.
Client Configuration Client Configuration
==================== ====================
@ -248,67 +253,68 @@ Client Configuration
[client] [client]
introducer.furl = (FURL string, mandatory) introducer.furl = (FURL string, mandatory)
This FURL tells the client how to connect to the introducer. Each Tahoe grid This FURL tells the client how to connect to the introducer. Each Tahoe
is defined by an introducer. The introducer's furl is created by the grid is defined by an introducer. The introducer's furl is created by the
introducer node and written into its base directory when it starts, introducer node and written into its base directory when it starts,
whereupon it should be published to everyone who wishes to attach a client whereupon it should be published to everyone who wishes to attach a
to that grid client to that grid
helper.furl = (FURL string, optional) helper.furl = (FURL string, optional)
If provided, the node will attempt to connect to and use the given helper If provided, the node will attempt to connect to and use the given helper
for uploads. See docs/helper.txt for details. for uploads. See docs/helper.txt for details.
key_generator.furl = (FURL string, optional) key_generator.furl = (FURL string, optional)
If provided, the node will attempt to connect to and use the given If provided, the node will attempt to connect to and use the given
key-generator service, using RSA keys from the external process rather than key-generator service, using RSA keys from the external process rather
generating its own. than generating its own.
stats_gatherer.furl = (FURL string, optional) stats_gatherer.furl = (FURL string, optional)
If provided, the node will connect to the given stats gatherer and provide If provided, the node will connect to the given stats gatherer and
it with operational statistics. provide it with operational statistics.
shares.needed = (int, optional) aka "k", default 3 shares.needed = (int, optional) aka "k", default 3
shares.total = (int, optional) aka "N", N >= k, default 10 shares.total = (int, optional) aka "N", N >= k, default 10
shares.happy = (int, optional) 1 <= happy <= N, default 7 shares.happy = (int, optional) 1 <= happy <= N, default 7
These three values set the default encoding parameters. Each time a new file These three values set the default encoding parameters. Each time a new
is uploaded, erasure-coding is used to break the ciphertext into separate file is uploaded, erasure-coding is used to break the ciphertext into
pieces. There will be "N" (i.e. shares.total) pieces created, and the file separate pieces. There will be "N" (i.e. shares.total) pieces created,
will be recoverable if any "k" (i.e. shares.needed) pieces are retrieved. and the file will be recoverable if any "k" (i.e. shares.needed) pieces
The default values are 3-of-10 (i.e. shares.needed = 3, shares.total = 10). are retrieved. The default values are 3-of-10 (i.e. shares.needed = 3,
Setting k to 1 is equivalent to simple replication (uploading N copies of shares.total = 10). Setting k to 1 is equivalent to simple replication
the file). (uploading N copies of the file).
These values control the tradeoff between storage overhead, performance, and These values control the tradeoff between storage overhead, performance,
reliability. To a first approximation, a 1MB file will use (1MB*N/k) of and reliability. To a first approximation, a 1MB file will use (1MB*N/k)
backend storage space (the actual value will be a bit more, because of other of backend storage space (the actual value will be a bit more, because of
forms of overhead). Up to N-k shares can be lost before the file becomes other forms of overhead). Up to N-k shares can be lost before the file
unrecoverable, so assuming there are at least N servers, up to N-k servers becomes unrecoverable, so assuming there are at least N servers, up to
can be offline without losing the file. So large N/k ratios are more N-k servers can be offline without losing the file. So large N/k ratios
reliable, and small N/k ratios use less disk space. Clearly, k must never be are more reliable, and small N/k ratios use less disk space. Clearly, k
smaller than N. must never be smaller than N.
Large values of N will slow down upload operations slightly, since more Large values of N will slow down upload operations slightly, since more
servers must be involved, and will slightly increase storage overhead due to servers must be involved, and will slightly increase storage overhead due
the hash trees that are created. Large values of k will cause downloads to to the hash trees that are created. Large values of k will cause
be marginally slower, because more servers must be involved. N cannot be downloads to be marginally slower, because more servers must be involved.
larger than 256, because of the 8-bit erasure-coding algorithm that Tahoe N cannot be larger than 256, because of the 8-bit erasure-coding
uses. algorithm that Tahoe uses.
shares.happy allows you control over the distribution of your immutable file. shares.happy allows you control over the distribution of your immutable
For a successful upload, shares are guaranteed to be initially placed on file. For a successful upload, shares are guaranteed to be initially
at least 'shares.happy' distinct servers, the correct functioning of any placed on at least 'shares.happy' distinct servers, the correct
k of which is sufficient to guarantee the availability of the uploaded file. functioning of any k of which is sufficient to guarantee the availability
This value should not be larger than the number of servers on your grid. of the uploaded file. This value should not be larger than the number of
servers on your grid.
A value of shares.happy <= k is allowed, but does not provide any redundancy
if some servers fail or lose shares. A value of shares.happy <= k is allowed, but does not provide any
redundancy if some servers fail or lose shares.
(Mutable files use a different share placement algorithm that does not
(Mutable files use a different share placement algorithm that does not
consider this parameter.) consider this parameter.)
@ -319,45 +325,47 @@ Storage Server Configuration
[storage] [storage]
enabled = (boolean, optional) enabled = (boolean, optional)
If this is True, the node will run a storage server, offering space to other If this is True, the node will run a storage server, offering space to
clients. If it is False, the node will not run a storage server, meaning other clients. If it is False, the node will not run a storage server,
that no shares will be stored on this node. Use False this for clients who meaning that no shares will be stored on this node. Use False this for
do not wish to provide storage service. The default value is True. clients who do not wish to provide storage service. The default value is
True.
readonly = (boolean, optional) readonly = (boolean, optional)
If True, the node will run a storage server but will not accept any shares, If True, the node will run a storage server but will not accept any
making it effectively read-only. Use this for storage servers which are shares, making it effectively read-only. Use this for storage servers
being decommissioned: the storage/ directory could be mounted read-only, which are being decommissioned: the storage/ directory could be mounted
while shares are moved to other servers. Note that this currently only read-only, while shares are moved to other servers. Note that this
affects immutable shares. Mutable shares (used for directories) will be currently only affects immutable shares. Mutable shares (used for
written and modified anyway. See ticket #390 for the current status of this directories) will be written and modified anyway. See ticket #390 for the
bug. The default value is False. current status of this bug. The default value is False.
reserved_space = (str, optional) reserved_space = (str, optional)
If provided, this value defines how much disk space is reserved: the storage If provided, this value defines how much disk space is reserved: the
server will not accept any share which causes the amount of free disk space storage server will not accept any share which causes the amount of free
to drop below this value. (The free space is measured by a call to statvfs(2) disk space to drop below this value. (The free space is measured by a
on Unix, or GetDiskFreeSpaceEx on Windows, and is the space available to the call to statvfs(2) on Unix, or GetDiskFreeSpaceEx on Windows, and is the
user account under which the storage server runs.) space available to the user account under which the storage server runs.)
This string contains a number, with an optional case-insensitive scale This string contains a number, with an optional case-insensitive scale
suffix like "K" or "M" or "G", and an optional "B" or "iB" suffix. So suffix like "K" or "M" or "G", and an optional "B" or "iB" suffix. So
"100MB", "100M", "100000000B", "100000000", and "100000kb" all mean the same "100MB", "100M", "100000000B", "100000000", and "100000kb" all mean the
thing. Likewise, "1MiB", "1024KiB", and "1048576B" all mean the same thing. same thing. Likewise, "1MiB", "1024KiB", and "1048576B" all mean the same
thing.
expire.enabled = expire.enabled =
expire.mode = expire.mode =
expire.override_lease_duration = expire.override_lease_duration =
expire.cutoff_date = expire.cutoff_date =
expire.immutable = expire.immutable =
expire.mutable = expire.mutable =
These settings control garbage-collection, in which the server will delete These settings control garbage-collection, in which the server will
shares that no longer have an up-to-date lease on them. Please see the delete shares that no longer have an up-to-date lease on them. Please see
neighboring "garbage-collection.txt" document for full details. the neighboring "garbage-collection.rst" document for full details.
Running A Helper Running A Helper
@ -370,12 +378,12 @@ service.
[helper] [helper]
enabled = (boolean, optional) enabled = (boolean, optional)
If True, the node will run a helper (see docs/helper.txt for details). The If True, the node will run a helper (see docs/helper.txt for details).
helper's contact FURL will be placed in private/helper.furl, from which it The helper's contact FURL will be placed in private/helper.furl, from
can be copied to any clients which wish to use it. Clearly nodes should not which it can be copied to any clients which wish to use it. Clearly nodes
both run a helper and attempt to use one: do not create both helper.furl and should not both run a helper and attempt to use one: do not create both
run_helper in the same node. The default is False. helper.furl and run_helper in the same node. The default is False.
Running An Introducer Running An Introducer
@ -384,8 +392,7 @@ Running An Introducer
The introducer node uses a different '.tac' file (named introducer.tac), and The introducer node uses a different '.tac' file (named introducer.tac), and
pays attention to the "[node]" section, but not the others. pays attention to the "[node]" section, but not the others.
The Introducer node maintains some different state than regular client The Introducer node maintains some different state than regular client nodes.
nodes.
BASEDIR/introducer.furl : This is generated the first time the introducer BASEDIR/introducer.furl : This is generated the first time the introducer
node is started, and used again on subsequent runs, to give the introduction node is started, and used again on subsequent runs, to give the introduction
@ -415,53 +422,51 @@ private/node.pem
other nodes. other nodes.
storage/ storage/
Nodes which host StorageServers will create this directory to hold Nodes which host StorageServers will create this directory to hold shares
shares of files on behalf of other clients. There will be a directory of files on behalf of other clients. There will be a directory underneath
underneath it for each StorageIndex for which this node is holding shares. it for each StorageIndex for which this node is holding shares. There is
There is also an "incoming" directory where partially-completed shares are also an "incoming" directory where partially-completed shares are held
held while they are being received. while they are being received.
client.tac client.tac
this file defines the client, by constructing the actual Client this file defines the client, by constructing the actual Client instance
instance each time the node is started. It is used by the 'twistd' each time the node is started. It is used by the 'twistd' daemonization
daemonization program (in the "-y" mode), which is run internally by the program (in the "-y" mode), which is run internally by the "tahoe start"
"tahoe start" command. This file is created by the "tahoe create-node" or command. This file is created by the "tahoe create-node" or "tahoe
"tahoe create-client" commands. create-client" commands.
private/control.furl private/control.furl
this file contains a FURL that provides access to a this file contains a FURL that provides access to a control port on the
control port on the client node, from which files can be uploaded and client node, from which files can be uploaded and downloaded. This file is
downloaded. This file is created with permissions that prevent anyone else created with permissions that prevent anyone else from reading it (on
from reading it (on operating systems that support such a concept), to insure operating systems that support such a concept), to insure that only the
that only the owner of the client node can use this feature. This port is owner of the client node can use this feature. This port is intended for
intended for debugging and testing use. debugging and testing use.
private/logport.furl private/logport.furl
this file contains a FURL that provides access to a this file contains a FURL that provides access to a 'log port' on the
'log port' on the client node, from which operational logs can be retrieved. client node, from which operational logs can be retrieved. Do not grant
Do not grant logport access to strangers, because occasionally secret logport access to strangers, because occasionally secret information may be
information may be placed in the logs. placed in the logs.
private/helper.furl private/helper.furl
if the node is running a helper (for use by other if the node is running a helper (for use by other clients), its contact
clients), its contact FURL will be placed here. See docs/helper.txt for more FURL will be placed here. See docs/helper.txt for more details.
details.
private/root_dir.cap (optional) private/root_dir.cap (optional)
The command-line tools will read a directory The command-line tools will read a directory cap out of this file and use
cap out of this file and use it, if you don't specify a '--dir-cap' option or it, if you don't specify a '--dir-cap' option or if you specify
if you specify '--dir-cap=root'. '--dir-cap=root'.
private/convergence (automatically generated) private/convergence (automatically generated)
An added secret for encrypting An added secret for encrypting immutable files. Everyone who has this same
immutable files. Everyone who has this same string in their string in their private/convergence file encrypts their immutable files in
private/convergence file encrypts their immutable files in the same way when the same way when uploading them. This causes identical files to "converge"
uploading them. This causes identical files to "converge" -- to share the -- to share the same storage space since they have identical ciphertext --
same storage space since they have identical ciphertext -- which conserves which conserves space and optimizes upload time, but it also exposes files
space and optimizes upload time, but it also exposes files to the possibility to the possibility of a brute-force attack by people who know that string.
of a brute-force attack by people who know that string. In this attack, if In this attack, if the attacker can guess most of the contents of a file,
the attacker can guess most of the contents of a file, then they can use then they can use brute-force to learn the remaining contents.
brute-force to learn the remaining contents.
So the set of people who know your private/convergence string is the set of So the set of people who know your private/convergence string is the set of
people who converge their storage space with you when you and they upload people who converge their storage space with you when you and they upload
@ -479,20 +484,21 @@ Other files
=========== ===========
logs/ logs/
Each Tahoe node creates a directory to hold the log messages produced Each Tahoe node creates a directory to hold the log messages produced as
as the node runs. These logfiles are created and rotated by the "twistd" the node runs. These logfiles are created and rotated by the "twistd"
daemonization program, so logs/twistd.log will contain the most recent daemonization program, so logs/twistd.log will contain the most recent
messages, logs/twistd.log.1 will contain the previous ones, logs/twistd.log.2 messages, logs/twistd.log.1 will contain the previous ones,
will be older still, and so on. twistd rotates logfiles after they grow logs/twistd.log.2 will be older still, and so on. twistd rotates logfiles
beyond 1MB in size. If the space consumed by logfiles becomes troublesome, after they grow beyond 1MB in size. If the space consumed by logfiles
they should be pruned: a cron job to delete all files that were created more becomes troublesome, they should be pruned: a cron job to delete all files
than a month ago in this logs/ directory should be sufficient. that were created more than a month ago in this logs/ directory should be
sufficient.
my_nodeid my_nodeid
this is written by all nodes after startup, and contains a this is written by all nodes after startup, and contains a base32-encoded
base32-encoded (i.e. human-readable) NodeID that identifies this specific (i.e. human-readable) NodeID that identifies this specific node. This
node. This NodeID is the same string that gets displayed on the web page (in NodeID is the same string that gets displayed on the web page (in the
the "which peers am I connected to" list), and the shortened form (the first "which peers am I connected to" list), and the shortened form (the first
characters) is recorded in various log messages. characters) is recorded in various log messages.
Backwards Compatibility Files Backwards Compatibility Files
@ -558,15 +564,15 @@ these are not the default values), merely a legal one.
timeout.disconnect = 1800 timeout.disconnect = 1800
ssh.port = 8022 ssh.port = 8022
ssh.authorized_keys_file = ~/.ssh/authorized_keys ssh.authorized_keys_file = ~/.ssh/authorized_keys
[client] [client]
introducer.furl = pb://ok45ssoklj4y7eok5c3xkmj@tahoe.example:44801/ii3uumo introducer.furl = pb://ok45ssoklj4y7eok5c3xkmj@tahoe.example:44801/ii3uumo
helper.furl = pb://ggti5ssoklj4y7eok5c3xkmj@helper.tahoe.example:7054/kk8lhr helper.furl = pb://ggti5ssoklj4y7eok5c3xkmj@helper.tahoe.example:7054/kk8lhr
[storage] [storage]
enabled = True enabled = True
readonly_storage = True readonly_storage = True
sizelimit = 10000000000 sizelimit = 10000000000
[helper] [helper]
run_helper = True run_helper = True

138
docs/garbage-collection.rst
View File

@ -49,21 +49,21 @@ Client-side Renewal
If all of the files and directories which you care about are reachable from a If all of the files and directories which you care about are reachable from a
single starting point (usually referred to as a "rootcap"), and you store single starting point (usually referred to as a "rootcap"), and you store
that rootcap as an alias (via "tahoe create-alias"), then the simplest way to that rootcap as an alias (via "tahoe create-alias"), then the simplest way to
renew these leases is with the following CLI command: renew these leases is with the following CLI command::
tahoe deep-check --add-lease ALIAS: tahoe deep-check --add-lease ALIAS:
This will recursively walk every directory under the given alias and renew This will recursively walk every directory under the given alias and renew
the leases on all files and directories. (You may want to add a --repair flag the leases on all files and directories. (You may want to add a ``--repair``
to perform repair at the same time). Simply run this command once a week (or flag to perform repair at the same time). Simply run this command once a week
whatever other renewal period your grid recommends) and make sure it (or whatever other renewal period your grid recommends) and make sure it
completes successfully. As a side effect, a manifest of all unique files and completes successfully. As a side effect, a manifest of all unique files and
directories will be emitted to stdout, as well as a summary of file sizes and directories will be emitted to stdout, as well as a summary of file sizes and
counts. It may be useful to track these statistics over time. counts. It may be useful to track these statistics over time.
Note that newly uploaded files (and newly created directories) get an initial Note that newly uploaded files (and newly created directories) get an initial
lease too: the --add-lease process is only needed to ensure that all older lease too: the ``--add-lease`` process is only needed to ensure that all
objects have up-to-date leases on them. older objects have up-to-date leases on them.
For larger systems (such as a commercial grid), a separate "maintenance For larger systems (such as a commercial grid), a separate "maintenance
daemon" is under development. This daemon will acquire manifests from daemon" is under development. This daemon will acquire manifests from
@ -84,12 +84,12 @@ below) and restarting the server node.
Each lease has two parameters: a create/renew timestamp and a duration. The Each lease has two parameters: a create/renew timestamp and a duration. The
timestamp is updated when the share is first uploaded (i.e. the file or timestamp is updated when the share is first uploaded (i.e. the file or
directory is created), and updated again each time the lease is renewed (i.e. directory is created), and updated again each time the lease is renewed (i.e.
"tahoe check --add-lease" is performed). The duration is currently fixed at "``tahoe check --add-lease``" is performed). The duration is currently fixed
31 days, and the "nominal lease expiration time" is simply $duration seconds at 31 days, and the "nominal lease expiration time" is simply $duration
after the $create_renew timestamp. (In a future release of Tahoe, the client seconds after the $create_renew timestamp. (In a future release of Tahoe, the
will get to request a specific duration, and the server will accept or reject client will get to request a specific duration, and the server will accept or
the request depending upon its local configuration, so that servers can reject the request depending upon its local configuration, so that servers
achieve better control over their storage obligations). can achieve better control over their storage obligations).
The lease-expiration code has two modes of operation. The first is age-based: The lease-expiration code has two modes of operation. The first is age-based:
leases are expired when their age is greater than their duration. This is the leases are expired when their age is greater than their duration. This is the
@ -123,89 +123,91 @@ The tahoe.cfg file uses the following keys to control lease expiration::
expire.enabled = (boolean, optional) expire.enabled = (boolean, optional)
If this is True, the storage server will delete shares on which all leases If this is True, the storage server will delete shares on which all
have expired. Other controls dictate when leases are considered to have leases have expired. Other controls dictate when leases are considered to
expired. The default is False. have expired. The default is False.
expire.mode = (string, "age" or "cutoff-date", required if expiration enabled) expire.mode = (string, "age" or "cutoff-date", required if expiration enabled)
If this string is "age", the age-based expiration scheme is used, and the If this string is "age", the age-based expiration scheme is used, and the
"expire.override_lease_duration" setting can be provided to influence the "expire.override_lease_duration" setting can be provided to influence the
lease ages. If it is "cutoff-date", the absolute-date-cutoff mode is used, lease ages. If it is "cutoff-date", the absolute-date-cutoff mode is
and the "expire.cutoff_date" setting must be provided to specify the cutoff used, and the "expire.cutoff_date" setting must be provided to specify
date. The mode setting currently has no default: you must provide a value. the cutoff date. The mode setting currently has no default: you must
provide a value.
In a future release, this setting is likely to default to "age", but in this In a future release, this setting is likely to default to "age", but in
release it was deemed safer to require an explicit mode specification. this release it was deemed safer to require an explicit mode
specification.
expire.override_lease_duration = (duration string, optional) expire.override_lease_duration = (duration string, optional)
When age-based expiration is in use, a lease will be expired if its When age-based expiration is in use, a lease will be expired if its
"lease.create_renew" timestamp plus its "lease.duration" time is "lease.create_renew" timestamp plus its "lease.duration" time is
earlier/older than the current time. This key, if present, overrides the earlier/older than the current time. This key, if present, overrides the
duration value for all leases, changing the algorithm from: duration value for all leases, changing the algorithm from:
if (lease.create_renew_timestamp + lease.duration) < now: if (lease.create_renew_timestamp + lease.duration) < now:
expire_lease() expire_lease()
to: to:
if (lease.create_renew_timestamp + override_lease_duration) < now: if (lease.create_renew_timestamp + override_lease_duration) < now:
expire_lease() expire_lease()
The value of this setting is a "duration string", which is a number of days, The value of this setting is a "duration string", which is a number of
months, or years, followed by a units suffix, and optionally separated by a days, months, or years, followed by a units suffix, and optionally
space, such as one of the following: separated by a space, such as one of the following:
7days 7days
31day 31day
60 days 60 days
2mo 2mo
3 month 3 month
12 months 12 months
2years 2years
This key is meant to compensate for the fact that clients do not yet have This key is meant to compensate for the fact that clients do not yet have
the ability to ask for leases that last longer than 31 days. A grid which the ability to ask for leases that last longer than 31 days. A grid which
wants to use faster or slower GC than a 31-day lease timer permits can use wants to use faster or slower GC than a 31-day lease timer permits can
this parameter to implement it. The current fixed 31-day lease duration use this parameter to implement it. The current fixed 31-day lease
makes the server behave as if "lease.override_lease_duration = 31days" had duration makes the server behave as if "lease.override_lease_duration =
been passed. 31days" had been passed.
This key is only valid when age-based expiration is in use (i.e. when This key is only valid when age-based expiration is in use (i.e. when
"expire.mode = age" is used). It will be rejected if cutoff-date expiration "expire.mode = age" is used). It will be rejected if cutoff-date
is in use. expiration is in use.
expire.cutoff_date = (date string, required if mode=cutoff-date) expire.cutoff_date = (date string, required if mode=cutoff-date)
When cutoff-date expiration is in use, a lease will be expired if its When cutoff-date expiration is in use, a lease will be expired if its
create/renew timestamp is older than the cutoff date. This string will be a create/renew timestamp is older than the cutoff date. This string will be
date in the following format: a date in the following format:
2009-01-16 (January 16th, 2009) 2009-01-16 (January 16th, 2009)
2008-02-02 2008-02-02
2007-12-25 2007-12-25
The actual cutoff time shall be midnight UTC at the beginning of the given The actual cutoff time shall be midnight UTC at the beginning of the
day. Lease timers should naturally be generous enough to not depend upon given day. Lease timers should naturally be generous enough to not depend
differences in timezone: there should be at least a few days between the upon differences in timezone: there should be at least a few days between
last renewal time and the cutoff date. the last renewal time and the cutoff date.
This key is only valid when cutoff-based expiration is in use (i.e. when This key is only valid when cutoff-based expiration is in use (i.e. when
"expire.mode = cutoff-date"). It will be rejected if age-based expiration is "expire.mode = cutoff-date"). It will be rejected if age-based expiration
in use. is in use.
expire.immutable = (boolean, optional) expire.immutable = (boolean, optional)
If this is False, then immutable shares will never be deleted, even if their If this is False, then immutable shares will never be deleted, even if
leases have expired. This can be used in special situations to perform GC on their leases have expired. This can be used in special situations to
mutable files but not immutable ones. The default is True. perform GC on mutable files but not immutable ones. The default is True.
expire.mutable = (boolean, optional) expire.mutable = (boolean, optional)
If this is False, then mutable shares will never be deleted, even if their If this is False, then mutable shares will never be deleted, even if
leases have expired. This can be used in special situations to perform GC on their leases have expired. This can be used in special situations to
immutable files but not mutable ones. The default is True. perform GC on immutable files but not mutable ones. The default is True.
Expiration Progress Expiration Progress
=================== ===================