tahoe-lafs/trac-2024-07-25
2
0
Issues 1.1k Wiki

remove hard limit on mutable file size #694

New Issue
Closed
opened 2009-05-04 16:27:37 +00:00 by sigmonsays · 8 comments
sigmonsays commented 2009-05-04 16:27:37 +00:00
Owner
Copy Link
<sigmonsays> the tahoe put returns this after several seconds "error, got 413 
             Request Entity Too Large"                                  [09:06] 
<sigmonsays> allmydata.interfaces.FileTooLargeError: SDMF is limited to one 
             segment, and 3500419 > 3500000                             [09:07] 
<sigmonsays> the file i'm trying to send is only 9k

---- configuration------
3 node cluster each w/ 2G (about 75% full on each)

---- [full exception]the------
12148151_1141322388.jpg
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 16  7234   16  1232    0     0   5401      0  0:00:01 --:--:--  0:00:01  5401waiting for file data on stdin..
100  7234  100  7234    0     0  22420      0 --:--:-- --:--:-- --:--:-- 63851
error, got 413 Request Entity Too Large
Traceback (most recent call last):
> File "/usr/local/src/tahoe-files/allmydata-tahoe-1.4.1/Twisted-8.1.0-py2.4-linux-x86_64.egg/twisted/internet/defer.py", line 312, in _startRunCallbacks
    self._runCallbacks()
> File "/usr/local/src/tahoe-files/allmydata-tahoe-1.4.1/Twisted-8.1.0-py2.4-linux-x86_64.egg/twisted/internet/defer.py", line 328, in _runCallbacks
    self.result = callback(self.result, *args, **kw)
> File "/usr/local/src/tahoe-files/allmydata-tahoe-1.4.1/Twisted-8.1.0-py2.4-linux-x86_64.egg/twisted/internet/defer.py", line 289, in _continue
    self.unpause()
> File "/usr/local/src/tahoe-files/allmydata-tahoe-1.4.1/Twisted-8.1.0-py2.4-linux-x86_64.egg/twisted/internet/defer.py", line 285, in unpause
    self._runCallbacks()
--- <exception caught here> ---
> File "/usr/local/src/tahoe-files/allmydata-tahoe-1.4.1/Twisted-8.1.0-py2.4-linux-x86_64.egg/twisted/internet/defer.py", line 328, in _runCallbacks
    self.result = callback(self.result, *args, **kw)
> File "/usr/local/src/tahoe-files/allmydata-tahoe-1.4.1/src/allmydata/mutable/filenode.py", line 403, in _apply
    return self._upload(new_contents, servermap)
> File "/usr/local/src/tahoe-files/allmydata-tahoe-1.4.1/src/allmydata/mutable/filenode.py", line 438, in _upload
    return p.publish(new_contents)
> File "/usr/local/src/tahoe-files/allmydata-tahoe-1.4.1/src/allmydata/mutable/publish.py", line 146, in publish
    raise [FileTooLargeError](wiki/FileTooLargeError)("SDMF is limited to one segment, and "
allmydata.interfaces.FileTooLargeError: SDMF is limited to one segment, and 3500419 > 3500000

12062355_1141263102.jpg
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 20  5998   20  1232    0     0   5319      0  0:00:01 --:--:--  0:00:01  5319waiting for file data on stdin..
100  5998  100  5998    0     0  19168      0 --:--:-- --:--:-- --:--:-- 58839
./send-to-tahoe.sh: line 19: 20708 Broken pipe             cat files.lst
``` <sigmonsays> the tahoe put returns this after several seconds "error, got 413 Request Entity Too Large" [09:06] <sigmonsays> allmydata.interfaces.FileTooLargeError: SDMF is limited to one segment, and 3500419 > 3500000 [09:07] <sigmonsays> the file i'm trying to send is only 9k ---- configuration------ 3 node cluster each w/ 2G (about 75% full on each) ---- [full exception]the------ 12148151_1141322388.jpg % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 16 7234 16 1232 0 0 5401 0 0:00:01 --:--:-- 0:00:01 5401waiting for file data on stdin.. 100 7234 100 7234 0 0 22420 0 --:--:-- --:--:-- --:--:-- 63851 error, got 413 Request Entity Too Large Traceback (most recent call last): > File "/usr/local/src/tahoe-files/allmydata-tahoe-1.4.1/Twisted-8.1.0-py2.4-linux-x86_64.egg/twisted/internet/defer.py", line 312, in _startRunCallbacks self._runCallbacks() > File "/usr/local/src/tahoe-files/allmydata-tahoe-1.4.1/Twisted-8.1.0-py2.4-linux-x86_64.egg/twisted/internet/defer.py", line 328, in _runCallbacks self.result = callback(self.result, *args, **kw) > File "/usr/local/src/tahoe-files/allmydata-tahoe-1.4.1/Twisted-8.1.0-py2.4-linux-x86_64.egg/twisted/internet/defer.py", line 289, in _continue self.unpause() > File "/usr/local/src/tahoe-files/allmydata-tahoe-1.4.1/Twisted-8.1.0-py2.4-linux-x86_64.egg/twisted/internet/defer.py", line 285, in unpause self._runCallbacks() --- <exception caught here> --- > File "/usr/local/src/tahoe-files/allmydata-tahoe-1.4.1/Twisted-8.1.0-py2.4-linux-x86_64.egg/twisted/internet/defer.py", line 328, in _runCallbacks self.result = callback(self.result, *args, **kw) > File "/usr/local/src/tahoe-files/allmydata-tahoe-1.4.1/src/allmydata/mutable/filenode.py", line 403, in _apply return self._upload(new_contents, servermap) > File "/usr/local/src/tahoe-files/allmydata-tahoe-1.4.1/src/allmydata/mutable/filenode.py", line 438, in _upload return p.publish(new_contents) > File "/usr/local/src/tahoe-files/allmydata-tahoe-1.4.1/src/allmydata/mutable/publish.py", line 146, in publish raise [FileTooLargeError](wiki/FileTooLargeError)("SDMF is limited to one segment, and " allmydata.interfaces.FileTooLargeError: SDMF is limited to one segment, and 3500419 > 3500000 12062355_1141263102.jpg % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 20 5998 20 1232 0 0 5319 0 0:00:01 --:--:-- 0:00:01 5319waiting for file data on stdin.. 100 5998 100 5998 0 0 19168 0 --:--:-- --:--:-- --:--:-- 58839 ./send-to-tahoe.sh: line 19: 20708 Broken pipe cat files.lst ```
tahoe-lafs added the
unknown
major
defect
1.4.1
 labels 2009-05-04 16:27:37 +00:00
tahoe-lafs added this to the undecided milestone 2009-05-04 16:27:37 +00:00
zooko commented 2009-05-04 16:57:16 +00:00
Copy Link

Reformatting for Trac wiki escape.

Reformatting for Trac wiki escape.
zooko commented 2009-05-04 17:15:05 +00:00
Copy Link

Thank you for the bug report, sigmonsays. There is a hardcoded limit on the maximum size of mutable files. Directories are stored inside mutable files. The directory into which you are linking your new file would grow beyond the limit by the addition of this link.

I believe the first thing to do is to remove the hardcoded limit, and I'm accordingly changing the title of this ticket to "remove hard limit on mutable file size". The line of code in question is [publish.py line 145]source:src/allmydata/mutable/publish.py@20090222233056-4233b-171d02bfd1df45fff4af7a4f64863755379e855a#L145. Someone go fix it! Just remove the MAX_SEGMENT_SIZE hardcoded parameter and all two places that it is used.

There is already a unit test in [test_mutable.py line 359]source:src/allmydata/test/test_mutable.py@20090218222301-4233b-49132283585996c7cee159d1ce2a9133bdd00aa7#L359 that makes sure that Tahoe raises a failure when you try to create a mutable file that is bigger than 3,500,000 bytes. Change that test to make sure that Tahoe doesn't raise a failure and instead that the file is created.

After that, however, you might start to learn why we put that limit in -- it is because modifying a mutable file requires downloading and re-uploading the entirety of that mutable file, and storing the entirety of it in RAM while changing it. So the more links you keep in that directory of yours, the slower it is going to be to read the directory or to change it, and the more RAM will be used.

Ultimately we need to implement efficient modification of mutable files without downloading and re-uploading the whole file -- that is the subject of #393 (mutable: implement MDMF).

In the mean-time, there are also some tickets about optimizing the CPU usage when processing large directories. Fixing these would not fix the problem that the entire directory has to be downloaded and re-uploaded, but these tickets might also be important: #327 (performance measurement of directories), #329
(dirnodes could cache encrypted/serialized entries for speed), #383 (large directories take a long time to modify), #414 (profiling on directory unpacking).

Thank you for the bug report, sigmonsays. There is a hardcoded limit on the maximum size of mutable files. Directories are stored inside mutable files. The directory into which you are linking your new file would grow beyond the limit by the addition of this link. I believe the first thing to do is to remove the hardcoded limit, and I'm accordingly changing the title of this ticket to "remove hard limit on mutable file size". The line of code in question is [publish.py line 145]source:src/allmydata/mutable/publish.py@20090222233056-4233b-171d02bfd1df45fff4af7a4f64863755379e855a#L145. Someone go fix it! Just remove the `MAX_SEGMENT_SIZE` hardcoded parameter and all two places that it is used. There is already a unit test in [test_mutable.py line 359]source:src/allmydata/test/test_mutable.py@20090218222301-4233b-49132283585996c7cee159d1ce2a9133bdd00aa7#L359 that makes sure that Tahoe raises a failure when you try to create a mutable file that is bigger than 3,500,000 bytes. Change that test to make sure that Tahoe *doesn't* raise a failure and instead that the file is created. After that, however, you might start to learn why we put that limit in -- it is because modifying a mutable file requires downloading and re-uploading the entirety of that mutable file, and storing the entirety of it in RAM while changing it. So the more links you keep in that directory of yours, the slower it is going to be to read the directory or to change it, and the more RAM will be used. Ultimately we need to implement efficient modification of mutable files without downloading and re-uploading the whole file -- that is the subject of #393 (mutable: implement MDMF). In the mean-time, there are also some tickets about optimizing the CPU usage when processing large directories. Fixing these would not fix the problem that the entire directory has to be downloaded and re-uploaded, but these tickets might also be important: #327 (performance measurement of directories), #329 (dirnodes could cache encrypted/serialized entries for speed), #383 (large directories take a long time to modify), #414 (profiling on directory unpacking).
zooko changed title from tahoe put reeturns "error, got 413 request entity too large" to remove hard limit on mutable file size 2009-05-04 17:15:05 +00:00
kevan commented 2009-06-20 02:53:41 +00:00
Author
Owner
Copy Link

Attachment 694.diff (15866 bytes) added

**Attachment** 694.diff (15866 bytes) added
694.diff
16 KiB
kevan commented 2009-06-20 02:53:53 +00:00
Author
Owner
Copy Link

I tried my hand at fixing this, and am attaching a patch. Comments?

I tried my hand at fixing this, and am attaching a patch. Comments?
zooko commented 2009-06-20 03:52:43 +00:00
Copy Link

Looks good! Please change the doc:

    # this used to be in Publish, but we removed it there. Some of the
    # tests in here still use it, though, so here it is.

to something like:

    # this used to be in Publish, but we removed the limit. Some of 
    # these tests test whether the new code correctly allows files 
    # larger than the limit

You can use darcs unrecord to undo the effect of your previous darcs record and then use darcs record again to make a new patch and attach it to this ticket. By the way, please name it with a trailing '.txt' this time.

Looks good! Please change the doc: ``` # this used to be in Publish, but we removed it there. Some of the # tests in here still use it, though, so here it is. ``` to something like: ``` # this used to be in Publish, but we removed the limit. Some of # these tests test whether the new code correctly allows files # larger than the limit ``` You can use `darcs unrecord` to undo the effect of your previous `darcs record` and then use `darcs record` again to make a new patch and attach it to this ticket. By the way, please name it with a trailing '.txt' this time.
kevan commented 2009-06-20 21:55:37 +00:00
Author
Owner
Copy Link

Attachment 694.txt (16323 bytes) added

**Attachment** 694.txt (16323 bytes) added
694.txt
16 KiB
kevan commented 2009-06-20 21:56:01 +00:00
Author
Owner
Copy Link

Done. Thanks for the feedback.

Done. Thanks for the feedback.
zooko commented 2009-06-21 05:28:47 +00:00
Copy Link

Fixed by changeset:db939750a8831c1e and changeset:efcc45951d3544ee. Thanks, Kevan!

Fixed by changeset:db939750a8831c1e and changeset:efcc45951d3544ee. Thanks, Kevan!
zooko added the
fixed
 label 2009-06-21 05:28:47 +00:00
zooko closed this issue 2009-06-21 05:28:47 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
0.2.0

   
0.3.0

   
0.4.0

   
0.5.0

   
0.5.1

   
0.6.0

   
0.6.1

   
0.7.0

   
0.8.0

   
0.9.0

   
1.0.0

   
1.1.0

   
1.10.0

   
1.10.1

   
1.10.2

   
1.10a2

   
1.11.0

   
1.12.0

   
1.12.1

   
1.13.0

   
1.14.0

   
1.15.0

   
1.15.1

   
1.2.0

   
1.3.0

   
1.4.1

   
1.5.0

   
1.6.0

   
1.6.1

   
1.7.0

   
1.7.1

   
1.7β

   
1.8.0

   
1.8.1

   
1.8.2

   
1.8.3

   
1.8β

   
1.9.0

   
1.9.0-s3branch

   
1.9.0a1

   
1.9.0a2

   
1.9.0b1

   
1.9.1

   
1.9.2

   
1.9.2a1

   
LeastAuthority.com automation

   
blocker

   
cannot reproduce

   
cloud-branch

   
code

catch-all for anything that can be fixed by writing code but isn't classified better

  
code-dirnodes

anything involving tahoe directories

  
code-encoding

immutable-file encoding format/layout, cap format, encoding/decoding algorithms

  
code-frontend

general frontend things: new user-facing interfaces, FTP/SFTP servers, FUSE-like interfaces

  
code-frontend-cli

CLI tools

  
code-frontend-ftp-sftp

   
code-frontend-magic-folder

   
code-frontend-web

webapi interfaces, user-facing "WUI" interfaces

  
code-mutable

anything involving mutable files: encoding/decoding, share format

  
code-network

Introducer, locating storage servers, establishing/maintaining connections to them

  
code-nodeadmin

node startup/shutdown, tools to create nodes, logging tools, management/monitoring tools, manhole, disk-space-limiting tools

  
code-peerselection

server selection/sorting algorithm, rebalancing code, allocation policy, swarming-download

  
code-storage

storage-server implementation, share-storage directory layout, client-to-server share transfer protocol

  
contrib

problems with non-core things that live in the source:contrib/ directory

  
critical

   
defect

   
dev-infrastructure

things that help us build Tahoe: buildbot, automated speed/memory/code-coverage tests and results-displayers, IRC channels, mailing lists, this Trac instance

  
documentation

docs items. Also use 'docs' keyword on tickets in more specific components.

  
duplicate

   
enhancement

   
fixed

   
invalid

   
major

   
minor

   
n/a

   
normal

   
operational

things involved in running a tahoe grid, external tools to monitor/maintain a grid (like "nodeadmin" but larger-scale). Was also used for issues with the allmydata.com grid.

  
packaging

setup.py script, library dependencies, binary packages, licensing, installation instructions

  
somebody else's problem

   
supercritical

   
task

   
trivial

   
unknown

Uncategorized. All tickets default to this component until someone updates it.

  
was already fixed

   
website

things about the tahoe-lafs.org website (and sometimes this Trac instance, shared with 'dev-infrastructure')

  
wontfix

   
worksforme
No Label
0.2.0
0.3.0
0.4.0
0.5.0
0.5.1
0.6.0
0.6.1
0.7.0
0.8.0
0.9.0
1.0.0
1.1.0
1.10.0
1.10.1
1.10.2
1.10a2
1.11.0
1.12.0
1.12.1
1.13.0
1.14.0
1.15.0
1.15.1
1.2.0
1.3.0
1.4.1
1.5.0
1.6.0
1.6.1
1.7.0
1.7.1
1.7β
1.8.0
1.8.1
1.8.2
1.8.3
1.8β
1.9.0
1.9.0-s3branch
1.9.0a1
1.9.0a2
1.9.0b1
1.9.1
1.9.2
1.9.2a1
LeastAuthority.com automation
blocker
cannot reproduce
cloud-branch
code
code-dirnodes
code-encoding
code-frontend
code-frontend-cli
code-frontend-ftp-sftp
code-frontend-magic-folder
code-frontend-web
code-mutable
code-network
code-nodeadmin
code-peerselection
code-storage
contrib
critical
defect
dev-infrastructure
documentation
duplicate
enhancement
fixed
invalid
major
minor
n/a
normal
operational
packaging
somebody else's problem
supercritical
task
trivial
unknown
was already fixed
website
wontfix
worksforme
Milestone
Clear milestone
No items
No Milestone
undecided
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Reference: tahoe-lafs/trac-2024-07-25#694
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?