LiteralFileURI instance has no attribute 'storage_index' #948
Labels
No Label
0.2.0
0.3.0
0.4.0
0.5.0
0.5.1
0.6.0
0.6.1
0.7.0
0.8.0
0.9.0
1.0.0
1.1.0
1.10.0
1.10.1
1.10.2
1.10a2
1.11.0
1.12.0
1.12.1
1.13.0
1.14.0
1.15.0
1.15.1
1.2.0
1.3.0
1.4.1
1.5.0
1.6.0
1.6.1
1.7.0
1.7.1
1.7β
1.8.0
1.8.1
1.8.2
1.8.3
1.8β
1.9.0
1.9.0-s3branch
1.9.0a1
1.9.0a2
1.9.0b1
1.9.1
1.9.2
1.9.2a1
LeastAuthority.com automation
blocker
cannot reproduce
cloud-branch
code
code-dirnodes
code-encoding
code-frontend
code-frontend-cli
code-frontend-ftp-sftp
code-frontend-magic-folder
code-frontend-web
code-mutable
code-network
code-nodeadmin
code-peerselection
code-storage
contrib
critical
defect
dev-infrastructure
documentation
duplicate
enhancement
fixed
invalid
major
minor
n/a
normal
operational
packaging
somebody else's problem
supercritical
task
trivial
unknown
was already fixed
website
wontfix
worksforme
No Milestone
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Reference: tahoe-lafs/trac-2024-07-25#948
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
A daily cronjob is running 'tahoe deep-check --repair' on files which were created by 'tahoe backup'. It began to fail with the following exception sometimes around the implementation of immutable directories.
As soon as this failure occurs, the deep-check operation is aborted. This might be dangerous because the remaining files are not being correctly checked.
[get_storage_index on DirectoryNode]source:src/allmydata/dirnode.py?rev=4195#L348 is implemented like this:
which makes unwarranted assumptions about attributes of URI objects. It should be
(This delegates to [get_storage_index on _DirectoryBaseURI]source:src/allmydata/uri.py?rev=4194#L448, which delegates to [get_storage_index on LiteralFileURI]source:src/allmydata/uri.py?rev=4194#L228, which correctly returns None. The bug probably occurred as a result of trying to skip one of these delegation steps, violating abstraction.)
Please check whether this is sufficient to fix the problem -- I haven't checked whether
tahoe check
handles a storage index of None correctly, although it should if it handles LIT files.(I meant
tahoe deep-check
, nottahoe check
.)This is a regression relative to 1.5.0, since
tahoe backup
didn't use immutable directories then, and only immutable directories can wrap a LiteralFileURI.Looking at where
get_storage_index
, it seems that several other operations on empty immutable directories that are small enough to fit in a LIT file are likely to fail. Viewing them in the WUI certainly provokes the same error:We should probably put out a point release (although there's time to fix other stuff and perhaps include #778.)
In general, a storage_index of None is used to indicate a "non-distributed file", which the Checker ought to skip entirely (non-distributed files are always healthy). I'd guess that we used to have about 90% support for these (i.e. of the code that I wrote, I probably made it tolerate LITs about 90% of the time, and/or we have test coverage of this for about 90% of the code paths). But it sounds like something changed recently.
As far as I can see, we have no tests for DIR2-LIT URIs, which is why this escaped notice in the 1.6.0 release cycle.
I'm writing tests.
Zooko is writing tests for test_dirnode.py. I'll write them for test_web.py, and write the actual fix. Note that there are other cases -- lots in test code and a few in non-test code -- where we directly access
.storage_index
on an URI object rather than calling.get_storage_index()
.Attachment fix-948-darcspatch.txt (71091 bytes) added
Change direct accesses to an_uri.storage_index to calls to .get_storage_index() (fixes #948)
Attachment test-web-948-darcspatch.txt (58273 bytes) added
Additions to test_web.py for #948
Attachment all-948-darcspatch.txt (78728 bytes) added
Additional fix for abbrev_si, with test (includes previous two patches)
Attachment test-dirnode-lit.darcspatch.txt (10695 bytes) added
add tests of literal dirnodes (the current code already passes these tests) This patch replaces the previous patch by the same name, which was buggy.
We should add a test of deepcheck (where this issue was first detected). I think all we need to do is add a
DIR2:LIT
to [DeepCheckWebGood.set_up_tree()]source:src/allmydata/test/test_deepcheck.py@4164#L206.DeepCheckWebGood.set_up_tree: agreed. I also want to check the WUI output for a DIR2:LIT, which is in test_web.py.
Attachment test-dirnode-lit-2-darcspatch.txt (68099 bytes) added
dirnode: add tests of literal dirnodes (current and fix for #948)
Attachment More Info for DIR2-LIT directory.html (18550 bytes) added
Nevow traceback showing bug when following More Info link of a DIR2-LIT directory (http://127.0.0.1:3456/uri/URI%3ADIR2-LIT%3Agqytunj2onug64tufqzdcosvkjetutcjkq5gw4tvm5vwszdgnz5hgyzufqydulbshj5x2lbm/?t=info)
Attachment test-dirnode-lit-in-deep-check.darcspatch.txt (17755 bytes) added
add tests of literal dirnodes to test_deepcheck.py (the current trunk fails these tests and so does the current all-948-darcspatch.txt! Of course I'm not sure if it is the tests or the code that is wrong)
Attachment test-dirnode-lit-in-deep-check-2-darcspatch.txt (71825 bytes) added
directories: add DIR2-LIT directories to test_deepcheck.py (#948) -- fixes test bugs
Attachment additional-fixes-for-948-darcspatch.txt (59841 bytes) added
Additional fixes for DIR2-LIT More Info page and deep-check/manifest operations (#948)
The patch bundles that need review are:
This does not include the test of WUI output, which is currently failing (the functionality seems to work, so I think it's a test problem -- will investigate tomorrow).
Attachment updates-to-NEWS-darcspatch.txt (55471 bytes) added
Updates to NEWS relevant to this ticket and #577
I think we need to add to some of these tests -- at least to test_deepcheck.py -- a large immutable directory. Large enough so that it isn't a LIT.
Replying to zooko:
I agree that would be useful, but strictly speaking this ticket is about DIR2-LITs, so let's get these patches reviewed first.
Attachment test-dirnode-lit-in-wui-darcspatch.txt (57850 bytes) added
Additional test for DIR2-LIT directories in test_web.py, fixed version (#948). Corrects a mistake in the use of Deferreds.
Okay I'm reviewing
Is that the right set to review?
I reviewed test-dirnode-lit-in-wui-darcspatch.txt and it looks good.
I reviewed test-dirnode-lit-in-deep-check-2-darcspatch.txt and it is good.
Replying to zooko:
Whoops, sorry I meant updates-to-NEWS-darcspatch.txt not test-dirnode-lit-in-deep-check-2-darcspatch.txt.
Okay, now I've reviewed test-dirnode-lit-in-deep-check-2-darcspatch.txt and approved it. Technically, it would be better if someone else reviewed test-dirnode-lit-in-deep-check-2-darcspatch.txt since I'm part author of it, but Brian's not around on IRC right now and I'm hoping to announce v1.6.1 tomorrow! :-) If Kevan or anyone else is reading this and wants to read over test-dirnode-lit-in-deep-check-2-darcspatch.txt and look for goofs, that would be great. It will probably take only a few minutes, especially if you don't mind being uncertain about exactly what all the tests do and just look for things that look suspiciously like mistakes to you. If you develop a confident understanding of what the tests actually do here, all the better!
Okay, I've reviewed test-dirnode-lit-2-darcspatch.txt and I like it. Again, I'm not the perfect reviewer since I'm the author of most of this one, so if you are reading this and you can double-check that patch, that would be great.
Okay, I've reviewed all-948-darcspatch.txt and could find nothing in it to change!
Okay I'm looking at additional-fixes-for-948-darcspatch.txt, which is the last patch from the set of six patches that constitute the fix for this ticket (per comment:75407). For this one I have some questions:
(Elsewhere we use either
""
or"<LIT>"
to be the string encoding of a null storage index.)""
when a storage index isNone
, or there already unit tests that fail when those changes are not applied?base32.b2a(m.origin_si or "")
feels slightly weird to me because it feels like a type error—we really mean that the expression should evaluate to""
not that the argument tob2a()
should be""
. However, the alternative is bigger and uglier:m.origin_si and base32.b2a(m.origin_si) or ""
. Maybe we should define a function:Attachment all-darcspatch.txt (125875 bytes) added
All patches to be applied for #948
all-darcspatch.txt includes a change to avoid
base32.b2a(... or "")
, although not using a separate function. This change has been reviewed by bigpig.The changes to emit
""
when the storage index isNone
are tested bytest_deepcheck.py
. We decided not to change "" for 1.6.1.Committed as changeset:973f0afdd32285a7, changeset:d29ec184a6e67623, changeset:fec9185f2a866523, changeset:187d837c1db532f6, changeset:e6aee33bb77f6f88, changeset:d7b50a3b86acf8c6, changeset:f8e2bab41ea6c38d, changeset:a2ed17f2a0d69ec3, changeset:40edf8f419947b80.