Add filesizes to magic-folder status API output #2886

New Issue

crwood · 2017-06-22T05:41:50Z

crwood commented

2017-06-22 05:41:50 +00:00

As an extension of #2885, it would be very useful if the magic-folder status API displayed each file's size in addition to the information it already provides. In conjunction with #2885, this would greatly facilitate the task of calculating the total overall progress and/or estimating the total remaning time of a set of magic-folder upload/download operations. Currently, applications that wish to make such calculations will need to manually query the readcaps of each individual magic-folder member (for download sizes) or get the filesize from the filesystem directly (for upload sizes), both of which add additional overhead and ideally shouldn't be necessary.

crwood added the

labels 2017-06-22 05:41:50 +00:00

crwood added this to the undecided milestone 2017-06-22 05:41:50 +00:00

meejah commented

2018-01-09 19:45:38 +00:00

Currently, the "tahoe magic-folder status" command does output file-sizes; these come from the metadata on the directory entries (i.e. "query the readcap for the dir") so you don't have to query each readcap for each file (as far as I understand).

That said, I think the only way for the "status API" to get this information would be to also read these dircaps .. so maybe it's better to leave that up to application? That is, some applications might not care about any metdata and so why should they have to wait for a bunch of dircap fetches? (Or: why not put all available metadata from the dircap into the status API?). For the upload sizes, these could be stat()-ed locally by the API so wouldn't put additional overhead.

Re-reading the "status" code in magic-folder, it fetches 3 URLs up front and can then display all the data (the dircaps for "upload_dircap" and "collective_dircap") .. oh and then also has to grab the dircap for each 'other' participant in the "collective_dircap" list. So it reads 3 URLs, and then one more for each participant in the magic-folder (and then has all metadata for all files that anyone has put into the magic-folder).

My conclusion: it would be pretty low-overhead to "do the stat call for you" and include file-sizes for uploads; for downloads that incurs additional dircap fetches so it might be best to leave that up to the application?

Currently, the "tahoe magic-folder status" command does output file-sizes; these come from the metadata on the directory entries (i.e. "query the readcap for the dir") so you don't have to query each readcap for each file (as far as I understand). That said, I think the only way for the "status API" to get this information would be to also read these dircaps .. so maybe it's better to leave that up to application? That is, some applications might not care about any metdata and so why should they have to wait for a bunch of dircap fetches? (Or: why not put *all* available metadata from the dircap into the status API?). For the upload sizes, these could be stat()-ed locally by the API so wouldn't put additional overhead. Re-reading the "status" code in magic-folder, it fetches 3 URLs up front and can then display all the data (the dircaps for "upload_dircap" and "collective_dircap") .. oh and then also has to grab the dircap for each 'other' participant in the "collective_dircap" list. So it reads 3 URLs, and then one more for each participant in the magic-folder (and then has all metadata for all files that anyone has put into the magic-folder). My conclusion: it would be pretty low-overhead to "do the stat call for you" and include file-sizes for uploads; for downloads that incurs additional dircap fetches so it might be best to leave that up to the application?

meejah commented

2018-01-09 21:55:25 +00:00

(...realized it's not just a stat call; for uploads you want to know the total size of all shares which will be uploaded -- usually a lot more than the filesize on disk)