tahoe-lafs/trac-2024-07-25
2
0
Issues 1.1k Wiki

Magic Folder on Mac OS X and other BSDs #1432

New Issue
Closed
opened 2011-07-17 02:13:59 +00:00 by davidsarah · 45 comments
davidsarah commented 2011-07-17 02:13:59 +00:00
Owner
Copy Link

The Magic Folder feature currently only works on Linux (where it uses inotify to detect filesystem changes), and Windows (where it uses ReadDirectoryChangesW). This ticket is about supporting the same feature on Mac OS X and possibly other BSD-based operating systems.

Mac OS X has the fsevents API, and all BSDs including Mac OS X have kqueue/kevent. If I understand correctly, neither fsevents nor kqueue/kevent give notifications of which files have changed, so the implementation will have to scan the directory and look at last-modified times to determine that. (This is occasionally also necessary for correctness on Linux and Windows; see #1430.)

The Magic Folder feature currently only works on Linux (where it uses inotify to detect filesystem changes), and Windows (where it uses `ReadDirectoryChangesW`). This ticket is about supporting the same feature on Mac OS X and possibly other BSD-based operating systems. Mac OS X has the [fsevents](http://en.wikipedia.org/wiki/FSEvents) API, and all BSDs including Mac OS X have kqueue/kevent. If I understand correctly, neither fsevents nor kqueue/kevent give notifications of which files have changed, so the implementation will have to scan the directory and look at last-modified times to determine that. (This is occasionally also necessary for correctness on Linux and Windows; see #1430.)
tahoe-lafs added the
code-frontend
major
defect
1.8.2
 labels 2011-07-17 02:13:59 +00:00
tahoe-lafs added this to the undecided milestone 2011-07-17 02:13:59 +00:00
tahoe-lafs changed title from drop-upload on Mac OS X to drop-upload on Mac OS X and other BSDs 2011-07-17 02:21:50 +00:00
tahoe-lafs added
enhancement
 and removed
defect
 labels 2011-07-25 12:26:43 +00:00
davidsarah commented 2011-08-15 19:38:42 +00:00
Author
Owner
Copy Link

Python's select module supports kqueue, which is a lot easier and less error-prone than accessing it via ctypes or an extension module. Note that a critical bug in that support was fixed in Python 2.6.5. (2.7 was released after 2.6.5 and has this fix.)

Python's [select](http://docs.python.org/library/select.html) module supports kqueue, which is a lot easier and less error-prone than accessing it via ctypes or an extension module. Note that a [critical bug](http://bugs.python.org/issue5910) in that support was fixed in Python 2.6.5. (2.7 was released after 2.6.5 and has this fix.)
tahoe-lafs added
normal
 and removed
major
 labels 2012-03-29 19:51:21 +00:00
malaparte commented 2012-11-15 06:07:59 +00:00
Author
Owner
Copy Link

I thought I'd just save this thought here, because I don't know if I have time to look at it closely. Have you considered 'watchdog' http://packages.python.org/watchdog/ to watch for changes on a folder across platforms?

I thought I'd just save this thought here, because I don't know if I have time to look at it closely. Have you considered 'watchdog' <http://packages.python.org/watchdog/> to watch for changes on a folder across platforms?
davidsarah commented 2012-11-15 23:39:14 +00:00
Author
Owner
Copy Link

watchdog uses a C extension module to access the fsevents API on OS X. I would strongly prefer to use ctypes for this (if needed, i.e. if we don't use kqueue instead) and not add any more native code dependencies.

watchdog uses a C [extension module](https://github.com/gorakhargosh/watchdog/blob/master/src/watchdog_fsevents.c) to access the fsevents API on OS X. I would strongly prefer to use ctypes for this (if needed, i.e. if we don't use kqueue instead) and not add any more native code dependencies.
davidsarah commented 2012-11-16 00:22:50 +00:00
Author
Owner
Copy Link

Also,

davidsarah@shinier:~/tahoe/git$ cloc --no3 --quiet watchdog

http://cloc.sourceforge.net v 1.09  T=4.0 s (15.0 files/s, 2563.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          36           1286           2238           4010
C                                3            168            473            738
Bourne Shell                    13             71            188            273
make                             3             46              5            193
DOS Batch                        2             27              1            172
XML                              1              0              0            155
C/C++ Header                     1             28             68             93
YAML                             1              0              0             21
-------------------------------------------------------------------------------
SUM:                            60           1626           2973           5655
-------------------------------------------------------------------------------

plus watchdog's PyYaml, argh and pathtools dependencies

compared to:

davidsarah@shinier:~/tahoe/git$ cloc --no3 --quiet trunk/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-x86_64.egg/twisted/internet/inotify.py

http://cloc.sourceforge.net v 1.09  T=0.5 s (2.0 files/s, 810.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                           1             84            140            181
-------------------------------------------------------------------------------
davidsarah@shinier:~/tahoe/git$ cloc --no3 --quiet trunk/src/allmydata/frontends/drop_upload.py

http://cloc.sourceforge.net v 1.09  T=0.5 s (2.0 files/s, 246.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                           1             25             10             88
-------------------------------------------------------------------------------

plus how ever much code it takes to support Windows (about 200-300 code lines), and BSD/OS X.

Also, ``` davidsarah@shinier:~/tahoe/git$ cloc --no3 --quiet watchdog http://cloc.sourceforge.net v 1.09 T=4.0 s (15.0 files/s, 2563.5 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Python 36 1286 2238 4010 C 3 168 473 738 Bourne Shell 13 71 188 273 make 3 46 5 193 DOS Batch 2 27 1 172 XML 1 0 0 155 C/C++ Header 1 28 68 93 YAML 1 0 0 21 ------------------------------------------------------------------------------- SUM: 60 1626 2973 5655 ------------------------------------------------------------------------------- ``` plus watchdog's [PyYaml, argh and pathtools dependencies](https://github.com/gorakhargosh/watchdog/blob/937d16b0be1495b1563353014b865e1234dacc9b/setup.py#L91) compared to: ``` davidsarah@shinier:~/tahoe/git$ cloc --no3 --quiet trunk/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-x86_64.egg/twisted/internet/inotify.py http://cloc.sourceforge.net v 1.09 T=0.5 s (2.0 files/s, 810.0 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Python 1 84 140 181 ------------------------------------------------------------------------------- davidsarah@shinier:~/tahoe/git$ cloc --no3 --quiet trunk/src/allmydata/frontends/drop_upload.py http://cloc.sourceforge.net v 1.09 T=0.5 s (2.0 files/s, 246.0 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Python 1 25 10 88 ------------------------------------------------------------------------------- ``` plus how ever much code it takes to support Windows (about [200-300 code lines](https://tahoe-lafs.org/trac/tahoe-lafs/attachment/ticket/1431/drop-upload-including-windows.darcs.patch#L788)), and BSD/OS X.
davidsarah commented 2012-11-16 00:29:27 +00:00
Author
Owner
Copy Link

I should mention that I don't care about lines of code per se; I'm just using it as a rough proxy measure for complexity and opportunity-for-bugs.

I should mention that I don't care about lines of code *per se*; I'm just using it as a rough proxy measure for complexity and opportunity-for-bugs.
warner removed the
code-frontend
 label 2014-12-02 19:48:43 +00:00
daira commented 2015-06-01 16:11:09 +00:00
Author
Owner
Copy Link

Add magic-folder keyword to all drop-upload tickets.

Add magic-folder keyword to all drop-upload tickets.
tahoe-lafs changed title from drop-upload on Mac OS X and other BSDs to Magic Folder on Mac OS X and other BSDs 2015-10-28 23:18:23 +00:00
daira commented 2015-12-04 21:14:33 +00:00
Author
Owner
Copy Link

SpiderOak has an fsevents-based OS X filesystem watcher which is open-source (GPL): https://github.com/SpiderOak/spideroak_osx_fsevents

[I did not mean that we should necessarily use this as-is.]Edit:

SpiderOak has an fsevents-based OS X filesystem watcher which is open-source (GPL): <https://github.com/SpiderOak/spideroak_osx_fsevents> [I did not mean that we should necessarily use this as-is.]Edit:
dawuud commented 2016-01-27 16:09:39 +00:00
Author
Owner
Copy Link

step number one seems to be: find an OSX VPS provider!

step number one seems to be: find an OSX VPS provider!
zooko commented 2016-04-07 14:03:34 +00:00
Copy Link

I think we should start with https://github.com/SpiderOak/spideroak_osx_fsevents and then consider changing it or switching to other techniques or whatever after we have it running.

I think we should start with <https://github.com/SpiderOak/spideroak_osx_fsevents> and then consider changing it or switching to other techniques or whatever after we have it running.
zooko commented 2016-04-07 16:50:18 +00:00
Copy Link

Okay I looked at https://github.com/SpiderOak/spideroak_osx_fsevents/blob/master/main.c and other code in watchdog and so on and I changed my mind, because https://github.com/SpiderOak/spideroak_osx_fsevents doesn't have a Python API.

We already have a ctypes-based API to Linux and Windows, so I guess our code would be more consistent, and easy for us to write, if we also wrote a ctypes-based API to OSX?

Okay I looked at <https://github.com/SpiderOak/spideroak_osx_fsevents/blob/master/main.c> and other code in watchdog and so on and I changed my mind, because <https://github.com/SpiderOak/spideroak_osx_fsevents> doesn't have a Python API. We already have a ctypes-based API to Linux and Windows, so I guess our code would be more consistent, and easy for us to write, if we also wrote a ctypes-based API to OSX?
dawuud commented 2016-04-13 12:03:40 +00:00
Author
Owner
Copy Link

Of the several steps needed to setup the Apple async filesystem events API we need to firstly call FSEventStreamCreate.

There's a snippet of code showing the FSEventStreamCreate being used:

https://developer.apple.com/library/mac/documentation/Darwin/Conceptual/FSEvents_ProgGuide/UsingtheFSEventsFramework/UsingtheFSEventsFramework.html

It doesn't look like ctypes allows the creation of c structs or objective c objects unless you redefine them and create them the ctypes way. So initially I was thinking that maybe recreating all the necessary structs/objective-c-objects in python ctypes manually is kinda tricky... and requires us to have the Apple source code or at least the .h header files with the struct/object definitions.

Perhaps the pyobjc library can help us to compose these objective-c structures so that we can use
ctypes to pass them into API functions. Here need a python equivalent for composing the CFArray of CFString which holds the filesystems paths to watch:

    CFStringRef mypath = CFSTR("/path/to/scan");

    CFArrayRef pathsToWatch = CFArrayCreate(NULL, (const void **)&mypath, 1, NULL);

    void *callbackInfo = NULL; // could put stream-specific data here.

    FSEventStreamRef stream;

    CFAbsoluteTime latency = 3.0; /* Latency in seconds */

The pyobjc module requies xcode and when I pip installed it after properly installed xcode it downloaded many more modules; I wonder if this thing is too heavy weight. We probably want to avoid heavyweight dependencies?

I also discovered that google wrote a ctypes wrapper for Apple objective-c structures in this project:

https://github.com/google/grr
https://github.com/google/grr/blob/master/grr/client/osx/objc.py

oh and i also found this other ctypes wrapper API here:
https://github.com/wbond/oscrypto/blob/master/oscrypto/_osx/_core_foundation_ctypes.py

Of the several steps needed to setup the Apple async filesystem events API we need to firstly call *FSEventStreamCreate*. There's a snippet of code showing the *FSEventStreamCreate* being used: <https://developer.apple.com/library/mac/documentation/Darwin/Conceptual/FSEvents_ProgGuide/UsingtheFSEventsFramework/UsingtheFSEventsFramework.html> It doesn't look like ctypes allows the creation of c structs or objective c objects unless you redefine them and create them the ctypes way. So initially I was thinking that maybe recreating all the necessary structs/objective-c-objects in python ctypes manually is kinda tricky... and requires us to have the Apple source code or at least the .h header files with the struct/object definitions. Perhaps the pyobjc library can help us to compose these objective-c structures so that we can use ctypes to pass them into API functions. Here need a python equivalent for composing the CFArray of CFString which holds the filesystems paths to watch: ``` CFStringRef mypath = CFSTR("/path/to/scan"); CFArrayRef pathsToWatch = CFArrayCreate(NULL, (const void **)&mypath, 1, NULL); void *callbackInfo = NULL; // could put stream-specific data here. FSEventStreamRef stream; CFAbsoluteTime latency = 3.0; /* Latency in seconds */ ``` The pyobjc module requies xcode and when I pip installed it after properly installed xcode it downloaded many more modules; I wonder if this thing is too heavy weight. We probably want to avoid heavyweight dependencies? I also discovered that google wrote a ctypes wrapper for Apple objective-c structures in this project: <https://github.com/google/grr> <https://github.com/google/grr/blob/master/grr/client/osx/objc.py> oh and i also found this other ctypes wrapper API here: <https://github.com/wbond/oscrypto/blob/master/oscrypto/_osx/_core_foundation_ctypes.py>
dawuud commented 2016-04-14 14:24:33 +00:00
Author
Owner
Copy Link

Currently I have a failed attempt at creating this CFArray of CFString`s; running it results in a segmentation fault... however the VM i'm using won't write a core file. What to do?

(virtenv-objc) xcloud114:~ Xcloud$ python test.py
Segmentation fault: 11


import ctypes
from ctypes import POINTER, byref, create_string_buffer, addressof
from ctypes import cdll, c_void_p
from ctypes.util import find_library

# kCFStringEncodingUTF8                                                                                                           
UTF8 = 134217984
# kCFAllocatorDefault                                                                                                             
CF_DEFAULT_ALLOCATOR = None

core_services = cdll.LoadLibrary(find_library('CoreServices'))
core_foundation = cdll.LoadLibrary(find_library('CoreFoundation'))

def PyStringToCFString(pystring):
    return core_foundation.CFStringCreateWithCString(CF_DEFAULT_ALLOCATOR,
                                  pystring.encode('utf8'),
                                  UTF8)

my_cfarray = core_foundation.CFArrayCreate(None, None, 0, ctypes.c_int.in_dll(core_services, "kCFTypeArrayCallBacks"))
my_cfstr = PyStringToCFString("/Users/Xcloud/magic-folder")

core_foundation.CFArrayAppendValue(my_cfarray, my_cfstr)
Currently I have a failed attempt at creating this CFArray of CFString`s; running it results in a segmentation fault... however the VM i'm using won't write a core file. What to do? ``` (virtenv-objc) xcloud114:~ Xcloud$ python test.py Segmentation fault: 11 ``` ```/usr/bin/env python import ctypes from ctypes import POINTER, byref, create_string_buffer, addressof from ctypes import cdll, c_void_p from ctypes.util import find_library # kCFStringEncodingUTF8 UTF8 = 134217984 # kCFAllocatorDefault CF_DEFAULT_ALLOCATOR = None core_services = cdll.LoadLibrary(find_library('CoreServices')) core_foundation = cdll.LoadLibrary(find_library('CoreFoundation')) def PyStringToCFString(pystring): return core_foundation.CFStringCreateWithCString(CF_DEFAULT_ALLOCATOR, pystring.encode('utf8'), UTF8) my_cfarray = core_foundation.CFArrayCreate(None, None, 0, ctypes.c_int.in_dll(core_services, "kCFTypeArrayCallBacks")) my_cfstr = PyStringToCFString("/Users/Xcloud/magic-folder") core_foundation.CFArrayAppendValue(my_cfarray, my_cfstr) ```
dawuud commented 2016-04-15 12:49:10 +00:00
Author
Owner
Copy Link

today i wrote a partial working prototype for Mac OS X using the python watchdog library, here in my dev branch:

https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog.0

many of the magic-folder tests pass. however it seems that test_magic_folder and test_move_tree fail due to watchdog not recursively watching directories or not adding a watch to newly added directories or some such thing.

anyway we should be able to replace this usage of the watchdog api with a ctypes equivalent for OSX if we want to do that.

today i wrote a partial working prototype for Mac OS X using the python watchdog library, here in my dev branch: <https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog.0> many of the magic-folder tests pass. however it seems that test_magic_folder and test_move_tree fail due to watchdog not recursively watching directories or not adding a watch to newly added directories or some such thing. anyway we should be able to replace this usage of the watchdog api with a ctypes equivalent for OSX if we want to do that.
zooko commented 2016-04-15 12:59:01 +00:00
Copy Link

Whoo-hoo! Way to go!

Whoo-hoo! Way to go!
dawuud commented 2016-04-16 15:46:29 +00:00
Author
Owner
Copy Link

this actually is a BSD compatibility change since getting magic-folder to work with this watchdog based inotify API actually makes it work on all the BSD operating systems using kqueue and it should also work on Mac OS X using the native apple filesystem events api.

we could of course replace it with two ctypes based inotify APIs. one for kqueue and one for Mac OS X fs events api...

currently there are only two or three unit tests that fail and these failures are not so terrible... it fails because it's emitting more events than the unit tests expect.

this actually is a BSD compatibility change since getting magic-folder to work with this watchdog based inotify API actually makes it work on all the BSD operating systems using kqueue and it should also work on Mac OS X using the native apple filesystem events api. we could of course replace it with two ctypes based inotify APIs. one for kqueue and one for Mac OS X fs events api... currently there are only two or three unit tests that fail and these failures are not so terrible... it fails because it's emitting more events than the unit tests expect.
dawuud commented 2016-05-02 13:56:31 +00:00
Author
Owner
Copy Link

the latest watchdog inotify magic-folder branch is here:
https://github.com/tahoe-lafs/tahoe-lafs/tree/1432.osx-watchdog.0

you may note that the travis ci tests fail but this is due to changes in the dot travis yaml file as we attempt to get it working in a linux container and an osx container.
this difficulty with the travis configuration was due to the removal of the config options specifying python and instead telling travis to run in either a linux or osx container. then we are required to manually install many package dependencies so that pip and tahoe-lafs function properly and so on. i have previously gotten the travis configuration file to work properly for linux and osx... but this indeed involved changing other configuration options such as enabling sudo.

what's the next step to getting this merged into tahoe master or magic-folder stable?

the latest watchdog inotify magic-folder branch is here: <https://github.com/tahoe-lafs/tahoe-lafs/tree/1432.osx-watchdog.0> you may note that the travis ci tests fail but this is due to changes in the dot travis yaml file as we attempt to get it working in a linux container and an osx container. this difficulty with the travis configuration was due to the removal of the config options specifying python and instead telling travis to run in either a linux or osx container. then we are required to manually install many package dependencies so that pip and tahoe-lafs function properly and so on. i have previously gotten the travis configuration file to work properly for linux and osx... but this indeed involved changing other configuration options such as enabling sudo. what's the next step to getting this merged into tahoe master or magic-folder stable?
daira commented 2016-05-02 14:59:20 +00:00
Author
Owner
Copy Link

dawuud: please paste the current test failures.

dawuud: please paste the current test failures.
dawuud commented 2016-05-03 15:33:25 +00:00
Author
Owner
Copy Link
[FAIL]
Traceback (most recent call last):
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 1104, in <lambda>
    d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_succeeded'), 4))
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual
    self.failUnlessEqual(a, b, msg=msg)
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 6 != 4

allmydata.test.test_magic_folder.RealTest.test_move_tree
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 687, in <lambda>
    d.addCallback(lambda ign: self._check_uploader_count('objects_not_uploaded', 1, magic=self.bob_magicfolder))
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 211, in _check_uploader_count
    expected)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual
    self.failUnlessEqual(a, b, msg=msg)
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 2 != 1

allmydata.test.test_magic_folder.RealTestAliceBob.test_alice_bob
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 169, in _check_file
    os.unlink(path_u)
exceptions.OSError: [Errno 2] No such file or directory: '/Users/Xcloud/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/MockTest/test_magic_fo
lder/gIRrj8/temp/local_dir/tempfile'

allmydata.test.test_magic_folder.MockTest.test_magic_folder
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 169, in _check_file
    os.unlink(path_u)
exceptions.OSError: [Errno 2] No such file or directory: '/Users/Xcloud/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/RealTest/test_magic_fo
lder/ff93bG/temp/local_dir/tempfile'

allmydata.test.test_magic_folder.RealTest.test_magic_folder
-------------------------------------------------------------------------------
Ran 35 tests in 171.881s

FAILED (failures=2, errors=2, successes=31)
``` [FAIL] Traceback (most recent call last): File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 1104, in <lambda> d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_succeeded'), 4)) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual self.failUnlessEqual(a, b, msg=msg) File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual super(_Assertions, self).assertEqual(first, second, msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual assertion_func(first, second, msg=msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual raise self.failureException(msg) twisted.trial.unittest.FailTest: 6 != 4 allmydata.test.test_magic_folder.RealTest.test_move_tree =============================================================================== [FAIL] Traceback (most recent call last): File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 687, in <lambda> d.addCallback(lambda ign: self._check_uploader_count('objects_not_uploaded', 1, magic=self.bob_magicfolder)) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 211, in _check_uploader_count expected) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual self.failUnlessEqual(a, b, msg=msg) File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual super(_Assertions, self).assertEqual(first, second, msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual assertion_func(first, second, msg=msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual raise self.failureException(msg) twisted.trial.unittest.FailTest: 2 != 1 allmydata.test.test_magic_folder.RealTestAliceBob.test_alice_bob =============================================================================== [ERROR] Traceback (most recent call last): File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks result = g.send(result) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 169, in _check_file os.unlink(path_u) exceptions.OSError: [Errno 2] No such file or directory: '/Users/Xcloud/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/MockTest/test_magic_fo lder/gIRrj8/temp/local_dir/tempfile' allmydata.test.test_magic_folder.MockTest.test_magic_folder =============================================================================== [ERROR] Traceback (most recent call last): File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks result = g.send(result) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 169, in _check_file os.unlink(path_u) exceptions.OSError: [Errno 2] No such file or directory: '/Users/Xcloud/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/RealTest/test_magic_fo lder/ff93bG/temp/local_dir/tempfile' allmydata.test.test_magic_folder.RealTest.test_magic_folder ------------------------------------------------------------------------------- Ran 35 tests in 171.881s FAILED (failures=2, errors=2, successes=31) ```
dawuud commented 2016-05-09 14:54:06 +00:00
Author
Owner
Copy Link

the above test results were actually from an earlier commit;
this commit id
8939fbf16baa2cbd9ede7b23037b61dfbe323cdd

seems to have fewer test errors and failures than the HEAD of that branch. This is of course because the code review "corrections" caused more tests to fail... I'll have to track down which changes cause the problems. I suspect they are unicode related changes.

Furthermore our inotify/watchdog API must implement two more features:

  1. allow the user to set an events mask to select which events to get notifications for

  2. event attributes. the event messages may contain information about the file object which caused the events such as: file type (e.g. file or directory).

HOWEVER, given that the watchdog API passes nearly all the magic-folder unit tests, this means either the watchdog api is good enough or our unit test coverage sucks.

the above test results were actually from an earlier commit; this commit id 8939fbf16baa2cbd9ede7b23037b61dfbe323cdd seems to have fewer test errors and failures than the HEAD of that branch. This is of course because the code review "corrections" caused more tests to fail... I'll have to track down which changes cause the problems. I suspect they are unicode related changes. Furthermore our inotify/watchdog API must implement two more features: 1. allow the user to set an events mask to select which events to get notifications for 2. event attributes. the event messages may contain information about the file object which caused the events such as: file type (e.g. file or directory). HOWEVER, given that the watchdog API passes nearly all the magic-folder unit tests, this means either the watchdog api is good enough or our unit test coverage sucks.
dawuud commented 2016-05-24 12:24:13 +00:00
Author
Owner
Copy Link

i made a new dev branch with the watchdogi inotify stable changes on top of the latest stable magic-folder branch (stable 13):
https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.1

pull request here:
https://github.com/tahoe-lafs/tahoe-lafs/pull/287

i made a new dev branch with the watchdogi inotify stable changes on top of the latest stable magic-folder branch (stable 13): <https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.1> pull request here: <https://github.com/tahoe-lafs/tahoe-lafs/pull/287>
dawuud commented 2016-05-25 14:36:49 +00:00
Author
Owner
Copy Link

new dev branch based off of meejah's magic-folder stable 18 branch here:
https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.3

no errors, two failures:

===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 183, in _check_file
    previously_disappeared + 1)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual
    self.failUnlessEqual(a, b, msg=msg)
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 0 != 1

allmydata.test.test_magic_folder.RealTest.test_magic_folder
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 1131, in <lambda>
    d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_succeeded'), 4))
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual
    self.failUnlessEqual(a, b, msg=msg)
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 6 != 4

allmydata.test.test_magic_folder.RealTest.test_move_tree
-------------------------------------------------------------------------------
Ran 35 tests in 182.773s

FAILED (failures=2, successes=33)
new dev branch based off of meejah's magic-folder stable 18 branch here: <https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.3> no errors, two failures: ``` =============================================================================== [FAIL] Traceback (most recent call last): File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks result = g.send(result) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 183, in _check_file previously_disappeared + 1) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual self.failUnlessEqual(a, b, msg=msg) File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual super(_Assertions, self).assertEqual(first, second, msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual assertion_func(first, second, msg=msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual raise self.failureException(msg) twisted.trial.unittest.FailTest: 0 != 1 allmydata.test.test_magic_folder.RealTest.test_magic_folder =============================================================================== [FAIL] Traceback (most recent call last): File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 1131, in <lambda> d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_succeeded'), 4)) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual self.failUnlessEqual(a, b, msg=msg) File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual super(_Assertions, self).assertEqual(first, second, msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual assertion_func(first, second, msg=msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual raise self.failureException(msg) twisted.trial.unittest.FailTest: 6 != 4 allmydata.test.test_magic_folder.RealTest.test_move_tree ------------------------------------------------------------------------------- Ran 35 tests in 182.773s FAILED (failures=2, successes=33) ```
dawuud commented 2016-05-25 15:29:30 +00:00
Author
Owner
Copy Link

however a simple change, ignore events outside the watched directory shows the score a bit closer to the target:

===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 183, in _check_file
    previously_disappeared + 1)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual
    self.failUnlessEqual(a, b, msg=msg)
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in 
assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in 
_baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 0 != 1

allmydata.test.test_magic_folder.RealTest.test_magic_folder
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 1131, in <lambda>
    d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_succeeded'), 4))
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual
    self.failUnlessEqual(a, b, msg=msg)
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in 
assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in 
_baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 3 != 4

allmydata.test.test_magic_folder.RealTest.test_move_tree
-------------------------------------------------------------------------------
Ran 35 tests in 188.915s

FAILED (failures=2, successes=33)
however a simple change, ignore events outside the watched directory shows the score a bit closer to the target: ``` =============================================================================== [FAIL] Traceback (most recent call last): File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks result = g.send(result) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 183, in _check_file previously_disappeared + 1) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual self.failUnlessEqual(a, b, msg=msg) File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual super(_Assertions, self).assertEqual(first, second, msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual assertion_func(first, second, msg=msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual raise self.failureException(msg) twisted.trial.unittest.FailTest: 0 != 1 allmydata.test.test_magic_folder.RealTest.test_magic_folder =============================================================================== [FAIL] Traceback (most recent call last): File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 1131, in <lambda> d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_succeeded'), 4)) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual self.failUnlessEqual(a, b, msg=msg) File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual super(_Assertions, self).assertEqual(first, second, msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual assertion_func(first, second, msg=msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual raise self.failureException(msg) twisted.trial.unittest.FailTest: 3 != 4 allmydata.test.test_magic_folder.RealTest.test_move_tree ------------------------------------------------------------------------------- Ran 35 tests in 188.915s FAILED (failures=2, successes=33) ```
dawuud commented 2016-05-26 14:29:05 +00:00
Author
Owner
Copy Link

new error output from tests show all tests pass except one:

    test_magic_folder ... set pending delay                                                                                                   [2/1850]
init INotifyEventHandler
START READING BEGIN
START READING END
PROCESS EVENT <FileCreatedEvent: src_path='/Users/Xcloud/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/RealTest/test_magic_folder/Q5TLUN/te$p/local_dir/short'>
PROCESS EVENT <DirModifiedEvent: src_path='/Users/Xcloud/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/RealTest/test_magic_folder/Q5TLUN/te$
p/local_dir'>
DO CALLBACKS
                                               [FAIL]
stopReading begin
stopReading end
wait until stopped

===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 188, in _check_file
    self.failUnlessReallyEqual(actual_data, data)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual
    self.failUnlessEqual(a, b, msg=msg)
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 'test' != 'different'

allmydata.test.test_magic_folder.RealTest.test_magic_folder
-------------------------------------------------------------------------------
Ran 2 tests in 5.686s

FAILED (failures=1, successes=1)
new error output from tests show all tests pass except one: ``` test_magic_folder ... set pending delay [2/1850] init INotifyEventHandler START READING BEGIN START READING END PROCESS EVENT <FileCreatedEvent: src_path='/Users/Xcloud/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/RealTest/test_magic_folder/Q5TLUN/te$p/local_dir/short'> PROCESS EVENT <DirModifiedEvent: src_path='/Users/Xcloud/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/RealTest/test_magic_folder/Q5TLUN/te$ p/local_dir'> DO CALLBACKS [FAIL] stopReading begin stopReading end wait until stopped =============================================================================== [FAIL] Traceback (most recent call last): File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks result = g.send(result) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 188, in _check_file self.failUnlessReallyEqual(actual_data, data) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual self.failUnlessEqual(a, b, msg=msg) File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual super(_Assertions, self).assertEqual(first, second, msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual assertion_func(first, second, msg=msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual raise self.failureException(msg) twisted.trial.unittest.FailTest: 'test' != 'different' allmydata.test.test_magic_folder.RealTest.test_magic_folder ------------------------------------------------------------------------------- Ran 2 tests in 5.686s FAILED (failures=1, successes=1) ```
dawuud commented 2016-05-30 15:26:12 +00:00
Author
Owner
Copy Link

in another dev branch i saved a commit with added debug print statements so i could track down the root cause of the above unit test failure:
https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.3.wip.1

what is happening is that this test case fails:
https://github.com/david415/tahoe-lafs/blob/e451e2eaa410d84aa3ae0b62a5aaf5a4d4aa786b/src/allmydata/test/test_magic_folder.py#L1240

_check_file is failing on the temporary file case; specifically here's the comparison that fails:

https://github.com/david415/tahoe-lafs/blob/e451e2eaa410d84aa3ae0b62a5aaf5a4d4aa786b/src/allmydata/test/test_magic_folder.py#L182

yield self.failUnlessReallyEqual(self._get_count('uploader.objects_disappeared'),
                                             previously_disappeared + 1)

uploader.objects_disappeared 0
previously_disappeared + 1 == 1

OK so it uploads the temporary file when it shouldn't. But is that really so bad? Seems better than not uploading enough.

in another dev branch i saved a commit with added debug print statements so i could track down the root cause of the above unit test failure: <https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.3.wip.1> what is happening is that this test case fails: <https://github.com/david415/tahoe-lafs/blob/e451e2eaa410d84aa3ae0b62a5aaf5a4d4aa786b/src/allmydata/test/test_magic_folder.py#L1240> _check_file is failing on the temporary file case; specifically here's the comparison that fails: <https://github.com/david415/tahoe-lafs/blob/e451e2eaa410d84aa3ae0b62a5aaf5a4d4aa786b/src/allmydata/test/test_magic_folder.py#L182> ``` yield self.failUnlessReallyEqual(self._get_count('uploader.objects_disappeared'), previously_disappeared + 1) ``` ``` uploader.objects_disappeared 0 previously_disappeared + 1 == 1 ``` OK so it uploads the temporary file when it shouldn't. But is that really so bad? Seems better than not uploading enough.
dawuud commented 2016-06-01 07:33:51 +00:00
Author
Owner
Copy Link

I have had two conversations about magic-folder unit test race conditions; one with Daira and one with Meejah. Currently the plan is for Meejah to fix/refactor all the uploader unit tests after figuring out a better API for uploader testing similar in spirit to how we refactored the downloader tests. Here's the ticket for that:

https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2790

I have had two conversations about magic-folder unit test race conditions; one with Daira and one with Meejah. Currently the plan is for Meejah to fix/refactor all the uploader unit tests after figuring out a better API for uploader testing similar in spirit to how we refactored the downloader tests. Here's the ticket for that: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2790>
dawuud commented 2016-06-22 15:59:41 +00:00
Author
Owner
Copy Link

here i've put the watchdog inotify changes on top of meejah's stable 18 branch:
https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.4

here i've put the watchdog inotify changes on top of meejah's stable 18 branch: <https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.4>
dawuud commented 2016-06-22 16:10:16 +00:00
Author
Owner
Copy Link

oh dear the linux build passes but the osx build fails with the current travis yaml configuration::

RuntimeError: You are linking against OpenSSL 0.9.8, which is no longer support by the OpenSSL
project. You need to upgrade to a newer version of OpenSSL.

there's a ticket regarding automated testing on OSX here:
https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2789

oh dear the linux build passes but the osx build fails with the current travis yaml configuration:: [RuntimeError](wiki/RuntimeError): You are linking against OpenSSL 0.9.8, which is no longer support by the OpenSSL project. You need to upgrade to a newer version of OpenSSL. there's a ticket regarding automated testing on OSX here: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2789>
warner commented 2016-08-16 17:55:57 +00:00
Copy Link

BTW, that openssl thing was fixed. The travis OS-X builder was using the wrong version of python, and failing to pick up the newer openssl library. If you update the PR, travis should run with the proper version.

BTW, that openssl thing was fixed. The travis OS-X builder was using the wrong version of python, and failing to pick up the newer openssl library. If you update the PR, travis should run with the proper version.
dawuud commented 2016-08-17 13:05:49 +00:00
Author
Owner
Copy Link

the current dev branch is:
https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.5

and current pull request is:
https://github.com/tahoe-lafs/tahoe-lafs/pull/308

the current dev branch is: <https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.5> and current pull request is: <https://github.com/tahoe-lafs/tahoe-lafs/pull/308>
dawuud commented 2016-08-29 13:31:04 +00:00
Author
Owner
Copy Link

currently on the darwin platform fails at least this test:
allmydata.test.test_magic_folder.RealTest.test_delete

I don't understand why this fails. I added a print statement to display the platform filesystem encoding which is UTF-8

test_delete ... set pending delay

init INotifyEventHandler

START READING BEGIN

START READING END

PROCESS EVENT <FileCreatedEvent: src_path='/Users/travis/build/tahoe-lafs/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/RealTest/test_delete/nvdp3V/temp/local_dir/foo'>

FILESYSTEM ENCODING: utf-8

Exception in thread Thread-8:

Traceback (most recent call last):

> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner

    self.run()

> File "/Users/travis/build/tahoe-lafs/tahoe-lafs/.tox/py27/lib/python2.7/site-packages/watchdog/observers/api.py", line 199, in run

    self.dispatch_events(self.event_queue, self.timeout)

> File "/Users/travis/build/tahoe-lafs/tahoe-lafs/.tox/py27/lib/python2.7/site-packages/watchdog/observers/api.py", line 368, in dispatch_events

    handler.dispatch(event)

> File "/Users/travis/build/tahoe-lafs/tahoe-lafs/.tox/py27/lib/python2.7/site-packages/watchdog/events.py", line 322, in dispatch

    self.on_any_event(event)

> File "/Users/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/watchdog/inotify.py", line 73, in on_any_event

    self.process(event)

> File "/Users/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/watchdog/inotify.py", line 47, in process

    event_filepath_u = abspath_expanduser_unicode(event_filepath_u, base=self._path)

> File "/Users/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/util/fileutil.py", line 310, in abspath_expanduser_unicode

    precondition_abspath(base)

> File "/Users/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/util/fileutil.py", line 276, in precondition_abspath

    raise [AssertionError](wiki/AssertionError)("an abspath must be a Unicode string")

[AssertionError](wiki/AssertionError): an abspath must be a Unicode string
currently on the darwin platform fails at least this test: ``allmydata.test.test_magic_folder.RealTest.test_delete`` I don't understand why this fails. I added a print statement to display the platform filesystem encoding which is UTF-8 ``` test_delete ... set pending delay init INotifyEventHandler START READING BEGIN START READING END PROCESS EVENT <FileCreatedEvent: src_path='/Users/travis/build/tahoe-lafs/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/RealTest/test_delete/nvdp3V/temp/local_dir/foo'> FILESYSTEM ENCODING: utf-8 Exception in thread Thread-8: Traceback (most recent call last): > File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner self.run() > File "/Users/travis/build/tahoe-lafs/tahoe-lafs/.tox/py27/lib/python2.7/site-packages/watchdog/observers/api.py", line 199, in run self.dispatch_events(self.event_queue, self.timeout) > File "/Users/travis/build/tahoe-lafs/tahoe-lafs/.tox/py27/lib/python2.7/site-packages/watchdog/observers/api.py", line 368, in dispatch_events handler.dispatch(event) > File "/Users/travis/build/tahoe-lafs/tahoe-lafs/.tox/py27/lib/python2.7/site-packages/watchdog/events.py", line 322, in dispatch self.on_any_event(event) > File "/Users/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/watchdog/inotify.py", line 73, in on_any_event self.process(event) > File "/Users/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/watchdog/inotify.py", line 47, in process event_filepath_u = abspath_expanduser_unicode(event_filepath_u, base=self._path) > File "/Users/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/util/fileutil.py", line 310, in abspath_expanduser_unicode precondition_abspath(base) > File "/Users/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/util/fileutil.py", line 276, in precondition_abspath raise [AssertionError](wiki/AssertionError)("an abspath must be a Unicode string") [AssertionError](wiki/AssertionError): an abspath must be a Unicode string ```
exarkun commented 2016-11-21 21:28:32 +00:00
Copy Link

Some changes built on david's last branch - https://github.com/exarkun/tahoe-lafs/tree/1432.osx-watchdog-stable.9

In particular, this adds a direct inotify-level test suite (copied from Twisted!) and makes some of the tests pass - including the IN_CLOSE_WRITE notification.

Some changes built on david's last branch - <https://github.com/exarkun/tahoe-lafs/tree/1432.osx-watchdog-stable.9> In particular, this adds a direct inotify-level test suite (copied from Twisted!) and makes some of the tests pass - including the IN_CLOSE_WRITE notification.
dawuud commented 2016-11-23 20:49:22 +00:00
Author
Owner
Copy Link

more progress here in this dev branch.
i've changed the inotify test suite to skip many of the tests for inotify
features that we do not use or need. I also modified some of the tests so
that they now pass when using either the watchdog inotify and the linux native inotify.
not yet tested with our windows inotify implementation.

https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.9

more progress here in this dev branch. i've changed the inotify test suite to skip many of the tests for inotify features that we do not use or need. I also modified some of the tests so that they now pass when using either the watchdog inotify and the linux native inotify. not yet tested with our windows inotify implementation. <https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.9>
dawuud commented 2016-11-23 22:45:46 +00:00
Author
Owner
Copy Link

now that we have a inotify test suite perhaps we can remove the "real" inotify magic folder tests?

in the above mentioned dev branch i've removed the real tests in this commit here:
https://github.com/david415/tahoe-lafs/commit/451dcf093b26b475378ef588e2cf68b40697cc8c

now that we have a inotify test suite perhaps we can remove the "real" inotify magic folder tests? in the above mentioned dev branch i've removed the real tests in this commit here: <https://github.com/david415/tahoe-lafs/commit/451dcf093b26b475378ef588e2cf68b40697cc8c>
meejah commented 2016-11-25 00:32:38 +00:00
Owner
Copy Link

Is the reactor.callFromThread() because the "watchdog" library is running the on_any_event from its own thread(s)?

Otherwise looks plausible. And yes I'd like to get rid of some of the "long" magic-folder tests especially. If our test show that all 3 underlying inotify-like implementations are doing the right thing, then yes it would be great to get rid of the "Real" vs "Mock" dual tests in magic-folder as well! (i.e. just have the Mock ones).

Is the `reactor.callFromThread()` because the "watchdog" library is running the `on_any_event` from its own thread(s)? Otherwise looks plausible. And yes I'd like to get rid of some of the "long" magic-folder tests especially. If our test show that all 3 underlying inotify-like implementations are doing the right thing, then yes it would be great to get rid of the "Real" vs "Mock" dual tests in magic-folder as well! (i.e. just have the Mock ones).
dawuud commented 2016-12-08 19:37:30 +00:00
Author
Owner
Copy Link

this latest dev branch is currently in a pull-request
https://github.com/tahoe-lafs/tahoe-lafs/pull/381

all tests pass except for the appveyor testing on windows.
currently i can see that the inotify tests are failing for windows;
the test_attrib fails and test_closeWrite timesout.

meejah and i plan to look at this soon.

this latest dev branch is currently in a pull-request <https://github.com/tahoe-lafs/tahoe-lafs/pull/381> all tests pass except for the appveyor testing on windows. currently i can see that the inotify tests are failing for windows; the test_attrib fails and test_closeWrite timesout. meejah and i plan to look at this soon.
dawuud commented 2017-01-09 16:02:08 +00:00
Author
Owner
Copy Link

i'm trying to make this branch pass all the tests.
in my latest commit i'm trying to make the test_inotify tests be skipped on windows...
we can make the tests pass on windows later.
for now i'd like to get the bsd changes merged if possible.

i'm trying to make this branch pass all the tests. in my latest commit i'm trying to make the test_inotify tests be skipped on windows... we can make the tests pass on windows later. for now i'd like to get the bsd changes merged if possible.
dawuud commented 2017-02-14 23:53:08 +00:00
Author
Owner
Copy Link

my latest dev branch:
https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.10

all tests pass. the magic-folder "real inotify" tests have been disabled.
the watch inotify implementation has it's own test suite that was borrowed and inspired from twisted's inotify test suite.

my latest dev branch: <https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.10> all tests pass. the magic-folder "real inotify" tests have been disabled. the watch inotify implementation has it's own test suite that was borrowed and inspired from twisted's inotify test suite.
dawuud commented 2017-02-15 18:13:55 +00:00
Author
Owner
Copy Link
No description provided.
dawuud commented 2017-02-20 18:31:25 +00:00
Author
Owner
Copy Link

attempting to skip the inotify tests on windows:
https://github.com/tahoe-lafs/tahoe-lafs/pull/401

the tests seem to be skipped but prints dirty reactor errors like this:

test_closeWrite ... SKIPPED
Traceback (most recent call last):
Failure: twisted.trial.util.DirtyReactorAggregateError: Reactor was unclean.
DelayedCalls: (set twisted.internet.base.DelayedCall.debug = True to debug)
<DelayedCall 0x4af3eb8 [0.00999999046326s] called=0 cancelled=0 LoopingCall<0.01>(INotify._poll, *(<function at 0x069DE770>, 1487535067.863), **{})()>
ERROR

why!? how to fix?

attempting to skip the inotify tests on windows: <https://github.com/tahoe-lafs/tahoe-lafs/pull/401> the tests seem to be skipped but prints dirty reactor errors like this: test_closeWrite ... SKIPPED Traceback (most recent call last): Failure: twisted.trial.util.DirtyReactorAggregateError: Reactor was unclean. [DelayedCalls](wiki/DelayedCalls): (set twisted.internet.base.DelayedCall.debug = True to debug) <DelayedCall 0x4af3eb8 [0.00999999046326s] called=0 cancelled=0 [LoopingCall](wiki/LoopingCall)<0.01>(INotify._poll, *(<function <lambda> at 0x069DE770>, 1487535067.863), **{})()> ERROR why!? how to fix?
dawuud commented 2017-02-20 20:50:29 +00:00
Author
Owner
Copy Link

fixed it

fixed it
crwood commented 2018-07-26 01:02:01 +00:00
Copy Link

Freshly resurrected in PR 516.

Freshly resurrected in [PR 516](https://github.com/tahoe-lafs/tahoe-lafs/pull/516).
exarkun commented 2019-01-31 19:47:50 +00:00
Copy Link

Bad news. Watchdog upstream have decided to stop supporting Python 2.

https://github.com/gorakhargosh/watchdog/issues/516

Bad news. Watchdog upstream have decided to stop supporting Python 2. <https://github.com/gorakhargosh/watchdog/issues/516>
exarkun commented 2019-02-22 19:21:54 +00:00
Copy Link

They changed their mind. There are some other issues, though, like events being missed with the fsevents backend on macOS.

They changed their mind. There are some other issues, though, like events being missed with the fsevents backend on macOS.
GitHub <noreply@github.com> commented 2019-03-14 17:02:13 +00:00
Author
Owner
Copy Link

In c1e6f08/trunk:

Merge pull request #558 from tahoe-lafs/1432.watchdog-magic-folder-with-eliot

Add support for macOS to Magic-Folders.

Fixes: ticket:1432
In [c1e6f08/trunk](/tahoe-lafs/trac-2024-07-25/commit/c1e6f0881331b6549337af651e87e2f1521c8945): ``` Merge pull request #558 from tahoe-lafs/1432.watchdog-magic-folder-with-eliot Add support for macOS to Magic-Folders. Fixes: ticket:1432 ```
tahoe-lafs added the
fixed
 label 2019-03-14 17:02:13 +00:00
GitHub <noreply@github.com> closed this issue 2019-03-14 17:02:13 +00:00
exarkun commented 2019-03-14 17:34:25 +00:00
Copy Link

Note to all, the above merge really did only add macOS support - not BSD. Sorry. Tickets should be about one thing, though.

Note to all, the above merge really did only add macOS support - not BSD. Sorry. Tickets should be about one thing, though.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
0.2.0

   
0.3.0

   
0.4.0

   
0.5.0

   
0.5.1

   
0.6.0

   
0.6.1

   
0.7.0

   
0.8.0

   
0.9.0

   
1.0.0

   
1.1.0

   
1.10.0

   
1.10.1

   
1.10.2

   
1.10a2

   
1.11.0

   
1.12.0

   
1.12.1

   
1.13.0

   
1.14.0

   
1.15.0

   
1.15.1

   
1.2.0

   
1.3.0

   
1.4.1

   
1.5.0

   
1.6.0

   
1.6.1

   
1.7.0

   
1.7.1

   
1.7β

   
1.8.0

   
1.8.1

   
1.8.2

   
1.8.3

   
1.8β

   
1.9.0

   
1.9.0-s3branch

   
1.9.0a1

   
1.9.0a2

   
1.9.0b1

   
1.9.1

   
1.9.2

   
1.9.2a1

   
LeastAuthority.com automation

   
blocker

   
cannot reproduce

   
cloud-branch

   
code

catch-all for anything that can be fixed by writing code but isn't classified better

  
code-dirnodes

anything involving tahoe directories

  
code-encoding

immutable-file encoding format/layout, cap format, encoding/decoding algorithms

  
code-frontend

general frontend things: new user-facing interfaces, FTP/SFTP servers, FUSE-like interfaces

  
code-frontend-cli

CLI tools

  
code-frontend-ftp-sftp

   
code-frontend-magic-folder

   
code-frontend-web

webapi interfaces, user-facing "WUI" interfaces

  
code-mutable

anything involving mutable files: encoding/decoding, share format

  
code-network

Introducer, locating storage servers, establishing/maintaining connections to them

  
code-nodeadmin

node startup/shutdown, tools to create nodes, logging tools, management/monitoring tools, manhole, disk-space-limiting tools

  
code-peerselection

server selection/sorting algorithm, rebalancing code, allocation policy, swarming-download

  
code-storage

storage-server implementation, share-storage directory layout, client-to-server share transfer protocol

  
contrib

problems with non-core things that live in the source:contrib/ directory

  
critical

   
defect

   
dev-infrastructure

things that help us build Tahoe: buildbot, automated speed/memory/code-coverage tests and results-displayers, IRC channels, mailing lists, this Trac instance

  
documentation

docs items. Also use 'docs' keyword on tickets in more specific components.

  
duplicate

   
enhancement

   
fixed

   
invalid

   
major

   
minor

   
n/a

   
normal

   
operational

things involved in running a tahoe grid, external tools to monitor/maintain a grid (like "nodeadmin" but larger-scale). Was also used for issues with the allmydata.com grid.

  
packaging

setup.py script, library dependencies, binary packages, licensing, installation instructions

  
somebody else's problem

   
supercritical

   
task

   
trivial

   
unknown

Uncategorized. All tickets default to this component until someone updates it.

  
was already fixed

   
website

things about the tahoe-lafs.org website (and sometimes this Trac instance, shared with 'dev-infrastructure')

  
wontfix

   
worksforme
No Label
0.2.0
0.3.0
0.4.0
0.5.0
0.5.1
0.6.0
0.6.1
0.7.0
0.8.0
0.9.0
1.0.0
1.1.0
1.10.0
1.10.1
1.10.2
1.10a2
1.11.0
1.12.0
1.12.1
1.13.0
1.14.0
1.15.0
1.15.1
1.2.0
1.3.0
1.4.1
1.5.0
1.6.0
1.6.1
1.7.0
1.7.1
1.7β
1.8.0
1.8.1
1.8.2
1.8.3
1.8β
1.9.0
1.9.0-s3branch
1.9.0a1
1.9.0a2
1.9.0b1
1.9.1
1.9.2
1.9.2a1
LeastAuthority.com automation
blocker
cannot reproduce
cloud-branch
code
code-dirnodes
code-encoding
code-frontend
code-frontend-cli
code-frontend-ftp-sftp
code-frontend-magic-folder
code-frontend-web
code-mutable
code-network
code-nodeadmin
code-peerselection
code-storage
contrib
critical
defect
dev-infrastructure
documentation
duplicate
enhancement
fixed
invalid
major
minor
n/a
normal
operational
packaging
somebody else's problem
supercritical
task
trivial
unknown
was already fixed
website
wontfix
worksforme
Milestone
Clear milestone
No items
No Milestone
undecided
Assignees
Clear assignees
No Assignees
6 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Reference: tahoe-lafs/trac-2024-07-25#1432
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?