PipelineError during backup #842

New Issue

tahoe-lafs · 2009-11-24T09:05:17Z

francois commented

2009-11-24 09:05:17 +00:00

I'm using a fresh darcs checkout as of November 24 with Alberto Berti's
patches for #641 applied.

A 'tahoe backup' run, sometimes crash with the following exception.

2009-11-24 09:38:21+0100 [Negotiation,client] Unhandled error in Deferred:
2009-11-24 09:38:21+0100 [Negotiation,client] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib/python2.4/site-packages/Twisted-8.1.0-py2.4-linux-i686.egg/twisted/internet/defer.py", line 312, in _startRunCallbacks
            self._runCallbacks()
          File "/usr/lib/python2.4/site-packages/Twisted-8.1.0-py2.4-linux-i686.egg/twisted/internet/defer.py", line 328, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/usr/lib/python2.4/site-packages/Twisted-8.1.0-py2.4-linux-i686.egg/twisted/internet/defer.py", line 289, in _continue
            self.unpause()
          File "/usr/lib/python2.4/site-packages/Twisted-8.1.0-py2.4-linux-i686.egg/twisted/internet/defer.py", line 285, in unpause
            self._runCallbacks()
        --- <exception caught here> ---
          File "/usr/lib/python2.4/site-packages/Twisted-8.1.0-py2.4-linux-i686.egg/twisted/internet/defer.py", line 328, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/root/tahoe/src/allmydata/immutable/encode.py", line 481, in _remove_shareholder
            level=log.UNUSUAL, failure=why)
          File "/root/tahoe/src/allmydata/immutable/encode.py", line 99, in log
            return log.msg(*args, **kwargs)
          File "/root/tahoe/support/lib/python2.4/site-packages/foolscap-0.4.2-py2.4.egg/foolscap/logging/log.py", line 197, in msg
            f2.setCopyableState(fs.getStateToCopy(f, FakeBroker))
          File "/root/tahoe/support/lib/python2.4/site-packages/foolscap-0.4.2-py2.4.egg/foolscap/call.py", line 796, in getStateToCopy
            state['value'] = str(obj.value) # Exception instance
        exceptions.AttributeError: PipelineError instance has no attribute 'args'

I'm using a fresh darcs checkout as of November 24 with Alberto Berti's patches for #641 applied. A 'tahoe backup' run, sometimes crash with the following exception. ``` 2009-11-24 09:38:21+0100 [Negotiation,client] Unhandled error in Deferred: 2009-11-24 09:38:21+0100 [Negotiation,client] Unhandled Error Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/Twisted-8.1.0-py2.4-linux-i686.egg/twisted/internet/defer.py", line 312, in _startRunCallbacks self._runCallbacks() File "/usr/lib/python2.4/site-packages/Twisted-8.1.0-py2.4-linux-i686.egg/twisted/internet/defer.py", line 328, in _runCallbacks self.result = callback(self.result, *args, **kw) File "/usr/lib/python2.4/site-packages/Twisted-8.1.0-py2.4-linux-i686.egg/twisted/internet/defer.py", line 289, in _continue self.unpause() File "/usr/lib/python2.4/site-packages/Twisted-8.1.0-py2.4-linux-i686.egg/twisted/internet/defer.py", line 285, in unpause self._runCallbacks() --- <exception caught here> --- File "/usr/lib/python2.4/site-packages/Twisted-8.1.0-py2.4-linux-i686.egg/twisted/internet/defer.py", line 328, in _runCallbacks self.result = callback(self.result, *args, **kw) File "/root/tahoe/src/allmydata/immutable/encode.py", line 481, in _remove_shareholder level=log.UNUSUAL, failure=why) File "/root/tahoe/src/allmydata/immutable/encode.py", line 99, in log return log.msg(*args, **kwargs) File "/root/tahoe/support/lib/python2.4/site-packages/foolscap-0.4.2-py2.4.egg/foolscap/logging/log.py", line 197, in msg f2.setCopyableState(fs.getStateToCopy(f, FakeBroker)) File "/root/tahoe/support/lib/python2.4/site-packages/foolscap-0.4.2-py2.4.egg/foolscap/call.py", line 796, in getStateToCopy state['value'] = str(obj.value) # Exception instance exceptions.AttributeError: PipelineError instance has no attribute 'args' ```

tahoe-lafs added the

labels 2009-11-24 09:05:17 +00:00

tahoe-lafs added this to the undecided milestone 2009-11-24 09:05:17 +00:00

zooko commented

2009-11-24 15:42:06 +00:00

Marking this as potentially a regression from v1.5.

zooko modified the milestone from undecided to 1.6.0

2009-11-24 15:42:06 +00:00

zooko commented

2009-11-24 15:44:23 +00:00

Brian, could you look at this?

I really don't understand this part:

          File "/root/tahoe/support/lib/python2.4/site-packages/foolscap-0.4.2-py2.4.egg/foolscap/call.py", line 796, in getStateToCopy
            state['value'] = str(obj.value) # Exception instance
        exceptions.AttributeError: PipelineError instance has no attribute 'args'

There is no attempt to access an attribute named 'args' on that line, so how can there be an AttributeError on 'args'?

Brian, could you look at this? I really don't understand this part: ``` File "/root/tahoe/support/lib/python2.4/site-packages/foolscap-0.4.2-py2.4.egg/foolscap/call.py", line 796, in getStateToCopy state['value'] = str(obj.value) # Exception instance exceptions.AttributeError: PipelineError instance has no attribute 'args' ``` There is no attempt to access an attribute named 'args' on that line, so how can there be an AttributeError on 'args'?

warner was assigned by zooko

2009-11-24 15:44:23 +00:00

warner commented

2009-11-24 20:07:27 +00:00

I think the problem is that PipelineError is not str-able (it overrides *init* and *repr* but not *str*).

If I had to guess, I'd say that a connection was lost during the upload, the encoder tried to log.msg the exception instance, but it wasn't str-able so it blew up. I'd also guess that this has been a problem since that Pipeline class went into 1.5.0, but only shows up if you get an exception during upload.

I'll add a test of str-ability and fix the class (source:src/allmydata/util/pipeline.py in case anyone's curious).

I think the problem is that `PipelineError` is not str-able (it overrides `*init*` and `*repr*` but not `*str*`). If I had to guess, I'd say that a connection was lost during the upload, the encoder tried to log.msg the exception instance, but it wasn't str-able so it blew up. I'd also guess that this has been a problem since that Pipeline class went into 1.5.0, but only shows up if you get an exception during upload. I'll add a test of str-ability and fix the class (source:src/allmydata/util/pipeline.py in case anyone's curious).

warner commented

2009-11-24 20:49:08 +00:00

hrm, I'm not seeing the test fail on my py2.6 system. Maybe py2.5 or py2.6 started implementing str() in terms of repr(), thus masking the problem?

warner commented

2009-11-24 21:34:10 +00:00

Ok, changeset:41bcc9f39ec52c7e should close this. Python2.5/2.6 don't suffer from the problem, but py2.4 does. francois: could you check that test_util.Pipeline passes with the new patch? If so, this problem should be resolved.

warner added the

fixed

label 2009-11-24 21:34:10 +00:00

warner closed this issue

2009-11-24 21:34:10 +00:00

francois commented

2009-11-25 17:58:11 +00:00

Wouah, that was fast, thanks Brian.

And, yes, the unit test passed on my old Debian etch machine.

Wouah, that was fast, thanks Brian. And, yes, the unit test passed on my old Debian etch machine.

Sign in to join this conversation.