multi backup error #2285

New Issue

tahoe-lafs · 2014-08-27T04:47:16Z

dawuud commented

2014-08-27 04:47:16 +00:00

When performing two backups at once (tahoe backup) one of them crashed
with a stack traceback ending in this error:
OperationalError: database is locked

I was running tahoe-lafs in Tails from wheezy-backports :

$ tahoe --version
allmydata-tahoe: 1.10.0
foolscap: 0.6.4
pycryptopp: 0.6.0.1206569328141510525648634803928199668821045408958
zfec: 1.4.5
Twisted: 12.0.0
Nevow: 0.10.0
zope.interface: unknown
python: 2.7.3
platform: Linux-debian_7.6-x86_64-32bit_ELF
pyOpenSSL: 0.13
simplejson: 2.5.2
pycrypto: 2.6
pyasn1: unknown
mock: 0.8.0
setuptools: 0.6 [distribute]

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/runner.py", line 156, in run
    rc = runner(sys.argv[1:], install_node_control=install_node_control)
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/runner.py", line 141, in runner
    rc = cli.dispatch[command](so)
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/cli.py", line 574, in backup
    rc = tahoe_backup.backup(options)
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/tahoe_backup.py", line 325, in backup
    return bu.run()
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/tahoe_backup.py", line 118, in run
    new_backup_dircap = self.process(options.from_dir)
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/tahoe_backup.py", line 188, in process
    childcap = self.process(childpath)
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/tahoe_backup.py", line 188, in process
    childcap = self.process(childpath)
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/tahoe_backup.py", line 188, in process
    childcap = self.process(childpath)
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/tahoe_backup.py", line 194, in process
    childcap, metadata = self.upload(childpath)
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/tahoe_backup.py", line 313, in upload
    bdb_results.did_upload(filecap)
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/backupdb.py", line 122, in did_upload
    self.mtime, self.ctime, self.size)
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/backupdb.py", line 264, in did_upload_file
    fileid = self.get_or_allocate_fileid_for_cap(filecap)
  File "/usr/lib/python2.7/dist-packages/allmydata/scripts/backupdb.py", line 256, in get_or_allocate_fileid_for_cap
    c.execute("SELECT fileid FROM caps WHERE filecap=?", (filecap,))
OperationalError: database is locked

When performing two backups at once (tahoe backup) one of them crashed with a stack traceback ending in this error: [OperationalError](wiki/OperationalError): database is locked I was running tahoe-lafs in Tails from wheezy-backports : ``` $ tahoe --version allmydata-tahoe: 1.10.0 foolscap: 0.6.4 pycryptopp: 0.6.0.1206569328141510525648634803928199668821045408958 zfec: 1.4.5 Twisted: 12.0.0 Nevow: 0.10.0 zope.interface: unknown python: 2.7.3 platform: Linux-debian_7.6-x86_64-32bit_ELF pyOpenSSL: 0.13 simplejson: 2.5.2 pycrypto: 2.6 pyasn1: unknown mock: 0.8.0 setuptools: 0.6 [distribute] ``` ``` Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/allmydata/scripts/runner.py", line 156, in run rc = runner(sys.argv[1:], install_node_control=install_node_control) File "/usr/lib/python2.7/dist-packages/allmydata/scripts/runner.py", line 141, in runner rc = cli.dispatch[command](so) File "/usr/lib/python2.7/dist-packages/allmydata/scripts/cli.py", line 574, in backup rc = tahoe_backup.backup(options) File "/usr/lib/python2.7/dist-packages/allmydata/scripts/tahoe_backup.py", line 325, in backup return bu.run() File "/usr/lib/python2.7/dist-packages/allmydata/scripts/tahoe_backup.py", line 118, in run new_backup_dircap = self.process(options.from_dir) File "/usr/lib/python2.7/dist-packages/allmydata/scripts/tahoe_backup.py", line 188, in process childcap = self.process(childpath) File "/usr/lib/python2.7/dist-packages/allmydata/scripts/tahoe_backup.py", line 188, in process childcap = self.process(childpath) File "/usr/lib/python2.7/dist-packages/allmydata/scripts/tahoe_backup.py", line 188, in process childcap = self.process(childpath) File "/usr/lib/python2.7/dist-packages/allmydata/scripts/tahoe_backup.py", line 194, in process childcap, metadata = self.upload(childpath) File "/usr/lib/python2.7/dist-packages/allmydata/scripts/tahoe_backup.py", line 313, in upload bdb_results.did_upload(filecap) File "/usr/lib/python2.7/dist-packages/allmydata/scripts/backupdb.py", line 122, in did_upload self.mtime, self.ctime, self.size) File "/usr/lib/python2.7/dist-packages/allmydata/scripts/backupdb.py", line 264, in did_upload_file fileid = self.get_or_allocate_fileid_for_cap(filecap) File "/usr/lib/python2.7/dist-packages/allmydata/scripts/backupdb.py", line 256, in get_or_allocate_fileid_for_cap c.execute("SELECT fileid FROM caps WHERE filecap=?", (filecap,)) OperationalError: database is locked ```

tahoe-lafs added the

labels 2014-08-27 04:47:16 +00:00

tahoe-lafs added this to the undecided milestone 2014-08-27 04:47:16 +00:00

daira commented

2014-08-27 14:27:49 +00:00

See also #2062.

zooko commented

2014-08-28 01:08:11 +00:00

I'll bet we could fix this simply by setting PRAGMA busy_timeout to a large value (like 3000 milliseconds). This would, if I understand correctly, cause the second one of the tahoe backup processes that attempted to write to a table to block until the first one finished its write and committed the transaction. This would only take a tiny fraction of a second (especially if we turned on journal_mode="WAL" and synchronous="NORMAL", like we do in the leasedb branch).

The only question I have is: how would we test a patch which made this change!

I'll bet we could fix this simply by setting [PRAGMA busy_timeout](https://www.sqlite.org/pragma.html#pragma_busy_timeout) to a large value (like 3000 milliseconds). This would, if I understand correctly, cause the second one of the `tahoe backup` processes that attempted to write to a table to block until the first one finished its write and committed the transaction. This would only take a tiny fraction of a second (especially if we turned on `journal_mode="WAL"` and `synchronous="NORMAL"`, like we do [in the leasedb branch](https://github.com/tahoe-lafs/tahoe-lafs/blame/006a04976eb42f56c118c34adf2ddb54c1605edb/src/allmydata/storage/leasedb.py#L130)). The only question I have is: how would we *test* a patch which made this change!

daira commented

2014-08-28 15:21:34 +00:00

Replying to zooko:

I'll bet we could fix this simply by setting PRAGMA busy_timeout to a large value (like 3000 milliseconds). This would, if I understand correctly, cause the second one of the tahoe backup processes that attempted to write to a table to block until the first one finished its write and committed the transaction.

I think that's correct.

The only question I have is: how would we test a patch which made this change!

I wouldn't mind having the test just check that the intended pragmas are set (it should also check the existing pragmas, which I don't think are currently tested).

Replying to [zooko](/tahoe-lafs/trac-2024-07-25/issues/2285#issuecomment-95645): > I'll bet we could fix this simply by setting [PRAGMA busy_timeout](https://www.sqlite.org/pragma.html#pragma_busy_timeout) to a large value (like 3000 milliseconds). This would, if I understand correctly, cause the second one of the `tahoe backup` processes that attempted to write to a table to block until the first one finished its write and committed the transaction. I think that's correct. > The only question I have is: how would we *test* a patch which made this change! I wouldn't mind having the test just check that the intended pragmas are set (it should also check the existing pragmas, which I don't think are currently tested).

Sign in to join this conversation.