make it work with bbfreeze #585

Closed
opened 2009-01-15 03:00:05 +00:00 by zooko · 29 comments

bbfreeze -- http://pypi.python.org/pypi/bbfreeze -- is an alternative to py2exe and (in the near future) to py2app. It is very easy to use:

easy_install bbfreeze
python ./setup.py bdist_bbfreeze

Except that it currently doesn't work for Tahoe on Windows -- something about our (very tricky and complicated) bin/tahoe hack is outwitting bbfreeze maybe?

bbfreeze -- <http://pypi.python.org/pypi/bbfreeze> -- is an alternative to py2exe and (in the near future) to py2app. It is very easy to use: ``` easy_install bbfreeze python ./setup.py bdist_bbfreeze ``` Except that it currently doesn't work for Tahoe on Windows -- something about our (very tricky and complicated) `bin/tahoe` hack is outwitting bbfreeze maybe?
zooko added the
packaging
major
defect
1.2.0
labels 2009-01-15 03:00:05 +00:00
zooko added this to the undecided milestone 2009-01-15 03:00:05 +00:00
Author

Unassigning cgalvan since he hasn't been active in a couple of months.

Unassigning cgalvan since he hasn't been active in a couple of months.
tahoe-lafs added
enhancement
and removed
defect
labels 2009-11-08 01:08:12 +00:00
Owner

This works for me on 1.6.0:

python setup.py build
python setup.py install
cd windows
bb-freeze tahoe.py

tahoe.exe is created in dist directory and runs okay.

This works for me on 1.6.0: ``` python setup.py build python setup.py install cd windows bb-freeze tahoe.py ``` tahoe.exe is created in dist directory and runs okay.
davidsarah commented 2010-03-02 00:57:05 +00:00
Owner

For me mids' recipe does produce an executable, but it references an .egg file that does not contain any resources, such as .xhtml files. This results in exceptions when trying to view the WUI of a node run using this executable.

For me mids' recipe does produce an executable, but it references an `.egg` file that does not contain any resources, such as `.xhtml` files. This results in exceptions when trying to view the WUI of a node run using this executable.
davidsarah commented 2010-03-02 01:01:29 +00:00
Owner

The exception when viewing the WUI was:

<type 'exceptions.WindowsError'>: [Error 2] The system cannot find the file specified:
'd:\\tahoe\\1.6.1\\tahoe-lafs\\windows\\dist\\allmydata_tahoe-1.6.0_r4256-py2.6.egg\\allmydata\\web\\directory.xhtml'

(That file indeed does not exist, although the directory does and contains .pyc files.)

zooko wrote on IRC:

[.egg]The is supposed to include [files]resource because (a) we said "include_package_data=True" as an argument to setup(), (b) that .xhtml file is registered under darcs revision control and (c) the setuptools_darcs plugin is installed.

The exception when viewing the WUI was: ``` <type 'exceptions.WindowsError'>: [Error 2] The system cannot find the file specified: 'd:\\tahoe\\1.6.1\\tahoe-lafs\\windows\\dist\\allmydata_tahoe-1.6.0_r4256-py2.6.egg\\allmydata\\web\\directory.xhtml' ``` (That file indeed does not exist, although the directory does and contains `.pyc` files.) zooko wrote on IRC: > [`.egg`]The is supposed to include [files]resource because (a) we said "include_package_data=True" as an argument to setup(), (b) that `.xhtml` file is registered under darcs revision control and (c) the setuptools_darcs plugin is installed.
Author

Replying to davidsarah:

For me mids' recipe does produce an executable, but it references an .egg file that does not contain any resources, such as .xhtml files. This results in exceptions when trying to view the WUI of a node run using this executable.

Hey! This was ticket #1033, which has been fixed. Maybe now mids's recipe will work! :-)

Replying to [davidsarah](/tahoe-lafs/trac-2024-07-25/issues/585#issuecomment-69020): > For me mids' recipe does produce an executable, but it references an `.egg` file that does not contain any resources, such as `.xhtml` files. This results in exceptions when trying to view the WUI of a node run using this executable. Hey! This was ticket #1033, which has been fixed. Maybe now mids's recipe will work! :-)
zooko modified the milestone from undecided to soon 2010-06-21 21:02:37 +00:00
davidsarah commented 2011-01-03 07:01:03 +00:00
Owner

Replying to [zooko]comment:7:

Replying to davidsarah:

For me mids' recipe does produce an executable, but it references an .egg file that does not contain any resources, such as .xhtml files. This results in exceptions when trying to view the WUI of a node run using this executable.

Hey! This was ticket #1033, which has been fixed. Maybe now mids's recipe will work! :-)

If it did before, it won't now, because the windows directory has been removed by #1282.

After much trial and error, I found the following recipe instead. First apply bb-freeze.darcs.patch to a checkout of trunk. Then from a cygwin prompt:

make clean
python setup.py build
sed 's/\\r//' <src/allmydata/_version.py >_version.new
mv -f _version.new src/allmydata/_version.py
export PYTHONPATH=support/Lib/site-packages
bb-freeze windows/tahoe.py

(cygwin is not essential, but we need sed to convert the CRLF line endings in _version.py, otherwise the compiler invoked by bb-freeze will fail to compile it. I don't know why it is so picky.)

The resulting dist\tahoe.exe has a bug that causes its argv array to be shifted, dropping sys.argv[0]. You can work around this by giving 'tahoe' as a dummy first argument.

I haven't tested it much, but it can run a functioning gateway node to the pubgrid, and list the public directory using the CLI. Unicode arguments and output even work!

Replying to [zooko]comment:7: > Replying to [davidsarah](/tahoe-lafs/trac-2024-07-25/issues/585#issuecomment-69020): > > For me mids' recipe does produce an executable, but it references an `.egg` file that does not contain any resources, such as `.xhtml` files. This results in exceptions when trying to view the WUI of a node run using this executable. > > Hey! This was ticket #1033, which has been fixed. Maybe now mids's recipe will work! :-) If it did before, it won't now, because the windows directory has been removed by #1282. After much trial and error, I found the following recipe instead. First apply [bb-freeze.darcs.patch](/tahoe-lafs/trac-2024-07-25/attachments/000078ac-a32c-5cfe-e764-98bee4072501) to a checkout of trunk. Then from a cygwin prompt: ``` make clean python setup.py build sed 's/\\r//' <src/allmydata/_version.py >_version.new mv -f _version.new src/allmydata/_version.py export PYTHONPATH=support/Lib/site-packages bb-freeze windows/tahoe.py ``` (cygwin is not essential, but we need sed to convert the CRLF line endings in _version.py, otherwise the compiler invoked by bb-freeze will fail to compile it. I don't know why it is so picky.) The resulting `dist\tahoe.exe` has a bug that causes its argv array to be shifted, dropping `sys.argv[0]`. You can work around this by giving 'tahoe' as a dummy first argument. I haven't tested it much, but it can run a functioning gateway node to the pubgrid, and list the public directory using the CLI. Unicode arguments and output even work!
davidsarah commented 2011-01-03 07:01:50 +00:00
Owner

Attachment bb-freeze.darcs.patch (13398 bytes) added

Various hacks to get bb-freeze to mostly work. Do not apply to trunk. refs #585

**Attachment** bb-freeze.darcs.patch (13398 bytes) added Various hacks to get bb-freeze to mostly work. Do not apply to trunk. refs #585
davidsarah commented 2011-01-03 07:14:24 +00:00
Owner

Attachment bb-freeze.darcs.2.patch (13928 bytes) added

Various hacks to get bb-freeze to mostly work. Definitely do not apply to trunk. This version of the patch includes the workaround of adding as the dummy 'tahoe' argument so you don't need to. refs #585

**Attachment** bb-freeze.darcs.2.patch (13928 bytes) added Various hacks to get bb-freeze to mostly work. Definitely do not apply to trunk. This version of the patch includes the workaround of adding as the dummy 'tahoe' argument so you don't need to. refs #585
davidsarah commented 2011-01-03 07:20:45 +00:00
Owner

Replying to [davidsarah]comment:8:

I haven't tested it much, but it can run a functioning gateway node to the pubgrid, and list the public directory using the CLI.

And the WUI works without the resources problem in comment:69020, I meant to say.

Replying to [davidsarah]comment:8: > I haven't tested it much, but it can run a functioning gateway node to the pubgrid, and list the public directory using the CLI. And the WUI works without the resources problem in [comment:69020](/tahoe-lafs/trac-2024-07-25/issues/585#issuecomment-69020), I meant to say.
davidsarah commented 2011-01-04 12:03:24 +00:00
Owner

I worked out why the arguments are off-by-one -- it is because the code to retrieve Unicode arguments in source:src/allmydata/windows/fixups.py depends on the script having been invoked via the Python interpreter, so that unicode_argv[0] (the python executable) needs to be skipped. The executable produced by bbfreeze, OTOH, is not invoked via the Python interpreter. This problem wouldn't have occurred before the fix to #1074 in Tahoe v1.8.0, which explains why it worked for mids.

bb-freeze.darcs.3.patch (which depends on the patch for #1287) has now been updated to solve this problem in a way that doesn't interfere with the behaviour of the usual bin/tahoe.

This attachment also includes a patch to the bundled darcsver, to make it write _version.py with LF line endings, by changing "wt+" to "wb+" here. This matches all of the other Python source files, so should not cause any problems. (We should make a darcsver release, this patch to the bundled version is just for testing.)

windows/tahoe.py has been moved to static/tahoe.py because it is not specific to Windows. It should probably work for any [http://www.freehackers.org/Packaging_a_python_program static Python executable packager], although I haven't tested any others.

I've changed the patch to only omit the setuptools version and path from the output of get_package_versions_and_locations() when running the bb-frozen executable (which does not actually have setuptools as a runtime dependency). The problem that this is addressing is that if the executable tries to import setuptools, it can fail to find the original site module and exit at that point.

I also refactored a bunch of stuff in __init__.py and _auto_deps.py. get_package_versions_and_locations() is now in _auto_deps.py because it needs to be updated whenever install_requires is. It suppresses deprecation warnings while importing modules, so that the bb-frozen executable doesn't print these warnings. Finally, we now correctly check that the versions of imported modules actually satisfy the requirements (except for pyasn1, where we don't know the version number). Previously, require_auto_deps() only checked that the distributions that pkg_resources ''attempted'' to put on the sys.path satisfy the requirements that it was told to satisfy. That is not sufficient due to #1246, #1258 and possibly other bugs. I needed to simplify the pycrypto requirement to >= 2.3, since the new check doesn't support disjunctive requirements.

The recipe is now:

make clean
python setup.py build
export PYTHONPATH=support/Lib/site-packages
bb-freeze static/tahoe.py

(This should work on a Windows command prompt, modulo availability of 'make', and set instead of export.)

We need to be able to test the resulting executable. I will file another ticket for that.

I worked out why the arguments are off-by-one -- it is because the code to retrieve Unicode arguments in source:src/allmydata/windows/fixups.py depends on the script having been invoked via the Python interpreter, so that `unicode_argv[0]` (the python executable) needs to be skipped. The executable produced by bbfreeze, OTOH, is not invoked via the Python interpreter. This problem wouldn't have occurred before the fix to #1074 in Tahoe v1.8.0, which explains why it worked for mids. [bb-freeze.darcs.3.patch](/tahoe-lafs/trac-2024-07-25/attachments/000078ac-a32c-5cfe-e764-1bbbfe2cb0a2) (which depends on the patch for #1287) has now been updated to solve this problem in a way that doesn't interfere with the behaviour of the usual `bin/tahoe`. This attachment also includes a patch to the bundled darcsver, to make it write `_version.py` with LF line endings, by changing "wt+" to "wb+" [here](http://tahoe-lafs.org/trac/darcsver/browser/trunk/darcsver/darcsvermodule.py#L50). This matches all of the other Python source files, so should not cause any problems. (We should make a darcsver release, this patch to the bundled version is just for testing.) `windows/tahoe.py` has been moved to `static/tahoe.py` because it is not specific to Windows. It should probably work for any [http://www.freehackers.org/Packaging_a_python_program static Python executable packager], although I haven't tested any others. I've changed the patch to only omit the setuptools version and path from the output of `get_package_versions_and_locations()` when running the bb-frozen executable (which does not actually have setuptools as a runtime dependency). The problem that this is addressing is that if the executable tries to import `setuptools`, it can fail to find the original `site` module and exit at that point. I also refactored a bunch of stuff in `__init__.py` and `_auto_deps.py`. `get_package_versions_and_locations()` is now in `_auto_deps.py` because it needs to be updated whenever `install_requires` is. It suppresses deprecation warnings while importing modules, so that the bb-frozen executable doesn't print these warnings. Finally, we now correctly check that the versions of imported modules actually satisfy the requirements (except for pyasn1, where we don't know the version number). Previously, `require_auto_deps()` only checked that the distributions that `pkg_resources` ''attempted'' to put on the `sys.path` satisfy the requirements that it was told to satisfy. That is not sufficient due to #1246, #1258 and possibly other bugs. I needed to simplify the pycrypto requirement to `>= 2.3`, since the new check doesn't support disjunctive requirements. The recipe is now: ``` make clean python setup.py build export PYTHONPATH=support/Lib/site-packages bb-freeze static/tahoe.py ``` (This should work on a Windows command prompt, modulo availability of 'make', and set instead of export.) We need to be able to test the resulting executable. I will file another ticket for that.
davidsarah commented 2011-01-04 12:05:58 +00:00
Owner

Attachment bb-freeze.darcs.3.patch (41381 bytes) added

Make bbfreeze work. This also refactors allmydata/init.py and _auto_deps.py and implements more robust checking of dependency requirements. It includes a change to the bundled darcsvar, to write the _version.py file with LF line endings.

**Attachment** bb-freeze.darcs.3.patch (41381 bytes) added Make bbfreeze work. This also refactors allmydata/*init*.py and _auto_deps.py and implements more robust checking of dependency requirements. It includes a change to the bundled darcsvar, to write the _version.py file with LF line endings.
tahoe-lafs modified the milestone from soon to 1.9.0 2011-01-04 12:08:58 +00:00
davidsarah commented 2011-01-07 10:41:19 +00:00
Owner

Attachment static-tahoe.darcs.patch (12074 bytes) added

static/tahoe.py: import mock (for 'tahoe debug trial') and xml.tree.minidom (to avoid noise from twisted plugins). refs #585

**Attachment** static-tahoe.darcs.patch (12074 bytes) added static/tahoe.py: import mock (for 'tahoe debug trial') and xml.tree.minidom (to avoid noise from twisted plugins). refs #585
davidsarah commented 2011-01-07 10:44:34 +00:00
Owner

With bb-freeze.darcs.3.patch + static-tahoe.darcs.patch + attachment🎫1296:tahoe-debug-trial.darcs.patch , we pass everything except for skipping a few runner tests (which I will fix tomorrow).

With [bb-freeze.darcs.3.patch](/tahoe-lafs/trac-2024-07-25/attachments/000078ac-a32c-5cfe-e764-1bbbfe2cb0a2) + [static-tahoe.darcs.patch](/tahoe-lafs/trac-2024-07-25/attachments/000078ac-a32c-5cfe-e764-84db4bea70c7) + attachment:ticket:1296:tahoe-debug-trial.darcs.patch , we pass everything except for skipping a few runner tests (which I will fix tomorrow).
davidsarah commented 2011-01-07 10:46:30 +00:00
Owner

Use dist\tahoe debug trial to test.

Use `dist\tahoe debug trial` to test.
david-sarah@jacaranda.org commented 2011-01-15 05:14:15 +00:00
Owner

In [4938/ticket1306]:

Add static/tahoe.py, to support bbfreeze, py2exe etc. refs #585
In [4938/ticket1306]: ``` Add static/tahoe.py, to support bbfreeze, py2exe etc. refs #585 ```
david-sarah@jacaranda.org commented 2011-01-15 05:14:16 +00:00
Owner

In [4939/ticket1306]:

static/tahoe.py: import mock (for 'tahoe debug trial') and xml.tree.minidom (to avoid noise from twisted plugins). refs #585
In [4939/ticket1306]: ``` static/tahoe.py: import mock (for 'tahoe debug trial') and xml.tree.minidom (to avoid noise from twisted plugins). refs #585 ```
david-sarah@jacaranda.org commented 2011-01-15 05:14:17 +00:00
Owner

In [4942/ticket1306]:

Make bb-freeze work. This also implements more robust checking of dependency requirements, as discussed in ticket:585#comment:10 . refs #585
In [4942/ticket1306]: ``` Make bb-freeze work. This also implements more robust checking of dependency requirements, as discussed in ticket:585#comment:10 . refs #585 ```
david-sarah@jacaranda.org commented 2011-01-15 05:14:17 +00:00
Owner

In [4943/ticket1306]:

Move get_package_versions_and_locations from __init__.py to _auto_deps.py (because it needs to be updated when dependencies change). refs #585
In [4943/ticket1306]: ``` Move get_package_versions_and_locations from __init__.py to _auto_deps.py (because it needs to be updated when dependencies change). refs #585 ```
tahoe-lafs modified the milestone from 1.9.0 to 1.8.2 2011-01-16 21:25:15 +00:00
david-sarah@jacaranda.org commented 2011-01-19 05:40:51 +00:00
Owner

In changeset:93c1909847e9dc44:

bin/tahoe-script.template, src/windows/fixups.py: simplify the method of stripping initial arguments in sys.argv on Windows. This helps with bb-freeze and running tahoe via 'coverage'. Also includes some wording changes and minor refactoring of bin/tahoe-script.template. refs #585, #1303
In changeset:93c1909847e9dc44: ``` bin/tahoe-script.template, src/windows/fixups.py: simplify the method of stripping initial arguments in sys.argv on Windows. This helps with bb-freeze and running tahoe via 'coverage'. Also includes some wording changes and minor refactoring of bin/tahoe-script.template. refs #585, #1303 ```
Author

I reviewed changeset:93c1909847e9dc44 and it looks good to me.

I reviewed changeset:93c1909847e9dc44 and it looks good to me.
Author

[20110113043040-93fa1-2a5ccfc48e2fe53b3fa5a91e913fbd300ccd98f4]/ticket1306 looks good. (I eyeballed it but didn't actually copy out the text and use diff to confirm that the text didn't get changed while it was getting moved.)

[20110113043040-93fa1-2a5ccfc48e2fe53b3fa5a91e913fbd300ccd98f4]/ticket1306 looks good. (I eyeballed it but didn't actually copy out the text and use `diff` to confirm that the text didn't get changed while it was getting moved.)
Author

[20110107085259-93fa1-386c985e5bebe82119025222c9c39a43aa5b40b4]/ticket1306 looks fine.

[20110107085259-93fa1-386c985e5bebe82119025222c9c39a43aa5b40b4]/ticket1306 looks fine.
Author

Okay I read through [20110113042457-93fa1-b0c23a181f94c19e75407448bdd776c9a065f1a1]/ticket1306. I can't claim to understand it 100%, but I couldn't see any defect or think of any design problem with it. +0.9 :-)

Okay I read through [20110113042457-93fa1-b0c23a181f94c19e75407448bdd776c9a065f1a1]/ticket1306. I can't claim to understand it 100%, but I couldn't see any defect or think of any design problem with it. +0.9 :-)
zooko removed their assignment 2011-01-19 07:51:03 +00:00
warner was assigned by zooko 2011-01-19 07:51:03 +00:00
Author

David-Sarah is going to merge these patches into trunk.

David-Sarah is going to merge these patches into trunk.
david-sarah@jacaranda.org commented 2011-01-21 09:00:03 +00:00
Owner

In changeset:8587c63bdc291faf:

Make bb-freeze (and probably other static packaging tools) work. This updates various places where we assumed that the tahoe process was executed via the Python interpreter. It also allows tests to recursively invoke the same tahoe.exe, rather than bin/tahoe. refs #585
In changeset:8587c63bdc291faf: ``` Make bb-freeze (and probably other static packaging tools) work. This updates various places where we assumed that the tahoe process was executed via the Python interpreter. It also allows tests to recursively invoke the same tahoe.exe, rather than bin/tahoe. refs #585 ```

I'm guessing that changeset:8587c63bdc291faf is meant to close this ticket. Please re-open if I'm wrong.

I'm guessing that changeset:8587c63bdc291faf is meant to close this ticket. Please re-open if I'm wrong.
warner added the
fixed
label 2011-01-21 17:22:45 +00:00

davidsarah mentioned on IRC about an hour ago that there's one more patch to go in (".tac files import pkg_resources, so that's still a dependency"). Reopening for that.

davidsarah mentioned on IRC about an hour ago that there's one more patch to go in (".tac files import pkg_resources, so that's still a dependency"). Reopening for that.
warner removed the
fixed
label 2011-01-21 23:54:09 +00:00
warner reopened this issue 2011-01-21 23:54:09 +00:00
david-sarah@jacaranda.org commented 2011-01-22 03:42:30 +00:00
Owner

In changeset:c8ccb297fa7f499d:

src/allmydata/_auto_deps.py: setuptools is still an install requirement even in frozen builds, because .tac files import pkg_resources. refs #585
In changeset:c8ccb297fa7f499d: ``` src/allmydata/_auto_deps.py: setuptools is still an install requirement even in frozen builds, because .tac files import pkg_resources. refs #585 ```
davidsarah commented 2011-01-22 04:57:17 +00:00
Owner

The PYTHONPATH is not right in the recipe in comment:69026. Using a Command Prompt, do:

python setup.py build
set PYTHONPATH=<tahoedir>\support\Lib\site-packages;<tahoedir>\src
bb-freeze static\tahoe.py

where <tahoedir> is the full path to the Tahoe source directory.

(make clean is only necessary if dependencies have changed.)

I'll add this to wiki/AdvancedInstall.

The PYTHONPATH is not right in the recipe in [comment:69026](/tahoe-lafs/trac-2024-07-25/issues/585#issuecomment-69026). Using a Command Prompt, do: ``` python setup.py build set PYTHONPATH=<tahoedir>\support\Lib\site-packages;<tahoedir>\src bb-freeze static\tahoe.py ``` where `<tahoedir>` is the full path to the Tahoe source directory. (`make clean` is only necessary if dependencies have changed.) I'll add this to [wiki/AdvancedInstall](wiki/AdvancedInstall).
tahoe-lafs added the
fixed
label 2011-01-22 04:57:17 +00:00
davidsarah closed this issue 2011-01-22 04:57:17 +00:00
Sign in to join this conversation.
No Milestone
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Reference: tahoe-lafs/trac-2024-07-25#585
No description provided.