Add binary builds to tahoe-lafs.org's buildbot/download page #2729

New Issue

crwood · 2016-02-09T08:21:44Z

crwood commented

2016-02-09 08:21:44 +00:00

I've recently put together some simple build scripts to help automate the creation binary distributions of Tahoe-LAFS for Windows and Mac using PyInstaller. Like bb-freeze, py2exe, py2app, and other related projects, with enough tinkering, Pyinstaller can be used to create self-contained distributions of python programs that can effectively be run "out of the box" by end users. Given the rather convoluted state of python packaging (and with it, the additional mess of horrors that typically comes with building Tahoe-LAFS from scratch), I think it would be desirable to have such packages available to end users as an alternative to the current offering of a manual install.

The aforementioned scripts can be found here (for Windows) and here (for OS X) and are currently being used by Gridsync's buildbot (which follows Tahoe-LAFS' upstream master branch and publishes the resultant binary packages here). It shouldn't be terribly difficult to port these over to tahoe-lafs.org's buildbot and I'd be happy to help out with the process (or even lend some buildslaves to it, if needed).

A few explanatory notes:

A ".spec" file is needed to inform Pyinstaller of any required files or modules that might not otherwise be detected by analyzing the package's import statements. The one I put together for tahoe here pulls in the static files needed for the web UI (as well as the 'characteristic' and 'cffi' -- which, for whatever reason, aren't automatically picked up by Pyinstaller). If you use this, the specified filepaths will probably need to be changed.
The above spec file generates one-folder bundles by choice rather than one-file binaries since the one-file approach will fail if the system tempdir is mounted with the NOEXEC flag (and one-file mode works by extracting bundled files to the system tempdir and running them from there).
On Windows, OpenSSL will need to be installed in accordance with Daira's instructions outlined here. The windows script will build OpenSSL if run with the --with-openssl flag but the other dependencies specified at the top of the file will still need to be installed manually. (This is no longer necessary)
The build scripts patch out the setuptools requirement from _auto_deps.py before running pyinstaller which allows tahoe to run frozen. Maybe this isn't necessary and maybe there's a better way to do it (I do see some code to this effect in _auto_deps.py) but I couldn't get tahoe running frozen without patching as init.py's check_requirement() fails with "PackagingError: no version info for setuptools". More investigation is needed..
Pyinstaller builds can be made reproducible (assuming the same platform, interpreter version, and architecture) by setting the PYTHONHASHSEED environment variable to a known/shared value. Note that this only affects the pythonic bits (and not, e.g., the C libraries); given the current dependencies, more work is needed to make these packages fully reproducible (but even so remains a worthy goal).
Pyinstaller builds are forwards-compatible but not (always) backwards-compatible; it's better to build on older operating system versions if possible.

I've recently put together some simple build [scripts](https://github.com/gridsync/frozen-tahoe) to help automate the creation binary distributions of Tahoe-LAFS for Windows and Mac using [PyInstaller](http://www.pyinstaller.org/). Like bb-freeze, py2exe, py2app, and other related projects, with enough tinkering, Pyinstaller can be used to create self-contained distributions of python programs that can effectively be run "out of the box" by end users. Given the rather convoluted state of python packaging (and with it, the additional mess of horrors that typically comes with building Tahoe-LAFS from scratch), I think it would be desirable to have such packages available to end users as an alternative to the current offering of a manual install. The aforementioned scripts can be found [here](https://github.com/gridsync/frozen-tahoe/blob/master/make-frozen-tahoe.bat) (for Windows) and [here](https://github.com/gridsync/frozen-tahoe/blob/master/make-frozen-tahoe.sh) (for OS X) and are currently being used by Gridsync's buildbot (which follows Tahoe-LAFS' upstream master branch and publishes the resultant binary packages [here](https://buildbot.gridsync.io/tahoe-lafs/)). It shouldn't be terribly difficult to port these over to tahoe-lafs.org's buildbot and I'd be happy to help out with the process (or even lend some buildslaves to it, if needed). A few explanatory notes: * A ".spec" file is needed to inform Pyinstaller of any required files or modules that might not otherwise be detected by analyzing the package's import statements. The one I put together for tahoe [here](https://github.com/gridsync/frozen-tahoe/blob/master/tahoe.spec) pulls in the static files needed for the web UI (as well as the 'characteristic' and 'cffi' -- which, for whatever reason, aren't automatically picked up by Pyinstaller). If you use this, the specified filepaths will probably need to be changed. * The above spec file generates one-folder bundles by choice rather than one-file binaries since the one-file approach will fail if the system tempdir is mounted with the NOEXEC flag (and one-file mode works by extracting bundled files to the system tempdir and running them from there). * ~~On Windows, OpenSSL will need to be installed in accordance with Daira's instructions outlined [here](https://github.com/tahoe-lafs/tahoe-lafs/blob/195.windows-packaging.10/docs/build/windows-installer.rst). The windows script will build OpenSSL if run with the `--with-openssl` flag but the other dependencies specified at the top of the file will still need to be installed manually.~~ (This is no longer necessary) * The build scripts patch out the setuptools requirement from _auto_deps.py before running pyinstaller which allows tahoe to run frozen. Maybe this isn't necessary and maybe there's a better way to do it (I do see [some code to this effect in _auto_deps.py](https://github.com/tahoe-lafs/tahoe-lafs/blob/master/src/allmydata/_auto_deps.py#L137)) but I couldn't get tahoe running frozen without patching as *init.py*'s check_requirement() fails with "PackagingError: no version info for setuptools". More investigation is needed.. * Pyinstaller builds can be made reproducible (assuming the same platform, interpreter version, and architecture) by setting the PYTHONHASHSEED environment variable to a known/shared value. Note that this only affects the pythonic bits (and not, e.g., the C libraries); given the current dependencies, more work is needed to make these packages fully reproducible (but even so remains a worthy goal). * Pyinstaller builds are forwards-compatible but not (always) backwards-compatible; it's better to build on older operating system versions if possible.

crwood added the

labels 2016-02-09 08:21:44 +00:00

crwood added this to the undecided milestone 2016-02-09 08:21:44 +00:00

warner commented

2016-12-14 01:11:24 +00:00

In today's devchat, we decided to try to get this done shortly after the 1.12 release, so hopefully in early january.

warner modified the milestone from undecided to 1.13.0

2016-12-14 01:11:24 +00:00

crwood commented

2017-06-22 14:22:18 +00:00

Submitted PR #421 for this.

Submitted [PR #421](https://github.com/tahoe-lafs/tahoe-lafs/pull/421) for this.

Brian Warner <warner@lothar.com> commented

2017-08-10 20:04:18 +00:00

In d91516a/trunk:

Merge PR421

closes tahoe-lafs/tahoe-lafs#421
refs ticket:2729

In [d91516a/trunk](/tahoe-lafs/trac-2024-07-25/commit/d91516a5c5c7d6c6a8d040aed4302aa2b9e3bf90): ``` Merge PR421 closes tahoe-lafs/tahoe-lafs#421 refs ticket:2729 ```

exarkun commented

2019-07-25 12:56:34 +00:00

The PR is closed. Why is the ticket still open?