tahoe-lafs
/
trac-2024-07-25
2
0
Issues 1.1k Wiki

setuptools/easy_install/pip-installed package lacks documentation #1659

New Issue
Open
opened 2012-01-18 03:23:59 +00:00 by nejucomo · 9 comments
nejucomo commented 2012-01-18 03:23:59 +00:00
Copy Link

I am a happy user of the pip installability feature, whereby I run "pip install allmydata-tahoe" to get a running tahoe installation.

This installation lacks documentation, at least where I sometimes find other python documentation in $VIRTUALENV_ROOT/share/doc/$PACKAGE_NAME.

Request: Fix the pypi distribution to bundle all documentation such that pip-installed tahoe provides documentation in this location.

If documentation is installed elsewhere, ensure that it is compatible with virtualenv. Specifically I should be able to install different versions of tahoe-lafs in different virtualenvs and have the correct copy of documentation for each install.

I am a happy user of the pip installability feature, whereby I run "pip install allmydata-tahoe" to get a running tahoe installation. This installation lacks documentation, at least where I sometimes find other python documentation in $VIRTUALENV_ROOT/share/doc/$PACKAGE_NAME. Request: Fix the pypi distribution to bundle all documentation such that pip-installed tahoe provides documentation in this location. If documentation is installed elsewhere, ensure that it is compatible with virtualenv. Specifically I should be able to install different versions of tahoe-lafs in different virtualenvs and have the correct copy of documentation for each install.
nejucomo added the
unknown
major
defect
1.9.0
 labels 2012-01-18 03:23:59 +00:00
nejucomo added this to the undecided milestone 2012-01-18 03:23:59 +00:00
nejucomo commented 2013-07-20 18:57:45 +00:00
Author
Copy Link

Note that the title of this ticket is misleading. This has nothing to do with pip. I added the pypi keyword, but this also is not specific to Tahoe-LAFS's PyPI page.

I just verified that if I download the pypi tarball or the self-hosted zip file, and then run:

$ python ./setup.py build
$ python ./setup.py install

-there are no doc files installed outside of the tarball/zipfile contents.

Note that the title of this ticket is misleading. This has nothing to do with `pip`. I added the `pypi` keyword, but this also is not specific to Tahoe-LAFS's [PyPI page](https://pypi.python.org/pypi/allmydata-tahoe/1.10.0). I just verified that if I download the [pypi tarball](https://pypi.python.org/pypi/allmydata-tahoe/1.10.0) or the [self-hosted zip file](https://tahoe-lafs.org/source/tahoe-lafs/releases/allmydata-tahoe-1.10.0.zip), and then run: ``` $ python ./setup.py build $ python ./setup.py install ``` -there are no doc files installed outside of the tarball/zipfile contents.
nejucomo commented 2013-07-20 19:01:55 +00:00
Author
Copy Link

I just added the proposed enhancement #2031 which is to serve ./docs from the web gateway. That would necessarily solve this issue by bundling all of ./docs as a package resource directory.

I just added the proposed enhancement #2031 which is to serve `./docs` from the web gateway. That would necessarily solve this issue by bundling all of `./docs` as a package resource directory.
tahoe-lafs changed title from pip installed-package lacks documentation. to setuptools/easy_install/pip-installed package lacks documentation 2013-07-22 23:02:40 +00:00
nejucomo commented 2013-07-26 05:48:49 +00:00
Author
Copy Link

I believe the summary should actually be changed to "packages installed from an extracted sdist tarball do not install documentation", but I haven't tested that thoroughly enough yet.

I believe the summary should actually be changed to "packages installed from an extracted sdist tarball do not install documentation", but I haven't tested that thoroughly enough yet.
nejucomo commented 2013-07-26 05:54:42 +00:00
Author
Copy Link

I just created this change to setup.py which causes all of ./docs and some other useful files like README.txt to be installed in $PREFIX/share/doc/allmydata_tahoe when installed with pip.

Next I'd like to test that this is also the case when I run python ./setup.py install --single-version-externally-managed from an unpacked sdist.

If that also works, then I would propose to merge that branch.

There are a few sticking points in that branch:

  • It replicates some information from MANIFEST.in. (I found while doing this that MANIFEST.in refers to non-existent paths, and that python ./setup.py sdist does not complain about these missing files, so it has accumulated some bitrot.)
  • It may use the wrong path format for all platforms... I suspect it should use distutils.util.convert_path() somewhere.
  • There's no test code. Testing build/install scaffolding is difficult!
I just created [this change to setup.py](https://github.com/nejucomo/tahoe-lafs/compare/1659-bundle-docs-in-sdist) which causes all of `./docs` and some other useful files like `README.txt` to be installed in `$PREFIX/share/doc/allmydata_tahoe` when installed with pip. Next I'd like to test that this is also the case when I run `python ./setup.py install --single-version-externally-managed` from an unpacked sdist. If that also works, then I would propose to merge that branch. There are a few sticking points in that branch: * It replicates some information from `MANIFEST.in`. (I found while doing this that `MANIFEST.in` refers to non-existent paths, and that `python ./setup.py sdist` does not complain about these missing files, so it has accumulated some bitrot.) * It may use the wrong path format for all platforms... I suspect it should use `distutils.util.convert_path()` somewhere. * There's no test code. Testing build/install scaffolding is difficult!
nejucomo commented 2013-07-26 06:18:16 +00:00
Author
Copy Link

Here's a demo of the 1659-bundle-docs-in-sdist branch using pip to install from a git repository:

 $ git rev-parse --abbrev-ref HEAD ; git rev-parse HEAD
1659-bundle-docs-in-sdist
b0ca93b732ba47cdd14d9b70996525c7cc8c85a1


$ TMP="$(mktemp --tmpdir -d 1659-bundle-docs-in-sdist.demo-XXXXXX)"


$ virtualenv "$TMP"
Using real prefix '/usr/local/bin/../stow/Python-2.7.3-setuptools-0.6c11'
New python executable in /home/n/tmp/1659-bundle-docs-in-sdist.demo-I4S29A/bin/python
Installing setuptools...............done.
Installing pip.................done.


$ source "$TMP/bin/activate"
(1659-bundle-docs-in-sdist.demo-I4S29A)

$ pip install twisted > $TMP/pip-install-twisted.log # necessary work around; see https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2032
(1659-bundle-docs-in-sdist.demo-I4S29A)

$ pip install . > $TMP/pip-install-dot.log
(1659-bundle-docs-in-sdist.demo-I4S29A)

$ cd "$VIRTUAL_ENV"
(1659-bundle-docs-in-sdist.demo-I4S29A)

$ ls ./share/doc/allmydata-tahoe/
COPYING.GPL  COPYING.TGPPL.rst  CREDITS  docs  NEWS.rst  README.txt  relnotes.txt
(1659-bundle-docs-in-sdist.demo-I4S29A)

$ find share/doc/ -type d
share/doc/
share/doc/zfec
share/doc/pyutil
share/doc/pycryptopp
share/doc/allmydata-tahoe
share/doc/allmydata-tahoe/docs
share/doc/allmydata-tahoe/docs/historical
share/doc/allmydata-tahoe/docs/frontends
share/doc/allmydata-tahoe/docs/proposed
share/doc/allmydata-tahoe/docs/specifications
share/doc/allmydata-tahoe/docs/specifications/backends
share/doc/allmydata-tahoe/docs/man
(1659-bundle-docs-in-sdist.demo-I4S29A)
Here's a demo of the [1659-bundle-docs-in-sdist](https://github.com/nejucomo/tahoe-lafs/tree/1659-bundle-docs-in-sdist) branch using `pip` to install from a `git` repository: ``` $ git rev-parse --abbrev-ref HEAD ; git rev-parse HEAD 1659-bundle-docs-in-sdist b0ca93b732ba47cdd14d9b70996525c7cc8c85a1 $ TMP="$(mktemp --tmpdir -d 1659-bundle-docs-in-sdist.demo-XXXXXX)" $ virtualenv "$TMP" Using real prefix '/usr/local/bin/../stow/Python-2.7.3-setuptools-0.6c11' New python executable in /home/n/tmp/1659-bundle-docs-in-sdist.demo-I4S29A/bin/python Installing setuptools...............done. Installing pip.................done. $ source "$TMP/bin/activate" (1659-bundle-docs-in-sdist.demo-I4S29A) $ pip install twisted > $TMP/pip-install-twisted.log # necessary work around; see https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2032 (1659-bundle-docs-in-sdist.demo-I4S29A) $ pip install . > $TMP/pip-install-dot.log (1659-bundle-docs-in-sdist.demo-I4S29A) $ cd "$VIRTUAL_ENV" (1659-bundle-docs-in-sdist.demo-I4S29A) $ ls ./share/doc/allmydata-tahoe/ COPYING.GPL COPYING.TGPPL.rst CREDITS docs NEWS.rst README.txt relnotes.txt (1659-bundle-docs-in-sdist.demo-I4S29A) $ find share/doc/ -type d share/doc/ share/doc/zfec share/doc/pyutil share/doc/pycryptopp share/doc/allmydata-tahoe share/doc/allmydata-tahoe/docs share/doc/allmydata-tahoe/docs/historical share/doc/allmydata-tahoe/docs/frontends share/doc/allmydata-tahoe/docs/proposed share/doc/allmydata-tahoe/docs/specifications share/doc/allmydata-tahoe/docs/specifications/backends share/doc/allmydata-tahoe/docs/man (1659-bundle-docs-in-sdist.demo-I4S29A) ```
nejucomo commented 2013-07-26 06:36:15 +00:00
Author
Copy Link

Here is a demo which does not use pip nor virtualenv:

$ git rev-parse --abbrev-ref HEAD ; git rev-parse HEAD
1659-bundle-docs-in-sdist
b0ca93b732ba47cdd14d9b70996525c7cc8c85a1

$ TMP="$(mktemp --tmpdir -d lafs-install-no-virtual-env-or-pip.demo-XXXXXX)"

$ python ./setup.py sdist > "$TMP/sdist.log"
warning: no previously-included files matching '*~' found anywhere in distribution

$ tar -xf ./dist/allmydata-tahoe-1.10.0.post32.tar.gz -C /usr/local/src

$ cd /usr/local/src/allmydata-tahoe-1.10.0.post32/

$ python ./setup.py install --single-version-externally-managed --record "$TMP/install-record" --prefix='/usr/local' > "$TMP/install.log"
warning: no previously-included files matching '*~' found anywhere in distribution

$ find /usr/local/share/doc/allmydata-tahoe/ -type d
/usr/local/share/doc/allmydata-tahoe/
/usr/local/share/doc/allmydata-tahoe/docs
/usr/local/share/doc/allmydata-tahoe/docs/historical
/usr/local/share/doc/allmydata-tahoe/docs/frontends
/usr/local/share/doc/allmydata-tahoe/docs/proposed
/usr/local/share/doc/allmydata-tahoe/docs/specifications
/usr/local/share/doc/allmydata-tahoe/docs/specifications/backends
/usr/local/share/doc/allmydata-tahoe/docs/man
Here is a demo which does not use `pip` nor `virtualenv`: ``` $ git rev-parse --abbrev-ref HEAD ; git rev-parse HEAD 1659-bundle-docs-in-sdist b0ca93b732ba47cdd14d9b70996525c7cc8c85a1 $ TMP="$(mktemp --tmpdir -d lafs-install-no-virtual-env-or-pip.demo-XXXXXX)" $ python ./setup.py sdist > "$TMP/sdist.log" warning: no previously-included files matching '*~' found anywhere in distribution $ tar -xf ./dist/allmydata-tahoe-1.10.0.post32.tar.gz -C /usr/local/src $ cd /usr/local/src/allmydata-tahoe-1.10.0.post32/ $ python ./setup.py install --single-version-externally-managed --record "$TMP/install-record" --prefix='/usr/local' > "$TMP/install.log" warning: no previously-included files matching '*~' found anywhere in distribution $ find /usr/local/share/doc/allmydata-tahoe/ -type d /usr/local/share/doc/allmydata-tahoe/ /usr/local/share/doc/allmydata-tahoe/docs /usr/local/share/doc/allmydata-tahoe/docs/historical /usr/local/share/doc/allmydata-tahoe/docs/frontends /usr/local/share/doc/allmydata-tahoe/docs/proposed /usr/local/share/doc/allmydata-tahoe/docs/specifications /usr/local/share/doc/allmydata-tahoe/docs/specifications/backends /usr/local/share/doc/allmydata-tahoe/docs/man ```
zooko commented 2013-07-26 14:08:21 +00:00
Copy Link

Replying to nejucomo:

  • It may use the wrong path format for all platforms... I suspect it should use distutils.util.convert_path() somewhere.
  • There's no test code. Testing build/install scaffolding is difficult!

We have a (kludgey) way to test such things. Here is the output on buildbot:

A test of ['python', '-c', "import glob, os, subprocess, sys; os.mkdir('egginstalldir'); tahoe_egg = glob.glob(os.path.join('dist', '*.egg'))[0]; sys.exit(subprocess.call(['easy_install', '-d', 'egginstalldir', tahoe_egg]))"]:

https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Kyle%20OpenBSD%20amd64/builds/235/steps/install-to-egg/logs/stdio

A test of ['python', '-c', "import subprocess, sys;sys.exit(subprocess.call(['setup.py', 'install', '--single-version-externally-managed', '--record=record.txt', '--prefix', 'prefixinstalldir']sys.executable,))"]:

https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Kyle%20OpenBSD%20amd64/builds/235/steps/install-to-prefix/logs/stdio

In both cases, they are followed by executing the unit tests from within the installed location:

https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Kyle%20OpenBSD%20amd64/builds/235/steps/test-from-egg

and

https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Kyle%20OpenBSD%20amd64/builds/235/steps/test-from-prefixdir

So to test that the docs are installed correctly, I guess we could add another Buildbot BuildStep which runs after the install and before the unit-tests-run-from-installed, which just asserts that the right files are in place.

Nathan: interested in contributing such a test?

Replying to [nejucomo](/tahoe-lafs/trac-2024-07-25/issues/1659#issuecomment-87161): > > * It may use the wrong path format for all platforms... I suspect it should use `distutils.util.convert_path()` somewhere. > * There's no test code. Testing build/install scaffolding is difficult! We have a (kludgey) way to test such things. Here is the output on buildbot: A test of `['python', '-c', "import glob, os, subprocess, sys; os.mkdir('egginstalldir'); tahoe_egg = glob.glob(os.path.join('dist', '*.egg'))[0]; sys.exit(subprocess.call(['easy_install', '-d', 'egginstalldir', tahoe_egg]))"]`: <https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Kyle%20OpenBSD%20amd64/builds/235/steps/install-to-egg/logs/stdio> A test of `['python', '-c', "import subprocess, sys;sys.exit(subprocess.call(['setup.py', 'install', '--single-version-externally-managed', '--record=record.txt', '--prefix', 'prefixinstalldir']sys.executable,))"]`: <https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Kyle%20OpenBSD%20amd64/builds/235/steps/install-to-prefix/logs/stdio> In both cases, they are followed by executing the unit tests from within the installed location: <https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Kyle%20OpenBSD%20amd64/builds/235/steps/test-from-egg> and <https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Kyle%20OpenBSD%20amd64/builds/235/steps/test-from-prefixdir> So to test that the docs are installed correctly, I guess we could add another Buildbot [BuildStep](wiki/BuildStep) which runs after the install and before the unit-tests-run-from-installed, which just asserts that the right files are in place. Nathan: interested in contributing such a test?
nejucomo commented 2013-07-26 17:09:14 +00:00
Author
Copy Link

Since I've already started a patch for this, I've assigned it to myself.

Since I've already started a patch for this, I've assigned it to myself.
daira commented 2014-09-07 17:31:09 +00:00
Owner
Copy Link

What's the status of this ticket?

What's the status of this ticket?
warner added
documentation
 and removed
unknown
 labels 2014-09-11 22:21:45 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
0.2.0

   
0.3.0

   
0.4.0

   
0.5.0

   
0.5.1

   
0.6.0

   
0.6.1

   
0.7.0

   
0.8.0

   
0.9.0

   
1.0.0

   
1.1.0

   
1.10.0

   
1.10.1

   
1.10.2

   
1.10a2

   
1.11.0

   
1.12.0

   
1.12.1

   
1.13.0

   
1.14.0

   
1.15.0

   
1.15.1

   
1.2.0

   
1.3.0

   
1.4.1

   
1.5.0

   
1.6.0

   
1.6.1

   
1.7.0

   
1.7.1

   
1.7β

   
1.8.0

   
1.8.1

   
1.8.2

   
1.8.3

   
1.8β

   
1.9.0

   
1.9.0-s3branch

   
1.9.0a1

   
1.9.0a2

   
1.9.0b1

   
1.9.1

   
1.9.2

   
1.9.2a1

   
LeastAuthority.com automation

   
blocker

   
cannot reproduce

   
cloud-branch

   
code

catch-all for anything that can be fixed by writing code but isn't classified better

  
code-dirnodes

anything involving tahoe directories

  
code-encoding

immutable-file encoding format/layout, cap format, encoding/decoding algorithms

  
code-frontend

general frontend things: new user-facing interfaces, FTP/SFTP servers, FUSE-like interfaces

  
code-frontend-cli

CLI tools

  
code-frontend-ftp-sftp

   
code-frontend-magic-folder

   
code-frontend-web

webapi interfaces, user-facing "WUI" interfaces

  
code-mutable

anything involving mutable files: encoding/decoding, share format

  
code-network

Introducer, locating storage servers, establishing/maintaining connections to them

  
code-nodeadmin

node startup/shutdown, tools to create nodes, logging tools, management/monitoring tools, manhole, disk-space-limiting tools

  
code-peerselection

server selection/sorting algorithm, rebalancing code, allocation policy, swarming-download

  
code-storage

storage-server implementation, share-storage directory layout, client-to-server share transfer protocol

  
contrib

problems with non-core things that live in the source:contrib/ directory

  
critical

   
defect

   
dev-infrastructure

things that help us build Tahoe: buildbot, automated speed/memory/code-coverage tests and results-displayers, IRC channels, mailing lists, this Trac instance

  
documentation

docs items. Also use 'docs' keyword on tickets in more specific components.

  
duplicate

   
enhancement

   
fixed

   
invalid

   
major

   
minor

   
n/a

   
normal

   
operational

things involved in running a tahoe grid, external tools to monitor/maintain a grid (like "nodeadmin" but larger-scale). Was also used for issues with the allmydata.com grid.

  
packaging

setup.py script, library dependencies, binary packages, licensing, installation instructions

  
somebody else's problem

   
supercritical

   
task

   
trivial

   
unknown

Uncategorized. All tickets default to this component until someone updates it.

  
was already fixed

   
website

things about the tahoe-lafs.org website (and sometimes this Trac instance, shared with 'dev-infrastructure')

  
wontfix

   
worksforme
No Label
0.2.0
0.3.0
0.4.0
0.5.0
0.5.1
0.6.0
0.6.1
0.7.0
0.8.0
0.9.0
1.0.0
1.1.0
1.10.0
1.10.1
1.10.2
1.10a2
1.11.0
1.12.0
1.12.1
1.13.0
1.14.0
1.15.0
1.15.1
1.2.0
1.3.0
1.4.1
1.5.0
1.6.0
1.6.1
1.7.0
1.7.1
1.7β
1.8.0
1.8.1
1.8.2
1.8.3
1.8β
1.9.0
1.9.0-s3branch
1.9.0a1
1.9.0a2
1.9.0b1
1.9.1
1.9.2
1.9.2a1
LeastAuthority.com automation
blocker
cannot reproduce
cloud-branch
code
code-dirnodes
code-encoding
code-frontend
code-frontend-cli
code-frontend-ftp-sftp
code-frontend-magic-folder
code-frontend-web
code-mutable
code-network
code-nodeadmin
code-peerselection
code-storage
contrib
critical
defect
dev-infrastructure
documentation
duplicate
enhancement
fixed
invalid
major
minor
n/a
normal
operational
packaging
somebody else's problem
supercritical
task
trivial
unknown
was already fixed
website
wontfix
worksforme
Milestone
Clear milestone
No items
No Milestone
undecided
Assignees
Clear assignees
No Assignees
nejucomo
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Reference: tahoe-lafs/trac-2024-07-25#1659
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?