test framework for web apps that use JavaScript #1001

New Issue

tahoe-lafs · 2010-03-17T02:02:39Z

davidsarah commented

2010-03-17 02:02:39 +00:00

If we add dependencies on JavaScript to Tahoe, such as the 'Tahoe Explorer' frontend as proposed in #1000, we will need a test framework for the JS code. Ideally, it should be able to test interactions between JavaScript and Python, and automatically run tests in multiple web browsers.

tahoe-lafs added the

labels 2010-03-17 02:02:39 +00:00

tahoe-lafs added this to the undecided milestone 2010-03-17 02:02:39 +00:00

zooko commented

2010-03-17 03:59:00 +00:00

I looked into what node.js uses since I consider node.js to be inspired by Twisted (at least by Twisted's event-based concurrency) and Twisted has an excellent tradition of testing. Here's what I found: http://debuggable.com/posts/unit-testing-with-node-js:4b647d40-34e4-435a-a880-3b04cbdd56cb

I looked into what node.js uses since I consider node.js to be inspired by Twisted (at least by Twisted's event-based concurrency) and Twisted has an excellent tradition of testing. Here's what I found: <http://debuggable.com/posts/unit-testing-with-node-js:4b647d40-34e4-435a-a880-3b04cbdd56cb>

davidsarah commented

2010-04-13 02:22:59 +00:00

I've been investigating Windmill, which seems to have similar functionality to Selenium in terms of web browser testing, but which uses Python as its scripting language (in addition to JavaScript). It looks like a very good fit, because we need to be able to script interactions between Python code and a web app (for example, setting up a temporary grid + directory structure in Python and then testing the web app against it, or hooking Python objects to make sure they are receiving the intended requests from the app).

I was initially concerned that Windmill might be overkill. However, if we need to be able to test Tahoe Explorer (#1000) as well as Josip Lisec's Music player and Hasini Gunasinghe's Blog for GSoC (if those projects are accepted), then I think we may end up needing a fair amount of the functionality that it provides over and above simpler JS test frameworks like JSUnit or node.js.

Windmill is Apache 2.0-licensed.

I've been investigating [Windmill](http://www.getwindmill.com/), which seems to have similar functionality to Selenium in terms of web browser testing, but which uses Python as its scripting language (in addition to JavaScript). It looks like a very good fit, because we need to be able to script interactions between Python code and a web app (for example, setting up a temporary grid + directory structure in Python and then testing the web app against it, or hooking Python objects to make sure they are receiving the intended requests from the app). I was initially concerned that Windmill might be overkill. However, if we need to be able to test Tahoe Explorer (#1000) as well as [Josip Lisec's Music player](http://socghop.appspot.com/gsoc/student_proposal/review/google/gsoc2010/josip_lisec/t127060317230) and [Hasini Gunasinghe's Blog](http://socghop.appspot.com/gsoc/student_proposal/review/google/gsoc2010/hasini_gunasinghe/t127081707686) for GSoC (if those projects are accepted), then I think we may end up needing a fair amount of the functionality that it provides over and above simpler JS test frameworks like JSUnit or node.js. Windmill is Apache 2.0-licensed.

tahoe-lafs modified the milestone from undecided to 1.8.0

2010-04-13 02:22:59 +00:00

tahoe-lafs changed title from ~~test framework for JavaScript~~ to test framework for web apps that use JavaScript

2010-04-13 02:31:09 +00:00

davidsarah commented

2010-04-13 02:36:58 +00:00

Attachment test_explorer.py (948 bytes) added

Dummy test that opens Firefox, IE, Safari and Chrome using Windmill

**Attachment** test_explorer.py (948 bytes) added Dummy test that opens Firefox, IE, Safari and Chrome using Windmill

test_explorer.py

948 B

davidsarah commented

2010-04-13 02:54:31 +00:00

Attachment tilting.py (2193 bytes) added

Simple hack to run Windmill tests from a trial test suite

**Attachment** tilting.py (2193 bytes) added Simple hack to run Windmill tests from a trial test suite

tilting.py

2.1 KiB

davidsarah commented

2010-04-13 02:56:35 +00:00

Attachment tilting.2.py (9 bytes) added

ignore me (file deleted)

**Attachment** tilting.2.py (9 bytes) added ignore me (file deleted)

tilting.2.py

9 B

josipl commented

2010-04-14 14:06:07 +00:00

I agree with David-Sarah that Windmill, in the long run, is better option.

It probably is a bit overkill for simpler tests, but assertion module from node.js just can't cover all of the functionality - with node.js one can't test the actual UI part since there is no DOM API provided by default. Module which provides DOM API exists but is far from complete and it has dependencies on other modules/tools, and thus I'm not sure if introducing all these dependencies is worth the trouble.

On the other hand, Windmill runs the tests directly in the browser and thus allows testing every aspect of app's functionality. Also, Windmill's IDE seems like an rather excellent tool.

I agree with David-Sarah that Windmill, in the long run, is better option.<br> It probably is a bit overkill for simpler tests, but assertion module from node.js just can't cover all of the functionality - with node.js one can't test the actual UI part since there is no DOM API provided by default. [Module](http://github.com/tmpvar/jsdom) which provides DOM API exists but is far from [complete](http://www.devcomments.com/Node-js-YUI-3-Dom-Manipulation-Oh-My-i24506.htm) and it has dependencies on other modules/tools, and thus I'm not sure if introducing all these dependencies is worth the trouble. On the other hand, Windmill runs the tests directly in the browser and thus allows testing every aspect of app's functionality. Also, [Windmill's IDE](http://www.youtube.com/watch?v=ZHBBtf3nKRQ) seems like an rather excellent tool.

davidsarah commented

2010-04-15 18:21:05 +00:00

Replying to josipl:

... with node.js one can't test the actual UI part since there is no DOM API provided by default. Module which provides DOM API exists but is far from complete and it has dependencies on other modules/tools, and thus I'm not sure if introducing all these dependencies is worth the trouble.

Also, that would only be testing against that particular implementation of the DOM, rather than the implementations used by Firefox, IE, Safari, and Chrome.

I'm going to attach a patch that adds a dependency on Windmill to source:_auto_deps.py . Please test that this doesn't break anything on any platform (it doesn't on win32), but don't commit to trunk yet.

Replying to [josipl](/tahoe-lafs/trac-2024-07-25/issues/1001#issuecomment-76553): > ... with node.js one can't test the actual UI part since there is no DOM API provided by default. [Module](http://github.com/tmpvar/jsdom) which provides DOM API exists but is far from [complete](http://www.devcomments.com/Node-js-YUI-3-Dom-Manipulation-Oh-My-i24506.htm) and it has dependencies on other modules/tools, and thus I'm not sure if introducing all these dependencies is worth the trouble. Also, that would only be testing against that particular implementation of the DOM, rather than the implementations used by Firefox, IE, Safari, and Chrome. I'm going to attach a patch that adds a dependency on Windmill to source:_auto_deps.py . Please test that this doesn't break anything on any platform (it doesn't on win32), but don't commit to trunk yet.

davidsarah commented

2010-04-15 18:33:12 +00:00

Oh, please test that after applying this patch and installing Tahoe using python setup.py install, that python -c "import windmill; help(windmill)" works. It should print a package description.

Oh, please test that after applying this patch and installing Tahoe using `python setup.py install`, that `python -c "import windmill; help(windmill)"` works. It should print a package description.

davidsarah commented

2010-04-16 19:16:10 +00:00

Attachment add-windmill-dependency-darcspatch.txt (1794 bytes) added

Add dependency on Windmill (fixed patch)

**Attachment** add-windmill-dependency-darcspatch.txt (1794 bytes) added Add dependency on Windmill (fixed patch)

add-windmill-dependency-darcspatch.txt

1.8 KiB

davidsarah commented

2010-04-23 23:58:13 +00:00

The patch has been applied, and hasn't caused any build failures so far (but not all of the buildslaves are on-line at the moment).

zooko commented

2010-05-04 10:14:06 +00:00

Now that more builders are working, we can see that this patch breaks the "Desert Island" test:

http://tahoe-lafs.org/buildbot/builders/clean/builds/2262/steps/test-desert-island/logs/stdio

This build step ends with:

python misc/check-build.py make.out no-downloads

Reading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-sdists/
Reading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/
Reading http://pypi.python.org/simple/windmill/
Reading http://www.getwindmill.com/
Reading http://windmill.osafoundation.org/
Downloading http://pypi.python.org/packages/source/w/windmill/windmill-1.3.tar.gz#md5=1f5a8fca648beeaacbfc8f0e01d41c8c
Failed: build tried to download files

To make it pass this test, add a copy of windmill into the file that it fetches at the beginning of the test:

test -f tahoe-deps.tar.gz || wget http://allmydata.org/source/tahoe/deps/tahoe-deps.tar.gz

Now that more builders are working, we can see that this patch breaks the "Desert Island" test: <http://tahoe-lafs.org/buildbot/builders/clean/builds/2262/steps/test-desert-island/logs/stdio> This build step ends with: ``` python misc/check-build.py make.out no-downloads Reading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-sdists/ Reading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/ Reading http://pypi.python.org/simple/windmill/ Reading http://www.getwindmill.com/ Reading http://windmill.osafoundation.org/ Downloading http://pypi.python.org/packages/source/w/windmill/windmill-1.3.tar.gz#md5=1f5a8fca648beeaacbfc8f0e01d41c8c Failed: build tried to download files ``` To make it pass this test, add a copy of windmill into the file that it fetches at the beginning of the test: ``` test -f tahoe-deps.tar.gz || wget http://allmydata.org/source/tahoe/deps/tahoe-deps.tar.gz ```

davidsarah commented

2010-05-04 16:14:14 +00:00

Since we don't actually use Windmill yet, and it's 3.2 MiB, I think we should just comment out the dependency for 1.7. We've already tested that it installs on all supported platforms, which was the intention of applying the patch.

davidsarah commented

2010-05-04 17:57:22 +00:00

Replying to davidsarah:

Since we don't actually use Windmill yet, and it's 3.2 MiB, I think we should just comment out the dependency for 1.7.

Done in changeset:43b092ffeb7cae9e.

Replying to [davidsarah](/tahoe-lafs/trac-2024-07-25/issues/1001#issuecomment-76558): > Since we don't actually use Windmill yet, and it's 3.2 MiB, I think we should just comment out the dependency for 1.7. Done in changeset:43b092ffeb7cae9e.

josipl commented

2010-05-13 19:55:21 +00:00

I'm attaching a modified version of tilting.py which was discussed the other day in IRC along with another dummy test.

TiltingMixin's setUp method now starts an introducer and storage nodes. Results of !JavaScript tests are being reported in a manner which test suite can easier pickup.

test_dummy.py file contains a dummy test which copies code of an imaginary application to public_html directory of a started storage node and then starts !JavaScript tests.

There are some unresolved issues with code: I couldn't find a way to disable Windmill from printing to stdout (someone on Windmill's IRC channel suggested that it should be simply used as a Nose plugin).
The other problem, is the way waiting for introducer.furl file is implemented - using a loop and sleep(), this will be fixed once Python API for creating/starting/stopping nodes gets implemented.

I'm attaching a modified version of tilting.py which was discussed the other day in IRC along with another dummy test. `TiltingMixin`'s `setUp` method now starts an introducer and storage nodes. Results of !JavaScript tests are being reported in a manner which test suite can easier pickup. `test_dummy.py` file contains a dummy test which copies code of an imaginary application to `public_html` directory of a started storage node and then starts !JavaScript tests. There are some unresolved issues with code: I couldn't find a way to disable Windmill from printing to stdout (someone on Windmill's IRC channel suggested that it should be simply used as a Nose plugin). The other problem, is the way waiting for introducer.furl file is implemented - using a loop and sleep(), this will be fixed once Python API for creating/starting/stopping nodes gets implemented.

josipl commented

2010-05-13 19:56:46 +00:00

Attachment tilting3.py (6421 bytes) added

**Attachment** tilting3.py (6421 bytes) added

tilting3.py

6.3 KiB

josipl commented

2010-05-13 19:58:17 +00:00

Attachment test_dummy.py (515 bytes) added

Dummy test which should run JS tests and open Firefox.

**Attachment** test_dummy.py (515 bytes) added Dummy test which should run JS tests and open Firefox.

test_dummy.py