We maintain three kinds of tests: unit tests, integration tests,
We maintain three kinds of tests: unit tests, integration tests,
and acceptance tests.
and acceptance tests.
Overall, you want to write the tests that **maximize coverage**
while **minimizing maintenance**.
In practice, this usually means investing heavily
in unit tests, which tend to be the most robust to changes in the code base.
The pyramid above shows the relative number of unit tests, integration tests,
and acceptance tests. Most of our tests are unit tests or integration tests.
### Unit Tests
### Unit Tests
* Each test case should be concise: setup, execute, check, and teardown.
* Each test case should be concise: setup, execute, check, and teardown.
...
@@ -31,8 +41,8 @@ You do not need to test every possible input--that's what unit
...
@@ -31,8 +41,8 @@ You do not need to test every possible input--that's what unit
tests are for. Instead, focus on testing the "happy path"
tests are for. Instead, focus on testing the "happy path"
to verify that the components work together correctly.
to verify that the components work together correctly.
* Many of our tests use the [Django test client](https://docs.djangoproject.com/en/dev/topics/testing/overview/) to simulate
* Many of our tests use the [Django test client](https://docs.djangoproject.com/en/dev/topics/testing/overview/)
HTTP requests to the server.
to simulate HTTP requests to the server.
### UI Acceptance Tests
### UI Acceptance Tests
* Use these to test that major program features are working correctly.
* Use these to test that major program features are working correctly.
...
@@ -41,15 +51,9 @@ HTTP requests to the server.
...
@@ -41,15 +51,9 @@ HTTP requests to the server.
these tests simulate user interactions through the browser using
these tests simulate user interactions through the browser using
[splinter](http://splinter.cobrateam.info/).
[splinter](http://splinter.cobrateam.info/).
Overall, you want to write the tests that **maximize coverage**
* We use [Bok Choy](http://bok-choy.readthedocs.org/en/latest/tutorial.html) to
while **minimizing maintenance**.
write end-user acceptance tests directly in Python, using the framework to
In practice, this usually means investing heavily
maximize reliability and maintainability.
in unit tests, which tend to be the most robust to changes in the code base.
The pyramid above shows the relative number of unit tests, integration tests,
and acceptance tests. Most of our tests are unit tests or integration tests.
## Test Locations
## Test Locations
...
@@ -65,9 +69,10 @@ and test. For example, the test for `src/views/module.coffee`
...
@@ -65,9 +69,10 @@ and test. For example, the test for `src/views/module.coffee`
should be written in `spec/views/module_spec.coffee`.
should be written in `spec/views/module_spec.coffee`.
* UI acceptance tests:
* UI acceptance tests:
- Set up and helper methods: `common/djangoapps/terrain`
- Set up and helper methods, and stubs for external services: `common/djangoapps/terrain`
- Tests: located in `features` subpackage within a Django app.
-Lettuce Tests: located in `features` subpackage within a Django app.
For example: `lms/djangoapps/courseware/features`
For example: `lms/djangoapps/courseware/features`
- Bok Choy Tests: Artifacts are located under `common/test/acceptance`
## Factories
## Factories
...
@@ -199,75 +204,56 @@ To run JavaScript tests in your default browser:
...
@@ -199,75 +204,56 @@ To run JavaScript tests in your default browser:
These rake commands call through to a custom test runner. For more info, see [js-test-tool](https://github.com/edx/js-test-tool).
These rake commands call through to a custom test runner. For more info, see [js-test-tool](https://github.com/edx/js-test-tool).
### Running Acceptance Tests (Bok Choy)
### Running Bok Choy Acceptance Tests
We use [Bok Choy](http://bok-choy.readthedocs.org/en/latest/tutorial.html) for acceptance testing.
We use [Bok Choy](http://bok-choy.readthedocs.org/en/latest/tutorial.html) for acceptance testing.
Bok Choy is a UI-level acceptance test framework for writing robust [Selenium](http://docs.seleniumhq.org/) tests in [Python](https://www.python.org/).
Bok Choy is a UI-level acceptance test framework for writing robust [Selenium](http://docs.seleniumhq.org/) tests in [Python](https://www.python.org/).
Bok Choy makes your acceptance tests reliable and maintainable by utilizing the Page Object and Promise design patterns.
Bok Choy makes your acceptance tests reliable and maintainable by utilizing the Page Object and Promise design patterns.
**Prerequisite**: You must have [ChromeDriver](https://code.google.com/p/selenium/wiki/ChromeDriver) installed to run the tests in Chrome. The tests are confirmed to run with Chrome (not Chromium) version 28.0.1500.71 with ChromeDriver version 2.1.210398.
**Prerequisites**:
* These prerequisites are all automatically installed and available in [Devstack](https://github.com/edx/configuration/wiki/edX-Developer-Stack),
the supported development enviornment for the edX Platform.
* Chromedriver and Chrome (see Running Lettuce Acceptance Tests below for the latest tested versions)
* Mongo
* Memcache
* mySQL
To run all the acceptance tests:
To run all the bok choy acceptance tests:
rake test:bok_choy
rake test:bok_choy
To run tests faster by not collecting static files, you can use:
Once the database has been set up and the static files collected, you can use the 'fast'
option to skip those tasks. This option can also be used with any of the test specs below:
rake test:bok_choy:fast
rake test:bok_choy:fast
To run single test:
To run single test, specify the name of the test file. For example:
rake test:bok_choy[TEST_FILE_NAME]
example
rake test:bok_choy[test_lms.py]
rake test:bok_choy[test_lms.py]
To run single test faster by not collecting static files:
To run single test faster by not repeating setup tasks:
rake test:bok_choy:fast[TEST_FILE_NAME]
rake test:bok_choy:fast[test_lms.py]
rake test:bok_choy:fast[test_lms.py]
To test only specific feature:
To test only a certain feature, specify the file and the testcase class:
