We maintain three kinds of tests: unit tests, integration tests,
and acceptance tests.
Overall, you want to write the tests that **maximize coverage**
while **minimizing maintenance**.
In practice, this usually means investing heavily
in unit tests, which tend to be the most robust to changes in the code base.
The pyramid above shows the relative number of unit tests, integration tests,
and acceptance tests. Most of our tests are unit tests or integration tests.
### Unit Tests
* Each test case should be concise: setup, execute, check, and teardown.
...
...
@@ -31,8 +41,8 @@ You do not need to test every possible input--that's what unit
tests are for. Instead, focus on testing the "happy path"
to verify that the components work together correctly.
* Many of our tests use the [Django test client](https://docs.djangoproject.com/en/dev/topics/testing/overview/) to simulate
HTTP requests to the server.
* Many of our tests use the [Django test client](https://docs.djangoproject.com/en/dev/topics/testing/overview/)
to simulate HTTP requests to the server.
### UI Acceptance Tests
* Use these to test that major program features are working correctly.
...
...
@@ -41,15 +51,9 @@ HTTP requests to the server.
these tests simulate user interactions through the browser using
[splinter](http://splinter.cobrateam.info/).
Overall, you want to write the tests that **maximize coverage**
while **minimizing maintenance**.
In practice, this usually means investing heavily
in unit tests, which tend to be the most robust to changes in the code base.
The pyramid above shows the relative number of unit tests, integration tests,
and acceptance tests. Most of our tests are unit tests or integration tests.
* We use [Bok Choy](http://bok-choy.readthedocs.org/en/latest/tutorial.html) to
write end-user acceptance tests directly in Python, using the framework to
maximize reliability and maintainability.
## Test Locations
...
...
@@ -65,9 +69,10 @@ and test. For example, the test for `src/views/module.coffee`
should be written in `spec/views/module_spec.coffee`.
* UI acceptance tests:
- Set up and helper methods: `common/djangoapps/terrain`
- Tests: located in `features` subpackage within a Django app.
- Set up and helper methods, and stubs for external services: `common/djangoapps/terrain`
-Lettuce Tests: located in `features` subpackage within a Django app.
For example: `lms/djangoapps/courseware/features`
- Bok Choy Tests: Artifacts are located under `common/test/acceptance`
## Factories
...
...
@@ -199,7 +204,56 @@ To run JavaScript tests in your default browser:
These rake commands call through to a custom test runner. For more info, see [js-test-tool](https://github.com/edx/js-test-tool).
### Running Acceptance Tests
### Running Bok Choy Acceptance Tests
We use [Bok Choy](http://bok-choy.readthedocs.org/en/latest/tutorial.html) for acceptance testing.
Bok Choy is a UI-level acceptance test framework for writing robust [Selenium](http://docs.seleniumhq.org/) tests in [Python](https://www.python.org/).
Bok Choy makes your acceptance tests reliable and maintainable by utilizing the Page Object and Promise design patterns.
**Prerequisites**:
* These prerequisites are all automatically installed and available in [Devstack](https://github.com/edx/configuration/wiki/edX-Developer-Stack),
the supported development enviornment for the edX Platform.
* Chromedriver and Chrome (see Running Lettuce Acceptance Tests below for the latest tested versions)
* Mongo
* Memcache
* mySQL
To run all the bok choy acceptance tests:
rake test:bok_choy
Once the database has been set up and the static files collected, you can use the 'fast'
option to skip those tasks. This option can also be used with any of the test specs below:
rake test:bok_choy:fast
To run single test, specify the name of the test file. For example:
rake test:bok_choy[test_lms.py]
To run single test faster by not repeating setup tasks:
rake test:bok_choy:fast[test_lms.py]
To test only a certain feature, specify the file and the testcase class:
