Skip to content

E
edx-platform
Commit 1fbe43a6 by Arthur Barrett
Options

Merge branch 'master' into feature/abarrett/lms-notes-app

parents 430e009b fca63f06
Showing with 1113 additions and 887 deletions
.gitignore
...@@ -36,3 +36,7 @@ chromedriver.log ...@@ -36,3 +36,7 @@ chromedriver.log
/nbproject /nbproject
ghostdriver.log ghostdriver.log
node_modules node_modules
.pip_download_cache/
.prereqs_cache
autodeploy.properties
.ws_migrations_complete
README.md
...@@ -8,7 +8,7 @@ Installation ...@@ -8,7 +8,7 @@ Installation
The installation process is a bit messy at the moment. Here's a high-level The installation process is a bit messy at the moment. Here's a high-level
overview of what you should do to get started. overview of what you should do to get started.
**TLDR:** There is a `create-dev-env.sh` script that will attempt to set all **TLDR:** There is a `scripts/create-dev-env.sh` script that will attempt to set all
of this up for you. If you're in a hurry, run that script. Otherwise, I suggest of this up for you. If you're in a hurry, run that script. Otherwise, I suggest
that you understand what the script is doing, and why, by reading this document. that you understand what the script is doing, and why, by reading this document.
...@@ -77,11 +77,16 @@ environment), and Node has a library installer called ...@@ -77,11 +77,16 @@ environment), and Node has a library installer called
Once you've got your languages and virtual environments set up, install Once you've got your languages and virtual environments set up, install
the libraries like so: the libraries like so:
$ pip install -r pre-requirements.txt $ pip install -r requirements/base.txt
$ pip install -r requirements.txt $ pip install -r requirements/post.txt
$ bundle install $ bundle install
$ npm install $ npm install
You can also use [`rake`](http://rake.rubyforge.org/) to get all of the prerequisites (or to update)
them if they've changed
$ rake install_prereqs
Other Dependencies Other Dependencies
------------------ ------------------
You'll also need to install [MongoDB](http://www.mongodb.org/), since our You'll also need to install [MongoDB](http://www.mongodb.org/), since our
...@@ -137,7 +142,7 @@ Studio, visit `127.0.0.1:8001` in your web browser; to view the LMS, visit ...@@ -137,7 +142,7 @@ Studio, visit `127.0.0.1:8001` in your web browser; to view the LMS, visit
There's also an older version of the LMS that saves its information in XML files There's also an older version of the LMS that saves its information in XML files
in the `data` directory, instead of in Mongo. To run this older version, run: in the `data` directory, instead of in Mongo. To run this older version, run:
$ rake lms $ rake lms
Further Documentation Further Documentation
===================== =====================
......
cms/djangoapps/contentstore/features/advanced-settings.feature
...@@ -11,7 +11,8 @@ Feature: Advanced (manual) course policy ...@@ -11,7 +11,8 @@ Feature: Advanced (manual) course policy
Given I am on the Advanced Course Settings page in Studio Given I am on the Advanced Course Settings page in Studio
Then the settings are alphabetized Then the settings are alphabetized
@skip-phantom # Skipped because Ubuntu ChromeDriver cannot click notification "Cancel"
@skip
Scenario: Test cancel editing key value Scenario: Test cancel editing key value
Given I am on the Advanced Course Settings page in Studio Given I am on the Advanced Course Settings page in Studio
When I edit the value of a policy key When I edit the value of a policy key
...@@ -20,7 +21,8 @@ Feature: Advanced (manual) course policy ...@@ -20,7 +21,8 @@ Feature: Advanced (manual) course policy
And I reload the page And I reload the page
Then the policy key value is unchanged Then the policy key value is unchanged
@skip-phantom # Skipped because Ubuntu ChromeDriver cannot click notification "Save"
@skip
Scenario: Test editing key value Scenario: Test editing key value
Given I am on the Advanced Course Settings page in Studio Given I am on the Advanced Course Settings page in Studio
When I edit the value of a policy key and save When I edit the value of a policy key and save
...@@ -28,7 +30,8 @@ Feature: Advanced (manual) course policy ...@@ -28,7 +30,8 @@ Feature: Advanced (manual) course policy
And I reload the page And I reload the page
Then the policy key value is changed Then the policy key value is changed
@skip-phantom # Skipped because Ubuntu ChromeDriver cannot edit CodeMirror input
@skip
Scenario: Test how multi-line input appears Scenario: Test how multi-line input appears
Given I am on the Advanced Course Settings page in Studio Given I am on the Advanced Course Settings page in Studio
When I create a JSON object as a value When I create a JSON object as a value
...@@ -36,7 +39,8 @@ Feature: Advanced (manual) course policy ...@@ -36,7 +39,8 @@ Feature: Advanced (manual) course policy
And I reload the page And I reload the page
Then it is displayed as formatted Then it is displayed as formatted
@skip-phantom # Skipped because Ubuntu ChromeDriver cannot edit CodeMirror input
@skip
Scenario: Test automatic quoting of non-JSON values Scenario: Test automatic quoting of non-JSON values
Given I am on the Advanced Course Settings page in Studio Given I am on the Advanced Course Settings page in Studio
When I create a non-JSON value not in quotes When I create a non-JSON value not in quotes
......
cms/djangoapps/contentstore/features/checklists.feature
...@@ -10,8 +10,6 @@ Feature: Course checklists ...@@ -10,8 +10,6 @@ Feature: Course checklists
Then I can check and uncheck tasks in a checklist Then I can check and uncheck tasks in a checklist
And They are correctly selected after I reload the page And They are correctly selected after I reload the page
@skip-phantom
@skip-firefox
Scenario: A task can link to a location within Studio Scenario: A task can link to a location within Studio
Given I have opened Checklists Given I have opened Checklists
When I select a link to the course outline When I select a link to the course outline
...@@ -19,8 +17,6 @@ Feature: Course checklists ...@@ -19,8 +17,6 @@ Feature: Course checklists
And I press the browser back button And I press the browser back button
Then I am brought back to the course outline in the correct state Then I am brought back to the course outline in the correct state
@skip-phantom
@skip-firefox
Scenario: A task can link to a location outside Studio Scenario: A task can link to a location outside Studio
Given I have opened Checklists Given I have opened Checklists
When I select a link to help page When I select a link to help page
......
cms/djangoapps/contentstore/features/course-settings.feature
Feature: Course Settings Feature: Course Settings
As a course author, I want to be able to configure my course settings. As a course author, I want to be able to configure my course settings.
@skip-phantom
Scenario: User can set course dates Scenario: User can set course dates
Given I have opened a new course in Studio Given I have opened a new course in Studio
When I select Schedule and Details When I select Schedule and Details
And I set course dates And I set course dates
Then I see the set dates on refresh Then I see the set dates on refresh
@skip-phantom
Scenario: User can clear previously set course dates (except start date) Scenario: User can clear previously set course dates (except start date)
Given I have set course dates Given I have set course dates
And I clear all the dates except start And I clear all the dates except start
Then I see cleared dates on refresh Then I see cleared dates on refresh
@skip-phantom
Scenario: User cannot clear the course start date Scenario: User cannot clear the course start date
Given I have set course dates Given I have set course dates
And I clear the course start date And I clear the course start date
......
cms/djangoapps/contentstore/features/section.feature
...@@ -3,7 +3,6 @@ Feature: Create Section ...@@ -3,7 +3,6 @@ Feature: Create Section
As a course author As a course author
I want to create and edit sections I want to create and edit sections
@skip-phantom
Scenario: Add a new section to a course Scenario: Add a new section to a course
Given I have opened a new course in Studio Given I have opened a new course in Studio
When I click the New Section link When I click the New Section link
...@@ -27,7 +26,8 @@ Feature: Create Section ...@@ -27,7 +26,8 @@ Feature: Create Section
And I save a new section release date And I save a new section release date
Then the section release date is updated Then the section release date is updated
@skip-phantom # Skipped because Ubuntu ChromeDriver hangs on alert
@skip
Scenario: Delete section Scenario: Delete section
Given I have opened a new course in Studio Given I have opened a new course in Studio
And I have added a new section And I have added a new section
......
cms/djangoapps/contentstore/features/signup.py
...@@ -18,10 +18,7 @@ def i_fill_in_the_registration_form(step): ...@@ -18,10 +18,7 @@ def i_fill_in_the_registration_form(step):
@step('I press the Create My Account button on the registration form$') @step('I press the Create My Account button on the registration form$')
def i_press_the_button_on_the_registration_form(step): def i_press_the_button_on_the_registration_form(step):
submit_css = 'form#register_form button#submit' submit_css = 'form#register_form button#submit'
# Workaround for click not working on ubuntu world.css_click(submit_css)
# for some unknown reason.
e = world.css_find(submit_css)
e.type(' ')
@step('I should see be on the studio home page$') @step('I should see be on the studio home page$')
......
cms/djangoapps/contentstore/features/studio-overview-togglesection.feature
...@@ -14,7 +14,6 @@ Feature: Overview Toggle Section ...@@ -14,7 +14,6 @@ Feature: Overview Toggle Section
When I navigate to the course overview page When I navigate to the course overview page
Then I do not see the "Collapse All Sections" link Then I do not see the "Collapse All Sections" link
@skip-phantom
Scenario: Collapse link appears after creating first section of a course Scenario: Collapse link appears after creating first section of a course
Given I have a course with no sections Given I have a course with no sections
When I navigate to the course overview page When I navigate to the course overview page
...@@ -22,7 +21,8 @@ Feature: Overview Toggle Section ...@@ -22,7 +21,8 @@ Feature: Overview Toggle Section
Then I see the "Collapse All Sections" link Then I see the "Collapse All Sections" link
And all sections are expanded And all sections are expanded
@skip-phantom # Skipped because Ubuntu ChromeDriver hangs on alert
@skip
Scenario: Collapse link is not removed after last section of a course is deleted Scenario: Collapse link is not removed after last section of a course is deleted
Given I have a course with 1 section Given I have a course with 1 section
And I navigate to the course overview page And I navigate to the course overview page
......
cms/djangoapps/contentstore/features/subsection.feature
...@@ -3,14 +3,12 @@ Feature: Create Subsection ...@@ -3,14 +3,12 @@ Feature: Create Subsection
As a course author As a course author
I want to create and edit subsections I want to create and edit subsections
@skip-phantom
Scenario: Add a new subsection to a section Scenario: Add a new subsection to a section
Given I have opened a new course section in Studio Given I have opened a new course section in Studio
When I click the New Subsection link When I click the New Subsection link
And I enter the subsection name and click save And I enter the subsection name and click save
Then I see my subsection on the Courseware page Then I see my subsection on the Courseware page
@skip-phantom
Scenario: Add a new subsection (with a name containing a quote) to a section (bug #216) Scenario: Add a new subsection (with a name containing a quote) to a section (bug #216)
Given I have opened a new course section in Studio Given I have opened a new course section in Studio
When I click the New Subsection link When I click the New Subsection link
...@@ -27,7 +25,6 @@ Feature: Create Subsection ...@@ -27,7 +25,6 @@ Feature: Create Subsection
And I reload the page And I reload the page
Then I see it marked as Homework Then I see it marked as Homework
@skip-phantom
Scenario: Set a due date in a different year (bug #256) Scenario: Set a due date in a different year (bug #256)
Given I have opened a new subsection in Studio Given I have opened a new subsection in Studio
And I have set a release date and due date in different years And I have set a release date and due date in different years
...@@ -35,7 +32,8 @@ Feature: Create Subsection ...@@ -35,7 +32,8 @@ Feature: Create Subsection
And I reload the page And I reload the page
Then I see the correct dates Then I see the correct dates
@skip-phantom # Skipped because Ubuntu ChromeDriver hangs on alert
@skip
Scenario: Delete a subsection Scenario: Delete a subsection
Given I have opened a new course section in Studio Given I have opened a new course section in Studio
And I have added a new subsection And I have added a new subsection
......
cms/djangoapps/contentstore/features/subsection.py
...@@ -63,14 +63,6 @@ def test_have_set_dates_in_different_years(step): ...@@ -63,14 +63,6 @@ def test_have_set_dates_in_different_years(step):
set_date_and_time('input#due_date', '01/02/2012', 'input#due_time', '04:00') set_date_and_time('input#due_date', '01/02/2012', 'input#due_time', '04:00')
@step('I see the correct dates$')
def i_see_the_correct_dates(step):
assert_equal('12/25/2011', world.css_find('input#start_date').first.value)
assert_equal('03:00', world.css_find('input#start_time').first.value)
assert_equal('01/02/2012', world.css_find('input#due_date').first.value)
assert_equal('04:00', world.css_find('input#due_time').first.value)
@step('I mark it as Homework$') @step('I mark it as Homework$')
def i_mark_it_as_homework(step): def i_mark_it_as_homework(step):
world.css_click('a.menu-toggle') world.css_click('a.menu-toggle')
...@@ -101,8 +93,20 @@ def the_subsection_does_not_exist(step): ...@@ -101,8 +93,20 @@ def the_subsection_does_not_exist(step):
assert world.browser.is_element_not_present_by_css(css) assert world.browser.is_element_not_present_by_css(css)
@step('I see the correct dates$')
def i_see_the_correct_dates(step):
assert_equal('12/25/2011', get_date('input#start_date'))
assert_equal('03:00', get_date('input#start_time'))
assert_equal('01/02/2012', get_date('input#due_date'))
assert_equal('04:00', get_date('input#due_time'))
############ HELPER METHODS ################### ############ HELPER METHODS ###################
def get_date(css):
return world.css_find(css).first.value.strip()
def save_subsection_name(name): def save_subsection_name(name):
name_css = 'input.new-subsection-name-input' name_css = 'input.new-subsection-name-input'
save_css = 'input.new-subsection-name-save' save_css = 'input.new-subsection-name-save'
......
cms/djangoapps/contentstore/tests/test_contentstore.py
...@@ -646,7 +646,7 @@ class ContentStoreTest(ModuleStoreTestCase): ...@@ -646,7 +646,7 @@ class ContentStoreTest(ModuleStoreTestCase):
resp = self.client.get(reverse('index')) resp = self.client.get(reverse('index'))
self.assertContains( self.assertContains(
resp, resp,
'<h1 class="title-1">My Courses</h1>', '<h1 class="page-header">My Courses</h1>',
status_code=200, status_code=200,
html=True html=True
) )
......
cms/djangoapps/contentstore/tests/test_i18n.py
...@@ -48,7 +48,7 @@ class InternationalizationTest(ModuleStoreTestCase): ...@@ -48,7 +48,7 @@ class InternationalizationTest(ModuleStoreTestCase):
resp = self.client.get(reverse('index')) resp = self.client.get(reverse('index'))
self.assertContains(resp, self.assertContains(resp,
'<h1 class="title-1">My Courses</h1>', '<h1 class="page-header">My Courses</h1>',
status_code=200, status_code=200,
html=True) html=True)
...@@ -63,7 +63,7 @@ class InternationalizationTest(ModuleStoreTestCase): ...@@ -63,7 +63,7 @@ class InternationalizationTest(ModuleStoreTestCase):
) )
self.assertContains(resp, self.assertContains(resp,
'<h1 class="title-1">My Courses</h1>', '<h1 class="page-header">My Courses</h1>',
status_code=200, status_code=200,
html=True) html=True)
......
cms/envs/acceptance.py
...@@ -8,27 +8,41 @@ from .test import * ...@@ -8,27 +8,41 @@ from .test import *
# otherwise the browser will not render the pages correctly # otherwise the browser will not render the pages correctly
DEBUG = True DEBUG = True
# Show the courses that are in the data directory # Disable warnings for acceptance tests, to make the logs readable
COURSES_ROOT = ENV_ROOT / "data" import logging
DATA_DIR = COURSES_ROOT logging.disable(logging.ERROR)
# MODULESTORE = {
# 'default': {
# 'ENGINE': 'xmodule.modulestore.xml.XMLModuleStore',
# 'OPTIONS': {
# 'data_dir': DATA_DIR,
# 'default_class': 'xmodule.hidden_module.HiddenDescriptor',
# }
# }
# }
MODULESTORE_OPTIONS = {
'default_class': 'xmodule.raw_module.RawDescriptor',
'host': 'localhost',
'db': 'test_xmodule',
'collection': 'acceptance_modulestore',
'fs_root': TEST_ROOT / "data",
'render_template': 'mitxmako.shortcuts.render_to_string',
}
MODULESTORE = {
'default': {
'ENGINE': 'xmodule.modulestore.mongo.MongoModuleStore',
'OPTIONS': MODULESTORE_OPTIONS
},
'direct': {
'ENGINE': 'xmodule.modulestore.mongo.MongoModuleStore',
'OPTIONS': MODULESTORE_OPTIONS
},
'draft': {
'ENGINE': 'xmodule.modulestore.mongo.DraftMongoModuleStore',
'OPTIONS': MODULESTORE_OPTIONS
}
}
# Set this up so that rake lms[acceptance] and running the # Set this up so that rake lms[acceptance] and running the
# harvest command both use the same (test) database # harvest command both use the same (test) database
# which they can flush without messing up your dev db # which they can flush without messing up your dev db
DATABASES = { DATABASES = {
'default': { 'default': {
'ENGINE': 'django.db.backends.sqlite3', 'ENGINE': 'django.db.backends.sqlite3',
'NAME': ENV_ROOT / "db" / "test_mitx.db", 'NAME': TEST_ROOT / "db" / "test_mitx.db",
'TEST_NAME': ENV_ROOT / "db" / "test_mitx.db", 'TEST_NAME': TEST_ROOT / "db" / "test_mitx.db",
} }
} }
......
cms/envs/test.py
...@@ -41,8 +41,8 @@ MODULESTORE_OPTIONS = { ...@@ -41,8 +41,8 @@ MODULESTORE_OPTIONS = {
'default_class': 'xmodule.raw_module.RawDescriptor', 'default_class': 'xmodule.raw_module.RawDescriptor',
'host': 'localhost', 'host': 'localhost',
'db': 'test_xmodule', 'db': 'test_xmodule',
'collection': 'modulestore', 'collection': 'test_modulestore',
'fs_root': GITHUB_REPO_ROOT, 'fs_root': TEST_ROOT / "data",
'render_template': 'mitxmako.shortcuts.render_to_string', 'render_template': 'mitxmako.shortcuts.render_to_string',
} }
......
cms/static/sass/_variables.scss
...@@ -184,6 +184,6 @@ $lightBluishGrey2: rgb(213, 220, 228); ...@@ -184,6 +184,6 @@ $lightBluishGrey2: rgb(213, 220, 228);
$error-red: rgb(253, 87, 87); $error-red: rgb(253, 87, 87);
// type // type
$sans-serif: $f-serif; $sans-serif: $f-sans-serif;
$body-line-height: golden-ratio(.875em, 1); $body-line-height: golden-ratio(.875em, 1);
common/djangoapps/terrain/browser.py
"""
Browser set up for acceptance tests.
"""
#pylint: disable=E1101
#pylint: disable=W0613
#pylint: disable=W0611
from lettuce import before, after, world from lettuce import before, after, world
from splinter.browser import Browser from splinter.browser import Browser
from logging import getLogger from logging import getLogger
from django.core.management import call_command from django.core.management import call_command
from django.conf import settings from django.conf import settings
from selenium.common.exceptions import WebDriverException
# Let the LMS and CMS do their one-time setup # Let the LMS and CMS do their one-time setup
# For example, setting up mongo caches # For example, setting up mongo caches
from lms import one_time_startup from lms import one_time_startup
from cms import one_time_startup from cms import one_time_startup
logger = getLogger(__name__) # There is an import issue when using django-staticfiles with lettuce
logger.info("Loading the lettuce acceptance testing terrain file...") # Lettuce assumes that we are using django.contrib.staticfiles,
# but the rest of the app assumes we are using django-staticfiles
# (in particular, django-pipeline and our mako implementation)
# To resolve this, we check whether staticfiles is installed,
# then redirect imports for django.contrib.staticfiles
# to use staticfiles.
try:
import staticfiles
except ImportError:
pass
else:
import sys
sys.modules['django.contrib.staticfiles'] = staticfiles
LOGGER = getLogger(__name__)
LOGGER.info("Loading the lettuce acceptance testing terrain file...")
MAX_VALID_BROWSER_ATTEMPTS = 20
@before.harvest @before.harvest
def initial_setup(server): def initial_setup(server):
''' """
Launch the browser once before executing the tests Launch the browser once before executing the tests.
''' """
browser_driver = getattr(settings, 'LETTUCE_BROWSER', 'chrome') browser_driver = getattr(settings, 'LETTUCE_BROWSER', 'chrome')
# There is an issue with ChromeDriver2 r195627 on Ubuntu
# in which we sometimes get an invalid browser session.
# This is a work-around to ensure that we get a valid session.
success = False
num_attempts = 0
while (not success) and num_attempts < MAX_VALID_BROWSER_ATTEMPTS:
# Get a browser session
world.browser = Browser(browser_driver) world.browser = Browser(browser_driver)
# Try to visit the main page
# If the browser session is invalid, this will
# raise a WebDriverException
try:
world.visit('/')
except WebDriverException:
world.browser.quit()
num_attempts += 1
else:
success = True
# If we were unable to get a valid session within the limit of attempts,
# then we cannot run the tests.
if not success:
raise IOError("Could not acquire valid ChromeDriver browser session.")
@before.each_scenario @before.each_scenario
def reset_data(scenario): def reset_data(scenario):
''' """
Clean out the django test database defined in the Clean out the django test database defined in the
envs/acceptance.py file: mitx_all/db/test_mitx.db envs/acceptance.py file: mitx_all/db/test_mitx.db
''' """
logger.debug("Flushing the test database...") LOGGER.debug("Flushing the test database...")
call_command('flush', interactive=False) call_command('flush', interactive=False)
@after.each_scenario @after.each_scenario
def screenshot_on_error(scenario): def screenshot_on_error(scenario):
''' """
Save a screenshot to help with debugging Save a screenshot to help with debugging.
''' """
if scenario.failed: if scenario.failed:
world.browser.driver.save_screenshot('/tmp/last_failed_scenario.png') world.browser.driver.save_screenshot('/tmp/last_failed_scenario.png')
@after.all @after.all
def teardown_browser(total): def teardown_browser(total):
''' """
Quit the browser after executing the tests Quit the browser after executing the tests.
''' """
world.browser.quit() world.browser.quit()
pass
common/djangoapps/terrain/course_helpers.py
...@@ -38,9 +38,11 @@ def create_user(uname): ...@@ -38,9 +38,11 @@ def create_user(uname):
@world.absorb @world.absorb
def log_in(username, password): def log_in(username, password):
''' """
Log the user in programatically Log the user in programatically.
''' This will delete any existing cookies to ensure that the user
logs in to the correct session.
"""
# Authenticate the user # Authenticate the user
user = authenticate(username=username, password=password) user = authenticate(username=username, password=password)
...@@ -60,14 +62,7 @@ def log_in(username, password): ...@@ -60,14 +62,7 @@ def log_in(username, password):
# Retrieve the sessionid and add it to the browser's cookies # Retrieve the sessionid and add it to the browser's cookies
cookie_dict = {settings.SESSION_COOKIE_NAME: request.session.session_key} cookie_dict = {settings.SESSION_COOKIE_NAME: request.session.session_key}
try: world.browser.cookies.delete()
world.browser.cookies.add(cookie_dict)
# WebDriver has an issue where we cannot set cookies
# before we make a GET request, so if we get an error,
# we load the '/' page and try again
except:
world.browser.visit(django_url('/'))
world.browser.cookies.add(cookie_dict) world.browser.cookies.add(cookie_dict)
......
common/djangoapps/terrain/ui_helpers.py
...@@ -53,12 +53,9 @@ def css_find(css): ...@@ -53,12 +53,9 @@ def css_find(css):
@world.absorb @world.absorb
def css_click(css_selector): def css_click(css_selector):
''' """
First try to use the regular click method, Perform a click on a CSS selector, retrying if it initially fails
but if clicking in the middle of an element """
doesn't work it might be that it thinks some other
element is on top of it there so click in the upper left
'''
try: try:
world.browser.find_by_css(css_selector).click() world.browser.find_by_css(css_selector).click()
......
common/lib/capa/capa/tests/test_responsetypes.py
...@@ -708,7 +708,7 @@ class JavascriptResponseTest(ResponseTest): ...@@ -708,7 +708,7 @@ class JavascriptResponseTest(ResponseTest):
def test_grade(self): def test_grade(self):
# Compile coffee files into javascript used by the response # Compile coffee files into javascript used by the response
coffee_file_path = os.path.dirname(__file__) + "/test_files/js/*.coffee" coffee_file_path = os.path.dirname(__file__) + "/test_files/js/*.coffee"
os.system("coffee -c %s" % (coffee_file_path)) os.system("node_modules/.bin/coffee -c %s" % (coffee_file_path))
problem = self.build_problem(generator_src="test_problem_generator.js", problem = self.build_problem(generator_src="test_problem_generator.js",
grader_src="test_problem_grader.js", grader_src="test_problem_grader.js",
......
common/lib/capa/jasmine_test_runner.html.erb 0 → 100644
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title>Jasmine Test Runner</title>
<link rel="stylesheet" type="text/css" href="<%= phantom_jasmine_path %>/vendor/jasmine-1.2.0/jasmine.css">
<script type="text/javascript" src="<%= phantom_jasmine_path %>/vendor/jasmine-1.2.0/jasmine.js"></script>
<script type="text/javascript" src="<%= phantom_jasmine_path %>/vendor/jasmine-1.2.0/jasmine-html.js"></script>
<script type="text/javascript" src="<%= phantom_jasmine_path %>/lib/console-runner.js"></script>
<script type="text/javascript" src="<%= common_coffee_root %>/ajax_prefix.js"></script>
<script type="text/javascript" src="<%= common_coffee_root %>/logger.js"></script>
<script type="text/javascript" src="<%= common_js_root %>/vendor/jquery.min.js"></script>
<script type="text/javascript" src="<%= common_js_root %>/vendor/jasmine-jquery.js"></script>
<script type="text/javascript" src="<%= common_js_root %>/vendor/jquery.cookie.js"></script>
<script type="text/javascript" src="<%= common_js_root %>/vendor/CodeMirror/codemirror.js"></script>
<script type="text/javascript" src="<%= common_js_root %>/vendor/tiny_mce/jquery.tinymce.js"></script>
<script type="text/javascript" src="<%= common_js_root %>/vendor/tiny_mce/tiny_mce.js"></script>
<script type="text/javascript" src="<%= common_js_root %>/vendor/RequireJS.js"></script>
<script type="text/javascript">
AjaxPrefix.addAjaxPrefix(jQuery, function() {
return "";
});
</script>
<!-- SOURCE FILES -->
<% for src in js_source %>
<script type="text/javascript" src="<%= src %>"></script>
<% end %>
<!-- SPEC FILES -->
<% for src in js_specs %>
<script type="text/javascript" src="<%= src %>"></script>
<% end %>
</head>
<body>
<script type="text/javascript">
var console_reporter = new jasmine.ConsoleReporter()
jasmine.getEnv().addReporter(new jasmine.TrivialReporter());
jasmine.getEnv().addReporter(console_reporter);
jasmine.getEnv().execute();
</script>
</body>
</html>
common/lib/capa/setup.py
...@@ -4,5 +4,5 @@ setup( ...@@ -4,5 +4,5 @@ setup(
name="capa", name="capa",
version="0.1", version="0.1",
packages=find_packages(exclude=["tests"]), packages=find_packages(exclude=["tests"]),
install_requires=['distribute==0.6.30', 'pyparsing==1.5.6'], install_requires=['distribute==0.6.28', 'pyparsing==1.5.6'],
) )
common/lib/xmodule/xmodule/templates.py
...@@ -4,8 +4,18 @@ These templates are used by the CMS to provide baseline content that ...@@ -4,8 +4,18 @@ These templates are used by the CMS to provide baseline content that
can be cloned when adding new modules to a course. can be cloned when adding new modules to a course.
`Template`s are defined in x_module. They contain 3 attributes: `Template`s are defined in x_module. They contain 3 attributes:
metadata: A dictionary with the template metadata metadata: A dictionary with the template metadata. This should contain
data: A JSON value that defines the template content any values for fields
* with scope Scope.settings
* that have values different than the field defaults
* and that are to be editable in Studio
data: A JSON value that defines the template content. This should be a dictionary
containing values for fields
* with scope Scope.content
* that have values different than the field defaults
* and that are to be editable in Studio
or, if the module uses a single Scope.content String field named `data`, this
should be a string containing the contents of that field
children: A list of Location urls that define the template children children: A list of Location urls that define the template children
Templates are defined on XModuleDescriptor types, in the template attribute. Templates are defined on XModuleDescriptor types, in the template attribute.
......
doc/development.md
...@@ -36,7 +36,7 @@ Check out the course data directories that you want to work with into the ...@@ -36,7 +36,7 @@ Check out the course data directories that you want to work with into the
To create your development environment, run the shell script in the root of To create your development environment, run the shell script in the root of
the repo: the repo:
create-dev-env.sh scripts/create-dev-env.sh
## Starting development servers ## Starting development servers
......
doc/testing.md
...@@ -161,36 +161,36 @@ try running `bundle install` to install the required ruby gems. ...@@ -161,36 +161,36 @@ try running `bundle install` to install the required ruby gems.
We use [Lettuce](http://lettuce.it/) for acceptance testing. We use [Lettuce](http://lettuce.it/) for acceptance testing.
Most of our tests use [Splinter](http://splinter.cobrateam.info/) Most of our tests use [Splinter](http://splinter.cobrateam.info/)
to simulate UI browser interactions. Splinter, in turn, to simulate UI browser interactions. Splinter, in turn,
uses [Selenium](http://docs.seleniumhq.org/) to control the browser. uses [Selenium](http://docs.seleniumhq.org/) to control the Chrome browser.
**Prerequisite**: You must have [ChromeDriver](https://code.google.com/p/selenium/wiki/ChromeDriver) **Prerequisite**: You must have [ChromeDriver](https://code.google.com/p/selenium/wiki/ChromeDriver)
installed to run the tests in Chrome. installed to run the tests in Chrome. The tests are confirmed to run
with Chrome (not Chromium) version 26.0.0.1410.63 with ChromeDriver
version r195636.
Before running the tests, you need to set up the test database: To run all the acceptance tests:
rm ../db/test_mitx.db rake test_acceptance_lms
rake django-admin[syncdb,lms,acceptance,--noinput] rake test_acceptance_cms
rake django-admin[migrate,lms,acceptance,--noinput]
To run the acceptance tests: To test only a specific feature:
1. Start the Django server locally using the settings in **acceptance.py**:
rake lms[acceptance] rake test_acceptance_lms[lms/djangoapps/courseware/features/problems.feature]
2. In another shell, run the tests: To start the debugger on failure, add the `--pdb` option:
django-admin.py harvest --no-server --settings=lms.envs.acceptance --pythonpath=. lms/djangoapps/portal/features/ rake test_acceptance_lms["lms/djangoapps/courseware/features/problems.feature --pdb"]
To test only a specific feature: To run tests faster by not collecting static files, you can use
`rake fasttest_acceptance_lms` and `rake fasttest_acceptance_cms`.
django-admin.py harvest --no-server --settings=lms.envs.acceptance --pythonpath=. lms/djangoapps/courseware/features/high-level-tabs.feature
**Troubleshooting**: If you get an error message that says something about harvest not being a command, you probably are missing a requirement. **Troubleshooting**: If you get an error message that says something about harvest not being a command, you probably are missing a requirement.
Try running: Try running:
pip install -r requirements.txt pip install -r requirements.txt
**Note**: The acceptance tests can *not* currently run in parallel.
## Viewing Test Coverage ## Viewing Test Coverage
......
fixtures/anonymize_fixtures.py deleted 100755 → 0
#! /usr/bin/env python
import sys
import json
import random
import copy
from collections import defaultdict
from argparse import ArgumentParser, FileType
from datetime import datetime
def generate_user(user_number):
return {
"pk": user_number,
"model": "auth.user",
"fields": {
"status": "w",
"last_name": "Last",
"gold": 0,
"is_staff": False,
"user_permissions": [],
"interesting_tags": "",
"email_key": None,
"date_joined": "2012-04-26 11:36:39",
"first_name": "",
"email_isvalid": False,
"avatar_type": "n",
"website": "",
"is_superuser": False,
"date_of_birth": None,
"last_login": "2012-04-26 11:36:48",
"location": "",
"new_response_count": 0,
"email": "user{num}@example.com".format(num=user_number),
"username": "user{num}".format(num=user_number),
"is_active": True,
"consecutive_days_visit_count": 0,
"email_tag_filter_strategy": 1,
"groups": [],
"password": "sha1$90e6f$562a1d783a0c47ce06ebf96b8c58123a0671bbf0",
"silver": 0,
"bronze": 0,
"questions_per_page": 10,
"about": "",
"show_country": True,
"country": "",
"display_tag_filter_strategy": 0,
"seen_response_count": 0,
"real_name": "",
"ignored_tags": "",
"reputation": 1,
"gravatar": "366d981a10116969c568a18ee090f44c",
"last_seen": "2012-04-26 11:36:39"
}
}
def parse_args(args=sys.argv[1:]):
parser = ArgumentParser()
parser.add_argument('-d', '--data', type=FileType('r'), default=sys.stdin)
parser.add_argument('-o', '--output', type=FileType('w'), default=sys.stdout)
parser.add_argument('count', type=int)
return parser.parse_args(args)
def main(args=sys.argv[1:]):
args = parse_args(args)
data = json.load(args.data)
unique_students = set(entry['fields']['student'] for entry in data)
if args.count > len(unique_students) * 0.1:
raise Exception("Can't be sufficiently anonymous selecting {count} of {unique} students".format(
count=args.count, unique=len(unique_students)))
by_problems = defaultdict(list)
for entry in data:
by_problems[entry['fields']['module_id']].append(entry)
out_data = []
out_pk = 1
for name, answers in by_problems.items():
for student_id in xrange(args.count):
sample = random.choice(answers)
data = copy.deepcopy(sample)
data["fields"]["student"] = student_id + 1
data["fields"]["created"] = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
data["fields"]["modified"] = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
data["pk"] = out_pk
out_pk += 1
out_data.append(data)
for student_id in xrange(args.count):
out_data.append(generate_user(student_id))
json.dump(out_data, args.output, indent=2)
if __name__ == "__main__":
sys.exit(main())
fixtures/pm.json deleted 100644 → 0
[{"pk": 1, "model": "user.userprofile", "fields": {"name": "pm", "language": "pm", "courseware": "course.xml", "meta": "", "location": "pm", "user": 1}}, {"pk": 1, "model": "auth.user", "fields": {"status": "w", "last_name": "", "gold": 0, "is_staff": true, "user_permissions": [], "interesting_tags": "", "email_key": null, "date_joined": "2012-01-23 17:03:54", "first_name": "", "email_isvalid": false, "avatar_type": "n", "website": "", "is_superuser": true, "date_of_birth": null, "last_login": "2012-01-23 17:04:16", "location": "", "new_response_count": 0, "email": "pmitros@csail.mit.edu", "username": "pm", "is_active": true, "consecutive_days_visit_count": 0, "email_tag_filter_strategy": 1, "groups": [], "password": "sha1$a3e96$dbabbd114f0da01bce2cc2adcafa2ca651c7ae0a", "silver": 0, "bronze": 0, "questions_per_page": 10, "about": "", "show_country": false, "country": "", "display_tag_filter_strategy": 0, "seen_response_count": 0, "real_name": "", "ignored_tags": "", "reputation": 1, "gravatar": "7a591afd0cc7972fdbe5e12e26af352a", "last_seen": "2012-01-23 17:04:41"}}, {"pk": 1, "model": "user.userprofile", "fields": {"name": "pm", "language": "pm", "courseware": "course.xml", "meta": "", "location": "pm", "user": 1}}, {"pk": 1, "model": "auth.user", "fields": {"status": "w", "last_name": "", "gold": 0, "is_staff": true, "user_permissions": [], "interesting_tags": "", "email_key": null, "date_joined": "2012-01-23 17:03:54", "first_name": "", "email_isvalid": false, "avatar_type": "n", "website": "", "is_superuser": true, "date_of_birth": null, "last_login": "2012-01-23 17:04:16", "location": "", "new_response_count": 0, "email": "pmitros@csail.mit.edu", "username": "pm", "is_active": true, "consecutive_days_visit_count": 0, "email_tag_filter_strategy": 1, "groups": [], "password": "sha1$a3e96$dbabbd114f0da01bce2cc2adcafa2ca651c7ae0a", "silver": 0, "bronze": 0, "questions_per_page": 10, "about": "", "show_country": false, "country": "", "display_tag_filter_strategy": 0, "seen_response_count": 0, "real_name": "", "ignored_tags": "", "reputation": 1, "gravatar": "7a591afd0cc7972fdbe5e12e26af352a", "last_seen": "2012-01-23 17:04:41"}}]
\ No newline at end of file
jenkins/base.sh deleted 100644 → 0
##
## requires >= 1.3.0 of the Jenkins git plugin
##
function github_status {
if [[ ! ${GIT_URL} =~ git@github.com:([^/]+)/([^\.]+).git ]]; then
echo "Cannot parse Github org or repo from URL, using defaults."
ORG="edx"
REPO="mitx"
else
ORG=${BASH_REMATCH[1]}
REPO=${BASH_REMATCH[2]}
fi
gcli status create $ORG $REPO $GIT_COMMIT \
--params=$1 \
target_url:$BUILD_URL \
description:"Build #$BUILD_NUMBER is running" \
-f csv
}
function github_mark_failed_on_exit {
trap '[ $? == "0" ] || github_status state:failed' EXIT
}
jenkins/test.sh
...@@ -3,8 +3,21 @@ ...@@ -3,8 +3,21 @@
set -e set -e
set -x set -x
##
## requires >= 1.3.0 of the Jenkins git plugin
##
function github_status { function github_status {
gcli status create edx edx-platform $GIT_COMMIT \ if [[ ! ${GIT_URL} =~ git@github.com:([^/]+)/([^\.]+).git ]]; then
echo "Cannot parse Github org or repo from URL, using defaults."
ORG="edx"
REPO="edx-platform"
else
ORG=${BASH_REMATCH[1]}
REPO=${BASH_REMATCH[2]}
fi
gcli status create $ORG $REPO $GIT_COMMIT \
--params=$1 \ --params=$1 \
target_url:$BUILD_URL \ target_url:$BUILD_URL \
description:"Build #$BUILD_NUMBER $2" \ description:"Build #$BUILD_NUMBER $2" \
...@@ -27,21 +40,32 @@ git submodule foreach 'git reset --hard HEAD' ...@@ -27,21 +40,32 @@ git submodule foreach 'git reset --hard HEAD'
export PYTHONIOENCODING=UTF-8 export PYTHONIOENCODING=UTF-8
GIT_BRANCH=${GIT_BRANCH/HEAD/master} GIT_BRANCH=${GIT_BRANCH/HEAD/master}
if [ ! -d /mnt/virtualenvs/"$JOB_NAME" ]; then
mkdir -p /mnt/virtualenvs/"$JOB_NAME" # When running in parallel on jenkins, workspace could be suffixed by @x
virtualenv /mnt/virtualenvs/"$JOB_NAME" # In that case, we want to use a separate virtualenv that matches up with
# workspace
#
# We need to handle both the case of /path/to/workspace
# and /path/to/workspace@2, which is why we use the following substitutions
#
# $WORKSPACE is the absolute path for the workspace
WORKSPACE_SUFFIX=$(expr "$WORKSPACE" : '.*\(@.*\)') || true
VIRTUALENV_DIR="/mnt/virtualenvs/${JOB_NAME}${WORKSPACE_SUFFIX}"
if [ ! -d "$VIRTUALENV_DIR" ]; then
mkdir -p "$VIRTUALENV_DIR"
virtualenv "$VIRTUALENV_DIR"
fi fi
export PIP_DOWNLOAD_CACHE=/mnt/pip-cache export PIP_DOWNLOAD_CACHE=/mnt/pip-cache
source /mnt/virtualenvs/"$JOB_NAME"/bin/activate # Allow django liveserver tests to use a range of ports
pip install -q -r pre-requirements.txt export DJANGO_LIVE_TEST_SERVER_ADDRESS=${DJANGO_LIVE_TEST_SERVER_ADDRESS-localhost:8000-9000}
yes w | pip install -q -r requirements.txt
bundle install source /mnt/virtualenvs/"$JOB_NAME"/bin/activate
npm install
rake install_prereqs
rake clobber rake clobber
rake pep8 > pep8.log || cat pep8.log rake pep8 > pep8.log || cat pep8.log
rake pylint > pylint.log || cat pylint.log rake pylint > pylint.log || cat pylint.log
...@@ -54,7 +78,7 @@ rake test_lms[false] || TESTS_FAILED=1 ...@@ -54,7 +78,7 @@ rake test_lms[false] || TESTS_FAILED=1
rake test_common/lib/capa || TESTS_FAILED=1 rake test_common/lib/capa || TESTS_FAILED=1
rake test_common/lib/xmodule || TESTS_FAILED=1 rake test_common/lib/xmodule || TESTS_FAILED=1
# Run the jaavascript unit tests # Run the javascript unit tests
rake phantomjs_jasmine_lms || TESTS_FAILED=1 rake phantomjs_jasmine_lms || TESTS_FAILED=1
rake phantomjs_jasmine_cms || TESTS_FAILED=1 rake phantomjs_jasmine_cms || TESTS_FAILED=1
rake phantomjs_jasmine_common/lib/xmodule || TESTS_FAILED=1 rake phantomjs_jasmine_common/lib/xmodule || TESTS_FAILED=1
......
jenkins/test_acceptance.sh 0 → 100755
#! /bin/bash
set -e
set -x
git remote prune origin
# Reset the submodule, in case it changed
git submodule foreach 'git reset --hard HEAD'
# Set the IO encoding to UTF-8 so that askbot will start
export PYTHONIOENCODING=UTF-8
if [ ! -d /mnt/virtualenvs/"$JOB_NAME" ]; then
mkdir -p /mnt/virtualenvs/"$JOB_NAME"
virtualenv /mnt/virtualenvs/"$JOB_NAME"
fi
export PIP_DOWNLOAD_CACHE=/mnt/pip-cache
source /mnt/virtualenvs/"$JOB_NAME"/bin/activate
rake install_prereqs
rake clobber
TESTS_FAILED=0
# Assumes that Xvfb has been started by upstart
# and is capturing display :1
# The command for this is:
# /usr/bin/Xvfb :1 -screen 0 1024x268x24
# This allows us to run Chrome without a display
export DISPLAY=:1
# Run the lms and cms acceptance tests
# (the -v flag turns off color in the output)
rake test_acceptance_lms["-v 3"] || TESTS_FAILED=1
rake test_acceptance_cms["-v 3"] || TESTS_FAILED=1
[ $TESTS_FAILED == '0' ]
lms/djangoapps/courseware/tests/tests.py
...@@ -631,8 +631,8 @@ class TestViewAuth(LoginEnrollmentTestCase): ...@@ -631,8 +631,8 @@ class TestViewAuth(LoginEnrollmentTestCase):
urls = reverse_urls(['info', 'progress'], course) urls = reverse_urls(['info', 'progress'], course)
urls.extend([ urls.extend([
reverse('book', kwargs={'course_id': course.id, reverse('book', kwargs={'course_id': course.id,
'book_index': book.title}) 'book_index': index})
for book in course.textbooks for index, book in enumerate(course.textbooks)
]) ])
return urls return urls
...@@ -643,8 +643,6 @@ class TestViewAuth(LoginEnrollmentTestCase): ...@@ -643,8 +643,6 @@ class TestViewAuth(LoginEnrollmentTestCase):
""" """
urls = reverse_urls(['about_course'], course) urls = reverse_urls(['about_course'], course)
urls.append(reverse('courses')) urls.append(reverse('courses'))
# Need separate test for change_enrollment, since it's a POST view
#urls.append(reverse('change_enrollment'))
return urls return urls
......
lms/envs/acceptance.py
...@@ -8,13 +8,17 @@ from .test import * ...@@ -8,13 +8,17 @@ from .test import *
# otherwise the browser will not render the pages correctly # otherwise the browser will not render the pages correctly
DEBUG = True DEBUG = True
# Disable warnings for acceptance tests, to make the logs readable
import logging
logging.disable(logging.ERROR)
# Use the mongo store for acceptance tests # Use the mongo store for acceptance tests
modulestore_options = { modulestore_options = {
'default_class': 'xmodule.raw_module.RawDescriptor', 'default_class': 'xmodule.raw_module.RawDescriptor',
'host': 'localhost', 'host': 'localhost',
'db': 'test_xmodule', 'db': 'test_xmodule',
'collection': 'modulestore', 'collection': 'acceptance_modulestore',
'fs_root': GITHUB_REPO_ROOT, 'fs_root': TEST_ROOT / "data",
'render_template': 'mitxmako.shortcuts.render_to_string', 'render_template': 'mitxmako.shortcuts.render_to_string',
} }
...@@ -33,7 +37,7 @@ CONTENTSTORE = { ...@@ -33,7 +37,7 @@ CONTENTSTORE = {
'ENGINE': 'xmodule.contentstore.mongo.MongoContentStore', 'ENGINE': 'xmodule.contentstore.mongo.MongoContentStore',
'OPTIONS': { 'OPTIONS': {
'host': 'localhost', 'host': 'localhost',
'db': 'test_xcontent', 'db': 'test_xmodule',
} }
} }
...@@ -43,8 +47,8 @@ CONTENTSTORE = { ...@@ -43,8 +47,8 @@ CONTENTSTORE = {
DATABASES = { DATABASES = {
'default': { 'default': {
'ENGINE': 'django.db.backends.sqlite3', 'ENGINE': 'django.db.backends.sqlite3',
'NAME': ENV_ROOT / "db" / "test_mitx.db", 'NAME': TEST_ROOT / "db" / "test_mitx.db",
'TEST_NAME': ENV_ROOT / "db" / "test_mitx.db", 'TEST_NAME': TEST_ROOT / "db" / "test_mitx.db",
} }
} }
......
lms/envs/common.py
...@@ -126,9 +126,7 @@ sys.path.append(COMMON_ROOT / 'lib') ...@@ -126,9 +126,7 @@ sys.path.append(COMMON_ROOT / 'lib')
# For Node.js # For Node.js
system_node_path = os.environ.get("NODE_PATH", None) system_node_path = os.environ.get("NODE_PATH", REPO_ROOT / 'node_modules')
if system_node_path is None:
system_node_path = "/usr/local/lib/node_modules"
node_paths = [COMMON_ROOT / "static/js/vendor", node_paths = [COMMON_ROOT / "static/js/vendor",
COMMON_ROOT / "static/coffee/src", COMMON_ROOT / "static/coffee/src",
......
pre-requirements.txt deleted 100644 → 0
# We use `scipy` in our project, which relies on `numpy`. `pip` apparently
# installs packages in a two-step process, where it will first try to build
# all packages, and then try to install all packages. As a result, if we simply
# added these packages to the top of `requirements.txt`, `pip` would try to
# build `scipy` before `numpy` has been installed, and it would fail. By
# separating this out into a `pre-requirements.txt` file, we can make sure
# that `numpy` is built *and* installed before we try to build `scipy`.
numpy==1.6.2
distribute>=0.6.28
rakefile
require 'rake/clean' require 'rake/clean'
require 'tempfile' require './rakefiles/helpers.rb'
require 'net/http'
require 'launchy' Dir['rakefiles/*.rake'].each do |rakefile|
require 'colorize' import rakefile
require 'erb' end
require 'tempfile'
# Build Constants # Build Constants
REPO_ROOT = File.dirname(__FILE__) REPO_ROOT = File.dirname(__FILE__)
BUILD_DIR = File.join(REPO_ROOT, "build")
REPORT_DIR = File.join(REPO_ROOT, "reports") REPORT_DIR = File.join(REPO_ROOT, "reports")
LMS_REPORT_DIR = File.join(REPORT_DIR, "lms")
# Packaging constants
DEPLOY_DIR = "/opt/wwc"
PACKAGE_NAME = "edx-platform"
PKG_VERSION = "0.1"
COMMIT = (ENV["GIT_COMMIT"] || `git rev-parse HEAD`).chomp()[0, 10]
BRANCH = (ENV["GIT_BRANCH"] || `git symbolic-ref -q HEAD`).chomp().gsub('refs/heads/', '').gsub('origin/', '')
BUILD_NUMBER = (ENV["BUILD_NUMBER"] || "dev").chomp()
# Set up the clean and clobber tasks
CLOBBER.include(BUILD_DIR, REPORT_DIR, 'test_root/*_repo', 'test_root/staticfiles')
CLEAN.include("#{BUILD_DIR}/*.deb", "#{BUILD_DIR}/util")
def select_executable(*cmds)
cmds.find_all{ |cmd| system("which #{cmd} > /dev/null 2>&1") }[0] || fail("No executables found from #{cmds.join(', ')}")
end
def django_admin(system, env, command, *args)
django_admin = ENV['DJANGO_ADMIN_PATH'] || select_executable('django-admin.py', 'django-admin')
return "#{django_admin} #{command} --traceback --settings=#{system}.envs.#{env} --pythonpath=. #{args.join(' ')}"
end
# Runs Process.spawn, and kills the process at the end of the rake process
# Expects the same arguments as Process.spawn
def background_process(*command)
pid = Process.spawn({}, *command, {:pgroup => true})
at_exit do
puts "Ending process and children"
pgid = Process.getpgid(pid)
begin
Timeout.timeout(5) do
puts "Terminating process group #{pgid}"
Process.kill(:SIGTERM, -pgid)
puts "Waiting on process group #{pgid}"
Process.wait(-pgid)
puts "Done waiting on process group #{pgid}"
end
rescue Timeout::Error
puts "Killing process group #{pgid}"
Process.kill(:SIGKILL, -pgid)
puts "Waiting on process group #{pgid}"
Process.wait(-pgid)
puts "Done waiting on process group #{pgid}"
end
end
end
def django_for_jasmine(system, django_reload)
if !django_reload
reload_arg = '--noreload'
end
port = 10000 + rand(40000)
jasmine_url = "http://localhost:#{port}/_jasmine/"
background_process(*django_admin(system, 'jasmine', 'runserver', '-v', '0', port.to_s, reload_arg).split(' '))
up = false
start_time = Time.now
until up do
if Time.now - start_time > 30
abort "Timed out waiting for server to start to run jasmine tests"
end
begin
response = Net::HTTP.get_response(URI(jasmine_url))
puts response.code
up = response.code == '200'
rescue => e
puts e.message
ensure
puts('Waiting server to start')
sleep(0.5)
end
end
yield jasmine_url
end
def template_jasmine_runner(lib)
coffee_files = Dir["#{lib}/**/js/**/*.coffee", "common/static/coffee/src/**/*.coffee"]
if !coffee_files.empty?
sh("node_modules/.bin/coffee -c #{coffee_files.join(' ')}")
end
phantom_jasmine_path = File.expand_path("node_modules/phantom-jasmine")
common_js_root = File.expand_path("common/static/js")
common_coffee_root = File.expand_path("common/static/coffee/src")
# Get arrays of spec and source files, ordered by how deep they are nested below the library
# (and then alphabetically) and expanded from a relative to an absolute path
spec_glob = File.join("#{lib}", "**", "spec", "**", "*.js")
src_glob = File.join("#{lib}", "**", "src", "**", "*.js")
js_specs = Dir[spec_glob].sort_by {|p| [p.split('/').length, p]} .map {|f| File.expand_path(f)}
js_source = Dir[src_glob].sort_by {|p| [p.split('/').length, p]} .map {|f| File.expand_path(f)}
template = ERB.new(File.read("#{lib}/jasmine_test_runner.html.erb"))
template_output = "#{lib}/jasmine_test_runner.html"
File.open(template_output, 'w') do |f|
f.write(template.result(binding))
end
yield File.expand_path(template_output)
end
def report_dir_path(dir)
return File.join(REPORT_DIR, dir.to_s)
end
def compile_assets(watch=false, debug=false)
xmodule_cmd = 'xmodule_assets common/static/xmodule'
if watch
xmodule_cmd = "watchmedo shell-command \
--patterns='*.js;*.coffee;*.sass;*.scss;*.css' \
--recursive \
--command='#{xmodule_cmd}' \
common/lib/xmodule"
end
coffee_cmd = "node_modules/.bin/coffee #{watch ? '--watch' : ''} --compile */static"
sass_cmd = "sass #{debug ? '--debug-info' : '--style compressed'} " +
"--load-path ./common/static/sass " +
"--require ./common/static/sass/bourbon/lib/bourbon.rb " +
"#{watch ? '--watch' : '--update --force'} */static"
[xmodule_cmd, coffee_cmd, sass_cmd].each do |cmd|
if watch
background_process(cmd)
else
pid = Process.spawn(cmd)
puts "Waiting for `#{cmd}` to complete (pid #{pid})"
Process.wait(pid)
puts "Completed"
if !$?.exited? || $?.exitstatus != 0
abort "`#{cmd}` failed"
end
end
end
end
task :default => [:test, :pep8, :pylint] task :default => [:test, :pep8, :pylint]
directory REPORT_DIR
default_options = {
:lms => '8000',
:cms => '8001',
}
desc "Install all prerequisites needed for the lms and cms"
task :install_prereqs => [:install_node_prereqs, :install_ruby_prereqs, :install_python_prereqs]
desc "Install all node prerequisites for the lms and cms"
task :install_node_prereqs do
sh('npm install')
end
desc "Install all ruby prerequisites for the lms and cms"
task :install_ruby_prereqs do
sh('bundle install')
end
desc "Install all python prerequisites for the lms and cms"
task :install_python_prereqs do
sh('pip install -r requirements.txt')
# Check for private-requirements.txt: used to install our libs as working dirs,
# or personal-use tools.
if File.file?("private-requirements.txt")
sh('pip install -r private-requirements.txt')
end
end
task :predjango do
sh("find . -type f -name *.pyc -delete")
sh('pip install -q --no-index -r local-requirements.txt')
end
task :clean_test_files do
sh("git clean -fqdx test_root")
end
[:lms, :cms, :common].each do |system|
report_dir = report_dir_path(system)
directory report_dir
desc "Run pep8 on all #{system} code"
task "pep8_#{system}" => report_dir do
sh("pep8 #{system} | tee #{report_dir}/pep8.report")
end
task :pep8 => "pep8_#{system}"
desc "Run pylint on all #{system} code"
task "pylint_#{system}" => report_dir do
apps = Dir["#{system}/*.py", "#{system}/djangoapps/*", "#{system}/lib/*"].map do |app|
File.basename(app)
end.select do |app|
app !=~ /.pyc$/
end.map do |app|
if app =~ /.py$/
app.gsub('.py', '')
else
app
end
end
pythonpath_prefix = "PYTHONPATH=#{system}:#{system}/djangoapps:#{system}/lib:common/djangoapps:common/lib"
sh("#{pythonpath_prefix} pylint --rcfile=.pylintrc -f parseable #{apps.join(' ')} | tee #{report_dir}/pylint.report")
end
task :pylint => "pylint_#{system}"
end
$failed_tests = 0
def run_under_coverage(cmd, root)
cmd0, cmd_rest = cmd.split(" ", 2)
# We use "python -m coverage" so that the proper python will run the importable coverage
# rather than the coverage that OS path finds.
cmd = "python -m coverage run --rcfile=#{root}/.coveragerc `which #{cmd0}` #{cmd_rest}"
return cmd
end
def run_tests(system, report_dir, stop_on_failure=true)
ENV['NOSE_XUNIT_FILE'] = File.join(report_dir, "nosetests.xml")
dirs = Dir["common/djangoapps/*"] + Dir["#{system}/djangoapps/*"]
cmd = django_admin(system, :test, 'test', '--logging-clear-handlers', *dirs.each)
sh(run_under_coverage(cmd, system)) do |ok, res|
if !ok and stop_on_failure
abort "Test failed!"
end
$failed_tests += 1 unless ok
end
end
TEST_TASK_DIRS = []
task :fastlms do
# this is >2 times faster that rake [lms], and does not need web, good for local dev
django_admin = ENV['DJANGO_ADMIN_PATH'] || select_executable('django-admin.py', 'django-admin')
sh("#{django_admin} runserver --traceback --settings=lms.envs.dev --pythonpath=.")
end
[:lms, :cms].each do |system|
report_dir = report_dir_path(system)
# Per System tasks
desc "Run all django tests on our djangoapps for the #{system}"
task "test_#{system}", [:stop_on_failure] => ["clean_test_files", :predjango, "#{system}:gather_assets:test", "fasttest_#{system}"]
# Have a way to run the tests without running collectstatic -- useful when debugging without
# messing with static files.
task "fasttest_#{system}", [:stop_on_failure] => [report_dir, :predjango] do |t, args|
args.with_defaults(:stop_on_failure => 'true')
run_tests(system, report_dir, args.stop_on_failure)
end
task :fasttest => "fasttest_#{system}"
TEST_TASK_DIRS << system
desc <<-desc
Start the #{system} locally with the specified environment (defaults to dev).
Other useful environments are devplus (for dev testing with a real local database)
desc
task system, [:env, :options] => [:predjango] do |t, args|
args.with_defaults(:env => 'dev', :options => default_options[system])
# Compile all assets first
compile_assets(watch=false, debug=true)
# Listen for any changes to assets
compile_assets(watch=true, debug=true)
sh(django_admin(system, args.env, 'runserver', args.options))
end
# Per environment tasks
Dir["#{system}/envs/**/*.py"].each do |env_file|
env = env_file.gsub("#{system}/envs/", '').gsub(/\.py/, '').gsub('/', '.')
desc "Attempt to import the settings file #{system}.envs.#{env} and report any errors"
task "#{system}:check_settings:#{env}" => :predjango do
sh("echo 'import #{system}.envs.#{env}' | #{django_admin(system, env, 'shell')}")
end
desc "Compile coffeescript and sass, and then run collectstatic in the specified environment"
task "#{system}:gather_assets:#{env}" do
compile_assets()
sh("#{django_admin(system, env, 'collectstatic', '--noinput')} > /dev/null") do |ok, status|
if !ok
abort "collectstatic failed!"
end
end
end
end
desc "Open jasmine tests for #{system} in your default browser"
task "browse_jasmine_#{system}" do
compile_assets()
django_for_jasmine(system, true) do |jasmine_url|
Launchy.open(jasmine_url)
puts "Press ENTER to terminate".red
$stdin.gets
end
end
desc "Use phantomjs to run jasmine tests for #{system} from the console"
task "phantomjs_jasmine_#{system}" do
compile_assets()
phantomjs = ENV['PHANTOMJS_PATH'] || 'phantomjs'
django_for_jasmine(system, false) do |jasmine_url|
sh("#{phantomjs} node_modules/phantom-jasmine/lib/run_jasmine_test.coffee #{jasmine_url}")
end
end
end
desc "Reset the relational database used by django. WARNING: this will delete all of your existing users"
task :resetdb, [:env] do |t, args|
args.with_defaults(:env => 'dev')
sh(django_admin(:lms, args.env, 'syncdb'))
sh(django_admin(:lms, args.env, 'migrate'))
end
desc "Update the relational database to the latest migration"
task :migrate, [:env] do |t, args|
args.with_defaults(:env => 'dev')
sh(django_admin(:lms, args.env, 'migrate'))
end
Dir["common/lib/*"].select{|lib| File.directory?(lib)}.each do |lib|
task_name = "test_#{lib}"
report_dir = report_dir_path(lib)
desc "Run tests for common lib #{lib}"
task task_name => report_dir do
ENV['NOSE_XUNIT_FILE'] = File.join(report_dir, "nosetests.xml")
cmd = "nosetests #{lib}"
sh(run_under_coverage(cmd, lib)) do |ok, res|
$failed_tests += 1 unless ok
end
end
TEST_TASK_DIRS << lib
desc "Run tests for common lib #{lib} (without coverage)"
task "fasttest_#{lib}" do
sh("nosetests #{lib}")
end
desc "Open jasmine tests for #{lib} in your default browser"
task "browse_jasmine_#{lib}" do
template_jasmine_runner(lib) do |f|
sh("python -m webbrowser -t 'file://#{f}'")
puts "Press ENTER to terminate".red
$stdin.gets
end
end
desc "Use phantomjs to run jasmine tests for #{lib} from the console"
task "phantomjs_jasmine_#{lib}" do
phantomjs = ENV['PHANTOMJS_PATH'] || 'phantomjs'
template_jasmine_runner(lib) do |f|
sh("#{phantomjs} node_modules/phantom-jasmine/lib/run_jasmine_test.coffee #{f}")
end
end
end
task :report_dirs
TEST_TASK_DIRS.each do |dir|
report_dir = report_dir_path(dir)
directory report_dir
task :report_dirs => [REPORT_DIR, report_dir]
end
task :test do
TEST_TASK_DIRS.each do |dir|
Rake::Task["test_#{dir}"].invoke(false)
end
if $failed_tests > 0
abort "Tests failed!"
end
end
namespace :coverage do
desc "Build the html coverage reports"
task :html => :report_dirs do
TEST_TASK_DIRS.each do |dir|
report_dir = report_dir_path(dir)
if !File.file?("#{report_dir}/.coverage")
next
end
sh("coverage html --rcfile=#{dir}/.coveragerc")
end
end
desc "Build the xml coverage reports"
task :xml => :report_dirs do
TEST_TASK_DIRS.each do |dir|
report_dir = report_dir_path(dir)
if !File.file?("#{report_dir}/.coverage")
next
end
# Why doesn't the rcfile control the xml output file properly??
sh("coverage xml -o #{report_dir}/coverage.xml --rcfile=#{dir}/.coveragerc")
end
end
end
task :runserver => :lms
desc "Run django-admin <action> against the specified system and environment"
task "django-admin", [:action, :system, :env, :options] do |t, args|
args.with_defaults(:env => 'dev', :system => 'lms', :options => '')
sh(django_admin(args.system, args.env, args.action, args.options))
end
desc "Set the staff bit for a user"
task :set_staff, [:user, :system, :env] do |t, args|
args.with_defaults(:env => 'dev', :system => 'lms', :options => '')
sh(django_admin(args.system, args.env, 'set_staff', args.user))
end
namespace :cms do
desc "Clone existing MongoDB based course"
task :clone do
if ENV['SOURCE_LOC'] and ENV['DEST_LOC']
sh(django_admin(:cms, :dev, :clone, ENV['SOURCE_LOC'], ENV['DEST_LOC']))
else
raise "You must pass in a SOURCE_LOC and DEST_LOC parameters"
end
end
desc "Delete existing MongoDB based course"
task :delete_course do
if ENV['LOC'] and ENV['COMMIT']
sh(django_admin(:cms, :dev, :delete_course, ENV['LOC'], ENV['COMMIT']))
elsif ENV['LOC']
sh(django_admin(:cms, :dev, :delete_course, ENV['LOC']))
else
raise "You must pass in a LOC parameter"
end
end
desc "Import course data within the given DATA_DIR variable"
task :import do
if ENV['DATA_DIR'] and ENV['COURSE_DIR']
sh(django_admin(:cms, :dev, :import, ENV['DATA_DIR'], ENV['COURSE_DIR']))
elsif ENV['DATA_DIR']
sh(django_admin(:cms, :dev, :import, ENV['DATA_DIR']))
else
raise "Please specify a DATA_DIR variable that point to your data directory.\n" +
"Example: \`rake cms:import DATA_DIR=../data\`"
end
end
desc "Imports all the templates from the code pack"
task :update_templates do
sh(django_admin(:cms, :dev, :update_templates))
end
desc "Import course data within the given DATA_DIR variable"
task :xlint do
if ENV['DATA_DIR'] and ENV['COURSE_DIR']
sh(django_admin(:cms, :dev, :xlint, ENV['DATA_DIR'], ENV['COURSE_DIR']))
elsif ENV['DATA_DIR']
sh(django_admin(:cms, :dev, :xlint, ENV['DATA_DIR']))
else
raise "Please specify a DATA_DIR variable that point to your data directory.\n" +
"Example: \`rake cms:import DATA_DIR=../data\`"
end
end
desc "Export course data to a tar.gz file"
task :export do
if ENV['COURSE_ID'] and ENV['OUTPUT_PATH']
sh(django_admin(:cms, :dev, :export, ENV['COURSE_ID'], ENV['OUTPUT_PATH']))
else
raise "Please specify a COURSE_ID and OUTPUT_PATH.\n" +
"Example: \`rake cms:export COURSE_ID=MITx/12345/name OUTPUT_PATH=foo.tar.gz\`"
end
end
end
desc "Build a properties file used to trigger autodeploy builds"
task :autodeploy_properties do
File.open("autodeploy.properties", "w") do |file|
file.puts("UPSTREAM_NOOP=false")
file.puts("UPSTREAM_BRANCH=#{BRANCH}")
file.puts("UPSTREAM_JOB=#{PACKAGE_NAME}")
file.puts("UPSTREAM_REVISION=#{COMMIT}")
end
end
# --- Internationalization tasks
namespace :i18n do
desc "Extract localizable strings from sources"
task :extract => "i18n:validate:gettext" do
sh(File.join(REPO_ROOT, "i18n", "extract.py"))
end
desc "Compile localizable strings from sources. With optional flag 'extract', will extract strings first."
task :generate => "i18n:validate:gettext" do
if ARGV.last.downcase == 'extract'
Rake::Task["i18n:extract"].execute
end
sh(File.join(REPO_ROOT, "i18n", "generate.py"))
end
desc "Simulate international translation by generating dummy strings corresponding to source strings."
task :dummy do
source_files = Dir["#{REPO_ROOT}/conf/locale/en/LC_MESSAGES/*.po"]
dummy_locale = 'fr'
cmd = File.join(REPO_ROOT, "i18n", "make_dummy.py")
for file in source_files do
sh("#{cmd} #{file} #{dummy_locale}")
end
end
namespace :validate do
desc "Make sure GNU gettext utilities are available"
task :gettext do
begin
select_executable('xgettext')
rescue
msg = "Cannot locate GNU gettext utilities, which are required by django for internationalization.\n"
msg += "(see https://docs.djangoproject.com/en/dev/topics/i18n/translation/#message-files)\n"
msg += "Try downloading them from http://www.gnu.org/software/gettext/"
abort(msg.red)
end
end
desc "Make sure config file with username/password exists"
task :transifex_config do
config_file = "#{Dir.home}/.transifexrc"
if !File.file?(config_file) or File.size(config_file)==0
msg ="Cannot connect to Transifex, config file is missing or empty: #{config_file}\n"
msg += "See http://help.transifex.com/features/client/#transifexrc"
abort(msg.red)
end
end
end
namespace :transifex do
desc "Push source strings to Transifex for translation"
task :push => "i18n:validate:transifex_config" do
cmd = File.join(REPO_ROOT, "i18n", "transifex.py")
sh("#{cmd} push")
end
desc "Pull translated strings from Transifex"
task :pull => "i18n:validate:transifex_config" do
cmd = File.join(REPO_ROOT, "i18n", "transifex.py")
sh("#{cmd} pull")
end
end
desc "Run tests for the internationalization library"
task :test => "i18n:validate:gettext" do
test = File.join(REPO_ROOT, "i18n", "tests")
sh("nosetests #{test}")
end
end
# --- Develop and public documentation ---
desc "Invoke sphinx 'make build' to generate docs."
task :builddocs, [:options] do |t, args|
if args.options == 'pub'
path = "doc/public"
else
path = "docs"
end
Dir.chdir(path) do
sh('make html')
end
end
desc "Show docs in browser (mac and ubuntu)."
task :showdocs, [:options] do |t, args|
if args.options == 'pub'
path = "doc/public"
else
path = "docs"
end
Dir.chdir("#{path}/build/html") do
if RUBY_PLATFORM.include? 'darwin' # mac os
sh('open index.html')
elsif RUBY_PLATFORM.include? 'linux' # make more ubuntu specific?
sh('sensible-browser index.html') # ubuntu
else
raise "\nUndefined how to run browser on your machine.
Please use 'rake builddocs' and then manually open
'mitx/#{path}/build/html/index.html."
end
end
end
desc "Build docs and show them in browser"
task :doc, [:options] => :builddocs do |t, args|
Rake::Task["showdocs"].invoke(args.options)
end
# --- Develop and public documentation ---
rakefiles/assets.rake 0 → 100644
def xmodule_cmd(watch=false, debug=false)
xmodule_cmd = 'xmodule_assets common/static/xmodule'
if watch
"watchmedo shell-command " +
"--patterns='*.js;*.coffee;*.sass;*.scss;*.css' " +
"--recursive " +
"--command='#{xmodule_cmd}' " +
"common/lib/xmodule"
else
xmodule_cmd
end
end
def coffee_cmd(watch=false, debug=false)
"node_modules/.bin/coffee #{watch ? '--watch' : ''} --compile */static"
end
def sass_cmd(watch=false, debug=false)
"sass #{debug ? '--debug-info' : '--style compressed'} " +
"--load-path ./common/static/sass " +
"--require ./common/static/sass/bourbon/lib/bourbon.rb " +
"#{watch ? '--watch' : '--update'} */static"
end
desc "Compile all assets"
multitask :assets => 'assets:all'
namespace :assets do
desc "Compile all assets in debug mode"
multitask :debug
desc "Watch all assets for changes and automatically recompile"
task :watch => 'assets:_watch' do
puts "Press ENTER to terminate".red
$stdin.gets
end
{:xmodule => :install_python_prereqs,
:coffee => :install_node_prereqs,
:sass => :install_ruby_prereqs}.each_pair do |asset_type, prereq_task|
desc "Compile all #{asset_type} assets"
task asset_type => prereq_task do
cmd = send(asset_type.to_s + "_cmd", watch=false, debug=false)
sh(cmd)
end
multitask :all => asset_type
multitask :debug => "assets:#{asset_type}:debug"
multitask :_watch => "assets:#{asset_type}:_watch"
namespace asset_type do
desc "Compile all #{asset_type} assets in debug mode"
task :debug => prereq_task do
cmd = send(asset_type.to_s + "_cmd", watch=false, debug=true)
sh(cmd)
end
desc "Watch all #{asset_type} assets and compile on change"
task :watch => "assets:#{asset_type}:_watch" do
puts "Press ENTER to terminate".red
$stdin.gets
end
task :_watch => prereq_task do
cmd = send(asset_type.to_s + "_cmd", watch=true, debug=true)
background_process(cmd)
end
end
end
multitask :sass => 'assets:xmodule'
namespace :sass do
# In watch mode, sass doesn't immediately compile out of date files,
# so force a recompile first
task :_watch => 'assets:sass:debug'
multitask :debug => 'assets:xmodule:debug'
end
multitask :coffee => 'assets:xmodule'
namespace :coffee do
multitask :debug => 'assets:xmodule:debug'
end
end
[:lms, :cms].each do |system|
# Per environment tasks
environments(system).each do |env|
desc "Compile coffeescript and sass, and then run collectstatic in the specified environment"
task "#{system}:gather_assets:#{env}" => :assets do
sh("#{django_admin(system, env, 'collectstatic', '--noinput')} > /dev/null") do |ok, status|
if !ok
abort "collectstatic failed!"
end
end
end
end
end
rakefiles/deploy.rake 0 → 100644
# Packaging constants
COMMIT = (ENV["GIT_COMMIT"] || `git rev-parse HEAD`).chomp()[0, 10]
PACKAGE_NAME = "mitx"
BRANCH = (ENV["GIT_BRANCH"] || `git symbolic-ref -q HEAD`).chomp().gsub('refs/heads/', '').gsub('origin/', '')
desc "Build a properties file used to trigger autodeploy builds"
task :autodeploy_properties do
File.open("autodeploy.properties", "w") do |file|
file.puts("UPSTREAM_NOOP=false")
file.puts("UPSTREAM_BRANCH=#{BRANCH}")
file.puts("UPSTREAM_JOB=#{PACKAGE_NAME}")
file.puts("UPSTREAM_REVISION=#{COMMIT}")
end
end
\ No newline at end of file
rakefiles/django.rake 0 → 100644
default_options = {
:lms => '8000',
:cms => '8001',
}
task :predjango => :install_python_prereqs do
sh("find . -type f -name *.pyc -delete")
sh('pip install -q --no-index -r requirements/local.txt')
end
task :fastlms do
# this is >2 times faster that rake [lms], and does not need web, good for local dev
django_admin = ENV['DJANGO_ADMIN_PATH'] || select_executable('django-admin.py', 'django-admin')
sh("#{django_admin} runserver --traceback --settings=lms.envs.dev --pythonpath=.")
end
[:lms, :cms].each do |system|
desc <<-desc
Start the #{system} locally with the specified environment (defaults to dev).
Other useful environments are devplus (for dev testing with a real local database)
desc
task system, [:env, :options] => [:install_prereqs, 'assets:_watch', :predjango] do |t, args|
args.with_defaults(:env => 'dev', :options => default_options[system])
sh(django_admin(system, args.env, 'runserver', args.options))
end
# Per environment tasks
environments(system).each do |env|
desc "Attempt to import the settings file #{system}.envs.#{env} and report any errors"
task "#{system}:check_settings:#{env}" => :predjango do
sh("echo 'import #{system}.envs.#{env}' | #{django_admin(system, env, 'shell')}")
end
end
end
desc "Reset the relational database used by django. WARNING: this will delete all of your existing users"
task :resetdb, [:env] do |t, args|
args.with_defaults(:env => 'dev')
sh(django_admin(:lms, args.env, 'syncdb'))
sh(django_admin(:lms, args.env, 'migrate'))
end
desc "Update the relational database to the latest migration"
task :migrate, [:env] do |t, args|
args.with_defaults(:env => 'dev')
sh(django_admin(:lms, args.env, 'migrate'))
end
task :runserver => :lms
desc "Run django-admin <action> against the specified system and environment"
task "django-admin", [:action, :system, :env, :options] do |t, args|
args.with_defaults(:env => 'dev', :system => 'lms', :options => '')
sh(django_admin(args.system, args.env, args.action, args.options))
end
desc "Set the staff bit for a user"
task :set_staff, [:user, :system, :env] do |t, args|
args.with_defaults(:env => 'dev', :system => 'lms', :options => '')
sh(django_admin(args.system, args.env, 'set_staff', args.user))
end
namespace :cms do
desc "Clone existing MongoDB based course"
task :clone do
if ENV['SOURCE_LOC'] and ENV['DEST_LOC']
sh(django_admin(:cms, :dev, :clone, ENV['SOURCE_LOC'], ENV['DEST_LOC']))
else
raise "You must pass in a SOURCE_LOC and DEST_LOC parameters"
end
end
desc "Delete existing MongoDB based course"
task :delete_course do
if ENV['LOC'] and ENV['COMMIT']
sh(django_admin(:cms, :dev, :delete_course, ENV['LOC'], ENV['COMMIT']))
elsif ENV['LOC']
sh(django_admin(:cms, :dev, :delete_course, ENV['LOC']))
else
raise "You must pass in a LOC parameter"
end
end
desc "Import course data within the given DATA_DIR variable"
task :import do
if ENV['DATA_DIR'] and ENV['COURSE_DIR']
sh(django_admin(:cms, :dev, :import, ENV['DATA_DIR'], ENV['COURSE_DIR']))
elsif ENV['DATA_DIR']
sh(django_admin(:cms, :dev, :import, ENV['DATA_DIR']))
else
raise "Please specify a DATA_DIR variable that point to your data directory.\n" +
"Example: \`rake cms:import DATA_DIR=../data\`"
end
end
desc "Imports all the templates from the code pack"
task :update_templates do
sh(django_admin(:cms, :dev, :update_templates))
end
desc "Import course data within the given DATA_DIR variable"
task :xlint do
if ENV['DATA_DIR'] and ENV['COURSE_DIR']
sh(django_admin(:cms, :dev, :xlint, ENV['DATA_DIR'], ENV['COURSE_DIR']))
elsif ENV['DATA_DIR']
sh(django_admin(:cms, :dev, :xlint, ENV['DATA_DIR']))
else
raise "Please specify a DATA_DIR variable that point to your data directory.\n" +
"Example: \`rake cms:import DATA_DIR=../data\`"
end
end
desc "Export course data to a tar.gz file"
task :export do
if ENV['COURSE_ID'] and ENV['OUTPUT_PATH']
sh(django_admin(:cms, :dev, :export, ENV['COURSE_ID'], ENV['OUTPUT_PATH']))
else
raise "Please specify a COURSE_ID and OUTPUT_PATH.\n" +
"Example: \`rake cms:export COURSE_ID=MITx/12345/name OUTPUT_PATH=foo.tar.gz\`"
end
end
end
\ No newline at end of file
rakefiles/docs.rake 0 → 100644
require 'launchy'
# --- Develop and public documentation ---
desc "Invoke sphinx 'make build' to generate docs."
task :builddocs, [:options] do |t, args|
if args.options == 'pub'
path = "doc/public"
else
path = "docs"
end
Dir.chdir(path) do
sh('make html')
end
end
desc "Show docs in browser (mac and ubuntu)."
task :showdocs, [:options] do |t, args|
if args.options == 'pub'
path = "doc/public"
else
path = "docs"
end
Dir.chdir("#{path}/build/html") do
Launchy.open('index.html')
end
end
desc "Build docs and show them in browser"
task :doc, [:options] => :builddocs do |t, args|
Rake::Task["showdocs"].invoke(args.options)
end
# --- Develop and public documentation ---
rakefiles/helpers.rb 0 → 100644
require 'digest/md5'
def select_executable(*cmds)
cmds.find_all{ |cmd| system("which #{cmd} > /dev/null 2>&1") }[0] || fail("No executables found from #{cmds.join(', ')}")
end
def django_admin(system, env, command, *args)
django_admin = ENV['DJANGO_ADMIN_PATH'] || select_executable('django-admin.py', 'django-admin')
return "#{django_admin} #{command} --traceback --settings=#{system}.envs.#{env} --pythonpath=. #{args.join(' ')}"
end
def report_dir_path(dir)
return File.join(REPORT_DIR, dir.to_s)
end
def when_changed(*files)
Rake::Task[PREREQS_MD5_DIR].invoke
cache_file = File.join(PREREQS_MD5_DIR, files.join('-').gsub(/\W+/, '-')) + '.md5'
digest = Digest::MD5.new()
Dir[*files].select{|file| File.file?(file)}.each do |file|
digest.file(file)
end
if !File.exists?(cache_file) or digest.hexdigest != File.read(cache_file)
yield
File.write(cache_file, digest.hexdigest)
end
end
# Runs Process.spawn, and kills the process at the end of the rake process
# Expects the same arguments as Process.spawn
def background_process(*command)
pid = Process.spawn({}, *command, {:pgroup => true})
at_exit do
puts "Ending process and children"
pgid = Process.getpgid(pid)
begin
Timeout.timeout(5) do
puts "Interrupting process group #{pgid}"
Process.kill(:SIGINT, -pgid)
puts "Waiting on process group #{pgid}"
Process.wait(-pgid)
puts "Done waiting on process group #{pgid}"
end
rescue Timeout::Error
begin
Timeout.timeout(5) do
puts "Terminating process group #{pgid}"
Process.kill(:SIGTERM, -pgid)
puts "Waiting on process group #{pgid}"
Process.wait(-pgid)
puts "Done waiting on process group #{pgid}"
end
rescue Timeout::Error
puts "Killing process group #{pgid}"
Process.kill(:SIGKILL, -pgid)
puts "Waiting on process group #{pgid}"
Process.wait(-pgid)
puts "Done waiting on process group #{pgid}"
end
end
end
end
def environments(system)
Dir["#{system}/envs/**/*.py"].select{|file| ! (/__init__.py$/ =~ file)}.map do |env_file|
env_file.gsub("#{system}/envs/", '').gsub(/\.py/, '').gsub('/', '.')
end
end
rakefiles/i18n.rake 0 → 100644
# --- Internationalization tasks
namespace :i18n do
desc "Extract localizable strings from sources"
task :extract => "i18n:validate:gettext" do
sh(File.join(REPO_ROOT, "i18n", "extract.py"))
end
desc "Compile localizable strings from sources. With optional flag 'extract', will extract strings first."
task :generate => "i18n:validate:gettext" do
if ARGV.last.downcase == 'extract'
Rake::Task["i18n:extract"].execute
end
sh(File.join(REPO_ROOT, "i18n", "generate.py"))
end
desc "Simulate international translation by generating dummy strings corresponding to source strings."
task :dummy do
source_files = Dir["#{REPO_ROOT}/conf/locale/en/LC_MESSAGES/*.po"]
dummy_locale = 'fr'
cmd = File.join(REPO_ROOT, "i18n", "make_dummy.py")
for file in source_files do
sh("#{cmd} #{file} #{dummy_locale}")
end
end
namespace :validate do
desc "Make sure GNU gettext utilities are available"
task :gettext do
begin
select_executable('xgettext')
rescue
msg = "Cannot locate GNU gettext utilities, which are required by django for internationalization.\n"
msg += "(see https://docs.djangoproject.com/en/dev/topics/i18n/translation/#message-files)\n"
msg += "Try downloading them from http://www.gnu.org/software/gettext/"
abort(msg.red)
end
end
desc "Make sure config file with username/password exists"
task :transifex_config do
config_file = "#{Dir.home}/.transifexrc"
if !File.file?(config_file) or File.size(config_file)==0
msg ="Cannot connect to Transifex, config file is missing or empty: #{config_file}\n"
msg += "See http://help.transifex.com/features/client/#transifexrc"
abort(msg.red)
end
end
end
namespace :transifex do
desc "Push source strings to Transifex for translation"
task :push => "i18n:validate:transifex_config" do
cmd = File.join(REPO_ROOT, "i18n", "transifex.py")
sh("#{cmd} push")
end
desc "Pull translated strings from Transifex"
task :pull => "i18n:validate:transifex_config" do
cmd = File.join(REPO_ROOT, "i18n", "transifex.py")
sh("#{cmd} pull")
end
end
desc "Run tests for the internationalization library"
task :test => "i18n:validate:gettext" do
test = File.join(REPO_ROOT, "i18n", "tests")
sh("nosetests #{test}")
end
end
rakefiles/jasmine.rake 0 → 100644
require 'colorize'
require 'erb'
require 'launchy'
require 'net/http'
def django_for_jasmine(system, django_reload)
if !django_reload
reload_arg = '--noreload'
end
port = 10000 + rand(40000)
jasmine_url = "http://localhost:#{port}/_jasmine/"
background_process(*django_admin(system, 'jasmine', 'runserver', '-v', '0', port.to_s, reload_arg).split(' '))
up = false
start_time = Time.now
until up do
if Time.now - start_time > 30
abort "Timed out waiting for server to start to run jasmine tests"
end
begin
response = Net::HTTP.get_response(URI(jasmine_url))
puts response.code
up = response.code == '200'
rescue => e
puts e.message
ensure
puts('Waiting server to start')
sleep(0.5)
end
end
yield jasmine_url
end
def template_jasmine_runner(lib)
coffee_files = Dir["#{lib}/**/js/**/*.coffee", "common/static/coffee/src/**/*.coffee"]
if !coffee_files.empty?
sh("node_modules/.bin/coffee -c #{coffee_files.join(' ')}")
end
phantom_jasmine_path = File.expand_path("node_modules/phantom-jasmine")
common_js_root = File.expand_path("common/static/js")
common_coffee_root = File.expand_path("common/static/coffee/src")
# Get arrays of spec and source files, ordered by how deep they are nested below the library
# (and then alphabetically) and expanded from a relative to an absolute path
spec_glob = File.join("#{lib}", "**", "spec", "**", "*.js")
src_glob = File.join("#{lib}", "**", "src", "**", "*.js")
js_specs = Dir[spec_glob].sort_by {|p| [p.split('/').length, p]} .map {|f| File.expand_path(f)}
js_source = Dir[src_glob].sort_by {|p| [p.split('/').length, p]} .map {|f| File.expand_path(f)}
template = ERB.new(File.read("#{lib}/jasmine_test_runner.html.erb"))
template_output = "#{lib}/jasmine_test_runner.html"
File.open(template_output, 'w') do |f|
f.write(template.result(binding))
end
yield File.expand_path(template_output)
end
[:lms, :cms].each do |system|
desc "Open jasmine tests for #{system} in your default browser"
task "browse_jasmine_#{system}" => :assets do
django_for_jasmine(system, true) do |jasmine_url|
Launchy.open(jasmine_url)
puts "Press ENTER to terminate".red
$stdin.gets
end
end
desc "Use phantomjs to run jasmine tests for #{system} from the console"
task "phantomjs_jasmine_#{system}" => :assets do
phantomjs = ENV['PHANTOMJS_PATH'] || 'phantomjs'
django_for_jasmine(system, false) do |jasmine_url|
sh("#{phantomjs} node_modules/phantom-jasmine/lib/run_jasmine_test.coffee #{jasmine_url}")
end
end
end
Dir["common/lib/*"].select{|lib| File.directory?(lib)}.each do |lib|
desc "Open jasmine tests for #{lib} in your default browser"
task "browse_jasmine_#{lib}" do
template_jasmine_runner(lib) do |f|
sh("python -m webbrowser -t 'file://#{f}'")
puts "Press ENTER to terminate".red
$stdin.gets
end
end
desc "Use phantomjs to run jasmine tests for #{lib} from the console"
task "phantomjs_jasmine_#{lib}" do
phantomjs = ENV['PHANTOMJS_PATH'] || 'phantomjs'
template_jasmine_runner(lib) do |f|
sh("#{phantomjs} node_modules/phantom-jasmine/lib/run_jasmine_test.coffee #{f}")
end
end
end
rakefiles/prereqs.rake 0 → 100644
require './rakefiles/helpers.rb'
PREREQS_MD5_DIR = ENV["PREREQ_CACHE_DIR"] || File.join(REPO_ROOT, '.prereqs_cache')
CLOBBER.include(PREREQS_MD5_DIR)
directory PREREQS_MD5_DIR
desc "Install all prerequisites needed for the lms and cms"
task :install_prereqs => [:install_node_prereqs, :install_ruby_prereqs, :install_python_prereqs]
desc "Install all node prerequisites for the lms and cms"
task :install_node_prereqs => "ws:migrate" do
when_changed('package.json') do
sh('npm install')
end unless ENV['NO_PREREQ_INSTALL']
end
desc "Install all ruby prerequisites for the lms and cms"
task :install_ruby_prereqs => "ws:migrate" do
when_changed('Gemfile') do
sh('bundle install')
end unless ENV['NO_PREREQ_INSTALL']
end
desc "Install all python prerequisites for the lms and cms"
task :install_python_prereqs => "ws:migrate" do
when_changed('requirements/**') do
ENV['PIP_DOWNLOAD_CACHE'] ||= '.pip_download_cache'
sh('pip install --exists-action w -r requirements/base.txt')
sh('pip install --exists-action w -r requirements/post.txt')
# Check for private-requirements.txt: used to install our libs as working dirs,
# or personal-use tools.
if File.file?("requirements/private.txt")
sh('pip install -r requirements/private.txt')
end
end unless ENV['NO_PREREQ_INSTALL']
end
\ No newline at end of file
rakefiles/quality.rake 0 → 100644
[:lms, :cms, :common].each do |system|
report_dir = report_dir_path(system)
directory report_dir
desc "Run pep8 on all #{system} code"
task "pep8_#{system}" => [report_dir, :install_python_prereqs] do
sh("pep8 #{system} | tee #{report_dir}/pep8.report")
end
task :pep8 => "pep8_#{system}"
desc "Run pylint on all #{system} code"
task "pylint_#{system}" => [report_dir, :install_python_prereqs] do
apps = Dir["#{system}/*.py", "#{system}/djangoapps/*", "#{system}/lib/*"].map do |app|
File.basename(app)
end.select do |app|
app !=~ /.pyc$/
end.map do |app|
if app =~ /.py$/
app.gsub('.py', '')
else
app
end
end
pythonpath_prefix = "PYTHONPATH=#{system}:#{system}/djangoapps:#{system}/lib:common/djangoapps:common/lib"
sh("#{pythonpath_prefix} pylint --rcfile=.pylintrc -f parseable #{apps.join(' ')} | tee #{report_dir}/pylint.report")
end
task :pylint => "pylint_#{system}"
end
\ No newline at end of file
rakefiles/tests.rake 0 → 100644
# Set up the clean and clobber tasks
CLOBBER.include(REPORT_DIR, 'test_root/*_repo', 'test_root/staticfiles')
$failed_tests = 0
def run_under_coverage(cmd, root)
cmd0, cmd_rest = cmd.split(" ", 2)
# We use "python -m coverage" so that the proper python will run the importable coverage
# rather than the coverage that OS path finds.
cmd = "python -m coverage run --rcfile=#{root}/.coveragerc `which #{cmd0}` #{cmd_rest}"
return cmd
end
def run_tests(system, report_dir, stop_on_failure=true)
ENV['NOSE_XUNIT_FILE'] = File.join(report_dir, "nosetests.xml")
dirs = Dir["common/djangoapps/*"] + Dir["#{system}/djangoapps/*"]
cmd = django_admin(system, :test, 'test', '--logging-clear-handlers', *dirs.each)
sh(run_under_coverage(cmd, system)) do |ok, res|
if !ok and stop_on_failure
abort "Test failed!"
end
$failed_tests += 1 unless ok
end
end
def run_acceptance_tests(system, report_dir, harvest_args)
sh(django_admin(system, 'acceptance', 'syncdb', '--noinput'))
sh(django_admin(system, 'acceptance', 'migrate', '--noinput'))
sh(django_admin(system, 'acceptance', 'harvest', '--debug-mode', '--tag -skip', harvest_args))
end
directory REPORT_DIR
task :clean_test_files do
sh("git clean -fqdx test_root")
end
TEST_TASK_DIRS = []
[:lms, :cms].each do |system|
report_dir = report_dir_path(system)
# Per System tasks
desc "Run all django tests on our djangoapps for the #{system}"
task "test_#{system}", [:stop_on_failure] => ["clean_test_files", :predjango, "#{system}:gather_assets:test", "fasttest_#{system}"]
# Have a way to run the tests without running collectstatic -- useful when debugging without
# messing with static files.
task "fasttest_#{system}", [:stop_on_failure] => [report_dir, :install_prereqs, :predjango] do |t, args|
args.with_defaults(:stop_on_failure => 'true')
run_tests(system, report_dir, args.stop_on_failure)
end
# Run acceptance tests
desc "Run acceptance tests"
task "test_acceptance_#{system}", [:harvest_args] => ["#{system}:gather_assets:acceptance", "fasttest_acceptance_#{system}"]
desc "Run acceptance tests without collectstatic"
task "fasttest_acceptance_#{system}", [:harvest_args] => ["clean_test_files", :predjango, report_dir] do |t, args|
args.with_defaults(:harvest_args => '')
run_acceptance_tests(system, report_dir, args.harvest_args)
end
task :fasttest => "fasttest_#{system}"
TEST_TASK_DIRS << system
end
Dir["common/lib/*"].select{|lib| File.directory?(lib)}.each do |lib|
task_name = "test_#{lib}"
report_dir = report_dir_path(lib)
desc "Run tests for common lib #{lib}"
task task_name => report_dir do
ENV['NOSE_XUNIT_FILE'] = File.join(report_dir, "nosetests.xml")
cmd = "nosetests #{lib}"
sh(run_under_coverage(cmd, lib)) do |ok, res|
$failed_tests += 1 unless ok
end
end
TEST_TASK_DIRS << lib
desc "Run tests for common lib #{lib} (without coverage)"
task "fasttest_#{lib}" do
sh("nosetests #{lib}")
end
end
task :report_dirs
TEST_TASK_DIRS.each do |dir|
report_dir = report_dir_path(dir)
directory report_dir
task :report_dirs => [REPORT_DIR, report_dir]
end
task :test do
TEST_TASK_DIRS.each do |dir|
Rake::Task["test_#{dir}"].invoke(false)
end
if $failed_tests > 0
abort "Tests failed!"
end
end
namespace :coverage do
desc "Build the html coverage reports"
task :html => :report_dirs do
TEST_TASK_DIRS.each do |dir|
report_dir = report_dir_path(dir)
if !File.file?("#{report_dir}/.coverage")
next
end
sh("coverage html --rcfile=#{dir}/.coveragerc")
end
end
desc "Build the xml coverage reports"
task :xml => :report_dirs do
TEST_TASK_DIRS.each do |dir|
report_dir = report_dir_path(dir)
if !File.file?("#{report_dir}/.coverage")
next
end
# Why doesn't the rcfile control the xml output file properly??
sh("coverage xml -o #{report_dir}/coverage.xml --rcfile=#{dir}/.coveragerc")
end
end
end
rakefiles/workspace.rake 0 → 100644
MIGRATION_MARKER_DIR = File.join(REPO_ROOT, '.ws_migrations_complete')
SKIP_MIGRATIONS = ENV['SKIP_WS_MIGRATIONS'] || false
directory MIGRATION_MARKER_DIR
namespace :ws do
task :migrate => MIGRATION_MARKER_DIR do
Dir['ws_migrations/*'].select{|m| File.executable?(m)}.each do |migration|
completion_file = File.join(MIGRATION_MARKER_DIR, File.basename(migration))
if ! File.exist?(completion_file)
sh(migration)
File.write(completion_file, "")
end
end unless SKIP_MIGRATIONS
end
end
\ No newline at end of file
repo-requirements.txt deleted 100644 → 0
-r github-requirements.txt
-r local-requirements.txt
\ No newline at end of file
requirements.txt → requirements/base.txt
-r repo-requirements.txt -r repo.txt
beautifulsoup4==4.1.3 beautifulsoup4==4.1.3
beautifulsoup==3.2.1 beautifulsoup==3.2.1
boto==2.6.0 boto==2.6.0
distribute==0.6.28
django-celery==3.0.11 django-celery==3.0.11
django-countries==1.5 django-countries==1.5
django-followit==0.0.3 django-followit==0.0.3
...@@ -21,11 +23,9 @@ feedparser==5.1.3 ...@@ -21,11 +23,9 @@ feedparser==5.1.3
fs==0.4.0 fs==0.4.0
GitPython==0.3.2.RC1 GitPython==0.3.2.RC1
glob2==0.3 glob2==0.3
http://sympy.googlecode.com/files/sympy-0.7.1.tar.gz
lxml==3.0.1 lxml==3.0.1
mako==0.7.3 mako==0.7.3
Markdown==2.2.1 Markdown==2.2.1
MySQL-python==1.2.4c1
networkx==1.7 networkx==1.7
nltk==2.0.4 nltk==2.0.4
numpy==1.6.2 numpy==1.6.2
...@@ -42,10 +42,10 @@ python-openid==2.2.5 ...@@ -42,10 +42,10 @@ python-openid==2.2.5
pytz==2012h pytz==2012h
PyYAML==3.10 PyYAML==3.10
requests==0.14.2 requests==0.14.2
scipy==0.11.0
Shapely==1.2.16 Shapely==1.2.16
sorl-thumbnail==11.12 sorl-thumbnail==11.12
South==0.7.6 South==0.7.6
sympy==0.7.1
xmltodict==0.4.1 xmltodict==0.4.1
# Used for debugging # Used for debugging
......
requirements/post.txt 0 → 100644
# This must be installed after distribute 0.6.28
MySQL-python==1.2.4c1
# This must be installed after numpy
scipy==0.11.0
requirements/repo.txt 0 → 100644
-r github.txt
-r local.txt
ws_migrations/README.rst 0 → 100644
Developer Workspace Migrations
==============================
This directory contains executable files which run once prior to
installation of pre-requisites to bring a developers workspace
into line.
Specifications
--------------
Each file in this directory should meet the following criteria
* Executable (`chmod +x ws_migrations/foo.sh`)
* Idempotent (ideally, each script is run only once, but no
guarantees are made by the caller, so the script must do
the right thing)
* Either fast or verbose (if the script is going to take
a long time, it should notify the user of that)
* A comment at the top of the file explaining the migration
Execution
---------
The scripts are run by the rake task `ws:migrate`. That task
only runs a given script if a corresponding marker file
in .completed-ws-migrations doesn't already exist.
If the SKIP_WS_MIGRATIONS environment variable is set, then
no workspace migrations will be run.
\ No newline at end of file
ws_migrations/clean_xmodule_assets.sh 0 → 100755
#! /bin/sh
# Remove all of the old xmodule coffee and sass directories
# in preparation to switching to use the xmodule_assets script
rm -rf cms/static/coffee/descriptor
rm -rf cms/static/coffee/module
rm -rf cms/static/sass/descriptor
rm -rf cms/static/sass/module
rm -rf lms/static/coffee/module
rm -rf lms/static/sass/module
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment