Skip to content

E
edx-platform
Commit 1fbe43a6 by Arthur Barrett
Options

Merge branch 'master' into feature/abarrett/lms-notes-app

parents 430e009b fca63f06
Showing with 1110 additions and 273 deletions
.gitignore
......@@ -36,3 +36,7 @@ chromedriver.log
/nbproject
ghostdriver.log
node_modules
.pip_download_cache/
.prereqs_cache
autodeploy.properties
.ws_migrations_complete
README.md
......@@ -8,7 +8,7 @@ Installation
The installation process is a bit messy at the moment. Here's a high-level
overview of what you should do to get started.
**TLDR:** There is a `create-dev-env.sh` script that will attempt to set all
**TLDR:** There is a `scripts/create-dev-env.sh` script that will attempt to set all
of this up for you. If you're in a hurry, run that script. Otherwise, I suggest
that you understand what the script is doing, and why, by reading this document.
......@@ -77,11 +77,16 @@ environment), and Node has a library installer called
Once you've got your languages and virtual environments set up, install
the libraries like so:
$ pip install -r pre-requirements.txt
$ pip install -r requirements.txt
$ pip install -r requirements/base.txt
$ pip install -r requirements/post.txt
$ bundle install
$ npm install
You can also use [`rake`](http://rake.rubyforge.org/) to get all of the prerequisites (or to update)
them if they've changed
$ rake install_prereqs
Other Dependencies
------------------
You'll also need to install [MongoDB](http://www.mongodb.org/), since our
......@@ -137,7 +142,7 @@ Studio, visit `127.0.0.1:8001` in your web browser; to view the LMS, visit
There's also an older version of the LMS that saves its information in XML files
in the `data` directory, instead of in Mongo. To run this older version, run:
$ rake lms
$ rake lms
Further Documentation
=====================
......
cms/djangoapps/contentstore/features/advanced-settings.feature
......@@ -11,7 +11,8 @@ Feature: Advanced (manual) course policy
Given I am on the Advanced Course Settings page in Studio
Then the settings are alphabetized
@skip-phantom
# Skipped because Ubuntu ChromeDriver cannot click notification "Cancel"
@skip
Scenario: Test cancel editing key value
Given I am on the Advanced Course Settings page in Studio
When I edit the value of a policy key
......@@ -20,7 +21,8 @@ Feature: Advanced (manual) course policy
And I reload the page
Then the policy key value is unchanged
@skip-phantom
# Skipped because Ubuntu ChromeDriver cannot click notification "Save"
@skip
Scenario: Test editing key value
Given I am on the Advanced Course Settings page in Studio
When I edit the value of a policy key and save
......@@ -28,7 +30,8 @@ Feature: Advanced (manual) course policy
And I reload the page
Then the policy key value is changed
@skip-phantom
# Skipped because Ubuntu ChromeDriver cannot edit CodeMirror input
@skip
Scenario: Test how multi-line input appears
Given I am on the Advanced Course Settings page in Studio
When I create a JSON object as a value
......@@ -36,7 +39,8 @@ Feature: Advanced (manual) course policy
And I reload the page
Then it is displayed as formatted
@skip-phantom
# Skipped because Ubuntu ChromeDriver cannot edit CodeMirror input
@skip
Scenario: Test automatic quoting of non-JSON values
Given I am on the Advanced Course Settings page in Studio
When I create a non-JSON value not in quotes
......
cms/djangoapps/contentstore/features/checklists.feature
......@@ -10,8 +10,6 @@ Feature: Course checklists
Then I can check and uncheck tasks in a checklist
And They are correctly selected after I reload the page
@skip-phantom
@skip-firefox
Scenario: A task can link to a location within Studio
Given I have opened Checklists
When I select a link to the course outline
......@@ -19,8 +17,6 @@ Feature: Course checklists
And I press the browser back button
Then I am brought back to the course outline in the correct state
@skip-phantom
@skip-firefox
Scenario: A task can link to a location outside Studio
Given I have opened Checklists
When I select a link to help page
......
cms/djangoapps/contentstore/features/course-settings.feature
Feature: Course Settings
As a course author, I want to be able to configure my course settings.
@skip-phantom
Scenario: User can set course dates
Given I have opened a new course in Studio
When I select Schedule and Details
And I set course dates
Then I see the set dates on refresh
@skip-phantom
Scenario: User can clear previously set course dates (except start date)
Given I have set course dates
And I clear all the dates except start
Then I see cleared dates on refresh
@skip-phantom
Scenario: User cannot clear the course start date
Given I have set course dates
And I clear the course start date
......
cms/djangoapps/contentstore/features/section.feature
......@@ -3,7 +3,6 @@ Feature: Create Section
As a course author
I want to create and edit sections
@skip-phantom
Scenario: Add a new section to a course
Given I have opened a new course in Studio
When I click the New Section link
......@@ -27,7 +26,8 @@ Feature: Create Section
And I save a new section release date
Then the section release date is updated
@skip-phantom
# Skipped because Ubuntu ChromeDriver hangs on alert
@skip
Scenario: Delete section
Given I have opened a new course in Studio
And I have added a new section
......
cms/djangoapps/contentstore/features/signup.py
......@@ -18,10 +18,7 @@ def i_fill_in_the_registration_form(step):
@step('I press the Create My Account button on the registration form$')
def i_press_the_button_on_the_registration_form(step):
submit_css = 'form#register_form button#submit'
# Workaround for click not working on ubuntu
# for some unknown reason.
e = world.css_find(submit_css)
e.type(' ')
world.css_click(submit_css)
@step('I should see be on the studio home page$')
......
cms/djangoapps/contentstore/features/studio-overview-togglesection.feature
......@@ -14,7 +14,6 @@ Feature: Overview Toggle Section
When I navigate to the course overview page
Then I do not see the "Collapse All Sections" link
@skip-phantom
Scenario: Collapse link appears after creating first section of a course
Given I have a course with no sections
When I navigate to the course overview page
......@@ -22,7 +21,8 @@ Feature: Overview Toggle Section
Then I see the "Collapse All Sections" link
And all sections are expanded
@skip-phantom
# Skipped because Ubuntu ChromeDriver hangs on alert
@skip
Scenario: Collapse link is not removed after last section of a course is deleted
Given I have a course with 1 section
And I navigate to the course overview page
......
cms/djangoapps/contentstore/features/subsection.feature
......@@ -3,14 +3,12 @@ Feature: Create Subsection
As a course author
I want to create and edit subsections
@skip-phantom
Scenario: Add a new subsection to a section
Given I have opened a new course section in Studio
When I click the New Subsection link
And I enter the subsection name and click save
Then I see my subsection on the Courseware page
@skip-phantom
Scenario: Add a new subsection (with a name containing a quote) to a section (bug #216)
Given I have opened a new course section in Studio
When I click the New Subsection link
......@@ -27,7 +25,6 @@ Feature: Create Subsection
And I reload the page
Then I see it marked as Homework
@skip-phantom
Scenario: Set a due date in a different year (bug #256)
Given I have opened a new subsection in Studio
And I have set a release date and due date in different years
......@@ -35,7 +32,8 @@ Feature: Create Subsection
And I reload the page
Then I see the correct dates
@skip-phantom
# Skipped because Ubuntu ChromeDriver hangs on alert
@skip
Scenario: Delete a subsection
Given I have opened a new course section in Studio
And I have added a new subsection
......
cms/djangoapps/contentstore/features/subsection.py
......@@ -63,14 +63,6 @@ def test_have_set_dates_in_different_years(step):
set_date_and_time('input#due_date', '01/02/2012', 'input#due_time', '04:00')
@step('I see the correct dates$')
def i_see_the_correct_dates(step):
assert_equal('12/25/2011', world.css_find('input#start_date').first.value)
assert_equal('03:00', world.css_find('input#start_time').first.value)
assert_equal('01/02/2012', world.css_find('input#due_date').first.value)
assert_equal('04:00', world.css_find('input#due_time').first.value)
@step('I mark it as Homework$')
def i_mark_it_as_homework(step):
world.css_click('a.menu-toggle')
......@@ -101,8 +93,20 @@ def the_subsection_does_not_exist(step):
assert world.browser.is_element_not_present_by_css(css)
@step('I see the correct dates$')
def i_see_the_correct_dates(step):
assert_equal('12/25/2011', get_date('input#start_date'))
assert_equal('03:00', get_date('input#start_time'))
assert_equal('01/02/2012', get_date('input#due_date'))
assert_equal('04:00', get_date('input#due_time'))
############ HELPER METHODS ###################
def get_date(css):
return world.css_find(css).first.value.strip()
def save_subsection_name(name):
name_css = 'input.new-subsection-name-input'
save_css = 'input.new-subsection-name-save'
......
cms/djangoapps/contentstore/tests/test_contentstore.py
......@@ -646,7 +646,7 @@ class ContentStoreTest(ModuleStoreTestCase):
resp = self.client.get(reverse('index'))
self.assertContains(
resp,
'<h1 class="title-1">My Courses</h1>',
'<h1 class="page-header">My Courses</h1>',
status_code=200,
html=True
)
......
cms/djangoapps/contentstore/tests/test_i18n.py
......@@ -48,7 +48,7 @@ class InternationalizationTest(ModuleStoreTestCase):
resp = self.client.get(reverse('index'))
self.assertContains(resp,
'<h1 class="title-1">My Courses</h1>',
'<h1 class="page-header">My Courses</h1>',
status_code=200,
html=True)
......@@ -63,7 +63,7 @@ class InternationalizationTest(ModuleStoreTestCase):
)
self.assertContains(resp,
'<h1 class="title-1">My Courses</h1>',
'<h1 class="page-header">My Courses</h1>',
status_code=200,
html=True)
......
cms/envs/acceptance.py
......@@ -8,27 +8,41 @@ from .test import *
# otherwise the browser will not render the pages correctly
DEBUG = True
# Show the courses that are in the data directory
COURSES_ROOT = ENV_ROOT / "data"
DATA_DIR = COURSES_ROOT
# MODULESTORE = {
# 'default': {
# 'ENGINE': 'xmodule.modulestore.xml.XMLModuleStore',
# 'OPTIONS': {
# 'data_dir': DATA_DIR,
# 'default_class': 'xmodule.hidden_module.HiddenDescriptor',
# }
# }
# }
# Disable warnings for acceptance tests, to make the logs readable
import logging
logging.disable(logging.ERROR)
MODULESTORE_OPTIONS = {
'default_class': 'xmodule.raw_module.RawDescriptor',
'host': 'localhost',
'db': 'test_xmodule',
'collection': 'acceptance_modulestore',
'fs_root': TEST_ROOT / "data",
'render_template': 'mitxmako.shortcuts.render_to_string',
}
MODULESTORE = {
'default': {
'ENGINE': 'xmodule.modulestore.mongo.MongoModuleStore',
'OPTIONS': MODULESTORE_OPTIONS
},
'direct': {
'ENGINE': 'xmodule.modulestore.mongo.MongoModuleStore',
'OPTIONS': MODULESTORE_OPTIONS
},
'draft': {
'ENGINE': 'xmodule.modulestore.mongo.DraftMongoModuleStore',
'OPTIONS': MODULESTORE_OPTIONS
}
}
# Set this up so that rake lms[acceptance] and running the
# harvest command both use the same (test) database
# which they can flush without messing up your dev db
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': ENV_ROOT / "db" / "test_mitx.db",
'TEST_NAME': ENV_ROOT / "db" / "test_mitx.db",
'NAME': TEST_ROOT / "db" / "test_mitx.db",
'TEST_NAME': TEST_ROOT / "db" / "test_mitx.db",
}
}
......
cms/envs/test.py
......@@ -41,8 +41,8 @@ MODULESTORE_OPTIONS = {
'default_class': 'xmodule.raw_module.RawDescriptor',
'host': 'localhost',
'db': 'test_xmodule',
'collection': 'modulestore',
'fs_root': GITHUB_REPO_ROOT,
'collection': 'test_modulestore',
'fs_root': TEST_ROOT / "data",
'render_template': 'mitxmako.shortcuts.render_to_string',
}
......
cms/static/sass/_variables.scss
......@@ -184,6 +184,6 @@ $lightBluishGrey2: rgb(213, 220, 228);
$error-red: rgb(253, 87, 87);
// type
$sans-serif: $f-serif;
$sans-serif: $f-sans-serif;
$body-line-height: golden-ratio(.875em, 1);
common/djangoapps/terrain/browser.py
"""
Browser set up for acceptance tests.
"""
#pylint: disable=E1101
#pylint: disable=W0613
#pylint: disable=W0611
from lettuce import before, after, world
from splinter.browser import Browser
from logging import getLogger
from django.core.management import call_command
from django.conf import settings
from selenium.common.exceptions import WebDriverException
# Let the LMS and CMS do their one-time setup
# For example, setting up mongo caches
from lms import one_time_startup
from cms import one_time_startup
logger = getLogger(__name__)
logger.info("Loading the lettuce acceptance testing terrain file...")
# There is an import issue when using django-staticfiles with lettuce
# Lettuce assumes that we are using django.contrib.staticfiles,
# but the rest of the app assumes we are using django-staticfiles
# (in particular, django-pipeline and our mako implementation)
# To resolve this, we check whether staticfiles is installed,
# then redirect imports for django.contrib.staticfiles
# to use staticfiles.
try:
import staticfiles
except ImportError:
pass
else:
import sys
sys.modules['django.contrib.staticfiles'] = staticfiles
LOGGER = getLogger(__name__)
LOGGER.info("Loading the lettuce acceptance testing terrain file...")
MAX_VALID_BROWSER_ATTEMPTS = 20
@before.harvest
def initial_setup(server):
'''
Launch the browser once before executing the tests
'''
"""
Launch the browser once before executing the tests.
"""
browser_driver = getattr(settings, 'LETTUCE_BROWSER', 'chrome')
world.browser = Browser(browser_driver)
# There is an issue with ChromeDriver2 r195627 on Ubuntu
# in which we sometimes get an invalid browser session.
# This is a work-around to ensure that we get a valid session.
success = False
num_attempts = 0
while (not success) and num_attempts < MAX_VALID_BROWSER_ATTEMPTS:
# Get a browser session
world.browser = Browser(browser_driver)
# Try to visit the main page
# If the browser session is invalid, this will
# raise a WebDriverException
try:
world.visit('/')
except WebDriverException:
world.browser.quit()
num_attempts += 1
else:
success = True
# If we were unable to get a valid session within the limit of attempts,
# then we cannot run the tests.
if not success:
raise IOError("Could not acquire valid ChromeDriver browser session.")
@before.each_scenario
def reset_data(scenario):
'''
"""
Clean out the django test database defined in the
envs/acceptance.py file: mitx_all/db/test_mitx.db
'''
logger.debug("Flushing the test database...")
"""
LOGGER.debug("Flushing the test database...")
call_command('flush', interactive=False)
@after.each_scenario
def screenshot_on_error(scenario):
'''
Save a screenshot to help with debugging
'''
"""
Save a screenshot to help with debugging.
"""
if scenario.failed:
world.browser.driver.save_screenshot('/tmp/last_failed_scenario.png')
@after.all
def teardown_browser(total):
'''
Quit the browser after executing the tests
'''
"""
Quit the browser after executing the tests.
"""
world.browser.quit()
pass
common/djangoapps/terrain/course_helpers.py
......@@ -38,9 +38,11 @@ def create_user(uname):
@world.absorb
def log_in(username, password):
'''
Log the user in programatically
'''
"""
Log the user in programatically.
This will delete any existing cookies to ensure that the user
logs in to the correct session.
"""
# Authenticate the user
user = authenticate(username=username, password=password)
......@@ -60,15 +62,8 @@ def log_in(username, password):
# Retrieve the sessionid and add it to the browser's cookies
cookie_dict = {settings.SESSION_COOKIE_NAME: request.session.session_key}
try:
world.browser.cookies.add(cookie_dict)
# WebDriver has an issue where we cannot set cookies
# before we make a GET request, so if we get an error,
# we load the '/' page and try again
except:
world.browser.visit(django_url('/'))
world.browser.cookies.add(cookie_dict)
world.browser.cookies.delete()
world.browser.cookies.add(cookie_dict)
@world.absorb
......
common/djangoapps/terrain/ui_helpers.py
......@@ -53,12 +53,9 @@ def css_find(css):
@world.absorb
def css_click(css_selector):
'''
First try to use the regular click method,
but if clicking in the middle of an element
doesn't work it might be that it thinks some other
element is on top of it there so click in the upper left
'''
"""
Perform a click on a CSS selector, retrying if it initially fails
"""
try:
world.browser.find_by_css(css_selector).click()
......
common/lib/capa/capa/tests/test_responsetypes.py
......@@ -708,7 +708,7 @@ class JavascriptResponseTest(ResponseTest):
def test_grade(self):
# Compile coffee files into javascript used by the response
coffee_file_path = os.path.dirname(__file__) + "/test_files/js/*.coffee"
os.system("coffee -c %s" % (coffee_file_path))
os.system("node_modules/.bin/coffee -c %s" % (coffee_file_path))
problem = self.build_problem(generator_src="test_problem_generator.js",
grader_src="test_problem_grader.js",
......
common/lib/capa/jasmine_test_runner.html.erb 0 → 100644
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title>Jasmine Test Runner</title>
<link rel="stylesheet" type="text/css" href="<%= phantom_jasmine_path %>/vendor/jasmine-1.2.0/jasmine.css">
<script type="text/javascript" src="<%= phantom_jasmine_path %>/vendor/jasmine-1.2.0/jasmine.js"></script>
<script type="text/javascript" src="<%= phantom_jasmine_path %>/vendor/jasmine-1.2.0/jasmine-html.js"></script>
<script type="text/javascript" src="<%= phantom_jasmine_path %>/lib/console-runner.js"></script>
<script type="text/javascript" src="<%= common_coffee_root %>/ajax_prefix.js"></script>
<script type="text/javascript" src="<%= common_coffee_root %>/logger.js"></script>
<script type="text/javascript" src="<%= common_js_root %>/vendor/jquery.min.js"></script>
<script type="text/javascript" src="<%= common_js_root %>/vendor/jasmine-jquery.js"></script>
<script type="text/javascript" src="<%= common_js_root %>/vendor/jquery.cookie.js"></script>
<script type="text/javascript" src="<%= common_js_root %>/vendor/CodeMirror/codemirror.js"></script>
<script type="text/javascript" src="<%= common_js_root %>/vendor/tiny_mce/jquery.tinymce.js"></script>
<script type="text/javascript" src="<%= common_js_root %>/vendor/tiny_mce/tiny_mce.js"></script>
<script type="text/javascript" src="<%= common_js_root %>/vendor/RequireJS.js"></script>
<script type="text/javascript">
AjaxPrefix.addAjaxPrefix(jQuery, function() {
return "";
});
</script>
<!-- SOURCE FILES -->
<% for src in js_source %>
<script type="text/javascript" src="<%= src %>"></script>
<% end %>
<!-- SPEC FILES -->
<% for src in js_specs %>
<script type="text/javascript" src="<%= src %>"></script>
<% end %>
</head>
<body>
<script type="text/javascript">
var console_reporter = new jasmine.ConsoleReporter()
jasmine.getEnv().addReporter(new jasmine.TrivialReporter());
jasmine.getEnv().addReporter(console_reporter);
jasmine.getEnv().execute();
</script>
</body>
</html>
common/lib/capa/setup.py
......@@ -4,5 +4,5 @@ setup(
name="capa",
version="0.1",
packages=find_packages(exclude=["tests"]),
install_requires=['distribute==0.6.30', 'pyparsing==1.5.6'],
install_requires=['distribute==0.6.28', 'pyparsing==1.5.6'],
)
common/lib/xmodule/xmodule/templates.py
......@@ -4,8 +4,18 @@ These templates are used by the CMS to provide baseline content that
can be cloned when adding new modules to a course.
`Template`s are defined in x_module. They contain 3 attributes:
metadata: A dictionary with the template metadata
data: A JSON value that defines the template content
metadata: A dictionary with the template metadata. This should contain
any values for fields
* with scope Scope.settings
* that have values different than the field defaults
* and that are to be editable in Studio
data: A JSON value that defines the template content. This should be a dictionary
containing values for fields
* with scope Scope.content
* that have values different than the field defaults
* and that are to be editable in Studio
or, if the module uses a single Scope.content String field named `data`, this
should be a string containing the contents of that field
children: A list of Location urls that define the template children
Templates are defined on XModuleDescriptor types, in the template attribute.
......
doc/development.md
......@@ -36,7 +36,7 @@ Check out the course data directories that you want to work with into the
To create your development environment, run the shell script in the root of
the repo:
create-dev-env.sh
scripts/create-dev-env.sh
## Starting development servers
......
doc/testing.md
......@@ -161,36 +161,36 @@ try running `bundle install` to install the required ruby gems.
We use [Lettuce](http://lettuce.it/) for acceptance testing.
Most of our tests use [Splinter](http://splinter.cobrateam.info/)
to simulate UI browser interactions. Splinter, in turn,
uses [Selenium](http://docs.seleniumhq.org/) to control the browser.
uses [Selenium](http://docs.seleniumhq.org/) to control the Chrome browser.
**Prerequisite**: You must have [ChromeDriver](https://code.google.com/p/selenium/wiki/ChromeDriver)
installed to run the tests in Chrome.
installed to run the tests in Chrome. The tests are confirmed to run
with Chrome (not Chromium) version 26.0.0.1410.63 with ChromeDriver
version r195636.
Before running the tests, you need to set up the test database:
To run all the acceptance tests:
rm ../db/test_mitx.db
rake django-admin[syncdb,lms,acceptance,--noinput]
rake django-admin[migrate,lms,acceptance,--noinput]
rake test_acceptance_lms
rake test_acceptance_cms
To run the acceptance tests:
1. Start the Django server locally using the settings in **acceptance.py**:
To test only a specific feature:
rake lms[acceptance]
rake test_acceptance_lms[lms/djangoapps/courseware/features/problems.feature]
2. In another shell, run the tests:
To start the debugger on failure, add the `--pdb` option:
django-admin.py harvest --no-server --settings=lms.envs.acceptance --pythonpath=. lms/djangoapps/portal/features/
rake test_acceptance_lms["lms/djangoapps/courseware/features/problems.feature --pdb"]
To test only a specific feature:
To run tests faster by not collecting static files, you can use
`rake fasttest_acceptance_lms` and `rake fasttest_acceptance_cms`.
django-admin.py harvest --no-server --settings=lms.envs.acceptance --pythonpath=. lms/djangoapps/courseware/features/high-level-tabs.feature
**Troubleshooting**: If you get an error message that says something about harvest not being a command, you probably are missing a requirement.
Try running:
pip install -r requirements.txt
**Note**: The acceptance tests can *not* currently run in parallel.
## Viewing Test Coverage
......
fixtures/anonymize_fixtures.py deleted 100755 → 0
#! /usr/bin/env python
import sys
import json
import random
import copy
from collections import defaultdict
from argparse import ArgumentParser, FileType
from datetime import datetime
def generate_user(user_number):
return {
"pk": user_number,
"model": "auth.user",
"fields": {
"status": "w",
"last_name": "Last",
"gold": 0,
"is_staff": False,
"user_permissions": [],
"interesting_tags": "",
"email_key": None,
"date_joined": "2012-04-26 11:36:39",
"first_name": "",
"email_isvalid": False,
"avatar_type": "n",
"website": "",
"is_superuser": False,
"date_of_birth": None,
"last_login": "2012-04-26 11:36:48",
"location": "",
"new_response_count": 0,
"email": "user{num}@example.com".format(num=user_number),
"username": "user{num}".format(num=user_number),
"is_active": True,
"consecutive_days_visit_count": 0,
"email_tag_filter_strategy": 1,
"groups": [],
"password": "sha1$90e6f$562a1d783a0c47ce06ebf96b8c58123a0671bbf0",
"silver": 0,
"bronze": 0,
"questions_per_page": 10,
"about": "",
"show_country": True,
"country": "",
"display_tag_filter_strategy": 0,
"seen_response_count": 0,
"real_name": "",
"ignored_tags": "",
"reputation": 1,
"gravatar": "366d981a10116969c568a18ee090f44c",
"last_seen": "2012-04-26 11:36:39"
}
}
def parse_args(args=sys.argv[1:]):
parser = ArgumentParser()
parser.add_argument('-d', '--data', type=FileType('r'), default=sys.stdin)
parser.add_argument('-o', '--output', type=FileType('w'), default=sys.stdout)
parser.add_argument('count', type=int)
return parser.parse_args(args)
def main(args=sys.argv[1:]):
args = parse_args(args)
data = json.load(args.data)
unique_students = set(entry['fields']['student'] for entry in data)
if args.count > len(unique_students) * 0.1:
raise Exception("Can't be sufficiently anonymous selecting {count} of {unique} students".format(
count=args.count, unique=len(unique_students)))
by_problems = defaultdict(list)
for entry in data:
by_problems[entry['fields']['module_id']].append(entry)
out_data = []
out_pk = 1
for name, answers in by_problems.items():
for student_id in xrange(args.count):
sample = random.choice(answers)
data = copy.deepcopy(sample)
data["fields"]["student"] = student_id + 1
data["fields"]["created"] = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
data["fields"]["modified"] = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
data["pk"] = out_pk
out_pk += 1
out_data.append(data)
for student_id in xrange(args.count):
out_data.append(generate_user(student_id))
json.dump(out_data, args.output, indent=2)
if __name__ == "__main__":
sys.exit(main())
fixtures/pm.json deleted 100644 → 0
[{"pk": 1, "model": "user.userprofile", "fields": {"name": "pm", "language": "pm", "courseware": "course.xml", "meta": "", "location": "pm", "user": 1}}, {"pk": 1, "model": "auth.user", "fields": {"status": "w", "last_name": "", "gold": 0, "is_staff": true, "user_permissions": [], "interesting_tags": "", "email_key": null, "date_joined": "2012-01-23 17:03:54", "first_name": "", "email_isvalid": false, "avatar_type": "n", "website": "", "is_superuser": true, "date_of_birth": null, "last_login": "2012-01-23 17:04:16", "location": "", "new_response_count": 0, "email": "pmitros@csail.mit.edu", "username": "pm", "is_active": true, "consecutive_days_visit_count": 0, "email_tag_filter_strategy": 1, "groups": [], "password": "sha1$a3e96$dbabbd114f0da01bce2cc2adcafa2ca651c7ae0a", "silver": 0, "bronze": 0, "questions_per_page": 10, "about": "", "show_country": false, "country": "", "display_tag_filter_strategy": 0, "seen_response_count": 0, "real_name": "", "ignored_tags": "", "reputation": 1, "gravatar": "7a591afd0cc7972fdbe5e12e26af352a", "last_seen": "2012-01-23 17:04:41"}}, {"pk": 1, "model": "user.userprofile", "fields": {"name": "pm", "language": "pm", "courseware": "course.xml", "meta": "", "location": "pm", "user": 1}}, {"pk": 1, "model": "auth.user", "fields": {"status": "w", "last_name": "", "gold": 0, "is_staff": true, "user_permissions": [], "interesting_tags": "", "email_key": null, "date_joined": "2012-01-23 17:03:54", "first_name": "", "email_isvalid": false, "avatar_type": "n", "website": "", "is_superuser": true, "date_of_birth": null, "last_login": "2012-01-23 17:04:16", "location": "", "new_response_count": 0, "email": "pmitros@csail.mit.edu", "username": "pm", "is_active": true, "consecutive_days_visit_count": 0, "email_tag_filter_strategy": 1, "groups": [], "password": "sha1$a3e96$dbabbd114f0da01bce2cc2adcafa2ca651c7ae0a", "silver": 0, "bronze": 0, "questions_per_page": 10, "about": "", "show_country": false, "country": "", "display_tag_filter_strategy": 0, "seen_response_count": 0, "real_name": "", "ignored_tags": "", "reputation": 1, "gravatar": "7a591afd0cc7972fdbe5e12e26af352a", "last_seen": "2012-01-23 17:04:41"}}]
\ No newline at end of file
jenkins/base.sh deleted 100644 → 0
##
## requires >= 1.3.0 of the Jenkins git plugin
##
function github_status {
if [[ ! ${GIT_URL} =~ git@github.com:([^/]+)/([^\.]+).git ]]; then
echo "Cannot parse Github org or repo from URL, using defaults."
ORG="edx"
REPO="mitx"
else
ORG=${BASH_REMATCH[1]}
REPO=${BASH_REMATCH[2]}
fi
gcli status create $ORG $REPO $GIT_COMMIT \
--params=$1 \
target_url:$BUILD_URL \
description:"Build #$BUILD_NUMBER is running" \
-f csv
}
function github_mark_failed_on_exit {
trap '[ $? == "0" ] || github_status state:failed' EXIT
}
jenkins/test.sh
......@@ -3,8 +3,21 @@
set -e
set -x
##
## requires >= 1.3.0 of the Jenkins git plugin
##
function github_status {
gcli status create edx edx-platform $GIT_COMMIT \
if [[ ! ${GIT_URL} =~ git@github.com:([^/]+)/([^\.]+).git ]]; then
echo "Cannot parse Github org or repo from URL, using defaults."
ORG="edx"
REPO="edx-platform"
else
ORG=${BASH_REMATCH[1]}
REPO=${BASH_REMATCH[2]}
fi
gcli status create $ORG $REPO $GIT_COMMIT \
--params=$1 \
target_url:$BUILD_URL \
description:"Build #$BUILD_NUMBER $2" \
......@@ -27,21 +40,32 @@ git submodule foreach 'git reset --hard HEAD'
export PYTHONIOENCODING=UTF-8
GIT_BRANCH=${GIT_BRANCH/HEAD/master}
if [ ! -d /mnt/virtualenvs/"$JOB_NAME" ]; then
mkdir -p /mnt/virtualenvs/"$JOB_NAME"
virtualenv /mnt/virtualenvs/"$JOB_NAME"
# When running in parallel on jenkins, workspace could be suffixed by @x
# In that case, we want to use a separate virtualenv that matches up with
# workspace
#
# We need to handle both the case of /path/to/workspace
# and /path/to/workspace@2, which is why we use the following substitutions
#
# $WORKSPACE is the absolute path for the workspace
WORKSPACE_SUFFIX=$(expr "$WORKSPACE" : '.*\(@.*\)') || true
VIRTUALENV_DIR="/mnt/virtualenvs/${JOB_NAME}${WORKSPACE_SUFFIX}"
if [ ! -d "$VIRTUALENV_DIR" ]; then
mkdir -p "$VIRTUALENV_DIR"
virtualenv "$VIRTUALENV_DIR"
fi
export PIP_DOWNLOAD_CACHE=/mnt/pip-cache
source /mnt/virtualenvs/"$JOB_NAME"/bin/activate
pip install -q -r pre-requirements.txt
yes w | pip install -q -r requirements.txt
# Allow django liveserver tests to use a range of ports
export DJANGO_LIVE_TEST_SERVER_ADDRESS=${DJANGO_LIVE_TEST_SERVER_ADDRESS-localhost:8000-9000}
bundle install
npm install
source /mnt/virtualenvs/"$JOB_NAME"/bin/activate
rake install_prereqs
rake clobber
rake pep8 > pep8.log || cat pep8.log
rake pylint > pylint.log || cat pylint.log
......@@ -54,7 +78,7 @@ rake test_lms[false] || TESTS_FAILED=1
rake test_common/lib/capa || TESTS_FAILED=1
rake test_common/lib/xmodule || TESTS_FAILED=1
# Run the jaavascript unit tests
# Run the javascript unit tests
rake phantomjs_jasmine_lms || TESTS_FAILED=1
rake phantomjs_jasmine_cms || TESTS_FAILED=1
rake phantomjs_jasmine_common/lib/xmodule || TESTS_FAILED=1
......
jenkins/test_acceptance.sh 0 → 100755
#! /bin/bash
set -e
set -x
git remote prune origin
# Reset the submodule, in case it changed
git submodule foreach 'git reset --hard HEAD'
# Set the IO encoding to UTF-8 so that askbot will start
export PYTHONIOENCODING=UTF-8
if [ ! -d /mnt/virtualenvs/"$JOB_NAME" ]; then
mkdir -p /mnt/virtualenvs/"$JOB_NAME"
virtualenv /mnt/virtualenvs/"$JOB_NAME"
fi
export PIP_DOWNLOAD_CACHE=/mnt/pip-cache
source /mnt/virtualenvs/"$JOB_NAME"/bin/activate
rake install_prereqs
rake clobber
TESTS_FAILED=0
# Assumes that Xvfb has been started by upstart
# and is capturing display :1
# The command for this is:
# /usr/bin/Xvfb :1 -screen 0 1024x268x24
# This allows us to run Chrome without a display
export DISPLAY=:1
# Run the lms and cms acceptance tests
# (the -v flag turns off color in the output)
rake test_acceptance_lms["-v 3"] || TESTS_FAILED=1
rake test_acceptance_cms["-v 3"] || TESTS_FAILED=1
[ $TESTS_FAILED == '0' ]
lms/djangoapps/courseware/tests/tests.py
......@@ -631,8 +631,8 @@ class TestViewAuth(LoginEnrollmentTestCase):
urls = reverse_urls(['info', 'progress'], course)
urls.extend([
reverse('book', kwargs={'course_id': course.id,
'book_index': book.title})
for book in course.textbooks
'book_index': index})
for index, book in enumerate(course.textbooks)
])
return urls
......@@ -643,8 +643,6 @@ class TestViewAuth(LoginEnrollmentTestCase):
"""
urls = reverse_urls(['about_course'], course)
urls.append(reverse('courses'))
# Need separate test for change_enrollment, since it's a POST view
#urls.append(reverse('change_enrollment'))
return urls
......
lms/envs/acceptance.py
......@@ -8,13 +8,17 @@ from .test import *
# otherwise the browser will not render the pages correctly
DEBUG = True
# Disable warnings for acceptance tests, to make the logs readable
import logging
logging.disable(logging.ERROR)
# Use the mongo store for acceptance tests
modulestore_options = {
'default_class': 'xmodule.raw_module.RawDescriptor',
'host': 'localhost',
'db': 'test_xmodule',
'collection': 'modulestore',
'fs_root': GITHUB_REPO_ROOT,
'collection': 'acceptance_modulestore',
'fs_root': TEST_ROOT / "data",
'render_template': 'mitxmako.shortcuts.render_to_string',
}
......@@ -33,7 +37,7 @@ CONTENTSTORE = {
'ENGINE': 'xmodule.contentstore.mongo.MongoContentStore',
'OPTIONS': {
'host': 'localhost',
'db': 'test_xcontent',
'db': 'test_xmodule',
}
}
......@@ -43,8 +47,8 @@ CONTENTSTORE = {
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': ENV_ROOT / "db" / "test_mitx.db",
'TEST_NAME': ENV_ROOT / "db" / "test_mitx.db",
'NAME': TEST_ROOT / "db" / "test_mitx.db",
'TEST_NAME': TEST_ROOT / "db" / "test_mitx.db",
}
}
......
lms/envs/common.py
......@@ -126,9 +126,7 @@ sys.path.append(COMMON_ROOT / 'lib')
# For Node.js
system_node_path = os.environ.get("NODE_PATH", None)
if system_node_path is None:
system_node_path = "/usr/local/lib/node_modules"
system_node_path = os.environ.get("NODE_PATH", REPO_ROOT / 'node_modules')
node_paths = [COMMON_ROOT / "static/js/vendor",
COMMON_ROOT / "static/coffee/src",
......
pre-requirements.txt deleted 100644 → 0
# We use `scipy` in our project, which relies on `numpy`. `pip` apparently
# installs packages in a two-step process, where it will first try to build
# all packages, and then try to install all packages. As a result, if we simply
# added these packages to the top of `requirements.txt`, `pip` would try to
# build `scipy` before `numpy` has been installed, and it would fail. By
# separating this out into a `pre-requirements.txt` file, we can make sure
# that `numpy` is built *and* installed before we try to build `scipy`.
numpy==1.6.2
distribute>=0.6.28
rakefiles/assets.rake 0 → 100644
def xmodule_cmd(watch=false, debug=false)
xmodule_cmd = 'xmodule_assets common/static/xmodule'
if watch
"watchmedo shell-command " +
"--patterns='*.js;*.coffee;*.sass;*.scss;*.css' " +
"--recursive " +
"--command='#{xmodule_cmd}' " +
"common/lib/xmodule"
else
xmodule_cmd
end
end
def coffee_cmd(watch=false, debug=false)
"node_modules/.bin/coffee #{watch ? '--watch' : ''} --compile */static"
end
def sass_cmd(watch=false, debug=false)
"sass #{debug ? '--debug-info' : '--style compressed'} " +
"--load-path ./common/static/sass " +
"--require ./common/static/sass/bourbon/lib/bourbon.rb " +
"#{watch ? '--watch' : '--update'} */static"
end
desc "Compile all assets"
multitask :assets => 'assets:all'
namespace :assets do
desc "Compile all assets in debug mode"
multitask :debug
desc "Watch all assets for changes and automatically recompile"
task :watch => 'assets:_watch' do
puts "Press ENTER to terminate".red
$stdin.gets
end
{:xmodule => :install_python_prereqs,
:coffee => :install_node_prereqs,
:sass => :install_ruby_prereqs}.each_pair do |asset_type, prereq_task|
desc "Compile all #{asset_type} assets"
task asset_type => prereq_task do
cmd = send(asset_type.to_s + "_cmd", watch=false, debug=false)
sh(cmd)
end
multitask :all => asset_type
multitask :debug => "assets:#{asset_type}:debug"
multitask :_watch => "assets:#{asset_type}:_watch"
namespace asset_type do
desc "Compile all #{asset_type} assets in debug mode"
task :debug => prereq_task do
cmd = send(asset_type.to_s + "_cmd", watch=false, debug=true)
sh(cmd)
end
desc "Watch all #{asset_type} assets and compile on change"
task :watch => "assets:#{asset_type}:_watch" do
puts "Press ENTER to terminate".red
$stdin.gets
end
task :_watch => prereq_task do
cmd = send(asset_type.to_s + "_cmd", watch=true, debug=true)
background_process(cmd)
end
end
end
multitask :sass => 'assets:xmodule'
namespace :sass do
# In watch mode, sass doesn't immediately compile out of date files,
# so force a recompile first
task :_watch => 'assets:sass:debug'
multitask :debug => 'assets:xmodule:debug'
end
multitask :coffee => 'assets:xmodule'
namespace :coffee do
multitask :debug => 'assets:xmodule:debug'
end
end
[:lms, :cms].each do |system|
# Per environment tasks
environments(system).each do |env|
desc "Compile coffeescript and sass, and then run collectstatic in the specified environment"
task "#{system}:gather_assets:#{env}" => :assets do
sh("#{django_admin(system, env, 'collectstatic', '--noinput')} > /dev/null") do |ok, status|
if !ok
abort "collectstatic failed!"
end
end
end
end
end
rakefiles/deploy.rake 0 → 100644
# Packaging constants
COMMIT = (ENV["GIT_COMMIT"] || `git rev-parse HEAD`).chomp()[0, 10]
PACKAGE_NAME = "mitx"
BRANCH = (ENV["GIT_BRANCH"] || `git symbolic-ref -q HEAD`).chomp().gsub('refs/heads/', '').gsub('origin/', '')
desc "Build a properties file used to trigger autodeploy builds"
task :autodeploy_properties do
File.open("autodeploy.properties", "w") do |file|
file.puts("UPSTREAM_NOOP=false")
file.puts("UPSTREAM_BRANCH=#{BRANCH}")
file.puts("UPSTREAM_JOB=#{PACKAGE_NAME}")
file.puts("UPSTREAM_REVISION=#{COMMIT}")
end
end
\ No newline at end of file
rakefiles/django.rake 0 → 100644
default_options = {
:lms => '8000',
:cms => '8001',
}
task :predjango => :install_python_prereqs do
sh("find . -type f -name *.pyc -delete")
sh('pip install -q --no-index -r requirements/local.txt')
end
task :fastlms do
# this is >2 times faster that rake [lms], and does not need web, good for local dev
django_admin = ENV['DJANGO_ADMIN_PATH'] || select_executable('django-admin.py', 'django-admin')
sh("#{django_admin} runserver --traceback --settings=lms.envs.dev --pythonpath=.")
end
[:lms, :cms].each do |system|
desc <<-desc
Start the #{system} locally with the specified environment (defaults to dev).
Other useful environments are devplus (for dev testing with a real local database)
desc
task system, [:env, :options] => [:install_prereqs, 'assets:_watch', :predjango] do |t, args|
args.with_defaults(:env => 'dev', :options => default_options[system])
sh(django_admin(system, args.env, 'runserver', args.options))
end
# Per environment tasks
environments(system).each do |env|
desc "Attempt to import the settings file #{system}.envs.#{env} and report any errors"
task "#{system}:check_settings:#{env}" => :predjango do
sh("echo 'import #{system}.envs.#{env}' | #{django_admin(system, env, 'shell')}")
end
end
end
desc "Reset the relational database used by django. WARNING: this will delete all of your existing users"
task :resetdb, [:env] do |t, args|
args.with_defaults(:env => 'dev')
sh(django_admin(:lms, args.env, 'syncdb'))
sh(django_admin(:lms, args.env, 'migrate'))
end
desc "Update the relational database to the latest migration"
task :migrate, [:env] do |t, args|
args.with_defaults(:env => 'dev')
sh(django_admin(:lms, args.env, 'migrate'))
end
task :runserver => :lms
desc "Run django-admin <action> against the specified system and environment"
task "django-admin", [:action, :system, :env, :options] do |t, args|
args.with_defaults(:env => 'dev', :system => 'lms', :options => '')
sh(django_admin(args.system, args.env, args.action, args.options))
end
desc "Set the staff bit for a user"
task :set_staff, [:user, :system, :env] do |t, args|
args.with_defaults(:env => 'dev', :system => 'lms', :options => '')
sh(django_admin(args.system, args.env, 'set_staff', args.user))
end
namespace :cms do
desc "Clone existing MongoDB based course"
task :clone do
if ENV['SOURCE_LOC'] and ENV['DEST_LOC']
sh(django_admin(:cms, :dev, :clone, ENV['SOURCE_LOC'], ENV['DEST_LOC']))
else
raise "You must pass in a SOURCE_LOC and DEST_LOC parameters"
end
end
desc "Delete existing MongoDB based course"
task :delete_course do
if ENV['LOC'] and ENV['COMMIT']
sh(django_admin(:cms, :dev, :delete_course, ENV['LOC'], ENV['COMMIT']))
elsif ENV['LOC']
sh(django_admin(:cms, :dev, :delete_course, ENV['LOC']))
else
raise "You must pass in a LOC parameter"
end
end
desc "Import course data within the given DATA_DIR variable"
task :import do
if ENV['DATA_DIR'] and ENV['COURSE_DIR']
sh(django_admin(:cms, :dev, :import, ENV['DATA_DIR'], ENV['COURSE_DIR']))
elsif ENV['DATA_DIR']
sh(django_admin(:cms, :dev, :import, ENV['DATA_DIR']))
else
raise "Please specify a DATA_DIR variable that point to your data directory.\n" +
"Example: \`rake cms:import DATA_DIR=../data\`"
end
end
desc "Imports all the templates from the code pack"
task :update_templates do
sh(django_admin(:cms, :dev, :update_templates))
end
desc "Import course data within the given DATA_DIR variable"
task :xlint do
if ENV['DATA_DIR'] and ENV['COURSE_DIR']
sh(django_admin(:cms, :dev, :xlint, ENV['DATA_DIR'], ENV['COURSE_DIR']))
elsif ENV['DATA_DIR']
sh(django_admin(:cms, :dev, :xlint, ENV['DATA_DIR']))
else
raise "Please specify a DATA_DIR variable that point to your data directory.\n" +
"Example: \`rake cms:import DATA_DIR=../data\`"
end
end
desc "Export course data to a tar.gz file"
task :export do
if ENV['COURSE_ID'] and ENV['OUTPUT_PATH']
sh(django_admin(:cms, :dev, :export, ENV['COURSE_ID'], ENV['OUTPUT_PATH']))
else
raise "Please specify a COURSE_ID and OUTPUT_PATH.\n" +
"Example: \`rake cms:export COURSE_ID=MITx/12345/name OUTPUT_PATH=foo.tar.gz\`"
end
end
end
\ No newline at end of file
rakefiles/docs.rake 0 → 100644
require 'launchy'
# --- Develop and public documentation ---
desc "Invoke sphinx 'make build' to generate docs."
task :builddocs, [:options] do |t, args|
if args.options == 'pub'
path = "doc/public"
else
path = "docs"
end
Dir.chdir(path) do
sh('make html')
end
end
desc "Show docs in browser (mac and ubuntu)."
task :showdocs, [:options] do |t, args|
if args.options == 'pub'
path = "doc/public"
else
path = "docs"
end
Dir.chdir("#{path}/build/html") do
Launchy.open('index.html')
end
end
desc "Build docs and show them in browser"
task :doc, [:options] => :builddocs do |t, args|
Rake::Task["showdocs"].invoke(args.options)
end
# --- Develop and public documentation ---
rakefiles/helpers.rb 0 → 100644
require 'digest/md5'
def select_executable(*cmds)
cmds.find_all{ |cmd| system("which #{cmd} > /dev/null 2>&1") }[0] || fail("No executables found from #{cmds.join(', ')}")
end
def django_admin(system, env, command, *args)
django_admin = ENV['DJANGO_ADMIN_PATH'] || select_executable('django-admin.py', 'django-admin')
return "#{django_admin} #{command} --traceback --settings=#{system}.envs.#{env} --pythonpath=. #{args.join(' ')}"
end
def report_dir_path(dir)
return File.join(REPORT_DIR, dir.to_s)
end
def when_changed(*files)
Rake::Task[PREREQS_MD5_DIR].invoke
cache_file = File.join(PREREQS_MD5_DIR, files.join('-').gsub(/\W+/, '-')) + '.md5'
digest = Digest::MD5.new()
Dir[*files].select{|file| File.file?(file)}.each do |file|
digest.file(file)
end
if !File.exists?(cache_file) or digest.hexdigest != File.read(cache_file)
yield
File.write(cache_file, digest.hexdigest)
end
end
# Runs Process.spawn, and kills the process at the end of the rake process
# Expects the same arguments as Process.spawn
def background_process(*command)
pid = Process.spawn({}, *command, {:pgroup => true})
at_exit do
puts "Ending process and children"
pgid = Process.getpgid(pid)
begin
Timeout.timeout(5) do
puts "Interrupting process group #{pgid}"
Process.kill(:SIGINT, -pgid)
puts "Waiting on process group #{pgid}"
Process.wait(-pgid)
puts "Done waiting on process group #{pgid}"
end
rescue Timeout::Error
begin
Timeout.timeout(5) do
puts "Terminating process group #{pgid}"
Process.kill(:SIGTERM, -pgid)
puts "Waiting on process group #{pgid}"
Process.wait(-pgid)
puts "Done waiting on process group #{pgid}"
end
rescue Timeout::Error
puts "Killing process group #{pgid}"
Process.kill(:SIGKILL, -pgid)
puts "Waiting on process group #{pgid}"
Process.wait(-pgid)
puts "Done waiting on process group #{pgid}"
end
end
end
end
def environments(system)
Dir["#{system}/envs/**/*.py"].select{|file| ! (/__init__.py$/ =~ file)}.map do |env_file|
env_file.gsub("#{system}/envs/", '').gsub(/\.py/, '').gsub('/', '.')
end
end
rakefiles/i18n.rake 0 → 100644
# --- Internationalization tasks
namespace :i18n do
desc "Extract localizable strings from sources"
task :extract => "i18n:validate:gettext" do
sh(File.join(REPO_ROOT, "i18n", "extract.py"))
end
desc "Compile localizable strings from sources. With optional flag 'extract', will extract strings first."
task :generate => "i18n:validate:gettext" do
if ARGV.last.downcase == 'extract'
Rake::Task["i18n:extract"].execute
end
sh(File.join(REPO_ROOT, "i18n", "generate.py"))
end
desc "Simulate international translation by generating dummy strings corresponding to source strings."
task :dummy do
source_files = Dir["#{REPO_ROOT}/conf/locale/en/LC_MESSAGES/*.po"]
dummy_locale = 'fr'
cmd = File.join(REPO_ROOT, "i18n", "make_dummy.py")
for file in source_files do
sh("#{cmd} #{file} #{dummy_locale}")
end
end
namespace :validate do
desc "Make sure GNU gettext utilities are available"
task :gettext do
begin
select_executable('xgettext')
rescue
msg = "Cannot locate GNU gettext utilities, which are required by django for internationalization.\n"
msg += "(see https://docs.djangoproject.com/en/dev/topics/i18n/translation/#message-files)\n"
msg += "Try downloading them from http://www.gnu.org/software/gettext/"
abort(msg.red)
end
end
desc "Make sure config file with username/password exists"
task :transifex_config do
config_file = "#{Dir.home}/.transifexrc"
if !File.file?(config_file) or File.size(config_file)==0
msg ="Cannot connect to Transifex, config file is missing or empty: #{config_file}\n"
msg += "See http://help.transifex.com/features/client/#transifexrc"
abort(msg.red)
end
end
end
namespace :transifex do
desc "Push source strings to Transifex for translation"
task :push => "i18n:validate:transifex_config" do
cmd = File.join(REPO_ROOT, "i18n", "transifex.py")
sh("#{cmd} push")
end
desc "Pull translated strings from Transifex"
task :pull => "i18n:validate:transifex_config" do
cmd = File.join(REPO_ROOT, "i18n", "transifex.py")
sh("#{cmd} pull")
end
end
desc "Run tests for the internationalization library"
task :test => "i18n:validate:gettext" do
test = File.join(REPO_ROOT, "i18n", "tests")
sh("nosetests #{test}")
end
end
rakefiles/jasmine.rake 0 → 100644
require 'colorize'
require 'erb'
require 'launchy'
require 'net/http'
def django_for_jasmine(system, django_reload)
if !django_reload
reload_arg = '--noreload'
end
port = 10000 + rand(40000)
jasmine_url = "http://localhost:#{port}/_jasmine/"
background_process(*django_admin(system, 'jasmine', 'runserver', '-v', '0', port.to_s, reload_arg).split(' '))
up = false
start_time = Time.now
until up do
if Time.now - start_time > 30
abort "Timed out waiting for server to start to run jasmine tests"
end
begin
response = Net::HTTP.get_response(URI(jasmine_url))
puts response.code
up = response.code == '200'
rescue => e
puts e.message
ensure
puts('Waiting server to start')
sleep(0.5)
end
end
yield jasmine_url
end
def template_jasmine_runner(lib)
coffee_files = Dir["#{lib}/**/js/**/*.coffee", "common/static/coffee/src/**/*.coffee"]
if !coffee_files.empty?
sh("node_modules/.bin/coffee -c #{coffee_files.join(' ')}")
end
phantom_jasmine_path = File.expand_path("node_modules/phantom-jasmine")
common_js_root = File.expand_path("common/static/js")
common_coffee_root = File.expand_path("common/static/coffee/src")
# Get arrays of spec and source files, ordered by how deep they are nested below the library
# (and then alphabetically) and expanded from a relative to an absolute path
spec_glob = File.join("#{lib}", "**", "spec", "**", "*.js")
src_glob = File.join("#{lib}", "**", "src", "**", "*.js")
js_specs = Dir[spec_glob].sort_by {|p| [p.split('/').length, p]} .map {|f| File.expand_path(f)}
js_source = Dir[src_glob].sort_by {|p| [p.split('/').length, p]} .map {|f| File.expand_path(f)}
template = ERB.new(File.read("#{lib}/jasmine_test_runner.html.erb"))
template_output = "#{lib}/jasmine_test_runner.html"
File.open(template_output, 'w') do |f|
f.write(template.result(binding))
end
yield File.expand_path(template_output)
end
[:lms, :cms].each do |system|
desc "Open jasmine tests for #{system} in your default browser"
task "browse_jasmine_#{system}" => :assets do
django_for_jasmine(system, true) do |jasmine_url|
Launchy.open(jasmine_url)
puts "Press ENTER to terminate".red
$stdin.gets
end
end
desc "Use phantomjs to run jasmine tests for #{system} from the console"
task "phantomjs_jasmine_#{system}" => :assets do
phantomjs = ENV['PHANTOMJS_PATH'] || 'phantomjs'
django_for_jasmine(system, false) do |jasmine_url|
sh("#{phantomjs} node_modules/phantom-jasmine/lib/run_jasmine_test.coffee #{jasmine_url}")
end
end
end
Dir["common/lib/*"].select{|lib| File.directory?(lib)}.each do |lib|
desc "Open jasmine tests for #{lib} in your default browser"
task "browse_jasmine_#{lib}" do
template_jasmine_runner(lib) do |f|
sh("python -m webbrowser -t 'file://#{f}'")
puts "Press ENTER to terminate".red
$stdin.gets
end
end
desc "Use phantomjs to run jasmine tests for #{lib} from the console"
task "phantomjs_jasmine_#{lib}" do
phantomjs = ENV['PHANTOMJS_PATH'] || 'phantomjs'
template_jasmine_runner(lib) do |f|
sh("#{phantomjs} node_modules/phantom-jasmine/lib/run_jasmine_test.coffee #{f}")
end
end
end
rakefiles/prereqs.rake 0 → 100644
require './rakefiles/helpers.rb'
PREREQS_MD5_DIR = ENV["PREREQ_CACHE_DIR"] || File.join(REPO_ROOT, '.prereqs_cache')
CLOBBER.include(PREREQS_MD5_DIR)
directory PREREQS_MD5_DIR
desc "Install all prerequisites needed for the lms and cms"
task :install_prereqs => [:install_node_prereqs, :install_ruby_prereqs, :install_python_prereqs]
desc "Install all node prerequisites for the lms and cms"
task :install_node_prereqs => "ws:migrate" do
when_changed('package.json') do
sh('npm install')
end unless ENV['NO_PREREQ_INSTALL']
end
desc "Install all ruby prerequisites for the lms and cms"
task :install_ruby_prereqs => "ws:migrate" do
when_changed('Gemfile') do
sh('bundle install')
end unless ENV['NO_PREREQ_INSTALL']
end
desc "Install all python prerequisites for the lms and cms"
task :install_python_prereqs => "ws:migrate" do
when_changed('requirements/**') do
ENV['PIP_DOWNLOAD_CACHE'] ||= '.pip_download_cache'
sh('pip install --exists-action w -r requirements/base.txt')
sh('pip install --exists-action w -r requirements/post.txt')
# Check for private-requirements.txt: used to install our libs as working dirs,
# or personal-use tools.
if File.file?("requirements/private.txt")
sh('pip install -r requirements/private.txt')
end
end unless ENV['NO_PREREQ_INSTALL']
end
\ No newline at end of file
rakefiles/quality.rake 0 → 100644
[:lms, :cms, :common].each do |system|
report_dir = report_dir_path(system)
directory report_dir
desc "Run pep8 on all #{system} code"
task "pep8_#{system}" => [report_dir, :install_python_prereqs] do
sh("pep8 #{system} | tee #{report_dir}/pep8.report")
end
task :pep8 => "pep8_#{system}"
desc "Run pylint on all #{system} code"
task "pylint_#{system}" => [report_dir, :install_python_prereqs] do
apps = Dir["#{system}/*.py", "#{system}/djangoapps/*", "#{system}/lib/*"].map do |app|
File.basename(app)
end.select do |app|
app !=~ /.pyc$/
end.map do |app|
if app =~ /.py$/
app.gsub('.py', '')
else
app
end
end
pythonpath_prefix = "PYTHONPATH=#{system}:#{system}/djangoapps:#{system}/lib:common/djangoapps:common/lib"
sh("#{pythonpath_prefix} pylint --rcfile=.pylintrc -f parseable #{apps.join(' ')} | tee #{report_dir}/pylint.report")
end
task :pylint => "pylint_#{system}"
end
\ No newline at end of file
rakefiles/tests.rake 0 → 100644
# Set up the clean and clobber tasks
CLOBBER.include(REPORT_DIR, 'test_root/*_repo', 'test_root/staticfiles')
$failed_tests = 0
def run_under_coverage(cmd, root)
cmd0, cmd_rest = cmd.split(" ", 2)
# We use "python -m coverage" so that the proper python will run the importable coverage
# rather than the coverage that OS path finds.
cmd = "python -m coverage run --rcfile=#{root}/.coveragerc `which #{cmd0}` #{cmd_rest}"
return cmd
end
def run_tests(system, report_dir, stop_on_failure=true)
ENV['NOSE_XUNIT_FILE'] = File.join(report_dir, "nosetests.xml")
dirs = Dir["common/djangoapps/*"] + Dir["#{system}/djangoapps/*"]
cmd = django_admin(system, :test, 'test', '--logging-clear-handlers', *dirs.each)
sh(run_under_coverage(cmd, system)) do |ok, res|
if !ok and stop_on_failure
abort "Test failed!"
end
$failed_tests += 1 unless ok
end
end
def run_acceptance_tests(system, report_dir, harvest_args)
sh(django_admin(system, 'acceptance', 'syncdb', '--noinput'))
sh(django_admin(system, 'acceptance', 'migrate', '--noinput'))
sh(django_admin(system, 'acceptance', 'harvest', '--debug-mode', '--tag -skip', harvest_args))
end
directory REPORT_DIR
task :clean_test_files do
sh("git clean -fqdx test_root")
end
TEST_TASK_DIRS = []
[:lms, :cms].each do |system|
report_dir = report_dir_path(system)
# Per System tasks
desc "Run all django tests on our djangoapps for the #{system}"
task "test_#{system}", [:stop_on_failure] => ["clean_test_files", :predjango, "#{system}:gather_assets:test", "fasttest_#{system}"]
# Have a way to run the tests without running collectstatic -- useful when debugging without
# messing with static files.
task "fasttest_#{system}", [:stop_on_failure] => [report_dir, :install_prereqs, :predjango] do |t, args|
args.with_defaults(:stop_on_failure => 'true')
run_tests(system, report_dir, args.stop_on_failure)
end
# Run acceptance tests
desc "Run acceptance tests"
task "test_acceptance_#{system}", [:harvest_args] => ["#{system}:gather_assets:acceptance", "fasttest_acceptance_#{system}"]
desc "Run acceptance tests without collectstatic"
task "fasttest_acceptance_#{system}", [:harvest_args] => ["clean_test_files", :predjango, report_dir] do |t, args|
args.with_defaults(:harvest_args => '')
run_acceptance_tests(system, report_dir, args.harvest_args)
end
task :fasttest => "fasttest_#{system}"
TEST_TASK_DIRS << system
end
Dir["common/lib/*"].select{|lib| File.directory?(lib)}.each do |lib|
task_name = "test_#{lib}"
report_dir = report_dir_path(lib)
desc "Run tests for common lib #{lib}"
task task_name => report_dir do
ENV['NOSE_XUNIT_FILE'] = File.join(report_dir, "nosetests.xml")
cmd = "nosetests #{lib}"
sh(run_under_coverage(cmd, lib)) do |ok, res|
$failed_tests += 1 unless ok
end
end
TEST_TASK_DIRS << lib
desc "Run tests for common lib #{lib} (without coverage)"
task "fasttest_#{lib}" do
sh("nosetests #{lib}")
end
end
task :report_dirs
TEST_TASK_DIRS.each do |dir|
report_dir = report_dir_path(dir)
directory report_dir
task :report_dirs => [REPORT_DIR, report_dir]
end
task :test do
TEST_TASK_DIRS.each do |dir|
Rake::Task["test_#{dir}"].invoke(false)
end
if $failed_tests > 0
abort "Tests failed!"
end
end
namespace :coverage do
desc "Build the html coverage reports"
task :html => :report_dirs do
TEST_TASK_DIRS.each do |dir|
report_dir = report_dir_path(dir)
if !File.file?("#{report_dir}/.coverage")
next
end
sh("coverage html --rcfile=#{dir}/.coveragerc")
end
end
desc "Build the xml coverage reports"
task :xml => :report_dirs do
TEST_TASK_DIRS.each do |dir|
report_dir = report_dir_path(dir)
if !File.file?("#{report_dir}/.coverage")
next
end
# Why doesn't the rcfile control the xml output file properly??
sh("coverage xml -o #{report_dir}/coverage.xml --rcfile=#{dir}/.coveragerc")
end
end
end
rakefiles/workspace.rake 0 → 100644
MIGRATION_MARKER_DIR = File.join(REPO_ROOT, '.ws_migrations_complete')
SKIP_MIGRATIONS = ENV['SKIP_WS_MIGRATIONS'] || false
directory MIGRATION_MARKER_DIR
namespace :ws do
task :migrate => MIGRATION_MARKER_DIR do
Dir['ws_migrations/*'].select{|m| File.executable?(m)}.each do |migration|
completion_file = File.join(MIGRATION_MARKER_DIR, File.basename(migration))
if ! File.exist?(completion_file)
sh(migration)
File.write(completion_file, "")
end
end unless SKIP_MIGRATIONS
end
end
\ No newline at end of file
repo-requirements.txt deleted 100644 → 0
-r github-requirements.txt
-r local-requirements.txt
\ No newline at end of file
requirements.txt → requirements/base.txt
-r repo-requirements.txt
-r repo.txt
beautifulsoup4==4.1.3
beautifulsoup==3.2.1
boto==2.6.0
distribute==0.6.28
django-celery==3.0.11
django-countries==1.5
django-followit==0.0.3
......@@ -21,11 +23,9 @@ feedparser==5.1.3
fs==0.4.0
GitPython==0.3.2.RC1
glob2==0.3
http://sympy.googlecode.com/files/sympy-0.7.1.tar.gz
lxml==3.0.1
mako==0.7.3
Markdown==2.2.1
MySQL-python==1.2.4c1
networkx==1.7
nltk==2.0.4
numpy==1.6.2
......@@ -42,10 +42,10 @@ python-openid==2.2.5
pytz==2012h
PyYAML==3.10
requests==0.14.2
scipy==0.11.0
Shapely==1.2.16
sorl-thumbnail==11.12
South==0.7.6
sympy==0.7.1
xmltodict==0.4.1
# Used for debugging
......
requirements/post.txt 0 → 100644
# This must be installed after distribute 0.6.28
MySQL-python==1.2.4c1
# This must be installed after numpy
scipy==0.11.0
requirements/repo.txt 0 → 100644
-r github.txt
-r local.txt
ws_migrations/README.rst 0 → 100644
Developer Workspace Migrations
==============================
This directory contains executable files which run once prior to
installation of pre-requisites to bring a developers workspace
into line.
Specifications
--------------
Each file in this directory should meet the following criteria
* Executable (`chmod +x ws_migrations/foo.sh`)
* Idempotent (ideally, each script is run only once, but no
guarantees are made by the caller, so the script must do
the right thing)
* Either fast or verbose (if the script is going to take
a long time, it should notify the user of that)
* A comment at the top of the file explaining the migration
Execution
---------
The scripts are run by the rake task `ws:migrate`. That task
only runs a given script if a corresponding marker file
in .completed-ws-migrations doesn't already exist.
If the SKIP_WS_MIGRATIONS environment variable is set, then
no workspace migrations will be run.
\ No newline at end of file
ws_migrations/clean_xmodule_assets.sh 0 → 100755
#! /bin/sh
# Remove all of the old xmodule coffee and sass directories
# in preparation to switching to use the xmodule_assets script
rm -rf cms/static/coffee/descriptor
rm -rf cms/static/coffee/module
rm -rf cms/static/sass/descriptor
rm -rf cms/static/sass/module
rm -rf lms/static/coffee/module
rm -rf lms/static/sass/module
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment