Merge remote-tracking branch 'origin/master' into feature/vik/oe-ui

AUTHORS

@@ -82,3 +82,5 @@ Adam Palay <adam@edx.org>
 Ian Hoover <ihoover@edx.org>
 Mukul Goyal <miki@edx.org>
 Robert Marks <rmarks@edx.org>
+Yarko Tymciurak <yarkot1@gmail.com>
+

CHANGELOG.rst

@@ -5,6 +5,16 @@ These are notable changes in edx-platform.  This is a rolling list of changes,
 in roughly chronological order, most recent first.  Add your entries at or near
 the top.  Include a label indicating the component affected.
 
+LMS: Added user preferences (arbitrary user/key/value tuples, for which
+which user/key is unique) and a REST API for reading users and
+preferences. Access to the REST API is restricted by use of the
+X-Edx-Api-Key HTTP header (which must match settings.EDX_API_KEY; if
+the setting is not present, the API is disabled).
+
+LMS: Added endpoints for AJAX requests to enable/disable notifications
+(which are not yet implemented) and a one-click unsubscribe page.
+
+Common: Add a manage.py that knows about edx-platform specific settings and projects
 
 Common: Added *experimental* support for jsinput type.
 

README.md

@@ -12,30 +12,35 @@ installation process.
 
 1. Make sure you have plenty of available disk space, >5GB
 2. Install Git: http://git-scm.com/downloads
-3. Install VirtualBox: https://www.virtualbox.org/wiki/Download_Old_Builds_4_2
-   (you need version 4.2.12, as later/earlier versions might not work well with 
-   Vagrant)
+3. Install VirtualBox: https://www.virtualbox.org/wiki/Downloads
+   See http://docs.vagrantup.com/v2/providers/index.html for a list of supported
+   Providers.  You should use VirtualBox >= 4.2.12.
+   (Windows: later/earlier VirtualBox versions than 4.2.12 have been reported to not work well with
+   Vagrant.  If this is still a problem, you can
+   install 4.2.12 from https://www.virtualbox.org/wiki/Download_Old_Builds_4_2).
 4. Install Vagrant: http://www.vagrantup.com/ (Vagrant 1.2.2 or later)
 5. Open a terminal
 6. Download the project: `git clone git://github.com/edx/edx-platform.git`
 7. Enter the project directory: `cd edx-platform/`
-8. (Windows only) Run the commands to 
+8. (Windows only) Run the commands to
    [deal with line endings and symlinks under Windows](https://github.com/edx/edx-platform/wiki/Simplified-install-with-vagrant#dealing-with-line-endings-and-symlinks-under-windows)
-9. Start: `vagrant up`
+9. Create the development environment and start it: `vagrant up`
 
-The last step might require your host machine's administrator password to setup NFS.
+The initial `vagrant up` will download a Linux image, then boot and ask for your
+host machine's administrator password to setup file sharing between your computer and the VM.
+Once file sharing is established, `edx-platform/scripts/create-dev-env.sh` will
+install dependencies and configure the VM.
+This will take a while; go grab a coffee.
 
-Afterwards, it will download an image, install all the dependencies and configure
-the VM. It will take a while, go grab a coffee.
+When complete, you should see a _"Success!"_ message.
+If not, refer to the
+[troubleshooting section](https://github.com/edx/edx-platform/wiki/Simplified-install-with-vagrant#troubleshooting).
 
-Once completed, hopefully you should see a "Success!" message indicating that the
-installation went fine. (If not, refer to the 
-[troubleshooting section](https://github.com/edx/edx-platform/wiki/Simplified-install-with-vagrant#troubleshooting).)
+Your development environment is initialized only on the first bring-up.
+Subsequently `vagrant up` commands will boot your virtual machine normally.
 
-Note: by default, the VM will get the IP `192.168.20.40`. If you need to use a 
-different IP, you can edit the file `Vagrantfile`. If you have already started the 
-VM with `vagrant up`, see "Stopping and restarting the VM" below to take the change 
-into account.
+Note: by default, the VM will get the IP `192.168.20.40`.
+You can change this in your `Vagrantfile` (the startup message will reflect your VM's actual IP).
 
 Accessing the VM
 ----------------
@@ -46,15 +51,24 @@ Once the installation is finished, to log into the virtual machine:
 $ vagrant ssh
 ```
 
-Note: This won't work from Windows, install install PuTTY from
-http://www.chiark.greenend.org.uk/%7Esgtatham/putty/download.html instead. Then
-connect to 127.0.0.1, port 2222, using vagrant/vagrant as a user/password.
+Note: This won't work from Windows. Instead, install PuTTY from
+http://www.chiark.greenend.org.uk/%7Esgtatham/putty/download.html. Then
+connect to 192.168.20.40, port 2222, using vagrant/vagrant as a user/password.
+
 
 Using edX
 ---------
 
-Once inside the VM, you can start Studio and LMS with the following commands
-(from the `/opt/edx/edx-platform` folder):
+When you login to your VM, you are in
+`/opt/edx/edx-platform` by default, which is shared from your host workspace.
+Your host computer contains the edx-project development code and repository.
+Your VM runs edx-platform code mounted from your host, so
+you can develop by editing on your host.
+
+After logging into your VM with `vagrant ssh`,
+start the _Studio_ and
+_Learning management system (LMS)_
+servers (run these from `/opt/edx/edx-platform`):
 
 Learning management system (LMS):
 
@@ -62,46 +76,85 @@ Learning management system (LMS):
 $ rake lms[cms.dev,0.0.0.0:8000]
 ```
 
-Studio:
+Studio (CMS):
 
 ```
 $ rake cms[dev,0.0.0.0:8001]
 ```
 
-Once started, open the following URLs in your browser:
+The servers will come up to these URLs:
 
-* Learning management system (LMS): http://192.168.20.40:8000/
-* Studio (CMS): http://192.168.20.40:8001/
+- LMS: http://192.168.20.40:8000/
+- CMS: http://192.168.20.40:8001/
 
-You can develop by editing the files directly in the `edx-platform/` directory you
-downloaded before, you don't need to connect to the VM to edit them (the VM uses
-those files to run edX, mirroring the folder in `/opt/edx/edx-platform`).
+Your VM's port 8000 is forwarded to host port 9000
+so you can also access the LMS with [http://localhost:9000/]().
+Similarly, VM port 8001 is forwarded to host port 9001.
+These are set in your `Vagrantfile`.
 
-You may also want to create a super-user with:
-
-```
-$ rake django-admin["createsuperuser"]
-```
-
-Also note that if you register a new user through the web interface,
-the activiation email will be posted to your VM's terminal window (search for
-lines similar to):
+Note that when you register a new user through the web interface,
+by default the activiation email will be appear on your VM's terminal.
+Search for lines similar to:
 
 ```
 Subject: Your account for edX Studio
 From: registration@edx.org
 ```
 
-and find the activation URL for the account you've created.
+and find the activation URL.
 
 See the [Frequently Asked Questions](https://github.com/edx/edx-platform/wiki/Frequently-Asked-Questions)
 for more usage tips.
 
+Django admin & debug toolbar
+-----------------------------
+
+You can enable admin logins and the debug_toolbar by editing
+`lms/envs/common.py`:
+
+- enable ADMIN login page by setting:
+  - ```
+     'ENABLE_DJANGO_ADMIN_SITE': True
+```
+
+
+- enable debug toolbar by uncommenting:
+  - ```
+    # 'debug_toolbar.middleware.DebugToolbarMiddleware',
+```
+
+These are also defined in `lms/envs/dev.py`,
+and usually active on localhost.
+
+To get at your VM's 127.0.0.1, explicitly forward one of VM's available localhost ports to your computer.
+Instead of `vagrant ssh`, login with:
+
+```
+$ ssh -L 6080:127.0.0.1:8080 vagrant@192.168.20.40
+```
+
+The password is _vagrant_.
+
+From your VM, start the LMS as a localhost instance:
+
+```
+$ rake lms[cms.dev,127.0.0.1:8080]
+```
+
+You should see the debug toolbar now on [http:/localhost:6080/]().
+You should now also see a login on [http://localhost:6080/admin/]()
+You will need a privileged user for the admin login.
+You can create a CMS/LMS super-user with:
+```
+$ ./manage.py lms createsuperuser
+```
+
+
 Stopping & starting
 -------------------
 
-To stop the VM (from your `edx-platform/` directory):
 
+To stop the VM (from your `edx-platform/` directory):
 ```
 $ vagrant halt
 ```
@@ -112,16 +165,27 @@ To restart:
 $ vagrant up
 ```
 
-or, to start without attempting to update the dependencies:
+To suspend and resume tasks in progress on your VM:
+```
+$ vagrant suspend
+$ # and later...
+$ vagrant resume
+```
 
+Your development environment is normally created once, on first `vagrant up`.
+You can continue to fetch changes in edx-platform
+as you work with your VM.
+To re-create your VM and create a fresh development environment:
 ```
-$ vagrant up --no-provision
+$ vagrant destroy
+$ vagrant up  # will make a new VM
 ```
 
+
 Troubleshooting
 ---------------
 
-If anything doesn't work as expected, see the 
+If anything doesn't work as expected, see the
 [troubleshooting section](https://github.com/edx/edx-platform/wiki/Simplified-install-with-vagrant#troubleshooting).
 
 Installation - Advanced
@@ -229,23 +293,12 @@ or any other process management tool.
 
 Configuring Your Project
 ------------------------
-We use [`rake`](http://rake.rubyforge.org/) to execute common tasks in our
-project. The `rake` tasks are defined in the `rakefile`, or you can run `rake -T`
-to view a summary.
-
 Before you run your project, you need to create a sqlite database, create
-tables in that database, run database migrations, and populate templates for
-CMS templates. Fortunately, `rake` will do all of this for you! Just run:
+tables in that database, and run database migrations. Fortunately, `django`
+will do all of this for you
 
-    $ rake django-admin[syncdb]
-    $ rake django-admin[migrate]
-
-If you are running these commands using the [`zsh`](http://www.zsh.org/) shell,
-zsh will assume that you are doing
-[shell globbing](https://en.wikipedia.org/wiki/Glob_%28programming%29), search for
-a file in your directory named `django-adminsyncdb` or `django-adminmigrate`,
-and fail. To fix this, just surround the argument with quotation marks, so that
-you're running `rake "django-admin[syncdb]"`.
+    $ ./manage.py lms syncdb --migrate
+    $ ./manage.py cms syncdb --migrate
 
 Run Your Project
 ----------------
@@ -253,6 +306,10 @@ edX has two components: Studio, the course authoring system; and the LMS
 (learning management system) used by students. These two systems communicate
 through the MongoDB database, which stores course information.
 
+We use [`rake`](http://rake.rubyforge.org/) to execute common tasks in our
+project. The `rake` tasks are defined in the `rakefile`, or you can run `rake -T`
+to view a summary.
+
 To run Studio, run:
 
     $ rake cms

cms/djangoapps/contentstore/features/checklists.feature

@@ -8,7 +8,7 @@ Feature: Course checklists
   Scenario: A course author can mark tasks as complete
     Given I have opened Checklists
     Then I can check and uncheck tasks in a checklist
-    And They are correctly selected after I reload the page
+    And They are correctly selected after reloading the page
 
   Scenario: A task can link to a location within Studio
     Given I have opened Checklists

cms/djangoapps/contentstore/features/checklists.py

@@ -45,7 +45,7 @@ def i_can_check_and_uncheck_tasks(step):
     verifyChecklist2Status(2, 7, 29)
 
 
-@step('They are correctly selected after I reload the page$')
+@step('They are correctly selected after reloading the page$')
 def tasks_correctly_selected_after_reload(step):
     reload_the_page(step)
     verifyChecklist2Status(2, 7, 29)

cms/djangoapps/contentstore/features/common.py

@@ -209,7 +209,8 @@ def i_created_a_video_component(step):
     world.create_component_instance(
         step, '.large-video-icon',
         'video',
-        '.xmodule_VideoModule'
+        '.xmodule_VideoModule',
+        has_multiple_templates=False
     )
 
 
@@ -238,6 +239,17 @@ def save_button_disabled(step):
     assert world.css_has_class(button_css, disabled)
 
 
+@step('I confirm the prompt')
+def confirm_the_prompt(step):
+    prompt_css = 'a.button.action-primary'
+    world.css_click(prompt_css)
+
+
+@step(u'I am shown a (.*)$')
+def i_am_shown_a_notification(step, notification_type):
+    assert world.is_css_present('.wrapper-%s' % notification_type)
+
+
 def type_in_codemirror(index, text):
     world.css_click(".CodeMirror", index=index)
     g = world.css_find("div.CodeMirror.CodeMirror-focused > div > textarea")

cms/djangoapps/contentstore/features/component.feature

@@ -67,3 +67,21 @@ Feature: Component Adding
         When I will confirm all alerts
         And I delete all components
         Then I see no components
+
+    Scenario: I see a prompt on delete
+        Given I have opened a new course in studio
+        And I am editing a new unit
+        And I add the following components:
+            | Component    |
+            | Discussion   |
+       And I delete a component
+       Then I am shown a prompt
+
+   Scenario: I see a notification on save
+        Given I have opened a new course in studio
+        And I am editing a new unit
+        And I add the following components:
+            | Component    |
+            | Discussion   |
+        And I edit and save a component
+        Then I am shown a notification

cms/djangoapps/contentstore/features/component.py

@@ -41,6 +41,17 @@ def see_no_components(steps):
     assert world.is_css_not_present('li.component')
 
 
+@step(u'I delete a component')
+def delete_one_component(step):
+    world.css_click('a.delete-button')
+
+
+@step(u'I edit and save a component')
+def edit_and_save_component(step):
+    world.css_click('.edit-button')
+    world.css_click('.save-button')
+
+
 def step_selector_list(data_type, path, index=1):
     selector_list = ['a[data-type="{}"]'.format(data_type)]
     if index != 1:

cms/djangoapps/contentstore/features/component_settings_editor_helpers.py

@@ -7,10 +7,16 @@ from terrain.steps import reload_the_page
 
 
 @world.absorb
-def create_component_instance(step, component_button_css, category, expected_css, boilerplate=None):
+def create_component_instance(step, component_button_css, category,
+                              expected_css, boilerplate=None,
+                              has_multiple_templates=True):
+
     click_new_component_button(step, component_button_css)
-    click_component_from_menu(category, boilerplate, expected_css)
 
+    if has_multiple_templates:
+        click_component_from_menu(category, boilerplate, expected_css)
+
+    assert_equal(1, len(world.css_find(expected_css)))
 
 @world.absorb
 def click_new_component_button(step, component_button_css):
@@ -34,7 +40,6 @@ def click_component_from_menu(category, boilerplate, expected_css):
     elements = world.css_find(elem_css)
     assert_equal(len(elements), 1)
     world.css_click(elem_css)
-    assert_equal(1, len(world.css_find(expected_css)))
 
 
 @world.absorb

cms/djangoapps/contentstore/features/studio-overview-togglesection.feature → cms/djangoapps/contentstore/features/course-overview.feature

-Feature: Overview Toggle Section
-    In order to quickly view the details of a course's section or to scan the inventory of sections
+Feature: Course Overview
+    In order to quickly view the details of a course's section and set release dates and grading
     As a course author
-    I want to toggle the visibility of each section's  subsection details in the overview listing
+    I want to use the course overview page
 
     Scenario: The default layout for the overview page is to show sections in expanded view
         Given I have a course with multiple sections
@@ -57,3 +57,9 @@ Feature: Overview Toggle Section
         And I click the "Expand All Sections" link
         Then I see the "Collapse All Sections" link
         And all sections are expanded
+
+   Scenario: Notification is shown on grading status changes
+        Given I have a course with 1 section
+        When I navigate to the course overview page
+        And I change an assignment's grading status
+        Then I am shown a notification

cms/djangoapps/contentstore/features/studio-overview-togglesection.py → cms/djangoapps/contentstore/features/course-overview.py

@@ -118,3 +118,9 @@ def all_sections_are_collapsed(step):
     subsections = world.css_find(subsection_locator)
     for index in range(len(subsections)):
         assert_false(world.css_visible(subsection_locator, index=index))
+
+
+@step(u"I change an assignment's grading status")
+def change_grading_status(step):
+    world.css_find('a.menu-toggle').click()
+    world.css_find('.menu li').first.click()

cms/djangoapps/contentstore/features/discussion-editor.py

@@ -9,7 +9,8 @@ def i_created_discussion_tag(step):
     world.create_component_instance(
         step, '.large-discussion-icon',
         'discussion',
-        '.xmodule_DiscussionModule'
+        '.xmodule_DiscussionModule',
+        has_multiple_templates=False
     )
 
 

cms/djangoapps/contentstore/features/html-editor.py

@@ -14,4 +14,4 @@ def i_created_blank_html_page(step):
 
 @step('I see only the HTML display name setting$')
 def i_see_only_the_html_display_name(step):
-    world.verify_all_setting_entries([['Display Name', "Blank HTML Page", False]])
+    world.verify_all_setting_entries([['Display Name', "Text", False]])

cms/djangoapps/contentstore/features/problem-editor.py

@@ -170,7 +170,8 @@ def edit_latex_source(step):
 @step('my change to the High Level Source is persisted')
 def high_level_source_persisted(step):
     def verify_text(driver):
-        return world.css_text('.problem') == 'hi'
+        css_sel = '.problem div>span'
+        return world.css_text(css_sel) == 'hi'
 
     world.wait_for(verify_text)
 

cms/djangoapps/contentstore/features/section.feature

@@ -33,4 +33,5 @@ Feature: Create Section
     And I have added a new section
     When I will confirm all alerts
     And I press the "section" delete icon
+    And I confirm the prompt
     Then the section does not exist

cms/djangoapps/contentstore/features/subsection.feature

@@ -38,4 +38,5 @@ Feature: Create Subsection
     And I see my subsection on the Courseware page
     When I will confirm all alerts
     And I press the "subsection" delete icon
+    And I confirm the prompt
     Then the subsection does not exist

cms/djangoapps/contentstore/features/video-editor.py

@@ -7,7 +7,7 @@ from lettuce import world, step
 @step('I see the correct settings and default values$')
 def i_see_the_correct_settings_and_values(step):
     world.verify_all_setting_entries([['Default Speed', 'OEoXaMPEzfM', False],
-                                      ['Display Name', 'Video Title', False],
+                                      ['Display Name', 'Video', False],
                                       ['Download Track', '', False],
                                       ['Download Video', '', False],
                                       ['Show Captions', 'True', False],

cms/djangoapps/contentstore/tests/test_contentstore.py

@@ -312,6 +312,23 @@ class ContentStoreToyCourseTest(ModuleStoreTestCase):
 
         self.assertGreater(len(course.textbooks), 0)
 
+    def test_default_tabs_on_create_course(self):
+        module_store = modulestore('direct')
+        CourseFactory.create(org='edX', course='999', display_name='Robot Super Course')
+        course_location = Location(['i4x', 'edX', '999', 'course', 'Robot_Super_Course', None])
+
+        course = module_store.get_item(course_location)
+
+        expected_tabs = []
+        expected_tabs.append({u'type': u'courseware'})
+        expected_tabs.append({u'type': u'course_info', u'name': u'Course Info'})
+        expected_tabs.append({u'type': u'textbooks'})
+        expected_tabs.append({u'type': u'discussion', u'name': u'Discussion'})
+        expected_tabs.append({u'type': u'wiki', u'name': u'Wiki'})
+        expected_tabs.append({u'type': u'progress', u'name': u'Progress'})
+
+        self.assertEqual(course.tabs, expected_tabs)
+
     def test_static_tab_reordering(self):
         module_store = modulestore('direct')
         CourseFactory.create(org='edX', course='999', display_name='Robot Super Course')

cms/djangoapps/contentstore/tests/test_course_updates.py

@@ -36,8 +36,11 @@ class CourseUpdateTest(CourseTestCase):
                                            'provided_id': payload['id']})
         content += '<div>div <p>p<br/></p></div>'
         payload['content'] = content
+        # POST requests were coming in w/ these header values causing an error; so, repro error here
         resp = self.client.post(first_update_url, json.dumps(payload),
-                                "application/json")
+                                "application/json",
+                                HTTP_X_HTTP_METHOD_OVERRIDE="PUT",
+                                REQUEST_METHOD="POST")
 
         self.assertHTMLEqual(content, json.loads(resp.content)['content'],
                              "iframe w/ div")

cms/djangoapps/contentstore/views/assets.py

@@ -13,7 +13,7 @@ from django_future.csrf import ensure_csrf_cookie
 from django.core.urlresolvers import reverse
 from django.core.servers.basehttp import FileWrapper
 from django.core.files.temp import NamedTemporaryFile
-from django.views.decorators.http import require_POST
+from django.views.decorators.http import require_POST, require_http_methods
 
 from mitxmako.shortcuts import render_to_response
 from cache_toolbox.core import del_cached_content
@@ -249,6 +249,7 @@ def remove_asset(request, org, course, name):
 
 
 @ensure_csrf_cookie
+@require_http_methods(("GET", "POST", "PUT"))
 @login_required
 def import_course(request, org, course, name):
     """
@@ -256,7 +257,7 @@ def import_course(request, org, course, name):
     """
     location = get_location_and_verify_access(request, org, course, name)
 
-    if request.method == 'POST':
+    if request.method in ('POST', 'PUT'):
         filename = request.FILES['course-data'].name
 
         if not filename.endswith('.tar.gz'):

cms/djangoapps/contentstore/views/checklist.py

@@ -37,6 +37,7 @@ def get_checklists(request, org, course, name):
 
     checklists, modified = expand_checklist_action_urls(course_module)
     if copied or modified:
+        course_module.save()
         modulestore.update_metadata(location, own_metadata(course_module))
     return render_to_response('checklists.html',
                               {
@@ -69,6 +70,7 @@ def update_checklist(request, org, course, name, checklist_index=None):
             # seeming noop which triggers kvs to record that the metadata is not default
             course_module.checklists = course_module.checklists
             checklists, _ = expand_checklist_action_urls(course_module)
+            course_module.save()
             modulestore.update_metadata(location, own_metadata(course_module))
             return JsonResponse(checklists[index])
         else:
@@ -79,6 +81,7 @@ def update_checklist(request, org, course, name, checklist_index=None):
         # In the JavaScript view initialize method, we do a fetch to get all the checklists.
         checklists, modified = expand_checklist_action_urls(course_module)
         if modified:
+            course_module.save()
             modulestore.update_metadata(location, own_metadata(course_module))
         return JsonResponse(checklists)
 

cms/djangoapps/contentstore/views/component.py

@@ -245,6 +245,7 @@ def edit_unit(request, location):
 
 @expect_json
 @login_required
+@require_http_methods(("GET", "POST", "PUT"))
 @ensure_csrf_cookie
 def assignment_type_update(request, org, course, category, name):
     '''
@@ -256,7 +257,7 @@ def assignment_type_update(request, org, course, category, name):
 
     if request.method == 'GET':
         return JsonResponse(CourseGradingModel.get_section_grader_type(location))
-    elif request.method == 'POST':  # post or put, doesn't matter.
+    elif request.method in ('POST', 'PUT'):  # post or put, doesn't matter.
         return JsonResponse(CourseGradingModel.update_section_grader_type(location, request.POST))
 
 

cms/djangoapps/contentstore/views/course.py

@@ -42,8 +42,7 @@ from .component import (
     ADVANCED_COMPONENT_POLICY_KEY)
 
 from django_comment_common.utils import seed_permissions_roles
-import datetime
-from django.utils.timezone import UTC
+
 from xmodule.html_module import AboutDescriptor
 __all__ = ['course_index', 'create_new_course', 'course_info',
            'course_info_updates', 'get_course_settings',
@@ -176,6 +175,7 @@ def course_info(request, org, course, name, provided_id=None):
 
 
 @expect_json
+@require_http_methods(("GET", "POST", "PUT", "DELETE"))
 @login_required
 @ensure_csrf_cookie
 def course_info_updates(request, org, course, provided_id=None):
@@ -206,7 +206,7 @@ def course_info_updates(request, org, course, provided_id=None):
         except:
             return HttpResponseBadRequest("Failed to delete",
                                           content_type="text/plain")
-    elif request.method == 'POST':
+    elif request.method in ('POST', 'PUT'):  # can be either and sometimes django is rewriting one to the other
         try:
             return JsonResponse(update_course_updates(location, request.POST, provided_id))
         except:
@@ -300,7 +300,7 @@ def course_settings_updates(request, org, course, name, section):
     if request.method == 'GET':
         # Cannot just do a get w/o knowing the course name :-(
         return JsonResponse(manager.fetch(Location(['i4x', org, course, 'course', name])), encoder=CourseSettingsEncoder)
-    elif request.method == 'POST':  # post or put, doesn't matter.
+    elif request.method in ('POST', 'PUT'):  # post or put, doesn't matter.
         return JsonResponse(manager.update_from_json(request.POST), encoder=CourseSettingsEncoder)
 
 
@@ -479,7 +479,7 @@ def textbook_index(request, org, course, name):
     if request.is_ajax():
         if request.method == 'GET':
             return JsonResponse(course_module.pdf_textbooks)
-        elif request.method == 'POST':
+        elif request.method in ('POST', 'PUT'):  # can be either and sometimes django is rewriting one to the other
             try:
                 textbooks = validate_textbooks_json(request.body)
             except TextbookValidationError as err:
@@ -580,7 +580,7 @@ def textbook_by_id(request, org, course, name, tid):
         if not textbook:
             return JsonResponse(status=404)
         return JsonResponse(textbook)
-    elif request.method in ('POST', 'PUT'):
+    elif request.method in ('POST', 'PUT'):  # can be either and sometimes django is rewriting one to the other
         try:
             new_textbook = validate_textbook_json(request.body)
         except TextbookValidationError as err:

cms/djangoapps/contentstore/views/user.py

@@ -83,6 +83,9 @@ def add_user(request, location):
         }
         return JsonResponse(msg, 400)
 
+    # remove leading/trailing whitespace if necessary
+    email = email.strip()
+
     # check that logged in user has admin permissions to this course
     if not has_access(request.user, location, role=INSTRUCTOR_ROLE_NAME):
         raise PermissionDenied()

cms/envs/aws.py

@@ -92,6 +92,7 @@ LOG_DIR = ENV_TOKENS['LOG_DIR']
 CACHES = ENV_TOKENS['CACHES']
 
 SESSION_COOKIE_DOMAIN = ENV_TOKENS.get('SESSION_COOKIE_DOMAIN')
+SESSION_ENGINE = ENV_TOKENS.get('SESSION_ENGINE', SESSION_ENGINE)
 
 # allow for environments to specify what cookie name our login subsystem should use
 # this is to fix a bug regarding simultaneous logins between edx.org and edge.edx.org which can
@@ -122,6 +123,10 @@ LOGGING = get_logger_config(LOG_DIR,
                             debug=False,
                             service_variant=SERVICE_VARIANT)
 
+#theming start:
+PLATFORM_NAME = ENV_TOKENS.get('PLATFORM_NAME', 'edX')
+
+
 ################ SECURE AUTH ITEMS ###############################
 # Secret things: passwords, access keys, etc.
 with open(ENV_ROOT / CONFIG_PREFIX + "auth.json") as auth_file:

cms/envs/dev.py

@@ -33,6 +33,10 @@ MODULESTORE = {
     'direct': {
         'ENGINE': 'xmodule.modulestore.mongo.MongoModuleStore',
         'OPTIONS': modulestore_options
+    },
+    'split': {
+        'ENGINE': 'xmodule.modulestore.split_mongo.SplitMongoModuleStore',
+        'OPTIONS': modulestore_options
     }
 }
 

cms/envs/test.py

@@ -63,6 +63,10 @@ MODULESTORE = {
     'draft': {
         'ENGINE': 'xmodule.modulestore.draft.DraftModuleStore',
         'OPTIONS': MODULESTORE_OPTIONS
+    },
+    'split': {
+        'ENGINE': 'xmodule.modulestore.split_mongo.SplitMongoModuleStore',
+        'OPTIONS': MODULESTORE_OPTIONS
     }
 }
 

cms/manage.py

-#!/usr/bin/env python
-from django.core.management import execute_manager
-import imp
-try:
-    imp.find_module('settings')   # Assumed to be in the same directory.
-except ImportError:
-    import sys
-    sys.stderr.write("Error: Can't find the file 'settings.py' in the directory containing %r. "
-                     "It appears you've customized things.\nYou'll have to run django-admin.py, "
-                     "passing it your settings module.\n" % __file__)
-    sys.exit(1)
-
-import settings
-
-if __name__ == "__main__":
-    execute_manager(settings)

cms/static/coffee/spec/views/overview_spec.coffee

@@ -40,17 +40,30 @@ describe "Course Overview", ->
         </div>
         """#"
 
+        appendSetFixtures """
+          <section class="courseware-section branch" data-id="a-location-goes-here">
+            <li class="branch collapsed id-holder" data-id="an-id-goes-here">
+              <a href="#" class="delete-section-button"></a>
+            </li>
+          </section>
+        """#"
+
         spyOn(window, 'saveSetSectionScheduleDate').andCallThrough()
         # Have to do this here, as it normally gets bound in document.ready()
         $('a.save-button').click(saveSetSectionScheduleDate)
+        $('a.delete-section-button').click(deleteSection)
+
         @notificationSpy = spyOn(CMS.Views.Notification.Mini.prototype, 'show').andCallThrough()
         window.analytics = jasmine.createSpyObj('analytics', ['track'])
         window.course_location_analytics = jasmine.createSpy()
-        sinon.useFakeXMLHttpRequest()
+        @xhr = sinon.useFakeXMLHttpRequest()
+        requests = @requests = []
+        @xhr.onCreate = (req) -> requests.push(req)
 
     afterEach ->
         delete window.analytics
         delete window.course_location_analytics
+        @notificationSpy.reset()
 
     it "should save model when save is clicked", ->
         $('a.edit-button').click()
@@ -61,3 +74,21 @@ describe "Course Overview", ->
         $('a.edit-button').click()
         $('a.save-button').click()
         expect(@notificationSpy).toHaveBeenCalled()
+
+    it "should delete model when delete is clicked", ->
+      deleteSpy = spyOn(window, '_deleteItem').andCallThrough()
+      $('a.delete-section-button').click()
+      $('a.action-primary').click()
+      expect(deleteSpy).toHaveBeenCalled()
+      expect(@requests[0].url).toEqual('/delete_item')
+
+    it "should not delete model when cancel is clicked", ->
+      deleteSpy = spyOn(window, '_deleteItem').andCallThrough()
+      $('a.delete-section-button').click()
+      $('a.action-secondary').click()
+      expect(@requests.length).toEqual(0)
+
+    it "should show a confirmation on delete", ->
+      $('a.delete-section-button').click()
+      $('a.action-primary').click()
+      expect(@notificationSpy).toHaveBeenCalled()

cms/static/coffee/src/views/module_edit.coffee

@@ -84,11 +84,15 @@ class CMS.Views.ModuleEdit extends Backbone.View
 
     data.metadata = _.extend(data.metadata || {}, @changedMetadata())
     @hideModal()
+    saving = new CMS.Views.Notification.Mini
+      title: gettext('Saving') + '…'
+    saving.show()
     @model.save(data).done( =>
     #   # showToastMessage("Your changes have been saved.", null, 3)
       @module = null
       @render()
       @$el.removeClass('editing')
+      saving.hide()
     ).fail( ->
       showToastMessage(gettext("There was an error saving your changes. Please try again."), null, 3)
     )

cms/static/coffee/src/views/unit.coffee

@@ -67,8 +67,8 @@ class CMS.Views.UnitEdit extends Backbone.View
     type = $(event.currentTarget).data('type')
     @$newComponentTypePicker.slideUp(250)
     @$(".new-component-#{type}").slideDown(250)
-    $('html, body').animate({ 
-      scrollTop: @$(".new-component-#{type}").offset().top 
+    $('html, body').animate({
+      scrollTop: @$(".new-component-#{type}").offset().top
     }, 500)
 
   closeNewComponent: (event) =>
@@ -115,27 +115,43 @@ class CMS.Views.UnitEdit extends Backbone.View
     @model.save()
 
   deleteComponent: (event) =>
-    if not confirm 'Are you sure you want to delete this component? This action cannot be undone.'
-      return
-    $component = $(event.currentTarget).parents('.component')
-    $.post('/delete_item', {
-      id: $component.data('id')
-    }, =>
-      analytics.track "Deleted a Component",
-        course: course_location_analytics
-        unit_id: unit_location_analytics
-        id: $component.data('id')
-
-      $component.remove()
-      # b/c we don't vigilantly keep children up to date
-      # get rid of it before it hurts someone
-      # sorry for the js, i couldn't figure out the coffee equivalent
-      `_this.model.save({children: _this.components()},
-          {success: function(model) {
-              model.unset('children');
-          }}
-      );`
+    msg = new CMS.Views.Prompt.Warning(
+      title: gettext('Delete this component?'),
+      message: gettext('Deleting this component is permanent and cannot be undone.'),
+      actions:
+        primary:
+          text: gettext('Yes, delete this component'),
+          click: (view) =>
+            view.hide()
+            deleting = new CMS.Views.Notification.Mini
+              title: gettext('Deleting') + '…',
+            deleting.show()
+            $component = $(event.currentTarget).parents('.component')
+            $.post('/delete_item', {
+              id: $component.data('id')
+            }, =>
+              deleting.hide()
+              analytics.track "Deleted a Component",
+                course: course_location_analytics
+                unit_id: unit_location_analytics
+                id: $component.data('id')
+
+              $component.remove()
+              # b/c we don't vigilantly keep children up to date
+              # get rid of it before it hurts someone
+              # sorry for the js, i couldn't figure out the coffee equivalent
+              `_this.model.save({children: _this.components()},
+                {success: function(model) {
+                model.unset('children');
+                }}
+              );`
+            )
+        secondary:
+          text: gettext('Cancel'),
+          click: (view) ->
+            view.hide()
     )
+    msg.show()
 
   deleteDraft: (event) ->
     @wait(true)
@@ -236,7 +252,7 @@ class CMS.Views.UnitEdit.NameEdit extends Backbone.View
 class CMS.Views.UnitEdit.LocationState extends Backbone.View
   initialize: =>
     @model.on('change:state', @render)
-  
+
   render: =>
     @$el.toggleClass("#{@model.previous('state')}-item #{@model.get('state')}-item")
 

cms/static/js/base.js

@@ -356,39 +356,61 @@ function createNewUnit(e) {
 
 function deleteUnit(e) {
     e.preventDefault();
-    _deleteItem($(this).parents('li.leaf'));
+    _deleteItem($(this).parents('li.leaf'), 'Unit');
 }
 
 function deleteSubsection(e) {
     e.preventDefault();
-    _deleteItem($(this).parents('li.branch'));
+    _deleteItem($(this).parents('li.branch'), 'Subsection');
 }
 
 function deleteSection(e) {
     e.preventDefault();
-    _deleteItem($(this).parents('section.branch'));
-}
-
-function _deleteItem($el) {
-    if (!confirm(gettext('Are you sure you wish to delete this item. It cannot be reversed!'))) return;
-
-    var id = $el.data('id');
-
-    analytics.track('Deleted an Item', {
-        'course': course_location_analytics,
-        'id': id
-    });
-
-
-    $.post('/delete_item', {
-        'id': id,
-        'delete_children': true,
-        'delete_all_versions': true
-    },
-
-    function(data) {
-        $el.remove();
+    _deleteItem($(this).parents('section.branch'), 'Section');
+}
+
+function _deleteItem($el, type) {
+    var confirm = new CMS.Views.Prompt.Warning({
+        title: gettext('Delete this ' + type + '?'),
+        message: gettext('Deleting this ' + type + ' is permanent and cannot be undone.'),
+        actions: {
+            primary: {
+                text: gettext('Yes, delete this ' + type),
+                click: function(view) {
+                    view.hide();
+
+                    var id = $el.data('id');
+
+                    analytics.track('Deleted an Item', {
+                        'course': course_location_analytics,
+                        'id': id
+                    });
+
+                    var deleting = new CMS.Views.Notification.Mini({
+                        title: gettext('Deleting') + '…'
+                    });
+                    deleting.show();
+
+                    $.post('/delete_item',
+                           {'id': id,
+                            'delete_children': true,
+                            'delete_all_versions': true},
+                           function(data) {
+                               $el.remove();
+                               deleting.hide();
+                           }
+                          );
+                }
+            },
+            secondary: {
+                text: gettext('Cancel'),
+                click: function(view) {
+                    view.hide();
+                }
+            }
+        }
     });
+    confirm.show();
 }
 
 function markAsLoaded() {
@@ -728,7 +750,7 @@ function saveSetSectionScheduleDate(e) {
         var $thisSection = $('.courseware-section[data-id="' + id + '"]');
         var html = _.template(
             '<span class="published-status">' +
-                '<strong>' + gettext("Will Release: ") + '</strong>' +
+                '<strong>' + gettext("Will Release:") + '&nbsp;</strong>' +
                 gettext("<%= date %> at <%= time %> UTC") +
             '</span>' +
             '<a href="#" class="edit-button" data-date="<%= date %>" data-time="<%= time %>" data-id="<%= id %>">' +

cms/static/js/views/assets.js

@@ -9,7 +9,7 @@ function removeAsset(e){
     e.preventDefault();
 
     var that = this;
-    var msg = new CMS.Views.Prompt.Confirmation({
+    var msg = new CMS.Views.Prompt.Warning({
         title: gettext("Delete File Confirmation"),
         message: gettext("Are you sure you wish to delete this item. It cannot be reversed!\n\nAlso any content that links/refers to this item will no longer work (e.g. broken images and/or links)"),
         actions: {

cms/static/js/views/grader-select-view.js

@@ -81,9 +81,18 @@ CMS.Views.OverviewAssignmentGrader = Backbone.View.extend({
 
 	      this.removeMenu(e);
 
+              var saving = new CMS.Views.Notification.Mini({
+                  title: gettext('Saving') + '…'
+              });
+              saving.show();
+
 	      // TODO I'm not happy with this string fetch via the html for what should be an id. I'd rather use the id attr
 	      // of the CourseGradingPolicy model or null for Not Graded (NOTE, change template's if check for is-selected accordingly)
-	      this.assignmentGrade.save('graderType', $(e.target).text());
+	      this.assignmentGrade.save(
+                  'graderType',
+                  $(e.target).text(),
+                  {success: function () { saving.hide(); }}
+              );
 
 	      this.render();
 	}

cms/static/js/views/settings/main_settings_view.js

@@ -107,6 +107,7 @@ CMS.Views.Settings.Details = CMS.Views.ValidatingView.extend({
         // to pick up when the date is typed directly in the field.
         datefield.change(setfield);
         timefield.on('changeTime', setfield);
+        timefield.on('input', setfield);
 
         datefield.datepicker('setDate', this.model.get(fieldName));
         // timepicker doesn't let us set null, so check that we have a time

cms/static/js/views/textbook.js

@@ -241,7 +241,7 @@ CMS.Views.EditChapter = Backbone.View.extend({
             asset_path: this.$("input.chapter-asset-path").val()
         });
         var msg = new CMS.Models.FileUpload({
-            title: _.template(gettext("Upload a new asset to “<%= name %>”"),
+            title: _.template(gettext("Upload a new PDF to “<%= name %>”"),
                 {name: section.escape('name')}),
             message: "Files must be in PDF format."
         });

cms/static/sass/views/_index.scss

@@ -71,8 +71,13 @@ body.index {
         color: $white;
       }
 
+      .wrapper-text-welcome, .logo {
+        display: inline-block;
+      }
+
       .logo {
         font-weight: 600;
+        margin-left: ($baseline/2);
       }
 
       .tagline {

cms/static/sass/views/_unit.scss

@@ -747,6 +747,7 @@ body.unit {
 
 // Unit Page Sidebar
 .unit-settings {
+
   .window-contents {
     padding: $baseline/2 $baseline;
   }
@@ -854,6 +855,24 @@ body.unit {
 }
 
 .unit-location {
+
+  // unit id
+  .wrapper-unit-id {
+
+    .unit-id {
+
+      .label {
+        @extend .t-title7;
+        margin-bottom: ($baseline/4);
+        color: $gray-d1;
+      }
+
+      .value {
+        margin-bottom: 0;
+      }
+    }
+  }
+
   .url {
     box-shadow: none;
     width: 100%;

cms/templates/edit-static-page.html

-<%inherit file="base.html" />
-<%! from django.core.urlresolvers import reverse %>
-<%block name="title">Editing Static Page</%block>
-<%block name="bodyclass">is-signedin course pages edit-static-page</%block>
-
-<%block name="content">
-<div class="main-wrapper">
-  <div class="inner-wrapper">
-    <div class="main-column">
-      <article class="static-page-details">
-        <div class="row">
-          <label>Display Name:</label>
-          <input type="text" value="Syllabus" class="page-display-name-input" data-metadata-name="display_name"/>
-        </div>
-        <div class="row">
-          <label>Page Content:</label>
-          <textarea class="page-contents"></textarea>
-        </div>
-      </article>
-    </div>
-    <div class="sidebar">
-      <div class="unit-settings window">
-        <h4 class="header">Page Settings</h4>
-        <div class="window-contents">
-          <div class="row visibility">
-            <label class="inline-label">Visibility:</label>
-            <select>
-              <option>Public</option>
-              <option>Private</option>
-            </select>
-          </div>
-          <div class="row unit-actions">
-            <a href="#" class="save-button">Save</a>
-            <a href="#" target="_blank" class="preview-button">Preview</a>
-          </div>
-        </div>
-      </div>
-    </div>
-  </div>
-</div>
-</%block>
\ No newline at end of file

cms/templates/editable_preview.html

-<section class='editable-preview'>
-${content}
-<div class="component-actions">
-  <a href="#" class="edit-button"><span class="edit-icon white"></span>Edit</a>
-  <a href="#" class="delete-button"><span class="delete-icon white"></span>Delete</a>	
-</div>
-<a data-tooltip="Drag to reorder" href="#" class="drag-handle"></a>
-<div class="component-editor">
-  <h5>Edit Video Component</h5>
-  <textarea class="component-source"><video youtube="1.50:q1xkuPsOY6Q,1.25:9WOY2dHz5i4,1.0:4rpg8Bq6hb4,0.75:KLim9Xkp7IY"/></textarea>
-  <a href="#" class="save-button">Save</a><a href="#" class="cancel-button">Cancel</a>
-</div>
-<section>

cms/templates/howitworks.html

@@ -11,7 +11,7 @@
   <section class="content content-header">
     <header>
       ## "edX Studio" should not be translated
-      <h1>${_('Welcome to')}<span class="logo">&nbsp;edX Studio</span></h1>
+      <h1><span class="wrapper-text-welcome">${_('Welcome to')}</span><span class="logo">edX Studio</span></h1>
       <p class="tagline">${_("Studio helps manage your courses online, so you can focus on teaching them")}</p>
     </header>
   </section>

cms/templates/js/edit-chapter.underscore

@@ -9,6 +9,6 @@
   <label for="chapter<%= order %>-asset-path"><%= gettext("Chapter Asset") %></label>
   <input id="chapter<%= order %>-asset-path" name="chapter<%= order %>-asset-path" class="chapter-asset-path" placeholder="<%= _.str.sprintf(gettext("path/to/introductionToCookieBaking-CH%d.pdf"), order) %>" value="<%= asset_path %>" type="text">
   <span class="tip tip-stacked"><%= gettext("upload a PDF file or provide the path to a Studio asset file") %></span>
-<button class="action action-upload"><%= gettext("Upload Asset") %></button>
+<button class="action action-upload"><%= gettext("Upload PDF") %></button>
 </div>
 <a href="" class="action action-close"><i class="icon-remove-sign"></i> <span class="sr"><%= gettext("delete chapter") %></span></a>

cms/templates/overview.html

-/<%! from django.utils.translation import ugettext as _ %>
+<%! from django.utils.translation import ugettext as _ %>
 <%inherit file="base.html" />
 <%!
    import logging

cms/templates/unit.html

@@ -171,10 +171,15 @@
         <div class="window unit-location">
           <h4 class="header">${_("Unit Location")}</h4>
           <div class="window-contents">
-            <div><input type="text" class="url" value="/courseware/${section.url_name}/${subsection.url_name}" disabled /></div>
+            <div class="row wrapper-unit-id">
+              <p class="unit-id">
+                <span class="label">${_("Unit Identifier:")}</span>
+                <input type="text" class="url value" value="${unit.location.name}" disabled />
+              </p>
+            </div>
             <ol>
               <li>
-                <a href="#" class="section-item">${section.display_name_with_default}</a>
+                <a href="${reverse('course_index', kwargs=dict(org=context_course.location.org, course=context_course.location.course, name=context_course.location.name))}" class="section-item">${section.display_name_with_default}</a>
                 <ol>
                   <li>
                     <a href="${reverse('edit_subsection', args=[subsection.location])}" class="section-item">

common/djangoapps/external_auth/models.py

@@ -4,9 +4,9 @@ WE'RE USING MIGRATIONS!
 If you make changes to this model, be sure to create an appropriate migration
 file and check it in at the same time as your model changes. To do that,
 
-1. Go to the mitx dir
-2. django-admin.py schemamigration student --auto --settings=lms.envs.dev --pythonpath=. description_of_your_change
-3. Add the migration file created in mitx/common/djangoapps/external_auth/migrations/
+1. Go to the edx-platform dir
+2. ./manage.py lms schemamigration student --auto description_of_your_change
+3. Add the migration file created in edx-platform/common/djangoapps/external_auth/migrations/
 """
 
 from django.db import models

common/djangoapps/heartbeat/views.py

 import json
 from datetime import datetime
+from pytz import UTC
 from django.http import HttpResponse
 from xmodule.modulestore.django import modulestore
 from dogapi import dog_stats_api
 
+
 @dog_stats_api.timed('edxapp.heartbeat')
 def heartbeat(request):
     """
     Simple view that a loadbalancer can check to verify that the app is up
     """
     output = {
-        'date': datetime.now().isoformat(),
+        'date': datetime.now(UTC).isoformat(),
         'courses': [course.location.url() for course in modulestore().get_courses()],
     }
     return HttpResponse(json.dumps(output, indent=4))

common/djangoapps/static_replace/__init__.py

@@ -43,6 +43,35 @@ def try_staticfiles_lookup(path):
     return url
 
 
+def replace_jump_to_id_urls(text, course_id, jump_to_id_base_url):
+    """
+    This will replace a link to another piece of courseware to a 'jump_to'
+    URL that will redirect to the right place in the courseware
+
+    NOTE: This is similar to replace_course_urls in terms of functionality
+    but it is intended to be used when we only have a 'id' that the
+    course author provides. This is much more helpful when using
+    Studio authored courses since they don't need to know the path. This
+    is also durable with respect to item moves.
+
+    text: The content over which to perform the subtitutions
+    course_id: The course_id in which this rewrite happens
+    jump_to_id_base_url: 
+        A app-tier (e.g. LMS) absolute path to the base of the handler that will perform the
+        redirect. e.g. /courses/<org>/<course>/<run>/jump_to_id. NOTE the <id> will be appended to
+        the end of this URL at re-write time
+
+    output: <text> after the link rewriting rules are applied
+    """
+
+    def replace_jump_to_id_url(match):
+        quote = match.group('quote')
+        rest = match.group('rest')
+        return "".join([quote, jump_to_id_base_url + rest, quote])
+
+    return re.sub(_url_replace_regex('/jump_to_id/'), replace_jump_to_id_url, text)
+
+
 def replace_course_urls(text, course_id):
     """
     Replace /course/$stuff urls with /courses/$course_id/$stuff urls
@@ -53,7 +82,6 @@ def replace_course_urls(text, course_id):
     returns: text with the links replaced
     """
 
-
     def replace_course_url(match):
         quote = match.group('quote')
         rest = match.group('rest')

common/djangoapps/student/management/commands/pearson_transfer.py

@@ -20,7 +20,7 @@ class Command(BaseCommand):
     files and then uploads over SFTP to Pearson and stuffs the entry in an
     S3 bucket for archive purposes.
 
-    Usage: django-admin.py pearson-transfer --mode [import|export|both]
+    Usage: ./manage.py pearson-transfer --mode [import|export|both]
     """
 
     option_list = BaseCommand.option_list + (

common/djangoapps/student/models.py

@@ -6,9 +6,9 @@ Migration Notes
 If you make changes to this model, be sure to create an appropriate migration
 file and check it in at the same time as your model changes. To do that,
 
-1. Go to the mitx dir
-2. django-admin.py schemamigration student --auto --settings=lms.envs.dev --pythonpath=. description_of_your_change
-3. Add the migration file created in mitx/common/djangoapps/student/migrations/
+1. Go to the edx-platform dir
+2. ./manage.py lms schemamigration student --auto description_of_your_change
+3. Add the migration file created in edx-platform/common/djangoapps/student/migrations/
 """
 from datetime import datetime
 import hashlib
@@ -69,30 +69,33 @@ class UserProfile(models.Model):
     location = models.CharField(blank=True, max_length=255, db_index=True)
 
     # Optional demographic data we started capturing from Fall 2012
-    this_year = datetime.now().year
+    this_year = datetime.now(UTC).year
     VALID_YEARS = range(this_year, this_year - 120, -1)
     year_of_birth = models.IntegerField(blank=True, null=True, db_index=True)
     GENDER_CHOICES = (('m', 'Male'), ('f', 'Female'), ('o', 'Other'))
-    gender = models.CharField(blank=True, null=True, max_length=6, db_index=True,
-                              choices=GENDER_CHOICES)
+    gender = models.CharField(
+        blank=True, null=True, max_length=6, db_index=True, choices=GENDER_CHOICES
+    )
 
     # [03/21/2013] removed these, but leaving comment since there'll still be
     # p_se and p_oth in the existing data in db.
     # ('p_se', 'Doctorate in science or engineering'),
     # ('p_oth', 'Doctorate in another field'),
-    LEVEL_OF_EDUCATION_CHOICES = (('p', 'Doctorate'),
-                                  ('m', "Master's or professional degree"),
-                                  ('b', "Bachelor's degree"),
-                                  ('a', "Associate's degree"),
-                                  ('hs', "Secondary/high school"),
-                                  ('jhs', "Junior secondary/junior high/middle school"),
-                                  ('el', "Elementary/primary school"),
-                                  ('none', "None"),
-                                  ('other', "Other"))
+    LEVEL_OF_EDUCATION_CHOICES = (
+        ('p', 'Doctorate'),
+        ('m', "Master's or professional degree"),
+        ('b', "Bachelor's degree"),
+        ('a', "Associate's degree"),
+        ('hs', "Secondary/high school"),
+        ('jhs', "Junior secondary/junior high/middle school"),
+        ('el', "Elementary/primary school"),
+        ('none', "None"),
+        ('other', "Other")
+    )
     level_of_education = models.CharField(
-                            blank=True, null=True, max_length=6, db_index=True,
-                            choices=LEVEL_OF_EDUCATION_CHOICES
-                         )
+        blank=True, null=True, max_length=6, db_index=True,
+        choices=LEVEL_OF_EDUCATION_CHOICES
+    )
     mailing_address = models.TextField(blank=True, null=True)
     goals = models.TextField(blank=True, null=True)
     allow_certificate = models.BooleanField(default=1)
@@ -307,18 +310,18 @@ class TestCenterUserForm(ModelForm):
 ACCOMMODATION_REJECTED_CODE = 'NONE'
 
 ACCOMMODATION_CODES = (
-                      (ACCOMMODATION_REJECTED_CODE, 'No Accommodation Granted'),
-                      ('EQPMNT', 'Equipment'),
-                      ('ET12ET', 'Extra Time - 1/2 Exam Time'),
-                      ('ET30MN', 'Extra Time - 30 Minutes'),
-                      ('ETDBTM', 'Extra Time - Double Time'),
-                      ('SEPRMM', 'Separate Room'),
-                      ('SRREAD', 'Separate Room and Reader'),
-                      ('SRRERC', 'Separate Room and Reader/Recorder'),
-                      ('SRRECR', 'Separate Room and Recorder'),
-                      ('SRSEAN', 'Separate Room and Service Animal'),
-                      ('SRSGNR', 'Separate Room and Sign Language Interpreter'),
-                      )
+    (ACCOMMODATION_REJECTED_CODE, 'No Accommodation Granted'),
+    ('EQPMNT', 'Equipment'),
+    ('ET12ET', 'Extra Time - 1/2 Exam Time'),
+    ('ET30MN', 'Extra Time - 30 Minutes'),
+    ('ETDBTM', 'Extra Time - Double Time'),
+    ('SEPRMM', 'Separate Room'),
+    ('SRREAD', 'Separate Room and Reader'),
+    ('SRRERC', 'Separate Room and Reader/Recorder'),
+    ('SRRECR', 'Separate Room and Recorder'),
+    ('SRSEAN', 'Separate Room and Service Animal'),
+    ('SRSGNR', 'Separate Room and Sign Language Interpreter'),
+)
 
 ACCOMMODATION_CODE_DICT = {code: name for (code, name) in ACCOMMODATION_CODES}
 
@@ -572,7 +575,6 @@ class TestCenterRegistrationForm(ModelForm):
         return code
 
 
-
 def get_testcenter_registration(user, course_id, exam_series_code):
     try:
         tcu = TestCenterUser.objects.get(user=user)

common/djangoapps/student/views.py

@@ -111,9 +111,9 @@ def get_date_for_press(publish_date):
     # strip off extra months, and just use the first:
     date = re.sub(multimonth_pattern, ", ", publish_date)
     if re.search(day_pattern, date):
-        date = datetime.datetime.strptime(date, "%B %d, %Y")
+        date = datetime.datetime.strptime(date, "%B %d, %Y").replace(tzinfo=UTC)
     else:
-        date = datetime.datetime.strptime(date, "%B, %Y")
+        date = datetime.datetime.strptime(date, "%B, %Y").replace(tzinfo=UTC)
     return date
 
 
@@ -1100,7 +1100,7 @@ def confirm_email_change(request, key):
         meta = up.get_meta()
         if 'old_emails' not in meta:
             meta['old_emails'] = []
-        meta['old_emails'].append([user.email, datetime.datetime.now().isoformat()])
+        meta['old_emails'].append([user.email, datetime.datetime.now(UTC).isoformat()])
         up.set_meta(meta)
         up.save()
         # Send it to the old email...
@@ -1198,7 +1198,7 @@ def accept_name_change_by_id(id):
     meta = up.get_meta()
     if 'old_names' not in meta:
         meta['old_names'] = []
-    meta['old_names'].append([up.name, pnc.rationale, datetime.datetime.now().isoformat()])
+    meta['old_names'].append([up.name, pnc.rationale, datetime.datetime.now(UTC).isoformat()])
     up.set_meta(meta)
 
     up.name = pnc.new_name

common/djangoapps/terrain/steps.py

@@ -129,6 +129,13 @@ def should_have_link_with_id_and_text(step, link_id, text):
     assert_equals(link.text, text)
 
 
+@step(r'should see a link to "([^"]*)" with the text "([^"]*)"$')
+def should_have_link_with_path_and_text(step, path, text):
+    link = world.browser.find_link_by_text(text)
+    assert len(link) > 0
+    assert_equals(link.first["href"], django_url(path))
+
+
 @step(r'should( not)? see "(.*)" (?:somewhere|anywhere) (?:in|on) (?:the|this) page')
 def should_see_in_the_page(step, doesnt_appear, text):
     if doesnt_appear:

common/djangoapps/xmodule_modifiers.py

@@ -42,6 +42,28 @@ def wrap_xmodule(get_html, module, template, context=None):
     return _get_html
 
 
+def replace_jump_to_id_urls(get_html, course_id, jump_to_id_base_url):
+    """
+    This will replace a link between courseware in the format
+    /jump_to/<id> with a URL for a page that will correctly redirect
+    This is similar to replace_course_urls, but much more flexible and
+    durable for Studio authored courses. See more comments in static_replace.replace_jump_to_urls
+
+    course_id: The course_id in which this rewrite happens
+    jump_to_id_base_url:
+        A app-tier (e.g. LMS) absolute path to the base of the handler that will perform the
+        redirect. e.g. /courses/<org>/<course>/<run>/jump_to_id. NOTE the <id> will be appended to
+        the end of this URL at re-write time
+
+    output: a wrapped get_html() function pointer, which, when called, will apply the
+        rewrite rules
+    """
+    @wraps(get_html)
+    def _get_html():
+        return static_replace.replace_jump_to_id_urls(get_html(), course_id, jump_to_id_base_url)
+    return _get_html
+
+
 def replace_course_urls(get_html, course_id):
     """
     Updates the supplied module with a new get_html function that wraps

common/lib/capa/capa/capa_problem.py

@@ -32,6 +32,8 @@ import capa.xqueue_interface as xqueue_interface
 import capa.responsetypes as responsetypes
 from capa.safe_exec import safe_exec
 
+from pytz import UTC
+
 # dict of tagname, Response Class -- this should come from auto-registering
 response_tag_dict = dict([(x.response_tag, x) for x in responsetypes.__all__])
 
@@ -42,13 +44,22 @@ solution_tags = ['solution']
 response_properties = ["codeparam", "responseparam", "answer", "openendedparam"]
 
 # special problem tags which should be turned into innocuous HTML
-html_transforms = {'problem': {'tag': 'div'},
-                   'text': {'tag': 'span'},
-                   'math': {'tag': 'span'},
-                   }
+html_transforms = {
+    'problem': {'tag': 'div'},
+    'text': {'tag': 'span'},
+    'math': {'tag': 'span'},
+}
 
 # These should be removed from HTML output, including all subelements
-html_problem_semantics = ["codeparam", "responseparam", "answer", "script", "hintgroup", "openendedparam", "openendedrubric"]
+html_problem_semantics = [
+    "codeparam",
+    "responseparam",
+    "answer",
+    "script",
+    "hintgroup",
+    "openendedparam",
+    "openendedrubric"
+]
 
 log = logging.getLogger(__name__)
 
@@ -242,11 +253,15 @@ class LoncapaProblem(object):
             return None
 
         # Get a list of timestamps of all queueing requests, then convert it to a DateTime object
-        queuetime_strs = [self.correct_map.get_queuetime_str(answer_id)
-                          for answer_id in self.correct_map
-                          if self.correct_map.is_queued(answer_id)]
-        queuetimes = [datetime.strptime(qt_str, xqueue_interface.dateformat)
-                      for qt_str in queuetime_strs]
+        queuetime_strs = [
+            self.correct_map.get_queuetime_str(answer_id)
+            for answer_id in self.correct_map
+            if self.correct_map.is_queued(answer_id)
+        ]
+        queuetimes = [
+            datetime.strptime(qt_str, xqueue_interface.dateformat).replace(tzinfo=UTC)
+            for qt_str in queuetime_strs
+        ]
 
         return max(queuetimes)
 
@@ -404,10 +419,16 @@ class LoncapaProblem(object):
                     # open using ModuleSystem OSFS filestore
                     ifp = self.system.filestore.open(filename)
                 except Exception as err:
-                    log.warning('Error %s in problem xml include: %s' % (
-                            err, etree.tostring(inc, pretty_print=True)))
-                    log.warning('Cannot find file %s in %s' % (
-                            filename, self.system.filestore))
+                    log.warning(
+                        'Error %s in problem xml include: %s' % (
+                            err, etree.tostring(inc, pretty_print=True)
+                        )
+                    )
+                    log.warning(
+                        'Cannot find file %s in %s' % (
+                            filename, self.system.filestore
+                        )
+                    )
                     # if debugging, don't fail - just log error
                     # TODO (vshnayder): need real error handling, display to users
                     if not self.system.get('DEBUG'):
@@ -418,8 +439,11 @@ class LoncapaProblem(object):
                     # read in and convert to XML
                     incxml = etree.XML(ifp.read())
                 except Exception as err:
-                    log.warning('Error %s in problem xml include: %s' % (
-                            err, etree.tostring(inc, pretty_print=True)))
+                    log.warning(
+                        'Error %s in problem xml include: %s' % (
+                            err, etree.tostring(inc, pretty_print=True)
+                        )
+                    )
                     log.warning('Cannot parse XML in %s' % (filename))
                     # if debugging, don't fail - just log error
                     # TODO (vshnayder): same as above
@@ -579,8 +603,9 @@ class LoncapaProblem(object):
         # let each Response render itself
         if problemtree in self.responders:
             overall_msg = self.correct_map.get_overall_message()
-            return self.responders[problemtree].render_html(self._extract_html,
-                                                response_msg=overall_msg)
+            return self.responders[problemtree].render_html(
+                self._extract_html, response_msg=overall_msg
+            )
 
         # let each custom renderer render itself:
         if problemtree.tag in customrender.registry.registered_tags():
@@ -628,9 +653,10 @@ class LoncapaProblem(object):
 
             answer_id = 1
             input_tags = inputtypes.registry.registered_tags()
-            inputfields = tree.xpath("|".join(['//' + response.tag + '[@id=$id]//' + x
-                                               for x in (input_tags + solution_tags)]),
-                                    id=response_id_str)
+            inputfields = tree.xpath(
+                "|".join(['//' + response.tag + '[@id=$id]//' + x for x in (input_tags + solution_tags)]),
+                id=response_id_str
+            )
 
             # assign one answer_id for each input type or solution type
             for entry in inputfields:

common/lib/capa/capa/correctmap.py

@@ -37,23 +37,27 @@ class CorrectMap(object):
         return self.cmap.__iter__()
 
     # See the documentation for 'set_dict' for the use of kwargs
-    def set(self,
-            answer_id=None,
-            correctness=None,
-            npoints=None,
-            msg='',
-            hint='',
-            hintmode=None,
-            queuestate=None, **kwargs):
+    def set(
+        self,
+        answer_id=None,
+        correctness=None,
+        npoints=None,
+        msg='',
+        hint='',
+        hintmode=None,
+        queuestate=None,
+        **kwargs
+    ):
 
         if answer_id is not None:
-            self.cmap[str(answer_id)] = {'correctness': correctness,
-                                    'npoints': npoints,
-                                    'msg': msg,
-                                    'hint': hint,
-                                    'hintmode': hintmode,
-                                    'queuestate': queuestate,
-                                    }
+            self.cmap[str(answer_id)] = {
+                'correctness': correctness,
+                'npoints': npoints,
+                'msg': msg,
+                'hint': hint,
+                'hintmode': hintmode,
+                'queuestate': queuestate,
+            }
 
     def __repr__(self):
         return repr(self.cmap)

common/lib/capa/capa/responsetypes.py

@@ -33,6 +33,7 @@ from shapely.geometry import Point, MultiPoint
 from calc import evaluator, UndefinedVariable
 from . import correctmap
 from datetime import datetime
+from pytz import UTC
 from .util import *
 from lxml import etree
 from lxml.html.soupparser import fromstring as fromstring_bs     # uses Beautiful Soup!!! FIXME?
@@ -1365,9 +1366,11 @@ class CodeResponse(LoncapaResponse):
             # Note that submission can be a file
             submission = student_answers[self.answer_id]
         except Exception as err:
-            log.error('Error in CodeResponse %s: cannot get student answer for %s;'
-                      ' student_answers=%s' %
-                     (err, self.answer_id, convert_files_to_filenames(student_answers)))
+            log.error(
+                'Error in CodeResponse %s: cannot get student answer for %s;'
+                ' student_answers=%s' %
+                (err, self.answer_id, convert_files_to_filenames(student_answers))
+            )
             raise Exception(err)
 
         # We do not support xqueue within Studio.
@@ -1381,19 +1384,20 @@ class CodeResponse(LoncapaResponse):
         #------------------------------------------------------------
 
         qinterface = self.system.xqueue['interface']
-        qtime = datetime.strftime(datetime.now(), xqueue_interface.dateformat)
+        qtime = datetime.strftime(datetime.now(UTC), xqueue_interface.dateformat)
 
         anonymous_student_id = self.system.anonymous_student_id
 
         # Generate header
-        queuekey = xqueue_interface.make_hashkey(str(self.system.seed) + qtime +
-                                                 anonymous_student_id +
-                                                 self.answer_id)
+        queuekey = xqueue_interface.make_hashkey(
+            str(self.system.seed) + qtime + anonymous_student_id + self.answer_id
+        )
         callback_url = self.system.xqueue['construct_callback']()
         xheader = xqueue_interface.make_xheader(
             lms_callback_url=callback_url,
             lms_key=queuekey,
-            queue_name=self.queue_name)
+            queue_name=self.queue_name
+        )
 
         # Generate body
         if is_list_of_files(submission):
@@ -1406,9 +1410,10 @@ class CodeResponse(LoncapaResponse):
 
         # Metadata related to the student submission revealed to the external
         # grader
-        student_info = {'anonymous_student_id': anonymous_student_id,
-                        'submission_time': qtime,
-                        }
+        student_info = {
+            'anonymous_student_id': anonymous_student_id,
+            'submission_time': qtime,
+        }
         contents.update({'student_info': json.dumps(student_info)})
 
         # Submit request. When successful, 'msg' is the prior length of the

common/lib/capa/capa/tests/test_responsetypes.py

@@ -18,6 +18,8 @@ from capa.correctmap import CorrectMap
 from capa.util import convert_files_to_filenames
 from capa.xqueue_interface import dateformat
 
+from pytz import UTC
+
 
 class ResponseTest(unittest.TestCase):
     """ Base class for tests of capa responses."""
@@ -333,8 +335,9 @@ class SymbolicResponseTest(ResponseTest):
 
         correct_map = problem.grade_answers(input_dict)
 
-        self.assertEqual(correct_map.get_correctness('1_2_1'),
-                        expected_correctness)
+        self.assertEqual(
+            correct_map.get_correctness('1_2_1'), expected_correctness
+        )
 
 
 class OptionResponseTest(ResponseTest):
@@ -702,7 +705,7 @@ class CodeResponseTest(ResponseTest):
         # Now we queue the LCP
         cmap = CorrectMap()
         for i, answer_id in enumerate(answer_ids):
-            queuestate = CodeResponseTest.make_queuestate(i, datetime.now())
+            queuestate = CodeResponseTest.make_queuestate(i, datetime.now(UTC))
             cmap.update(CorrectMap(answer_id=answer_ids[i], queuestate=queuestate))
         self.problem.correct_map.update(cmap)
 
@@ -718,7 +721,7 @@ class CodeResponseTest(ResponseTest):
         old_cmap = CorrectMap()
         for i, answer_id in enumerate(answer_ids):
             queuekey = 1000 + i
-            queuestate = CodeResponseTest.make_queuestate(queuekey, datetime.now())
+            queuestate = CodeResponseTest.make_queuestate(queuekey, datetime.now(UTC))
             old_cmap.update(CorrectMap(answer_id=answer_ids[i], queuestate=queuestate))
 
         # Message format common to external graders
@@ -778,13 +781,15 @@ class CodeResponseTest(ResponseTest):
         cmap = CorrectMap()
         for i, answer_id in enumerate(answer_ids):
             queuekey = 1000 + i
-            latest_timestamp = datetime.now()
+            latest_timestamp = datetime.now(UTC)
             queuestate = CodeResponseTest.make_queuestate(queuekey, latest_timestamp)
             cmap.update(CorrectMap(answer_id=answer_id, queuestate=queuestate))
         self.problem.correct_map.update(cmap)
 
         # Queue state only tracks up to second
-        latest_timestamp = datetime.strptime(datetime.strftime(latest_timestamp, dateformat), dateformat)
+        latest_timestamp = datetime.strptime(
+            datetime.strftime(latest_timestamp, dateformat), dateformat
+        ).replace(tzinfo=UTC)
 
         self.assertEquals(self.problem.get_recentmost_queuetime(), latest_timestamp)
 

common/lib/capa/capa/xqueue_interface.py

@@ -30,9 +30,11 @@ def make_xheader(lms_callback_url, lms_key, queue_name):
           'queue_name': designate a specific queue within xqueue server, e.g. 'MITx-6.00x' (string)
         }
     """
-    return json.dumps({'lms_callback_url': lms_callback_url,
-                        'lms_key': lms_key,
-                        'queue_name': queue_name})
+    return json.dumps({
+        'lms_callback_url': lms_callback_url,
+        'lms_key': lms_key,
+        'queue_name': queue_name
+    })
 
 
 def parse_xreply(xreply):
@@ -60,7 +62,7 @@ class XQueueInterface(object):
     '''
 
     def __init__(self, url, django_auth, requests_auth=None):
-        self.url  = url
+        self.url = url
         self.auth = django_auth
         self.session = requests.session(auth=requests_auth)
 
@@ -95,13 +97,13 @@ class XQueueInterface(object):
 
         return (error, msg)
 
-
     def _login(self):
-        payload = {'username': self.auth['username'],
-                    'password': self.auth['password']}
+        payload = {
+            'username': self.auth['username'],
+            'password': self.auth['password']
+        }
         return self._http_post(self.url + '/xqueue/login/', payload)
 
-
     def _send_to_queue(self, header, body, files_to_upload):
         payload = {'xqueue_header': header,
                    'xqueue_body': body}
@@ -112,7 +114,6 @@ class XQueueInterface(object):
 
         return self._http_post(self.url + '/xqueue/submit/', payload, files=files)
 
-
     def _http_post(self, url, data, files=None):
         try:
             r = self.session.post(url, data=data, files=files)

common/lib/xmodule/xmodule/capa_module.py

@@ -309,7 +309,13 @@ class CapaModule(CapaFields, XModule):
         d = self.get_score()
         score = d['score']
         total = d['total']
+
         if total > 0:
+            if self.weight is not None:
+                # scale score and total by weight/total:
+                score = score * self.weight / total
+                total = self.weight
+
             try:
                 return Progress(score, total)
             except (TypeError, ValueError):
@@ -321,11 +327,13 @@ class CapaModule(CapaFields, XModule):
         """
         Return some html with data about the module
         """
+        progress = self.get_progress()
         return self.system.render_template('problem_ajax.html', {
             'element_id': self.location.html_id(),
             'id': self.id,
             'ajax_url': self.system.ajax_url,
-            'progress': Progress.to_js_status_str(self.get_progress())
+            'progress_status': Progress.to_js_status_str(progress),
+            'progress_detail': Progress.to_js_detail_str(progress),
         })
 
     def check_button_name(self):
@@ -485,8 +493,7 @@ class CapaModule(CapaFields, XModule):
         """
         Return html for the problem.
 
-        Adds check, reset, save buttons as necessary based on the problem config
-        and state.
+        Adds check, reset, save buttons as necessary based on the problem config and state.
         """
 
         try:
@@ -516,13 +523,12 @@ class CapaModule(CapaFields, XModule):
                    'reset_button': self.should_show_reset_button(),
                    'save_button': self.should_show_save_button(),
                    'answer_available': self.answer_available(),
-                   'ajax_url': self.system.ajax_url,
                    'attempts_used': self.attempts,
                    'attempts_allowed': self.max_attempts,
-                   'progress': self.get_progress(),
                    }
 
         html = self.system.render_template('problem.html', context)
+
         if encapsulate:
             html = u'<div id="problem_{id}" class="problem" data-url="{ajax_url}">'.format(
                 id=self.location.html_id(), ajax_url=self.system.ajax_url
@@ -584,6 +590,7 @@ class CapaModule(CapaFields, XModule):
         result.update({
             'progress_changed': after != before,
             'progress_status': Progress.to_js_status_str(after),
+            'progress_detail': Progress.to_js_detail_str(after),
         })
 
         return json.dumps(result, cls=ComplexEncoder)
@@ -614,6 +621,7 @@ class CapaModule(CapaFields, XModule):
         Problem can be completely wrong.
         Pressing RESET button makes this function to return False.
         """
+        # used by conditional module
         return self.lcp.done
 
     def is_attempted(self):
@@ -757,6 +765,7 @@ class CapaModule(CapaFields, XModule):
         """
         return {'html': self.get_problem_html(encapsulate=False)}
 
+
     @staticmethod
     def make_dict_of_responses(data):
         """
@@ -1117,8 +1126,12 @@ class CapaDescriptor(CapaFields, RawDescriptor):
     mako_template = "widgets/problem-edit.html"
     js = {'coffee': [resource_string(__name__, 'js/src/problem/edit.coffee')]}
     js_module_name = "MarkdownEditingDescriptor"
-    css = {'scss': [resource_string(__name__, 'css/editor/edit.scss'),
-                    resource_string(__name__, 'css/problem/edit.scss')]}
+    css = {
+        'scss': [
+            resource_string(__name__, 'css/editor/edit.scss'),
+            resource_string(__name__, 'css/problem/edit.scss')
+        ]
+    }
 
     # Capa modules have some additional metadata:
     # TODO (vshnayder): do problems have any other metadata?  Do they
@@ -1146,19 +1159,6 @@ class CapaDescriptor(CapaFields, RawDescriptor):
             path[8:],
         ]
 
-    @classmethod
-    def from_xml(cls, xml_data, system, org=None, course=None):
-        """
-        Augment regular translation w/ setting the pre-Studio defaults.
-        """
-        problem = super(CapaDescriptor, cls).from_xml(xml_data, system, org, course)
-        # pylint: disable=W0212
-        if 'showanswer' not in problem._model_data:
-            problem.showanswer = "closed"
-        if 'rerandomize' not in problem._model_data:
-            problem.rerandomize = "always"
-        return problem
-
     @property
     def non_editable_metadata_fields(self):
         non_editable_fields = super(CapaDescriptor, self).non_editable_metadata_fields

common/lib/xmodule/xmodule/course_module.py

@@ -15,6 +15,7 @@ import json
 
 from xblock.core import Scope, List, String, Dict, Boolean
 from .fields import Date
+from xmodule.modulestore.locator import CourseLocator
 from django.utils.timezone import UTC
 from xmodule.util import date_utils
 
@@ -192,9 +193,8 @@ class CourseFields(object):
             }},
             scope=Scope.content)
     show_calculator = Boolean(help="Whether to show the calculator in this course", default=False, scope=Scope.settings)
-    display_name = String(
-        help="Display name for this module", default="Empty",
-        display_name="Display Name", scope=Scope.settings)
+    display_name = String(help="Display name for this module", default="Empty", display_name="Display Name", scope=Scope.settings)
+    show_chat = Boolean(help="Whether to show the chat widget in this course", default=False, scope=Scope.settings)
     tabs = List(help="List of tabs to enable in this course", scope=Scope.settings)
     end_of_course_survey_url = String(help="Url for the end-of-course survey", scope=Scope.settings)
     discussion_blackouts = List(help="List of pairs of start/end dates for discussion blackouts", scope=Scope.settings)
@@ -373,7 +373,10 @@ class CourseDescriptor(CourseFields, SequenceDescriptor):
         super(CourseDescriptor, self).__init__(*args, **kwargs)
 
         if self.wiki_slug is None:
-            self.wiki_slug = self.location.course
+            if isinstance(self.location, Location):
+                self.wiki_slug = self.location.course
+            elif isinstance(self.location, CourseLocator):
+                self.wiki_slug = self.location.course_id or self.display_name
 
         msg = None
 
@@ -407,7 +410,7 @@ class CourseDescriptor(CourseFields, SequenceDescriptor):
                     continue
 
         # TODO check that this is still needed here and can't be by defaults.
-        if self.tabs is None:
+        if not self.tabs:
             # When calling the various _tab methods, can omit the 'type':'blah' from the
             # first arg, since that's only used for dispatch
             tabs = []

common/lib/xmodule/xmodule/css/capa/display.scss

@@ -3,6 +3,7 @@ h2 {
   margin-bottom: 15px;
 
   &.problem-header {
+    display: inline-block;
     section.staff {
       margin-top: 30px;
       font-size: 80%;
@@ -28,6 +29,13 @@ iframe[seamless]{
   color: darken($error-red, 11%);
 }
 
+section.problem-progress {
+  display: inline-block;
+  color: #999;
+  font-size: em(16);
+  font-weight: 100;
+  padding-left: 5px;
+}
 
 section.problem {
   @media print {

common/lib/xmodule/xmodule/css/sequence/display.scss

@@ -211,6 +211,8 @@ nav.sequence-nav {
           @include transition(all .1s $ease-in-out-quart 0s);
           white-space: pre;
           z-index: 99;
+          visibility: hidden;
+          pointer-events: none;
 
           &:empty {
             background: none;
@@ -238,6 +240,7 @@ nav.sequence-nav {
             display: block;
             margin-top: 4px;
             opacity: 1.0;
+            visibility: visible;
           }
         }
       }

common/lib/xmodule/xmodule/discussion_module.py

@@ -12,10 +12,14 @@ class DiscussionFields(object):
     display_name = String(
         display_name="Display Name",
         help="Display name for this module",
-        default="Discussion Tag",
-        scope=Scope.settings)
-    data = String(help="XML data for the problem", scope=Scope.content,
-        default="<discussion></discussion>")
+        default="Discussion",
+        scope=Scope.settings
+    )
+    data = String(
+        help="XML data for the problem",
+        scope=Scope.content,
+        default="<discussion></discussion>"
+    )
     discussion_category = String(
         display_name="Category",
         default="Week 1",

common/lib/xmodule/xmodule/error_module.py

@@ -79,8 +79,10 @@ class ErrorDescriptor(ErrorFields, JSONEditingDescriptor):
     @classmethod
     def _construct(cls, system, contents, error_msg, location):
 
-        if location.name is None:
-            location = location._replace(
+        if isinstance(location, dict) and 'course' in location:
+            location = Location(location)
+        if isinstance(location, Location) and location.name is None:
+            location = location.replace(
                 category='error',
                 # Pick a unique url_name -- the sha1 hash of the contents.
                 # NOTE: We could try to pull out the url_name of the errored descriptor,
@@ -94,7 +96,7 @@ class ErrorDescriptor(ErrorFields, JSONEditingDescriptor):
         model_data = {
             'error_msg': str(error_msg),
             'contents': contents,
-            'display_name': 'Error: ' + location.name,
+            'display_name': 'Error: ' + location.url(),
             'location': location,
             'category': 'error'
         }

common/lib/xmodule/xmodule/fields.py

@@ -80,6 +80,7 @@ class Date(ModelType):
 
 TIMEDELTA_REGEX = re.compile(r'^((?P<days>\d+?) day(?:s?))?(\s)?((?P<hours>\d+?) hour(?:s?))?(\s)?((?P<minutes>\d+?) minute(?:s)?)?(\s)?((?P<seconds>\d+?) second(?:s)?)?$')
 
+
 class Timedelta(ModelType):
     def from_json(self, time_str):
         """

common/lib/xmodule/xmodule/foldit_module.py

@@ -91,15 +91,18 @@ class FolditModule(FolditFields, XModule):
             PuzzleComplete.completed_puzzles(self.system.anonymous_student_id),
             key=lambda d: (d['set'], d['subset']))
 
-    def puzzle_leaders(self, n=10):
+    def puzzle_leaders(self, n=10, courses=None):
         """
         Returns a list of n pairs (user, score) corresponding to the top
         scores; the pairs are in descending order of score.
         """
         from foldit.models import Score
 
-        leaders = [(e['username'], e['score']) for e in Score.get_tops_n(10)]
-        leaders.sort(key=lambda x:-x[1])
+        if courses is None:
+            courses = [self.location.course_id]
+
+        leaders = [(leader['username'], leader['score']) for leader in Score.get_tops_n(10, course_list=courses)]
+        leaders.sort(key=lambda x: -x[1])
 
         return leaders
 

common/lib/xmodule/xmodule/gst_module.py

@@ -19,10 +19,45 @@ from xblock.core import String, Scope
 
 log = logging.getLogger(__name__)
 
+DEFAULT_RENDER="""
+    <h2>Graphic slider tool: Dynamic range and implicit functions.</h2>
+
+    <p>You can make the range of the x axis (but not ticks of x axis) of
+      functions depend on a parameter value. This can be useful when the
+      function domain needs to be variable.</p>
+    <p>Implicit functions like a circle can be plotted as 2 separate
+        functions of the same color.</p>
+     <div style="height:50px;">
+     <slider var='r' style="width:400px;float:left;"/>
+     <textbox var='r' style="float:left;width:60px;margin-left:15px;"/>
+   </div>
+    <plot style="margin-top:15px;margin-bottom:15px;"/>
+"""
+DEFAULT_CONFIGURATION="""
+    <parameters>
+        <param var="r" min="5" max="25" step="0.5" initial="12.5" />
+    </parameters>
+    <functions>
+      <function color="red">Math.sqrt(r * r - x * x)</function>
+      <function color="red">-Math.sqrt(r * r - x * x)</function>
+    </functions>
+    <plot>
+      <xrange>
+        <!-- dynamic range -->
+          <min>-r</min>
+          <max>r</max>
+      </xrange>
+      <num_points>1000</num_points>
+      <xticks>-30, 6, 30</xticks>
+      <yticks>-30, 6, 30</yticks>
+    </plot>
+"""
+
+
 
 class GraphicalSliderToolFields(object):
-    render = String(scope=Scope.content)
-    configuration = String(scope=Scope.content)
+    render = String(scope=Scope.content, default=DEFAULT_RENDER)
+    configuration = String(scope=Scope.content, default=DEFAULT_CONFIGURATION)
 
 
 class GraphicalSliderToolModule(GraphicalSliderToolFields, XModule):

common/lib/xmodule/xmodule/html_module.py

@@ -25,9 +25,9 @@ class HtmlFields(object):
         scope=Scope.settings,
         # it'd be nice to have a useful default but it screws up other things; so,
         # use display_name_with_default for those
-        default="Blank HTML Page"
+        default="Text"
     )
-    data = String(help="Html contents to display for this module", default="", scope=Scope.content)
+    data = String(help="Html contents to display for this module", default=u"", scope=Scope.content)
     source_code = String(help="Source code for LaTeX documents. This feature is not well-supported.", scope=Scope.settings)
 
 

common/lib/xmodule/xmodule/js/fixtures/problem.html

 <section class='xmodule_display xmodule_CapaModule' data-type='Problem'>
   <section id='problem_1' 
-           class='problems-wrapper' 
+           class='problems-wrapper'
            data-problem-id='i4x://edX/101/problem/Problem1' 
            data-url='/problem/Problem1'>
   </section>

common/lib/xmodule/xmodule/js/fixtures/problem_content.html

 <h2 class="problem-header">Problem Header</h2>
 
+<section class='problem-progress'>
+</section>
+
 <section class="problem">
   <p>Problem Content</p>
 

common/lib/xmodule/xmodule/js/spec/capa/display_spec.coffee

@@ -77,6 +77,25 @@ describe 'Problem', ->
         [@problem.updateMathML, @stubbedJax, $('#input_example_1').get(0)]
       ]
 
+  describe 'renderProgressState', ->
+    beforeEach ->
+      @problem = new Problem($('.xmodule_display'))
+      #@renderProgressState = @problem.renderProgressState
+
+    describe 'with a status of "none"', ->
+      it 'reports the number of points possible', ->
+        @problem.el.data('progress_status', 'none')
+        @problem.el.data('progress_detail', '0/1')
+        @problem.renderProgressState()
+        expect(@problem.$('.problem-progress').html()).toEqual "(1 point possible)"
+
+    describe 'with any other valid status', ->
+      it 'reports the current score', ->
+        @problem.el.data('progress_status', 'foo')
+        @problem.el.data('progress_detail', '1/1')
+        @problem.renderProgressState()
+        expect(@problem.$('.problem-progress').html()).toEqual "(1/1 points)"
+
   describe 'render', ->
     beforeEach ->
       @problem = new Problem($('.xmodule_display'))

common/lib/xmodule/xmodule/js/src/capa/display.coffee

@@ -35,15 +35,34 @@ class @Problem
       @$('input.math').each (index, element) =>
         MathJax.Hub.Queue [@refreshMath, null, element]
 
+  renderProgressState: =>
+    detail = @el.data('progress_detail')
+    status = @el.data('progress_status')
+    # i18n
+    progress = "(#{detail} points)"
+    if status == 'none' and detail? and detail.indexOf('/') > 0
+        a = detail.split('/')
+        possible = parseInt(a[1])
+        if possible == 1
+            # i18n
+            progress = "(#{possible} point possible)"
+        else
+            # i18n
+            progress = "(#{possible} points possible)"
+    @$('.problem-progress').html(progress)
+
   updateProgress: (response) =>
     if response.progress_changed
-        @el.attr progress: response.progress_status
+        @el.data('progress_status', response.progress_status)
+        @el.data('progress_detail', response.progress_detail)
         @el.trigger('progressChanged')
+    @renderProgressState()
 
   forceUpdate: (response) =>
-    @el.attr progress: response.progress_status
+    @el.data('progress_status', response.progress_status)
+    @el.data('progress_detail', response.progress_detail)
     @el.trigger('progressChanged')
-
+    @renderProgressState()
 
   queueing: =>
     @queued_items = @$(".xqueue")
@@ -113,7 +132,7 @@ class @Problem
           @setupInputTypes()
           @bind()
           @queueing()
-
+          @forceUpdate response
 
   # TODO add hooks for problem types here by inspecting response.html and doing
   # stuff if a div w a class is found

common/lib/xmodule/xmodule/js/src/sequence/display.coffee

@@ -45,7 +45,7 @@ class @Sequence
     new_progress = "NA"
     _this = this
     $('.problems-wrapper').each (index) ->
-      progress = $(this).attr 'progress'
+      progress = $(this).data 'progress_status'
       new_progress = _this.mergeProgress progress, new_progress
 
     @progressTable[@position] = new_progress

common/lib/xmodule/xmodule/modulestore/exceptions.py

@@ -7,10 +7,18 @@ class ItemNotFoundError(Exception):
     pass
 
 
+class ItemWriteConflictError(Exception):
+    pass
+
+
 class InsufficientSpecificationError(Exception):
     pass
 
 
+class OverSpecificationError(Exception):
+    pass
+
+
 class InvalidLocationError(Exception):
     pass
 
@@ -21,3 +29,13 @@ class NoPathToItem(Exception):
 
 class DuplicateItemError(Exception):
     pass
+
+
+class VersionConflictError(Exception):
+    """
+    The caller asked for either draft or published head and gave a version which conflicted with it.
+    """
+    def __init__(self, requestedLocation, currentHead):
+        super(VersionConflictError, self).__init__()
+        self.requestedLocation = requestedLocation
+        self.currentHead = currentHead

common/lib/xmodule/xmodule/modulestore/inheritance.py

@@ -50,6 +50,8 @@ def inherit_metadata(descriptor, model_data):
 
 
 def own_metadata(module):
+    # IN SPLIT MONGO this is just ['metadata'] as it keeps ['_inherited_metadata'] separate!
+    # FIXME move into kvs? will that work for xml mongo?
     """
     Return a dictionary that contains only non-inherited field keys,
     mapped to their values

common/lib/xmodule/xmodule/modulestore/mongo/base.py

@@ -105,15 +105,6 @@ class MongoKeyValueStore(KeyValueStore):
         else:
             raise InvalidScopeError(key.scope)
 
-    def set_many(self, update_dict):
-        """set_many method. Implementations should accept an `update_dict` of
-        key-value pairs, and set all the `keys` to the given `value`s."""
-        # `set` simply updates an in-memory db, rather than calling down to a real db,
-        # as mongo bulk save is handled elsewhere. A future improvement would be to pull
-        # the mongo-specific bulk save logic into this method.
-        for key, value in update_dict.iteritems():
-            self.set(key, value)
-
     def delete(self, key):
         if key.scope == Scope.children:
             self._children = []

common/lib/xmodule/xmodule/modulestore/parsers.py

+import re
+
+URL_RE = re.compile(r'^edx://(.+)$', re.IGNORECASE)
+
+
+def parse_url(string):
+    """
+    A url must begin with 'edx://' (case-insensitive match),
+    followed by either a version_guid or a course_id.
+
+    Examples:
+        'edx://@0123FFFF'
+        'edx://edu.mit.eecs.6002x'
+        'edx://edu.mit.eecs.6002x;published'
+        'edx://edu.mit.eecs.6002x;published#HW3'
+
+    This returns None if string cannot be parsed.
+
+    If it can be parsed as a version_guid, returns a dict
+    with key 'version_guid' and the value,
+
+    If it can be parsed as a course_id, returns a dict
+    with keys 'id' and 'revision' (value of 'revision' may be None),
+
+    """
+    match = URL_RE.match(string)
+    if not match:
+        return None
+    path = match.group(1)
+    if path[0] == '@':
+        return parse_guid(path[1:])
+    return parse_course_id(path)
+
+
+BLOCK_RE = re.compile(r'^\w+$', re.IGNORECASE)
+
+
+def parse_block_ref(string):
+    r"""
+    A block_ref is a string of word_chars.
+
+    <word_chars> matches one or more Unicode word characters; this includes most
+    characters that can be part of a word in any language, as well as numbers
+    and the underscore. (see definition of \w in python regular expressions,
+    at http://docs.python.org/dev/library/re.html)
+
+    If string is a block_ref, returns a dict with key 'block_ref' and the value,
+    otherwise returns None.
+    """
+    if len(string) > 0 and BLOCK_RE.match(string):
+        return {'block': string}
+    return None
+
+
+GUID_RE = re.compile(r'^(?P<version_guid>[A-F0-9]+)(#(?P<block>\w+))?$', re.IGNORECASE)
+
+
+def parse_guid(string):
+    """
+    A version_guid is a string of hex digits (0-F).
+
+    If string is a version_guid, returns a dict with key 'version_guid' and the value,
+    otherwise returns None.
+    """
+    m = GUID_RE.match(string)
+    if m is not None:
+        return m.groupdict()
+    else:
+        return None
+
+
+COURSE_ID_RE = re.compile(r'^(?P<id>(\w+)(\.\w+\w*)*)(;(?P<revision>\w+))?(#(?P<block>\w+))?$', re.IGNORECASE)
+
+
+def parse_course_id(string):
+    r"""
+
+    A course_id has a main id component.
+    There may also be an optional revision (;published or ;draft).
+    There may also be an optional block (#HW3 or #Quiz2).
+
+    Examples of valid course_ids:
+
+      'edu.mit.eecs.6002x'
+      'edu.mit.eecs.6002x;published'
+      'edu.mit.eecs.6002x#HW3'
+      'edu.mit.eecs.6002x;published#HW3'
+
+
+    Syntax:
+
+      course_id = main_id [; revision] [# block]
+
+      main_id = name [. name]*
+
+      revision = name
+
+      block = name
+
+      name = <word_chars>
+
+    <word_chars> matches one or more Unicode word characters; this includes most
+    characters that can be part of a word in any language, as well as numbers
+    and the underscore. (see definition of \w in python regular expressions,
+    at http://docs.python.org/dev/library/re.html)
+
+    If string is a course_id, returns a dict with keys 'id', 'revision', and 'block'.
+    Revision is optional: if missing returned_dict['revision'] is None.
+    Block is optional: if missing returned_dict['block'] is None.
+    Else returns None.
+    """
+    match = COURSE_ID_RE.match(string)
+    if not match:
+        return None
+    return match.groupdict()

common/lib/xmodule/xmodule/modulestore/split_mongo/__init__.py

+from split import SplitMongoModuleStore

common/lib/xmodule/xmodule/modulestore/split_mongo/caching_descriptor_system.py

+import sys
+import logging
+from xmodule.mako_module import MakoDescriptorSystem
+from xmodule.x_module import XModuleDescriptor
+from xmodule.modulestore.locator import BlockUsageLocator
+from xmodule.error_module import ErrorDescriptor
+from xmodule.errortracker import exc_info_to_str
+from xblock.runtime import DbModel
+from ..exceptions import ItemNotFoundError
+from .split_mongo_kvs import SplitMongoKVS, SplitMongoKVSid
+
+log = logging.getLogger(__name__)
+
+# TODO should this be here or w/ x_module or ???
+class CachingDescriptorSystem(MakoDescriptorSystem):
+    """
+    A system that has a cache of a course version's json that it will use to load modules
+    from, with a backup of calling to the underlying modulestore for more data.
+
+    Computes the metadata inheritance upon creation.
+    """
+    def __init__(self, modulestore, course_entry, module_data, lazy,
+        default_class, error_tracker, render_template):
+        """
+        Computes the metadata inheritance and sets up the cache.
+
+        modulestore: the module store that can be used to retrieve additional
+        modules
+
+        module_data: a dict mapping Location -> json that was cached from the
+            underlying modulestore
+
+        default_class: The default_class to use when loading an
+            XModuleDescriptor from the module_data
+
+        resources_fs: a filesystem, as per MakoDescriptorSystem
+
+        error_tracker: a function that logs errors for later display to users
+
+        render_template: a function for rendering templates, as per
+            MakoDescriptorSystem
+        """
+        # TODO find all references to resources_fs and make handle None
+        super(CachingDescriptorSystem, self).__init__(
+                self._load_item, None, error_tracker, render_template)
+        self.modulestore = modulestore
+        self.course_entry = course_entry
+        self.lazy = lazy
+        self.module_data = module_data
+        self.default_class = default_class
+        # TODO see if self.course_id is needed: is already in course_entry but could be > 1 value
+        # Compute inheritance
+        modulestore.inherit_metadata(course_entry.get('blocks', {}),
+            course_entry.get('blocks', {})
+            .get(course_entry.get('root')))
+
+    def _load_item(self, usage_id, course_entry_override=None):
+        # TODO ensure all callers of system.load_item pass just the id
+        json_data = self.module_data.get(usage_id)
+        if json_data is None:
+            # deeper than initial descendant fetch or doesn't exist
+            self.modulestore.cache_items(self, [usage_id], lazy=self.lazy)
+            json_data = self.module_data.get(usage_id)
+            if json_data is None:
+                raise ItemNotFoundError
+
+        class_ = XModuleDescriptor.load_class(
+            json_data.get('category'),
+            self.default_class
+        )
+        return self.xblock_from_json(class_, usage_id, json_data, course_entry_override)
+
+    def xblock_from_json(self, class_, usage_id, json_data, course_entry_override=None):
+        if course_entry_override is None:
+            course_entry_override = self.course_entry
+        # most likely a lazy loader but not the id directly
+        definition = json_data.get('definition', {})
+        metadata = json_data.get('metadata', {})
+
+        block_locator = BlockUsageLocator(
+            version_guid=course_entry_override['_id'],
+            usage_id=usage_id,
+            course_id=course_entry_override.get('course_id'),
+            revision=course_entry_override.get('revision')
+        )
+
+        kvs = SplitMongoKVS(
+            definition,
+            json_data.get('children', []),
+            metadata,
+            json_data.get('_inherited_metadata'),
+            block_locator,
+            json_data.get('category'))
+        model_data = DbModel(kvs, class_, None,
+            SplitMongoKVSid(
+                # DbModel req's that these support .url()
+                block_locator,
+                self.modulestore.definition_locator(definition)))
+
+        try:
+            module = class_(self, model_data)
+        except Exception:
+            log.warning("Failed to load descriptor", exc_info=True)
+            if usage_id is None:
+                usage_id = "MISSING"
+            return ErrorDescriptor.from_json(
+                json_data,
+                self,
+                BlockUsageLocator(version_guid=course_entry_override['_id'],
+                    usage_id=usage_id),
+                error_msg=exc_info_to_str(sys.exc_info())
+            )
+
+        module.edited_by = json_data.get('edited_by')
+        module.edited_on = json_data.get('edited_on')
+        module.previous_version = json_data.get('previous_version')
+        module.update_version = json_data.get('update_version')
+        module.definition_locator = self.modulestore.definition_locator(definition)
+        return module

common/lib/xmodule/xmodule/modulestore/split_mongo/definition_lazy_loader.py

+from xmodule.modulestore.locator import DescriptionLocator
+
+
+class DefinitionLazyLoader(object):
+    """
+    A placeholder to put into an xblock in place of its definition which
+    when accessed knows how to get its content. Only useful if the containing
+    object doesn't force access during init but waits until client wants the
+    definition. Only works if the modulestore is a split mongo store.
+    """
+    def __init__(self, modulestore, definition_id):
+        """
+        Simple placeholder for yet-to-be-fetched data
+        :param modulestore: the pymongo db connection with the definitions
+        :param definition_locator: the id of the record in the above to fetch
+        """
+        self.modulestore = modulestore
+        self.definition_locator = DescriptionLocator(definition_id)
+
+    def fetch(self):
+        """
+        Fetch the definition. Note, the caller should replace this lazy
+        loader pointer with the result so as not to fetch more than once
+        """
+        return self.modulestore.definitions.find_one(
+            {'_id': self.definition_locator.definition_id})

common/lib/xmodule/xmodule/modulestore/split_mongo/split_mongo_kvs.py

+import copy
+from xblock.core import Scope
+from collections import namedtuple
+from xblock.runtime import KeyValueStore, InvalidScopeError
+from .definition_lazy_loader import DefinitionLazyLoader
+
+# id is a BlockUsageLocator, def_id is the definition's guid
+SplitMongoKVSid = namedtuple('SplitMongoKVSid', 'id, def_id')
+
+
+# TODO should this be here or w/ x_module or ???
+class SplitMongoKVS(KeyValueStore):
+    """
+    A KeyValueStore that maps keyed data access to one of the 3 data areas
+    known to the MongoModuleStore (data, children, and metadata)
+    """
+    def __init__(self, definition, children, metadata, _inherited_metadata, location, category):
+        """
+
+        :param definition:
+        :param children:
+        :param metadata: the locally defined value for each metadata field
+        :param _inherited_metadata: the value of each inheritable field from above this.
+            Note, metadata may override and disagree w/ this b/c this says what the value
+            should be if metadata is undefined for this field.
+        """
+        # ensure kvs's don't share objects w/ others so that changes can't appear in separate ones
+        # the particular use case was that changes to kvs's were polluting caches. My thinking was
+        # that kvs's should be independent thus responsible for the isolation.
+        if isinstance(definition, DefinitionLazyLoader):
+            self._definition = definition
+        else:
+            self._definition = copy.copy(definition)
+        self._children = copy.copy(children)
+        self._metadata = copy.copy(metadata)
+        self._inherited_metadata = _inherited_metadata
+        self._location = location
+        self._category = category
+
+    def get(self, key):
+        if key.scope == Scope.children:
+            return self._children
+        elif key.scope == Scope.parent:
+            return None
+        elif key.scope == Scope.settings:
+            if key.field_name in self._metadata:
+                return self._metadata[key.field_name]
+            elif key.field_name in self._inherited_metadata:
+                return self._inherited_metadata[key.field_name]
+            else:
+                raise KeyError()
+        elif key.scope == Scope.content:
+            if key.field_name == 'location':
+                return self._location
+            elif key.field_name == 'category':
+                return self._category
+            else:
+                if isinstance(self._definition, DefinitionLazyLoader):
+                    self._definition = self._definition.fetch()
+                if (key.field_name == 'data' and
+                        not isinstance(self._definition.get('data'), dict)):
+                    return self._definition.get('data')
+                elif 'data' not in self._definition or key.field_name not in self._definition['data']:
+                    raise KeyError()
+                else:
+                    return self._definition['data'][key.field_name]
+        else:
+            raise InvalidScopeError(key.scope)
+
+    def set(self, key, value):
+        # TODO cache db update implications & add method to invoke
+        if key.scope == Scope.children:
+            self._children = value
+            # TODO remove inheritance from any orphaned exchildren
+            # TODO add inheritance to any new children
+        elif key.scope == Scope.settings:
+            # TODO if inheritable, push down to children who don't override
+            self._metadata[key.field_name] = value
+        elif key.scope == Scope.content:
+            if key.field_name == 'location':
+                self._location = value
+            elif key.field_name == 'category':
+                self._category = value
+            else:
+                if isinstance(self._definition, DefinitionLazyLoader):
+                    self._definition = self._definition.fetch()
+                if (key.field_name == 'data' and
+                        not isinstance(self._definition.get('data'), dict)):
+                    self._definition.get('data')
+                else:
+                    self._definition.setdefault('data', {})[key.field_name] = value
+        else:
+            raise InvalidScopeError(key.scope)
+
+    def delete(self, key):
+        # TODO cache db update implications & add method to invoke
+        if key.scope == Scope.children:
+            self._children = []
+        elif key.scope == Scope.settings:
+            # TODO if inheritable, ensure _inherited_metadata has value from above and
+            # revert children to that value
+            if key.field_name in self._metadata:
+                del self._metadata[key.field_name]
+        elif key.scope == Scope.content:
+            # don't allow deletion of location nor category
+            if key.field_name == 'location':
+                pass
+            elif key.field_name == 'category':
+                pass
+            else:
+                if isinstance(self._definition, DefinitionLazyLoader):
+                    self._definition = self._definition.fetch()
+                if (key.field_name == 'data' and
+                        not isinstance(self._definition.get('data'), dict)):
+                    self._definition.setdefault('data', None)
+                else:
+                    try:
+                        del self._definition['data'][key.field_name]
+                    except KeyError:
+                        pass
+        else:
+            raise InvalidScopeError(key.scope)
+
+    def has(self, key):
+        if key.scope in (Scope.children, Scope.parent):
+            return True
+        elif key.scope == Scope.settings:
+            return key.field_name in self._metadata or key.field_name in self._inherited_metadata
+        elif key.scope == Scope.content:
+            if key.field_name == 'location':
+                return True
+            elif key.field_name == 'category':
+                return self._category is not None
+            else:
+                if isinstance(self._definition, DefinitionLazyLoader):
+                    self._definition = self._definition.fetch()
+                if (key.field_name == 'data' and
+                        not isinstance(self._definition.get('data'), dict)):
+                    return self._definition.get('data') is not None
+                else:
+                    return key.field_name in self._definition.get('data', {})
+        else:
+            return False
+
+    def get_data(self):
+        """
+        Intended only for use by persistence layer to get the native definition['data'] rep
+        """
+        if isinstance(self._definition, DefinitionLazyLoader):
+            self._definition = self._definition.fetch()
+        return self._definition.get('data')
+
+    def get_own_metadata(self):
+        """
+        Get the metadata explicitly set on this element.
+        """
+        return self._metadata
+
+    def get_inherited_metadata(self):
+        """
+        Get the metadata set by the ancestors (which own metadata may override or not)
+        """
+        return self._inherited_metadata

common/lib/xmodule/xmodule/modulestore/tests/factories.py

@@ -38,16 +38,6 @@ class XModuleCourseFactory(Factory):
             new_course.display_name = display_name
 
         new_course.lms.start = datetime.datetime.now(UTC).replace(microsecond=0)
-        new_course.tabs = kwargs.pop(
-            'tabs',
-            [
-                {"type": "courseware"},
-                {"type": "course_info", "name": "Course Info"},
-                {"type": "discussion", "name": "Discussion"},
-                {"type": "wiki", "name": "Wiki"},
-                {"type": "progress", "name": "Progress"}
-            ]
-        )
 
         # The rest of kwargs become attributes on the course:
         for k, v in kwargs.iteritems():

common/lib/xmodule/xmodule/modulestore/tests/persistent_factories.py

+from xmodule.modulestore.django import modulestore
+from xmodule.course_module import CourseDescriptor
+from xmodule.x_module import XModuleDescriptor
+import factory
+
+
+# [dhm] I'm not sure why we're using factory_boy if we're not following its pattern. If anyone
+# assumes they can call build, it will completely fail, for example.
+# pylint: disable=W0232
+class PersistentCourseFactory(factory.Factory):
+    """
+    Create a new course (not a new version of a course, but a whole new index entry).
+
+    keywords:
+    * org: defaults to textX
+    * prettyid: defaults to 999
+    * display_name
+    * user_id
+    * data (optional) the data payload to save in the course item
+    * metadata (optional) the metadata payload. If display_name is in the metadata, that takes
+    precedence over any display_name provided directly.
+    """
+    FACTORY_FOR = CourseDescriptor
+
+    org = 'testX'
+    prettyid = '999'
+    display_name = 'Robot Super Course'
+    user_id = "test_user"
+    data = None
+    metadata = None
+    master_version = 'draft'
+
+    # pylint: disable=W0613
+    @classmethod
+    def _create(cls, target_class, *args, **kwargs):
+
+        org = kwargs.get('org')
+        prettyid = kwargs.get('prettyid')
+        display_name = kwargs.get('display_name')
+        user_id = kwargs.get('user_id')
+        data = kwargs.get('data')
+        metadata = kwargs.get('metadata', {})
+        if metadata is None:
+            metadata = {}
+        if 'display_name' not in metadata:
+            metadata['display_name'] = display_name
+
+        # Write the data to the mongo datastore
+        new_course = modulestore('split').create_course(
+            org, prettyid, user_id, metadata=metadata, course_data=data, id_root=prettyid,
+            master_version=kwargs.get('master_version'))
+
+        return new_course
+
+    @classmethod
+    def _build(cls, target_class, *args, **kwargs):
+        raise NotImplementedError()
+
+
+class ItemFactory(factory.Factory):
+    FACTORY_FOR = XModuleDescriptor
+
+    category = 'chapter'
+    user_id = 'test_user'
+    display_name = factory.LazyAttributeSequence(lambda o, n: "{} {}".format(o.category, n))
+
+    # pylint: disable=W0613
+    @classmethod
+    def _create(cls, target_class, *args, **kwargs):
+        """
+        Uses *kwargs*:
+
+        *parent_location* (required): the location of the course & possibly parent
+
+        *category* (defaults to 'chapter')
+
+        *data* (optional): the data for the item
+
+        definition_locator (optional): the DescriptorLocator for the definition this uses or branches
+
+        *display_name* (optional): the display name of the item
+
+        *metadata* (optional): dictionary of metadata attributes (display_name here takes
+        precedence over the above attr)
+        """
+        metadata = kwargs.get('metadata', {})
+        if 'display_name' not in metadata and 'display_name' in kwargs:
+            metadata['display_name'] = kwargs['display_name']
+
+        return modulestore('split').create_item(kwargs['parent_location'], kwargs['category'],
+            kwargs['user_id'], definition_locator=kwargs.get('definition_locator'),
+            new_def_data=kwargs.get('data'), metadata=metadata)
+
+    @classmethod
+    def _build(cls, target_class, *args, **kwargs):
+        raise NotImplementedError()

common/lib/xmodule/xmodule/modulestore/xml_importer.py

@@ -138,7 +138,7 @@ def import_module_from_xml(modulestore, static_content_store, course_data_path,
             # For example, what I'm seeing is <img src='foo.jpg' />   ->   <img src='bar.jpg'>
             # Note the dropped element closing tag. This causes the LMS to fail when rendering modules - that's
             # no good, so we have to do this kludge
-            if isinstance(module.data, str) or isinstance(module.data, unicode):   # some module 'data' fields are non strings which blows up the link traversal code
+            if isinstance(module.data, str) or isinstance(module.data, unicode):  # some module 'data' fields are non strings which blows up the link traversal code
                 lxml_rewrite_links(module.data, lambda link: verify_content_links(module, course_data_path, static_content_store, link, remap_dict))
 
                 for key in remap_dict.keys():
@@ -315,7 +315,7 @@ def import_module(module, store, course_data_path, static_content_store, allow_n
             # For example, what I'm seeing is <img src='foo.jpg' />   ->   <img src='bar.jpg'>
             # Note the dropped element closing tag. This causes the LMS to fail when rendering modules - that's
             # no good, so we have to do this kludge
-            if isinstance(module_data, str) or isinstance(module_data, unicode):   # some module 'data' fields are non strings which blows up the link traversal code
+            if isinstance(module_data, str) or isinstance(module_data, unicode):  # some module 'data' fields are non strings which blows up the link traversal code
                 lxml_rewrite_links(module_data, lambda link: verify_content_links(module, course_data_path, static_content_store, link, remap_dict))
 
                 for key in remap_dict.keys():
@@ -523,6 +523,26 @@ def validate_data_source_paths(data_dir, course_dir):
     return err_cnt, warn_cnt
 
 
+def validate_course_policy(module_store, course_id):
+    """
+    Validate that the course explicitly sets values for any fields whose defaults may have changed between
+    the export and the import.
+
+    Does not add to error count as these are just warnings.
+    """
+    # is there a reliable way to get the module location just given the course_id?
+    warn_cnt = 0
+    for module in module_store.modules[course_id].itervalues():
+        if module.location.category == 'course':
+            if not 'rerandomize' in module._model_data:
+                warn_cnt += 1
+                print 'WARN: course policy does not specify value for "rerandomize" whose default is now "never". The behavior of your course may change.'
+            if not 'showanswer' in module._model_data:
+                warn_cnt += 1
+                print 'WARN: course policy does not specify value for "showanswer" whose default is now "finished". The behavior of your course may change.'
+    return warn_cnt
+
+
 def perform_xlint(data_dir, course_dirs,
                   default_class='xmodule.raw_module.RawDescriptor',
                   load_error_modules=True):
@@ -568,6 +588,8 @@ def perform_xlint(data_dir, course_dirs,
         err_cnt += validate_category_hierarchy(module_store, course_id, "chapter", "sequential")
         # constrain that sequentials only have 'verticals'
         err_cnt += validate_category_hierarchy(module_store, course_id, "sequential", "vertical")
+        # validate the course policy overrides any defaults which have changed over time
+        warn_cnt += validate_course_policy(module_store, course_id)
         # don't allow metadata on verticals, since we can't edit them in studio
         err_cnt += validate_no_non_editable_metadata(module_store, course_id, "vertical")
         # don't allow metadata on chapters, since we can't edit them in studio

common/lib/xmodule/xmodule/open_ended_grading_classes/open_ended_module.py

@@ -19,6 +19,7 @@ import openendedchild
 from numpy import median
 
 from datetime import datetime
+from pytz import UTC
 
 from .combined_open_ended_rubric import CombinedOpenEndedRubric
 
@@ -170,7 +171,7 @@ class OpenEndedModule(openendedchild.OpenEndedChild):
         if xqueue is None:
             return {'success': False, 'msg': "Couldn't submit feedback."}
         qinterface = xqueue['interface']
-        qtime = datetime.strftime(datetime.now(), xqueue_interface.dateformat)
+        qtime = datetime.strftime(datetime.now(UTC), xqueue_interface.dateformat)
         anonymous_student_id = system.anonymous_student_id
         queuekey = xqueue_interface.make_hashkey(str(system.seed) + qtime +
                                                  anonymous_student_id +
@@ -224,7 +225,7 @@ class OpenEndedModule(openendedchild.OpenEndedChild):
         if xqueue is None:
             return False
         qinterface = xqueue['interface']
-        qtime = datetime.strftime(datetime.now(), xqueue_interface.dateformat)
+        qtime = datetime.strftime(datetime.now(UTC), xqueue_interface.dateformat)
 
         anonymous_student_id = system.anonymous_student_id
 

common/lib/xmodule/xmodule/open_ended_grading_classes/openendedchild.py

@@ -5,6 +5,7 @@ import re
 
 import open_ended_image_submission
 from xmodule.progress import Progress
+import capa.xqueue_interface as xqueue_interface
 from capa.util import *
 from .peer_grading_service import PeerGradingService, MockPeerGradingService
 import controller_query_service
@@ -334,12 +335,15 @@ class OpenEndedChild(object):
             log.exception("Could not create image and check it.")
 
         if image_ok:
-            image_key = image_data.name + datetime.now().strftime("%Y%m%d%H%M%S")
+            image_key = image_data.name + datetime.now(UTC).strftime(
+                xqueue_interface.dateformat
+            )
 
             try:
                 image_data.seek(0)
-                success, s3_public_url = open_ended_image_submission.upload_to_s3(image_data, image_key,
-                                                                                  self.s3_interface)
+                success, s3_public_url = open_ended_image_submission.upload_to_s3(
+                    image_data, image_key, self.s3_interface
+                )
             except:
                 log.exception("Could not upload image to S3.")
 

common/lib/xmodule/xmodule/progress.py

@@ -146,7 +146,7 @@ class Progress(object):
         sending Progress objects to js to limit dependencies.
         '''
         if progress is None:
-            return "NA"
+            return "0"
         return progress.ternary_str()
 
     @staticmethod
@@ -157,5 +157,5 @@ class Progress(object):
         passing Progress objects to js to limit dependencies.
         '''
         if progress is None:
-            return "NA"
+            return "0"
         return str(progress)

common/lib/xmodule/xmodule/schematic_module.py

-
-from .x_module import XModule, XModuleDescriptor
-
-
-class ModuleDescriptor(XModuleDescriptor):
-    pass
-
-
-class Module(XModule):
-    def get_html(self):
-        return '<input type="hidden" class="schematic" name="{item_id}" height="480" width="640">'.format(item_id=self.item_id)

common/lib/xmodule/xmodule/tests/test_capa_module.py

@@ -1233,6 +1233,37 @@ class CapaModuleTest(unittest.TestCase):
             mock_log.exception.assert_called_once_with('Got bad progress')
             mock_log.reset_mock()
 
+    @patch('xmodule.capa_module.Progress')
+    def test_get_progress_calculate_progress_fraction(self, mock_progress):
+        """
+        Check that score and total are calculated correctly for the progress fraction.
+        """
+        module = CapaFactory.create()
+        module.weight = 1
+        module.get_progress()
+        mock_progress.assert_called_with(0, 1)
+
+        other_module = CapaFactory.create(correct=True)
+        other_module.weight = 1
+        other_module.get_progress()
+        mock_progress.assert_called_with(1, 1)
+
+    def test_get_html(self):
+        """
+        Check that get_html() calls get_progress() with no arguments.
+        """
+        module = CapaFactory.create()
+        module.get_progress = Mock(wraps=module.get_progress)
+        module.get_html()
+        module.get_progress.assert_called_once_with()
+
+    def test_get_problem(self):
+        """
+        Check that get_problem() returns the expected dictionary.
+        """
+        module = CapaFactory.create()
+        self.assertEquals(module.get_problem("data"), {'html': module.get_problem_html(encapsulate=False)})
+
 
 class ComplexEncoderTest(unittest.TestCase):
     def test_default(self):

common/lib/xmodule/xmodule/tests/test_combined_open_ended.py

@@ -14,6 +14,7 @@ from xmodule.modulestore import Location
 from lxml import etree
 import capa.xqueue_interface as xqueue_interface
 from datetime import datetime
+from pytz import UTC
 import logging
 
 log = logging.getLogger(__name__)
@@ -212,7 +213,7 @@ class OpenEndedModuleTest(unittest.TestCase):
                'submission_id': '1',
                'grader_id': '1',
                'score': 3}
-        qtime = datetime.strftime(datetime.now(), xqueue_interface.dateformat)
+        qtime = datetime.strftime(datetime.now(UTC), xqueue_interface.dateformat)
         student_info = {'anonymous_student_id': self.test_system.anonymous_student_id,
                         'submission_time': qtime}
         contents = {
@@ -233,7 +234,7 @@ class OpenEndedModuleTest(unittest.TestCase):
 
     def test_send_to_grader(self):
         submission = "This is a student submission"
-        qtime = datetime.strftime(datetime.now(), xqueue_interface.dateformat)
+        qtime = datetime.strftime(datetime.now(UTC), xqueue_interface.dateformat)
         student_info = {'anonymous_student_id': self.test_system.anonymous_student_id,
                         'submission_time': qtime}
         contents = self.openendedmodule.payload.copy()
@@ -632,6 +633,7 @@ class OpenEndedModuleXmlTest(unittest.TestCase, DummyModulestore):
         module.handle_ajax("reset", {})
         self.assertEqual(module.state, "initial")
 
+
 class OpenEndedModuleXmlAttemptTest(unittest.TestCase, DummyModulestore):
     """
     Test if student is able to reset the problem

common/lib/xmodule/xmodule/tests/test_progress.py

@@ -90,15 +90,15 @@ class ProgressTest(unittest.TestCase):
         self.assertEqual(Progress.to_js_status_str(self.not_started), "none")
         self.assertEqual(Progress.to_js_status_str(self.half_done), "in_progress")
         self.assertEqual(Progress.to_js_status_str(self.done), "done")
-        self.assertEqual(Progress.to_js_status_str(None), "NA")
+        self.assertEqual(Progress.to_js_status_str(None), "0")
 
     def test_to_js_detail_str(self):
         '''Test the Progress.to_js_detail_str() method'''
         f = Progress.to_js_detail_str
         for p in (self.not_started, self.half_done, self.done):
             self.assertEqual(f(p), str(p))
-        # But None should be encoded as NA
-        self.assertEqual(f(None), "NA")
+        # But None should be encoded as 0
+        self.assertEqual(f(None), "0")
 
     def test_add(self):
         '''Test the Progress.add_counts() method'''

common/lib/xmodule/xmodule/tests/test_xblock_wrappers.py

+"""
+Tests for the wrapping layer that provides the XBlock API using XModule/Descriptor
+functionality
+"""
+
+from nose.tools import assert_equal
+from unittest.case import SkipTest
+from mock import Mock
+
+from xmodule.annotatable_module import AnnotatableDescriptor
+from xmodule.capa_module import CapaDescriptor
+from xmodule.course_module import CourseDescriptor
+from xmodule.combined_open_ended_module import CombinedOpenEndedDescriptor
+from xmodule.discussion_module import DiscussionDescriptor
+from xmodule.gst_module import GraphicalSliderToolDescriptor
+from xmodule.html_module import HtmlDescriptor
+from xmodule.peer_grading_module import PeerGradingDescriptor
+from xmodule.poll_module import PollDescriptor
+from xmodule.video_module import VideoDescriptor
+from xmodule.word_cloud_module import WordCloudDescriptor
+from xmodule.crowdsource_hinter import CrowdsourceHinterDescriptor
+from xmodule.videoalpha_module import VideoAlphaDescriptor
+from xmodule.seq_module import SequenceDescriptor
+from xmodule.conditional_module import ConditionalDescriptor
+from xmodule.randomize_module import RandomizeDescriptor
+from xmodule.vertical_module import VerticalDescriptor
+from xmodule.wrapper_module import WrapperDescriptor
+
+LEAF_XMODULES = (
+    AnnotatableDescriptor,
+    CapaDescriptor,
+    CombinedOpenEndedDescriptor,
+    DiscussionDescriptor,
+    GraphicalSliderToolDescriptor,
+    HtmlDescriptor,
+    PeerGradingDescriptor,
+    PollDescriptor,
+    VideoDescriptor,
+    # This is being excluded because it has dependencies on django
+    #VideoAlphaDescriptor,
+    WordCloudDescriptor,
+)
+
+
+CONTAINER_XMODULES = (
+    CrowdsourceHinterDescriptor,
+    CourseDescriptor,
+    SequenceDescriptor,
+    ConditionalDescriptor,
+    RandomizeDescriptor,
+    VerticalDescriptor,
+    WrapperDescriptor,
+    CourseDescriptor,
+)
+
+
+class TestXBlockWrapper(object):
+
+    @property
+    def leaf_module_runtime(self):
+        runtime = Mock()
+        runtime.render_template = lambda *args, **kwargs: unicode((args, kwargs))
+        runtime.anonymous_student_id = 'anonymous_student_id'
+        runtime.open_ended_grading_interface = {}
+        runtime.seed = 5
+        runtime.get = lambda x: getattr(runtime, x)
+        runtime.position = 2
+        runtime.ajax_url = 'ajax_url'
+        runtime.xblock_model_data = lambda d: d._model_data
+        return runtime
+
+    @property
+    def leaf_descriptor_runtime(self):
+        runtime = Mock()
+        runtime.render_template = lambda *args, **kwargs: unicode((args, kwargs))
+        return runtime
+
+    def leaf_descriptor(self, descriptor_cls):
+        return descriptor_cls(
+            self.leaf_descriptor_runtime,
+            {'location': 'i4x://org/course/catagory/name'}
+        )
+
+    def leaf_module(self, descriptor_cls):
+        return self.leaf_descriptor(descriptor_cls).xmodule(self.leaf_module_runtime)
+
+    def container_module_runtime(self, depth):
+        runtime = self.leaf_module_runtime
+        if depth == 0:
+            runtime.get_module.side_effect = lambda x: self.leaf_module(HtmlDescriptor)
+        else:
+            runtime.get_module.side_effect = lambda x: self.container_module(VerticalDescriptor, depth-1)
+        return runtime
+
+    @property
+    def container_descriptor_runtime(self):
+        runtime = Mock()
+        runtime.render_template = lambda *args, **kwargs: unicode((args, kwargs))
+        return runtime
+
+    def container_descriptor(self, descriptor_cls):
+        return descriptor_cls(
+            self.container_descriptor_runtime,
+            {
+                'location': 'i4x://org/course/catagory/name',
+                'children': range(3)
+            }
+        )
+
+    def container_module(self, descriptor_cls, depth):
+        return self.container_descriptor(descriptor_cls).xmodule(self.container_module_runtime(depth))
+
+class TestStudentView(TestXBlockWrapper):
+
+    # Test that for all of the leaf XModule Descriptors,
+    # the student_view wrapper returns the same thing in its content
+    # as get_html returns
+    def test_student_view_leaf_node(self):
+        for descriptor_cls in LEAF_XMODULES:
+            yield self.check_student_view_leaf_node, descriptor_cls
+
+    # Check that when an xmodule is instantiated from descriptor_cls
+    # it generates the same thing from student_view that it does from get_html
+    def check_student_view_leaf_node(self, descriptor_cls):
+        xmodule = self.leaf_module(descriptor_cls)
+        assert_equal(xmodule.get_html(), xmodule.student_view(None).content)
+
+
+    # Test that for all container XModule Descriptors,
+    # their corresponding XModule renders the same thing using student_view
+    # as it does using get_html, under the following conditions:
+    # a) All of its descendents are xmodules
+    # b) Some of its descendents are xmodules and some are xblocks
+    # c) All of its descendents are xblocks
+    def test_student_view_container_node(self):
+        for descriptor_cls in CONTAINER_XMODULES:
+            yield self.check_student_view_container_node_xmodules_only, descriptor_cls
+            yield self.check_student_view_container_node_mixed, descriptor_cls
+            yield self.check_student_view_container_node_xblocks_only, descriptor_cls
+
+
+    # Check that when an xmodule is generated from descriptor_cls
+    # with only xmodule children, it generates the same html from student_view
+    # as it does using get_html
+    def check_student_view_container_node_xmodules_only(self, descriptor_cls):
+        xmodule = self.container_module(descriptor_cls, 2)
+        assert_equal(xmodule.get_html(), xmodule.student_view(None).content)
+
+    # Check that when an xmodule is generated from descriptor_cls
+    # with mixed xmodule and xblock children, it generates the same html from student_view
+    # as it does using get_html
+    def check_student_view_container_node_mixed(self, descriptor_cls):
+        raise SkipTest("XBlock support in XDescriptor not yet fully implemented")
+
+    # Check that when an xmodule is generated from descriptor_cls
+    # with only xblock children, it generates the same html from student_view
+    # as it does using get_html
+    def check_student_view_container_node_xblocks_only(self, descriptor_cls):
+        raise SkipTest("XBlock support in XModules not yet fully implemented")
+

common/lib/xmodule/xmodule/video_module.py

@@ -27,11 +27,13 @@ class VideoFields(object):
         scope=Scope.settings,
         # it'd be nice to have a useful default but it screws up other things; so,
         # use display_name_with_default for those
-        default="Video Title"
+        default="Video"
     )
-    data = String(help="XML data for the problem",
+    data = String(
+        help="XML data for the problem",
         default='',
-        scope=Scope.content)
+        scope=Scope.content
+    )
     position = Integer(help="Current position in the video", scope=Scope.user_state, default=0)
     show_captions = Boolean(help="This controls whether or not captions are shown by default.", display_name="Show Captions", scope=Scope.settings, default=True)
     youtube_id_1_0 = String(help="This is the Youtube ID reference for the normal speed video.", display_name="Default Speed", scope=Scope.settings, default="OEoXaMPEzfM")
@@ -125,7 +127,7 @@ class VideoDescriptor(VideoFields,
             url identifiers
         """
         video = super(VideoDescriptor, cls).from_xml(xml_data, system, org, course)
-        _parse_video_xml(video, xml_data)
+        _parse_video_xml(video, video.data)
         return video
 
     def definition_to_xml(self, resource_fs):
@@ -146,10 +148,6 @@ def _parse_video_xml(video, xml_data):
     display_name = xml.get('display_name')
     if display_name:
         video.display_name = display_name
-    elif video.url_name is not None:
-        # copies the logic of display_name_with_default in order that studio created videos will have an
-        # initial non guid name
-        video.display_name = video.url_name.replace('_', ' ')
 
     youtube = xml.get('youtube')
     if youtube:

common/lib/xmodule/xmodule/x_module.py

@@ -8,9 +8,11 @@ from collections import namedtuple
 from pkg_resources import resource_listdir, resource_string, resource_isdir
 
 from xmodule.modulestore import inheritance, Location
-from xmodule.modulestore.exceptions import ItemNotFoundError, InsufficientSpecificationError
+from xmodule.modulestore.exceptions import ItemNotFoundError, InsufficientSpecificationError, InvalidLocationError
 
 from xblock.core import XBlock, Scope, String, Integer, Float, ModelType
+from xblock.fragment import Fragment
+from xmodule.modulestore.locator import BlockUsageLocator
 
 log = logging.getLogger(__name__)
 
@@ -27,7 +29,13 @@ class LocationField(ModelType):
         """
         Parse the json value as a Location
         """
-        return Location(value)
+        try:
+            return Location(value)
+        except InvalidLocationError:
+            if isinstance(value, BlockUsageLocator):
+                return value
+            else:
+                return BlockUsageLocator(value)
 
     def to_json(self, value):
         """
@@ -166,6 +174,10 @@ class XModule(XModuleFields, HTMLSnippet, XBlock):
             self.url_name = self.location.name
             if not hasattr(self, 'category'):
                 self.category = self.location.category
+        elif isinstance(self.location, BlockUsageLocator):
+            self.url_name = self.location.usage_id
+            if not hasattr(self, 'category'):
+                raise InsufficientSpecificationError()
         else:
             raise InsufficientSpecificationError()
         self._loaded_children = None
@@ -191,6 +203,13 @@ class XModule(XModuleFields, HTMLSnippet, XBlock):
         '''
         if self._loaded_children is None:
             child_descriptors = self.get_child_descriptors()
+
+            # This deliberately uses system.get_module, rather than runtime.get_block,
+            # because we're looking at XModule children, rather than XModuleDescriptor children.
+            # That means it can use the deprecated XModule apis, rather than future XBlock apis
+
+            # TODO: Once we're in a system where this returns a mix of XModuleDescriptors
+            # and XBlocks, we're likely to have to change this more
             children = [self.system.get_module(descriptor) for descriptor in child_descriptors]
             # get_module returns None if the current user doesn't have access
             # to the location.
@@ -296,6 +315,19 @@ class XModule(XModuleFields, HTMLSnippet, XBlock):
         return ""
 
 
+    # ~~~~~~~~~~~~~~~ XBlock API Wrappers ~~~~~~~~~~~~~~~~
+    def student_view(self, context):
+        """
+        Return a fragment with the html from this XModule
+
+        Doesn't yet add any of the javascript to the fragment, nor the css.
+        Also doesn't expect any javascript binding, yet.
+
+        Makes no use of the context parameter
+        """
+        return Fragment(self.get_html())
+
+
 def policy_key(location):
     """
     Get the key for a location in a policy file.  (Since the policy file is
@@ -436,8 +468,17 @@ class XModuleDescriptor(XModuleFields, HTMLSnippet, ResourceTemplates, XBlock):
             self.url_name = self.location.name
             if not hasattr(self, 'category'):
                 self.category = self.location.category
+        elif isinstance(self.location, BlockUsageLocator):
+            self.url_name = self.location.usage_id
+            if not hasattr(self, 'category'):
+                raise InsufficientSpecificationError()
         else:
             raise InsufficientSpecificationError()
+        # update_version is the version which last updated this xblock v prev being the penultimate updater
+        # leaving off original_version since it complicates creation w/o any obv value yet and is computable
+        # by following previous until None
+        # definition_locator is only used by mongostores which separate definitions from blocks
+        self.edited_by = self.edited_on = self.previous_version = self.update_version = self.definition_locator = None
         self._child_instances = None
 
     @property
@@ -473,7 +514,7 @@ class XModuleDescriptor(XModuleFields, HTMLSnippet, ResourceTemplates, XBlock):
                     child = child_loc
                 else:
                     try:
-                        child = self.system.load_item(child_loc)
+                        child = self.runtime.get_block(child_loc)
                     except ItemNotFoundError:
                         log.exception('Unable to load item {loc}, skipping'.format(loc=child_loc))
                         continue
@@ -514,22 +555,30 @@ class XModuleDescriptor(XModuleFields, HTMLSnippet, ResourceTemplates, XBlock):
 
     # ================================= JSON PARSING ===========================
     @staticmethod
-    def load_from_json(json_data, system, default_class=None):
+    def load_from_json(json_data, system, default_class=None, parent_xblock=None):
         """
         This method instantiates the correct subclass of XModuleDescriptor based
-        on the contents of json_data.
+        on the contents of json_data. It does not persist it and can create one which
+        has no usage id.
 
-        json_data must contain a 'location' element, and must be suitable to be
-        passed into the subclasses `from_json` method as model_data
+        parent_xblock is used to compute inherited metadata as well as to append the new xblock.
+
+        json_data:
+        - 'location' : must have this field
+        - 'category': the xmodule category (required or location must be a Location)
+        - 'metadata': a dict of locally set metadata (not inherited)
+        - 'children': a list of children's usage_ids w/in this course
+        - 'definition':
+        - '_id' (optional): the usage_id of this. Will generate one if not given one.
         """
         class_ = XModuleDescriptor.load_class(
-            json_data['location']['category'],
+            json_data.get('category', json_data.get('location', {}).get('category')),
             default_class
         )
-        return class_.from_json(json_data, system)
+        return class_.from_json(json_data, system, parent_xblock)
 
     @classmethod
-    def from_json(cls, json_data, system):
+    def from_json(cls, json_data, system, parent_xblock=None):
         """
         Creates an instance of this descriptor from the supplied json_data.
         This may be overridden by subclasses
@@ -547,28 +596,25 @@ class XModuleDescriptor(XModuleFields, HTMLSnippet, ResourceTemplates, XBlock):
                Otherwise, it contains the single field 'data'
             4) Any value later in this list overrides a value earlier in this list
 
-        system: A DescriptorSystem for interacting with external resources
-        """
-        model_data = {}
-
-        for key, value in json_data.get('metadata', {}).items():
-            model_data[cls._translate(key)] = value
-
-        model_data.update(json_data.get('metadata', {}))
-
-        definition = json_data.get('definition', {})
-        if 'children' in definition:
-            model_data['children'] = definition['children']
-
-        if 'data' in definition:
-            if isinstance(definition['data'], dict):
-                model_data.update(definition['data'])
-            else:
-                model_data['data'] = definition['data']
-
-        model_data['location'] = json_data['location']
-
-        return cls(system, model_data)
+        json_data:
+        - 'category': the xmodule category (required)
+        - 'metadata': a dict of locally set metadata (not inherited)
+        - 'children': a list of children's usage_ids w/in this course
+        - 'definition':
+        - '_id' (optional): the usage_id of this. Will generate one if not given one.
+        """
+        usage_id = json_data.get('_id', None)
+        if not '_inherited_metadata' in json_data and parent_xblock is not None:
+            json_data['_inherited_metadata'] = parent_xblock.xblock_kvs.get_inherited_metadata().copy()
+            json_metadata = json_data.get('metadata', {})
+            for field in inheritance.INHERITABLE_METADATA:
+                if field in json_metadata:
+                    json_data['_inherited_metadata'][field] = json_metadata[field]
+
+        new_block = system.xblock_from_json(cls, usage_id, json_data)
+        if parent_xblock is not None:
+            parent_xblock.children.append(new_block)
+        return new_block
 
     @classmethod
     def _translate(cls, key):
@@ -649,6 +695,8 @@ class XModuleDescriptor(XModuleFields, HTMLSnippet, ResourceTemplates, XBlock):
         """
         Use w/ caution. Really intended for use by the persistence layer.
         """
+        # if caller wants kvs, caller's assuming it's up to date; so, decache it
+        self.save()
         return self._model_data._kvs
 
     # =============================== BUILTIN METHODS ==========================
@@ -780,6 +828,10 @@ class DescriptorSystem(object):
         self.resources_fs = resources_fs
         self.error_tracker = error_tracker
 
+    def get_block(self, block_id):
+        """See documentation for `xblock.runtime:Runtime.get_block`"""
+        return self.load_item(block_id)
+
 
 class XMLParsingSystem(DescriptorSystem):
     def __init__(self, load_item, resources_fs, error_tracker, process_xml, policy, **kwargs):
@@ -868,8 +920,8 @@ class ModuleSystem(object):
 
         publish(event) - A function that allows XModules to publish events (such as grade changes)
 
-        xblock_model_data - A dict-like object containing the all data available to this
-            xblock
+        xblock_model_data - A function that constructs a model_data for an xblock from its
+            corresponding descriptor
 
         cache - A cache object with two methods:
             .get(key) returns an object from the cache or None.

common/lib/xmodule/xmodule/xml_module.py

@@ -306,6 +306,7 @@ class XmlDescriptor(XModuleDescriptor):
         org and course are optional strings that will be used in the generated modules
             url identifiers
         """
+
         xml_object = etree.fromstring(xml_data)
         # VS[compat] -- just have the url_name lookup, once translation is done
         url_name = xml_object.get('url_name', xml_object.get('slug'))
@@ -318,7 +319,8 @@ class XmlDescriptor(XModuleDescriptor):
             filepath = cls._format_filepath(xml_object.tag, name_to_pathname(url_name))
             definition_xml = cls.load_file(filepath, system.resources_fs, location)
         else:
-            definition_xml = xml_object  # this is just a pointer, not the real definition content
+            definition_xml = xml_object
+            filepath = None
 
         definition, children = cls.load_definition(definition_xml, system, location)  # note this removes metadata
 

common/test/data/splitmongo_json/active_versions.json

+[{"_id" : "GreekHero", 
+  "org" : "testx", 
+  "prettyid" : "test_course", 
+  "versions" : {
+    "draft" : { "$oid" : "1d00000000000000dddd0000" }
+   },
+  "edited_on" : {"$date" : 1364481713238},
+  "edited_by" : "test@edx.org"},
+
+ {"_id" : "wonderful", 
+  "org" : "testx", 
+  "prettyid" : "another_course", 
+  "versions" : {
+    "draft" : { "$oid" : "1d00000000000000dddd2222" },
+    "published" : { "$oid" : "1d00000000000000eeee0000" }
+    },
+  "edited_on" : {"$date" : 1364481313238},
+  "edited_by" : "test@edx.org"},
+  
+ {"_id" : "contender", 
+  "org" : "guestx", 
+  "prettyid" : "test_course", 
+  "versions" : {
+        "draft" : { "$oid" : "1d00000000000000dddd5555" }},
+  "edited_on" : {"$date" : 1364491313238},
+  "edited_by" : "test@guestx.edu"}
+]