Merge branch 'master' into ormsbee/verifyuser_func

cms/djangoapps/contentstore/features/advanced-settings.feature

@@ -2,6 +2,7 @@ Feature: Advanced (manual) course policy
   In order to specify course policy settings for which no custom user interface exists
   I want to be able to manually enter JSON key /value pairs
 
+
   Scenario: A course author sees default advanced settings
     Given I have opened a new course in Studio
     When I select the Advanced Settings
@@ -11,6 +12,8 @@ Feature: Advanced (manual) course policy
     Given I am on the Advanced Course Settings page in Studio
     Then the settings are alphabetized
 
+  # Sauce labs does not play nicely with CodeMirror
+  @skip_sauce
   Scenario: Test cancel editing key value
     Given I am on the Advanced Course Settings page in Studio
     When I edit the value of a policy key
@@ -19,6 +22,8 @@ Feature: Advanced (manual) course policy
     And I reload the page
     Then the policy key value is unchanged
 
+  # Sauce labs does not play nicely with CodeMirror
+  @skip_sauce
   Scenario: Test editing key value
     Given I am on the Advanced Course Settings page in Studio
     When I edit the value of a policy key and save
@@ -26,6 +31,8 @@ Feature: Advanced (manual) course policy
     And I reload the page
     Then the policy key value is changed
 
+  # Sauce labs does not play nicely with CodeMirror
+  @skip_sauce
   Scenario: Test how multi-line input appears
     Given I am on the Advanced Course Settings page in Studio
     When I create a JSON object as a value for "discussion_topics"
@@ -33,6 +40,8 @@ Feature: Advanced (manual) course policy
     And I reload the page
     Then it is displayed as formatted
 
+  # Sauce labs does not play nicely with CodeMirror
+  @skip_sauce
   Scenario: Test error if value supplied is of the wrong type
     Given I am on the Advanced Course Settings page in Studio
     When I create a JSON object as a value for "display_name"
@@ -41,6 +50,8 @@ Feature: Advanced (manual) course policy
     Then the policy key value is unchanged
 
   # This feature will work in Firefox only when Firefox is the active window
+  # Sauce labs does not play nicely with CodeMirror
+  @skip_sauce
   Scenario: Test automatic quoting of non-JSON values
     Given I am on the Advanced Course Settings page in Studio
     When I create a non-JSON value not in quotes
@@ -48,6 +59,8 @@ Feature: Advanced (manual) course policy
     And I reload the page
     Then it is displayed as a string
 
+  # Sauce labs does not play nicely with CodeMirror
+  @skip_sauce
   Scenario: Confirmation is shown on save
     Given I am on the Advanced Course Settings page in Studio
     When I edit the value of a policy key

cms/djangoapps/contentstore/features/checklists.feature

@@ -10,7 +10,10 @@ Feature: Course checklists
     Then I can check and uncheck tasks in a checklist
     And They are correctly selected after reloading the page
 
-  # CHROME ONLY, due to issues getting link to be active in firefox
+  # There are issues getting link to be active in browsers other than chrome
+  @skip_firefox
+  @skip_internetexplorer
+  @skip_safari
   Scenario: A task can link to a location within Studio
     Given I have opened Checklists
     When I select a link to the course outline
@@ -18,7 +21,10 @@ Feature: Course checklists
     And I press the browser back button
     Then I am brought back to the course outline in the correct state
 
-  # CHROME ONLY, due to issues getting link to be active in firefox
+  # There are issues getting link to be active in browsers other than chrome
+  @skip_firefox
+  @skip_internetexplorer
+  @skip_safari
   Scenario: A task can link to a location outside Studio
     Given I have opened Checklists
     When I select a link to help page

cms/djangoapps/contentstore/features/course-overview.feature

@@ -64,6 +64,10 @@ Feature: Course Overview
         And I change an assignment's grading status
         Then I am shown a notification
 
+    # Notification is not shown on reorder for IE
+    # Safari does not have moveMouseTo implemented
+    @skip_internetexplorer
+    @skip_safari
    Scenario: Notification is shown on subsection reorder
         Given I have opened a new course section in Studio
         And I have added a new subsection

cms/djangoapps/contentstore/features/course-settings.feature

 Feature: Course Settings
   As a course author, I want to be able to configure my course settings.
 
+  # Safari has trouble keeps dates on refresh
+  @skip_safari
   Scenario: User can set course dates
     Given I have opened a new course in Studio
     When I select Schedule and Details
@@ -8,12 +10,16 @@ Feature: Course Settings
     And I press the "Save" notification button
     Then I see the set dates on refresh
 
+    # IE has trouble with saving information
+    @skip_internetexplorer
   Scenario: User can clear previously set course dates (except start date)
     Given I have set course dates
     And I clear all the dates except start
     And I press the "Save" notification button
     Then I see cleared dates on refresh
 
+    # IE has trouble with saving information
+    @skip_internetexplorer
   Scenario: User cannot clear the course start date
     Given I have set course dates
     And I press the "Save" notification button
@@ -21,6 +27,10 @@ Feature: Course Settings
     Then I receive a warning about course start date
     And The previously set start date is shown on refresh
 
+    # IE has trouble with saving information
+    # Safari gets CSRF token errors
+    @skip_internetexplorer
+    @skip_safari
   Scenario: User can correct the course start date warning
     Given I have tried to clear the course start
     And I have entered a new course start date
@@ -28,12 +38,16 @@ Feature: Course Settings
     Then The warning about course start date goes away
     And My new course start date is shown on refresh
 
+  # Safari does not save + refresh properly through sauce labs
+  @skip_safari
   Scenario: Settings are only persisted when saved
     Given I have set course dates
     And I press the "Save" notification button
     When I change fields
     Then I do not see the new changes persisted on refresh
 
+  # Safari does not save + refresh properly through sauce labs
+  @skip_safari
   Scenario: Settings are reset on cancel
     Given I have set course dates
     And I press the "Save" notification button
@@ -41,6 +55,8 @@ Feature: Course Settings
     And I press the "Cancel" notification button
     Then I do not see the changes
 
+  # Safari gets CSRF token errors
+  @skip_safari
   Scenario: Confirmation is shown on save
     Given I have opened a new course in Studio
     When I select Schedule and Details

cms/djangoapps/contentstore/features/course-team.py

@@ -91,7 +91,7 @@ def remove_course_team_admin(_step, outer_capture, name):
 
 @step(u'"([^"]*)" logs in$')
 def other_user_login(_step, name):
-    world.browser.cookies.delete()
+    world.visit('logout')
     world.visit('/')
 
     signin_css = 'a.action-signin'

cms/djangoapps/contentstore/features/course-updates.feature

 Feature: Course updates
     As a course author, I want to be able to provide updates to my students
 
+    # Internet explorer can't select all so the update appears weirdly
+    @skip_internetexplorer
     Scenario: Users can add updates
         Given I have opened a new course in Studio
         And I go to the course updates page
@@ -8,6 +10,8 @@ Feature: Course updates
         Then I should see the update "Hello"
         And I see a "saving" notification
 
+    # Internet explorer can't select all so the update appears weirdly
+    @skip_internetexplorer
     Scenario: Users can edit updates
         Given I have opened a new course in Studio
         And I go to the course updates page
@@ -33,6 +37,8 @@ Feature: Course updates
         Then I should see the date "June 1, 2013"
         And I see a "saving" notification
 
+    # Internet explorer can't select all so the update appears weirdly
+    @skip_internetexplorer
     Scenario: Users can change handouts
         Given I have opened a new course in Studio
         And I go to the course updates page

cms/djangoapps/contentstore/features/discussion-editor.feature

@@ -6,6 +6,8 @@ Feature: Discussion Component Editor
     And I edit and select Settings
     Then I see three alphabetized settings and their expected values
 
+  # Safari doesn't save the name properly
+  @skip_safari
   Scenario: User can modify display name
     Given I have created a Discussion Tag
     And I edit and select Settings

cms/djangoapps/contentstore/features/grading.feature

@@ -13,7 +13,7 @@ Feature: Course Grading
         When I add "6" new grades
         Then I see I now have "5" grades
 
-    #Cannot reliably make the delete button appear so using javascript instead
+    # Cannot reliably make the delete button appear so using javascript instead
     Scenario: Users can delete grading ranges
         Given I have opened a new course in Studio
         And I am viewing the grading settings
@@ -21,6 +21,9 @@ Feature: Course Grading
         And I delete a grade
         Then I see I now have "2" grades
 
+    # IE and Safari cannot reliably drag and drop through selenium
+    @skip_internetexplorer
+    @skip_safari
     Scenario: Users can move grading ranges
         Given I have opened a new course in Studio
         And I am viewing the grading settings
@@ -85,6 +88,9 @@ Feature: Course Grading
         When I change assignment type "Homework" to ""
         Then the save button is disabled
 
+    # IE and Safari cannot type in grade range name
+    @skip_internetexplorer
+    @skip_safari
     Scenario: User can edit grading range names
         Given I have opened a new course in Studio
         And I have populated the course

cms/djangoapps/contentstore/features/grading.py

@@ -112,10 +112,10 @@ def changes_not_persisted(step):
 
 @step(u'I see the assignment type "(.*)"$')
 def i_see_the_assignment_type(_step, name):
-      assignment_css = '#course-grading-assignment-name'
-      assignments = world.css_find(assignment_css)
-      types = [ele['value'] for ele in assignments]
-      assert name in types
+    assignment_css = '#course-grading-assignment-name'
+    assignments = world.css_find(assignment_css)
+    types = [ele['value'] for ele in assignments]
+    assert name in types
 
 
 @step(u'I change the highest grade range to "(.*)"$')
@@ -144,6 +144,7 @@ def cannot_edit_fail(_step):
         pass  # We should get this exception on failing to edit the element
 
 
+
 @step(u'I change the grace period to "(.*)"$')
 def i_change_grace_period(_step, grace_period):
     grace_period_css = '#course-grading-graceperiod'

cms/djangoapps/contentstore/features/html-editor.feature

@@ -6,6 +6,8 @@ Feature: HTML Editor
     And I edit and select Settings
     Then I see only the HTML display name setting
 
+  # Safari doesn't save the name properly
+  @skip_safari
   Scenario: User can modify display name
     Given I have created a Blank HTML Page
     And I edit and select Settings

cms/djangoapps/contentstore/features/problem-editor.feature

@@ -7,12 +7,16 @@ Feature: Problem Editor
     Then I see five alphabetized settings and their expected values
     And Edit High Level Source is not visible
 
+  # Safari is having trouble saving the values on sauce
+  @skip_safari
   Scenario: User can modify String values
     Given I have created a Blank Common Problem
     When I edit and select Settings
     Then I can modify the display name
     And my display name change is persisted on save
 
+  # Safari is having trouble saving the values on sauce
+  @skip_safari
   Scenario: User can specify special characters in String values
     Given I have created a Blank Common Problem
     When I edit and select Settings
@@ -25,6 +29,8 @@ Feature: Problem Editor
     Then I can revert the display name to unset
     And my display name is unset on save
 
+  # IE will not click the revert button properly
+  @skip_internetexplorer
   Scenario: User can select values in a Select
     Given I have created a Blank Common Problem
     When I edit and select Settings
@@ -32,6 +38,8 @@ Feature: Problem Editor
     And my change to randomization is persisted
     And I can revert to the default value for randomization
 
+  # Safari will input it as 35.
+  @skip_safari
   Scenario: User can modify float input values
     Given I have created a Blank Common Problem
     When I edit and select Settings
@@ -44,16 +52,22 @@ Feature: Problem Editor
     When I edit and select Settings
     Then if I set the weight to "abc", it remains unset
 
+  # Safari will input it as 234.
+  @skip_safari
   Scenario: User cannot type decimal values integer number field
     Given I have created a Blank Common Problem
     When I edit and select Settings
     Then if I set the max attempts to "2.34", it will persist as a valid integer
 
+  # Safari will input it incorrectly
+  @skip_safari
   Scenario: User cannot type out of range values in an integer number field
     Given I have created a Blank Common Problem
     When I edit and select Settings
     Then if I set the max attempts to "-3", it will persist as a valid integer
 
+  # Safari will input it as 35.
+  @skip_safari
   Scenario: Settings changes are not saved on Cancel
     Given I have created a Blank Common Problem
     When I edit and select Settings
@@ -67,6 +81,8 @@ Feature: Problem Editor
     Then Edit High Level Source is visible
 
   # This feature will work in Firefox only when Firefox is the active window
+  # IE will not interact with the high level source in sauce labs
+  @skip_internetexplorer
   Scenario: High Level source is persisted for LaTeX problem (bug STUD-280)
     Given I have created a LaTeX Problem
     When I edit and compile the High Level Source

cms/djangoapps/contentstore/features/static-pages.feature

@@ -15,6 +15,8 @@ Feature: Static Pages
         And I "delete" the "Empty" page
         Then I should not see a "Empty" static page
 
+    # Safari won't update the name properly
+    @skip_safari
     Scenario: Users can edit static pages
         Given I have opened a new course in Studio
         And I go to the static pages page

cms/djangoapps/contentstore/features/subsection.feature

@@ -25,6 +25,8 @@ Feature: Create Subsection
     And I reload the page
     Then I see it marked as Homework
 
+  # Safari has trouble saving the date in Sauce
+  @skip_safari
   Scenario: Set a due date in a different year (bug #256)
     Given I have opened a new subsection in Studio
     And I set the subsection release date to 12/25/2011 03:00

cms/djangoapps/contentstore/features/textbooks.feature

@@ -5,6 +5,9 @@ Feature: Textbooks
     When I go to the textbooks page
     Then I should see a message telling me to create a new textbook
 
+  # IE and Safari on sauce labs will not upload the textbook correctly resulting in an error
+  @skip_internetexplorer
+  @skip_safari
   Scenario: Create a textbook
     Given I have opened a new course in Studio
     And I go to the textbooks page

cms/djangoapps/contentstore/features/upload.feature

 Feature: Upload Files
     As a course author, I want to be able to upload files for my students
 
+    # Uploading isn't working on safari with sauce labs
+    @skip_safari
     Scenario: Users can upload files
         Given I have opened a new course in Studio
         And I go to the files and uploads page
@@ -8,6 +10,8 @@ Feature: Upload Files
         Then I should see the file "test" was uploaded
         And The url for the file "test" is valid
 
+    # Uploading isn't working on safari with sauce labs
+    @skip_safari
     Scenario: Users can update files
         Given I have opened a new course in studio
         And I go to the files and uploads page
@@ -15,6 +19,8 @@ Feature: Upload Files
         And I upload the file "test"
         Then I should see only one "test"
 
+    # Uploading isn't working on safari with sauce labs
+    @skip_safari
     Scenario: Users can delete uploaded files
         Given I have opened a new course in studio
         And I go to the files and uploads page
@@ -23,12 +29,16 @@ Feature: Upload Files
         Then I should not see the file "test" was uploaded
         And I see a confirmation that the file was deleted
 
+    # Uploading isn't working on safari with sauce labs
+    @skip_safari
     Scenario: Users can download files
         Given I have opened a new course in studio
         And I go to the files and uploads page
         When I upload the file "test"
         Then I can download the correct "test" file
 
+    # Uploading isn't working on safari with sauce labs
+    @skip_safari
     Scenario: Users can download updated files
         Given I have opened a new course in studio
         And I go to the files and uploads page

cms/djangoapps/contentstore/features/upload.py

@@ -10,6 +10,7 @@ import os
 
 TEST_ROOT = settings.COMMON_TEST_DATA_ROOT
 
+
 @step(u'I go to the files and uploads page')
 def go_to_uploads(_step):
     menu_css = 'li.nav-course-courseware'
@@ -106,8 +107,8 @@ def get_index(file_name):
 def get_file(file_name):
     index = get_index(file_name)
     assert index != -1
-
     url_css = 'a.filename'
+
     def get_url():
         return world.css_find(url_css)[index]._element.get_attribute('href')
     url = world.retry_on_exception(get_url)

cms/djangoapps/contentstore/features/video-editor.feature

@@ -6,17 +6,23 @@ Feature: Video Component Editor
     And I edit the component
     Then I see the correct video settings and default values
 
+  # Safari has trouble saving values on Sauce
+  @skip_safari
   Scenario: User can modify Video display name
     Given I have created a Video component
     And I edit the component
     Then I can modify the display name
     And my video display name change is persisted on save
 
+  # Sauce Labs cannot delete cookies
+  @skip_sauce
   Scenario: Captions are hidden when "show captions" is false
     Given I have created a Video component
     And I have set "show captions" to False
     Then when I view the video it does not show the captions
 
+  # Sauce Labs cannot delete cookies
+  @skip_sauce
   Scenario: Captions are shown when "show captions" is true
     Given I have created a Video component
     And I have set "show captions" to True

cms/djangoapps/contentstore/features/video.feature

@@ -10,15 +10,21 @@ Feature: Video Component
     Given I have clicked the new unit button
     Then creating a video takes a single click
 
+  # Sauce Labs cannot delete cookies
+  @skip_sauce
   Scenario: Captions are hidden correctly
     Given I have created a Video component
     And I have hidden captions
     Then when I view the video it does not show the captions
 
+  # Sauce Labs cannot delete cookies
+  @skip_sauce
   Scenario: Captions are shown correctly
     Given I have created a Video component
     Then when I view the video it does show the captions
 
+  # Sauce Labs cannot delete cookies
+  @skip_sauce
   Scenario: Captions are toggled correctly
     Given I have created a Video component
     And I have toggled captions

cms/djangoapps/contentstore/management/commands/import.py

@@ -2,7 +2,7 @@
 Script for importing courseware from XML format
 """
 
-from django.core.management.base import BaseCommand, CommandError
+from django.core.management.base import BaseCommand, CommandError, make_option
 from xmodule.modulestore.xml_importer import import_from_xml
 from xmodule.modulestore.django import modulestore
 from xmodule.contentstore.django import contentstore
@@ -14,18 +14,26 @@ class Command(BaseCommand):
     """
     help = 'Import the specified data directory into the default ModuleStore'
 
+    option_list = BaseCommand.option_list + (
+        make_option('--nostatic',
+                    action='store_true',
+                    help='Skip import of static content'),
+    )
+
     def handle(self, *args, **options):
         "Execute the command"
         if len(args) == 0:
-            raise CommandError("import requires at least one argument: <data directory> [<course dir>...]")
+            raise CommandError("import requires at least one argument: <data directory> [--nostatic] [<course dir>...]")
 
         data_dir = args[0]
+        do_import_static = not (options.get('nostatic', False))
         if len(args) > 1:
             course_dirs = args[1:]
         else:
             course_dirs = None
         print("Importing.  Data_dir={data}, course_dirs={courses}".format(
             data=data_dir,
-            courses=course_dirs))
+            courses=course_dirs,
+            dis=do_import_static))
         import_from_xml(modulestore('direct'), data_dir, course_dirs, load_error_modules=False,
-                        static_content_store=contentstore(), verbose=True)
+                        static_content_store=contentstore(), verbose=True, do_import_static=do_import_static)

cms/djangoapps/contentstore/tests/test_contentstore.py

@@ -476,7 +476,7 @@ class ContentStoreToyCourseTest(ModuleStoreTestCase):
         content_store = contentstore()
 
         module_store = modulestore('direct')
-        import_from_xml(module_store, 'common/test/data/', ['toy'], static_content_store=content_store)
+        import_from_xml(module_store, 'common/test/data/', ['toy'], static_content_store=content_store, verbose=True)
 
         course_location = CourseDescriptor.id_to_location('edX/toy/2012_Fall')
         course = module_store.get_item(course_location)
@@ -945,8 +945,17 @@ class ContentStoreToyCourseTest(ModuleStoreTestCase):
                                                   'vertical', 'vertical_test', None]), depth=1)
 
         self.assertTrue(getattr(vertical, 'is_draft', False))
+        self.assertNotIn('index_in_children_list', child.xml_attributes)
+        self.assertNotIn('parent_sequential_url', vertical.xml_attributes)
+                
         for child in vertical.get_children():
             self.assertTrue(getattr(child, 'is_draft', False))
+            self.assertNotIn('index_in_children_list', child.xml_attributes)
+            if hasattr(child, 'data'):
+                self.assertNotIn('index_in_children_list', child.data)
+            self.assertNotIn('parent_sequential_url', child.xml_attributes)
+            if hasattr(child, 'data'):
+                self.assertNotIn('parent_sequential_url', child.data)
 
         # make sure that we don't have a sequential that is in draft mode
         sequential = draft_store.get_item(Location(['i4x', 'edX', 'toy',
@@ -1057,6 +1066,38 @@ class ContentStoreToyCourseTest(ModuleStoreTestCase):
         # It should now contain empty data
         self.assertEquals(imported_word_cloud.data, '')
 
+    def test_html_export_roundtrip(self):
+        """
+        Test that a course which has HTML that has style formatting is preserved in export/import
+        """
+        module_store = modulestore('direct')
+        content_store = contentstore()
+
+        import_from_xml(module_store, 'common/test/data/', ['toy'])
+
+        location = CourseDescriptor.id_to_location('edX/toy/2012_Fall')
+
+        # Export the course
+        root_dir = path(mkdtemp_clean())
+        export_to_xml(module_store, content_store, location, root_dir, 'test_roundtrip')
+
+        # Reimport and get the video back
+        import_from_xml(module_store, root_dir)
+
+        # get the sample HTML with styling information
+        html_module = module_store.get_instance(
+            'edX/toy/2012_Fall',
+            Location(['i4x', 'edX', 'toy', 'html', 'with_styling'])
+        )
+        self.assertIn('<p style="font:italic bold 72px/30px Georgia, serif; color: red; ">', html_module.data)
+
+        # get the sample HTML with just a simple <img> tag information
+        html_module = module_store.get_instance(
+            'edX/toy/2012_Fall',
+            Location(['i4x', 'edX', 'toy', 'html', 'just_img'])
+        )
+        self.assertIn('<img src="/static/foo_bar.jpg" />', html_module.data)
+
     def test_course_handouts_rewrites(self):
         module_store = modulestore('direct')
 

cms/djangoapps/contentstore/tests/test_import_nostatic.py

+#pylint: disable=E1101
+'''
+Tests for importing with no static
+'''
+
+from django.test.client import Client
+from django.test.utils import override_settings
+from django.conf import settings
+from path import path
+import copy
+
+from django.contrib.auth.models import User
+
+from xmodule.modulestore.tests.django_utils import ModuleStoreTestCase
+
+from xmodule.modulestore import Location
+from xmodule.modulestore.django import modulestore
+from xmodule.contentstore.django import contentstore
+from xmodule.modulestore.xml_importer import import_from_xml
+from xmodule.contentstore.content import StaticContent
+
+from xmodule.course_module import CourseDescriptor
+
+from xmodule.exceptions import NotFoundError
+from uuid import uuid4
+
+
+TEST_DATA_CONTENTSTORE = copy.deepcopy(settings.CONTENTSTORE)
+TEST_DATA_CONTENTSTORE['OPTIONS']['db'] = 'test_xcontent_%s' % uuid4().hex
+
+
+@override_settings(CONTENTSTORE=TEST_DATA_CONTENTSTORE)
+class ContentStoreImportNoStaticTest(ModuleStoreTestCase):
+    """
+    Tests that rely on the toy and test_import_course courses.
+    NOTE: refactor using CourseFactory so they do not.
+    """
+    def setUp(self):
+
+        settings.MODULESTORE['default']['OPTIONS']['fs_root'] = path('common/test/data')
+        settings.MODULESTORE['direct']['OPTIONS']['fs_root'] = path('common/test/data')
+        uname = 'testuser'
+        email = 'test+courses@edx.org'
+        password = 'foo'
+
+        # Create the use so we can log them in.
+        self.user = User.objects.create_user(uname, email, password)
+
+        # Note that we do not actually need to do anything
+        # for registration if we directly mark them active.
+        self.user.is_active = True
+        # Staff has access to view all courses
+        self.user.is_staff = True
+
+        # Save the data that we've just changed to the db.
+        self.user.save()
+
+        self.client = Client()
+        self.client.login(username=uname, password=password)
+
+    def load_test_import_course(self):
+        '''
+        Load the standard course used to test imports (for do_import_static=False behavior).
+        '''
+        content_store = contentstore()
+        module_store = modulestore('direct')
+        import_from_xml(module_store, 'common/test/data/', ['test_import_course'], static_content_store=content_store, do_import_static=False, verbose=True)
+        course_location = CourseDescriptor.id_to_location('edX/test_import_course/2012_Fall')
+        course = module_store.get_item(course_location)
+        self.assertIsNotNone(course)
+
+        return module_store, content_store, course, course_location
+
+    def test_static_import(self):
+        '''
+        Stuff in static_import should always be imported into contentstore
+        '''
+        _, content_store, course, course_location = self.load_test_import_course()
+
+        # make sure we have ONE asset in our contentstore ("should_be_imported.html")
+        all_assets = content_store.get_all_content_for_course(course_location)
+        print "len(all_assets)=%d" % len(all_assets)
+        self.assertEqual(len(all_assets), 1)
+
+        content = None
+        try:
+            location = StaticContent.get_location_from_path('/c4x/edX/test_import_course/asset/should_be_imported.html')
+            content = content_store.find(location)
+        except NotFoundError:
+            pass
+
+        self.assertIsNotNone(content)
+
+        # make sure course.lms.static_asset_path is correct
+        print "static_asset_path = {0}".format(course.lms.static_asset_path)
+        self.assertEqual(course.lms.static_asset_path, 'test_import_course')
+
+    def test_asset_import_nostatic(self):
+        '''
+        This test validates that an image asset is NOT imported when do_import_static=False
+        '''
+        content_store = contentstore()
+
+        module_store = modulestore('direct')
+        import_from_xml(module_store, 'common/test/data/', ['toy'], static_content_store=content_store, do_import_static=False, verbose=True)
+
+        course_location = CourseDescriptor.id_to_location('edX/toy/2012_Fall')
+        module_store.get_item(course_location)
+
+        # make sure we have NO assets in our contentstore
+        all_assets = content_store.get_all_content_for_course(course_location)
+        print "len(all_assets)=%d" % len(all_assets)
+        self.assertEqual(len(all_assets), 0)
+
+    def test_no_static_link_rewrites_on_import(self):
+        module_store = modulestore('direct')
+        import_from_xml(module_store, 'common/test/data/', ['toy'], do_import_static=False, verbose=True)
+
+        handouts = module_store.get_item(Location(['i4x', 'edX', 'toy', 'course_info', 'handouts', None]))
+        self.assertIn('/static/', handouts.data)
+
+        handouts = module_store.get_item(Location(['i4x', 'edX', 'toy', 'html', 'toyhtml', None]))
+        self.assertIn('/static/', handouts.data)

cms/envs/acceptance.py

@@ -8,6 +8,7 @@ so that we can run the lettuce acceptance tests.
 # pylint: disable=W0401, W0614
 
 from .test import *
+from lms.envs.sauce import *
 
 # You need to start the server in debug mode,
 # otherwise the browser will not render the pages correctly
@@ -17,7 +18,7 @@ DEBUG = True
 import logging
 logging.disable(logging.ERROR)
 import os
-import random
+from random import choice, randint
 
 
 def seed():
@@ -75,7 +76,6 @@ DATABASES = {
 # Use the auto_auth workflow for creating users and logging them in
 MITX_FEATURES['AUTOMATIC_AUTH_FOR_TESTING'] = True
 
-
 # HACK
 # Setting this flag to false causes imports to not load correctly in the lettuce python files
 # We do not yet understand why this occurs. Setting this to true is a stopgap measure
@@ -84,5 +84,5 @@ USE_I18N = True
 # Include the lettuce app for acceptance testing, including the 'harvest' django-admin command
 INSTALLED_APPS += ('lettuce.django',)
 LETTUCE_APPS = ('contentstore',)
-LETTUCE_SERVER_PORT = random.randint(1024, 65535)
-LETTUCE_BROWSER = 'chrome'
+LETTUCE_SERVER_PORT = choice(PORTS) if SAUCE.get('SAUCE_ENABLED') else randint(1024, 65535)
+LETTUCE_BROWSER = os.environ.get('LETTUCE_BROWSER', 'chrome')

common/djangoapps/static_replace/__init__.py

@@ -90,7 +90,7 @@ def replace_course_urls(text, course_id):
     return re.sub(_url_replace_regex('/course/'), replace_course_url, text)
 
 
-def replace_static_urls(text, data_directory, course_id=None):
+def replace_static_urls(text, data_directory, course_id=None, static_asset_path=''):
     """
     Replace /static/$stuff urls either with their correct url as generated by collectstatic,
     (/static/$md5_hashed_stuff) or by the course-specific content static url
@@ -100,6 +100,7 @@ def replace_static_urls(text, data_directory, course_id=None):
     text: The source text to do the substitution in
     data_directory: The directory in which course data is stored
     course_id: The course identifier used to distinguish static content for this course in studio
+    static_asset_path: Path for static assets, which overrides data_directory and course_namespace, if nonempty
     """
 
     def replace_static_url(match):
@@ -116,7 +117,7 @@ def replace_static_urls(text, data_directory, course_id=None):
         if settings.DEBUG and finders.find(rest, True):
             return original
         # if we're running with a MongoBacked store course_namespace is not None, then use studio style urls
-        elif course_id and modulestore().get_modulestore_type(course_id) != XML_MODULESTORE_TYPE:
+        elif (not static_asset_path) and course_id and modulestore().get_modulestore_type(course_id) != XML_MODULESTORE_TYPE:
             # first look in the static file pipeline and see if we are trying to reference
             # a piece of static content which is in the mitx repo (e.g. JS associated with an xmodule)
 
@@ -135,7 +136,7 @@ def replace_static_urls(text, data_directory, course_id=None):
                 url = StaticContent.convert_legacy_static_url_with_course_id(rest, course_id)
         # Otherwise, look the file up in staticfiles_storage, and append the data directory if needed
         else:
-            course_path = "/".join((data_directory, rest))
+            course_path = "/".join((static_asset_path or data_directory, rest))
 
             try:
                 if staticfiles_storage.exists(rest):
@@ -152,7 +153,7 @@ def replace_static_urls(text, data_directory, course_id=None):
 
 
     return re.sub(
-        _url_replace_regex('/static/(?!{data_dir})'.format(data_dir=data_directory)),
+        _url_replace_regex('/static/(?!{data_dir})'.format(data_dir=static_asset_path or data_directory)),
         replace_static_url,
         text
     )

common/djangoapps/terrain/browser.py

@@ -11,6 +11,10 @@ from logging import getLogger
 from django.core.management import call_command
 from django.conf import settings
 from selenium.common.exceptions import WebDriverException
+from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
+from requests import put
+from base64 import encodestring
+from json import dumps
 
 # Let the LMS and CMS do their one-time setup
 # For example, setting up mongo caches
@@ -42,43 +46,93 @@ LOGGER.info("Loading the lettuce acceptance testing terrain file...")
 MAX_VALID_BROWSER_ATTEMPTS = 20
 
 
+def get_username_and_key():
+    """
+    Returns the Sauce Labs username and access ID as set by environment variables
+    """
+    return {"username": settings.SAUCE.get('USERNAME'), "access-key": settings.SAUCE.get('ACCESS_ID')}
+
+
+def set_job_status(jobid, passed=True):
+    """
+    Sets the job status on sauce labs
+    """
+    body_content = dumps({"passed": passed})
+    config = get_username_and_key()
+    base64string = encodestring('{}:{}'.format(config['username'], config['access-key']))[:-1]
+    result = put('http://saucelabs.com/rest/v1/{}/jobs/{}'.format(config['username'], world.jobid),
+        data=body_content,
+        headers={"Authorization": "Basic {}".format(base64string)})
+    return result.status_code == 200
+
+
+def make_desired_capabilities():
+    """
+    Returns a DesiredCapabilities object corresponding to the environment sauce parameters
+    """
+    desired_capabilities = settings.SAUCE.get('BROWSER', DesiredCapabilities.CHROME)
+    desired_capabilities['platform'] = settings.SAUCE.get('PLATFORM')
+    desired_capabilities['version'] = settings.SAUCE.get('VERSION')
+    desired_capabilities['device-type'] = settings.SAUCE.get('DEVICE')
+    desired_capabilities['name'] = settings.SAUCE.get('SESSION')
+    desired_capabilities['build'] = settings.SAUCE.get('BUILD')
+    desired_capabilities['video-upload-on-pass'] = False
+    desired_capabilities['sauce-advisor'] = False
+    desired_capabilities['record-screenshots'] = False
+    desired_capabilities['selenium-version'] = "2.34.0"
+    desired_capabilities['max-duration'] = 3600
+    desired_capabilities['public'] = 'public restricted'
+    return desired_capabilities
+
+
 @before.harvest
 def initial_setup(server):
     """
     Launch the browser once before executing the tests.
     """
-    browser_driver = getattr(settings, 'LETTUCE_BROWSER', 'chrome')
+    world.absorb(settings.SAUCE.get('SAUCE_ENABLED'), 'SAUCE_ENABLED')
 
-    # There is an issue with ChromeDriver2 r195627 on Ubuntu
-    # in which we sometimes get an invalid browser session.
-    # This is a work-around to ensure that we get a valid session.
-    success = False
-    num_attempts = 0
-    while (not success) and num_attempts < MAX_VALID_BROWSER_ATTEMPTS:
+    if not world.SAUCE_ENABLED:
+        browser_driver = getattr(settings, 'LETTUCE_BROWSER', 'chrome')
 
-        # Get a browser session
-        world.browser = Browser(browser_driver)
+        # There is an issue with ChromeDriver2 r195627 on Ubuntu
+        # in which we sometimes get an invalid browser session.
+        # This is a work-around to ensure that we get a valid session.
+        success = False
+        num_attempts = 0
+        while (not success) and num_attempts < MAX_VALID_BROWSER_ATTEMPTS:
+            world.browser = Browser(browser_driver)
 
-        # Try to visit the main page
-        # If the browser session is invalid, this will
-        # raise a WebDriverException
-        try:
-            world.visit('/')
+            # Try to visit the main page
+            # If the browser session is invalid, this will
+            # raise a WebDriverException
+            try:
+                world.visit('/')
 
-        except WebDriverException:
-            world.browser.quit()
-            num_attempts += 1
+            except WebDriverException:
+                world.browser.quit()
+                num_attempts += 1
 
-        else:
-            success = True
+            else:
+                success = True
 
-    # If we were unable to get a valid session within the limit of attempts,
-    # then we cannot run the tests.
-    if not success:
-        raise IOError("Could not acquire valid {driver} browser session.".format(driver=browser_driver))
+        # If we were unable to get a valid session within the limit of attempts,
+        # then we cannot run the tests.
+        if not success:
+            raise IOError("Could not acquire valid {driver} browser session.".format(driver=browser_driver))
 
-    # Set the browser size to 1280x1024
-    world.browser.driver.set_window_size(1280, 1024)
+        world.browser.driver.set_window_size(1280, 1024)
+
+    else:
+        config = get_username_and_key()
+        world.browser = Browser(
+            'remote',
+            url="http://{}:{}@ondemand.saucelabs.com:80/wd/hub".format(config['username'], config['access-key']),
+            **make_desired_capabilities()
+        )
+        world.browser.driver.implicitly_wait(30)
+
+    world.absorb(world.browser.driver.session_id, 'jobid')
 
 
 @before.each_scenario
@@ -97,7 +151,6 @@ def clear_data(scenario):
     world.spew('scenario_dict')
 
 
-
 @after.each_scenario
 def reset_databases(scenario):
     '''
@@ -128,4 +181,6 @@ def teardown_browser(total):
     """
     Quit the browser after executing the tests.
     """
+    if world.SAUCE_ENABLED:
+        set_job_status(world.jobid, total.scenarios_ran == total.scenarios_passed)
     world.browser.quit()

common/djangoapps/terrain/steps.py

@@ -99,7 +99,7 @@ def i_am_logged_in_user(step):
 
 @step('I am not logged in$')
 def i_am_not_logged_in(step):
-    world.browser.cookies.delete()
+    world.visit('logout')
 
 
 @step('I am staff for course "([^"]*)"$')
@@ -138,10 +138,13 @@ def should_have_link_with_path_and_text(step, path, text):
 
 @step(r'should( not)? see "(.*)" (?:somewhere|anywhere) (?:in|on) (?:the|this) page')
 def should_see_in_the_page(step, doesnt_appear, text):
+    multiplier = 1
+    if world.SAUCE_ENABLED:
+        multiplier = 2
     if doesnt_appear:
-        assert world.browser.is_text_not_present(text, wait_time=5)
+        assert world.browser.is_text_not_present(text, wait_time=5*multiplier)
     else:
-        assert world.browser.is_text_present(text, wait_time=5)
+        assert world.browser.is_text_present(text, wait_time=5*multiplier)
 
 
 @step('I am logged in$')
@@ -150,7 +153,7 @@ def i_am_logged_in(step):
     world.log_in(username='robot', password='test')
     world.browser.visit(django_url('/'))
     # You should not see the login link
-    assert_equals(world.browser.find_by_css('a#login'), [])
+    assert world.is_css_not_present('a#login')
 
 
 @step(u'I am an edX user$')

common/djangoapps/xmodule_modifiers.py

@@ -76,7 +76,7 @@ def replace_course_urls(get_html, course_id):
     return _get_html
 
 
-def replace_static_urls(get_html, data_dir, course_id=None):
+def replace_static_urls(get_html, data_dir, course_id=None, static_asset_path=''):
     """
     Updates the supplied module with a new get_html function that wraps
     the old get_html function and substitutes urls of the form /static/...
@@ -85,7 +85,7 @@ def replace_static_urls(get_html, data_dir, course_id=None):
 
     @wraps(get_html)
     def _get_html():
-        return static_replace.replace_static_urls(get_html(), data_dir, course_id)
+        return static_replace.replace_static_urls(get_html(), data_dir, course_id, static_asset_path=static_asset_path)
     return _get_html
 
 

common/lib/xmodule/xmodule/html_module.py

@@ -33,11 +33,13 @@ class HtmlFields(object):
 
 
 class HtmlModule(HtmlFields, XModule):
-    js = {'coffee': [resource_string(__name__, 'js/src/javascript_loader.coffee'),
-                     resource_string(__name__, 'js/src/collapsible.coffee'),
-                     resource_string(__name__, 'js/src/html/display.coffee')
-                    ]
-         }
+    js = {
+        'coffee': [
+            resource_string(__name__, 'js/src/javascript_loader.coffee'),
+            resource_string(__name__, 'js/src/collapsible.coffee'),
+            resource_string(__name__, 'js/src/html/display.coffee')
+        ]
+    }
     js_module_name = "HTMLModule"
     css = {'scss': [resource_string(__name__, 'css/html/display.scss')]}
 
@@ -118,8 +120,10 @@ class HtmlDescriptor(HtmlFields, XmlDescriptor, EditingDescriptor):
             # from .html
             # 'filename' in html pointers is a relative path
             # (not same as 'html/blah.html' when the pointer is in a directory itself)
-            pointer_path = "{category}/{url_path}".format(category='html',
-                                                  url_path=name_to_pathname(location.name))
+            pointer_path = "{category}/{url_path}".format(
+                category='html',
+                url_path=name_to_pathname(location.name)
+            )
             base = path(pointer_path).dirname()
             # log.debug("base = {0}, base.dirname={1}, filename={2}".format(base, base.dirname(), filename))
             filepath = "{base}/{name}.html".format(base=base, name=filename)
@@ -164,19 +168,16 @@ class HtmlDescriptor(HtmlFields, XmlDescriptor, EditingDescriptor):
     # TODO (vshnayder): make export put things in the right places.
 
     def definition_to_xml(self, resource_fs):
-        '''If the contents are valid xml, write them to filename.xml.  Otherwise,
-        write just <html filename="" [meta-attrs="..."]> to filename.xml, and the html
+        ''' Write <html filename="" [meta-attrs="..."]> to filename.xml, and the html
         string to filename.html.
         '''
-        try:
-            return etree.fromstring(self.data)
-        except etree.XMLSyntaxError:
-            pass
 
-        # Not proper format.  Write html to file, return an empty tag
+        # Write html to file, return an empty tag
         pathname = name_to_pathname(self.url_name)
-        filepath = u'{category}/{pathname}.html'.format(category=self.category,
-                                                    pathname=pathname)
+        filepath = u'{category}/{pathname}.html'.format(
+            category=self.category,
+            pathname=pathname
+        )
 
         resource_fs.makedir(os.path.dirname(filepath), recursive=True, allow_recreate=True)
         with resource_fs.open(filepath, 'w') as filestream:
@@ -190,6 +191,7 @@ class HtmlDescriptor(HtmlFields, XmlDescriptor, EditingDescriptor):
         elt.set("filename", relname)
         return elt
 
+
 class AboutFields(object):
     display_name = String(
         help="Display name for this module",
@@ -202,12 +204,14 @@ class AboutFields(object):
         scope=Scope.content
     )
 
+
 class AboutModule(AboutFields, HtmlModule):
     """
     Overriding defaults but otherwise treated as HtmlModule.
     """
     pass
 
+
 class AboutDescriptor(AboutFields, HtmlDescriptor):
     """
     These pieces of course content are treated as HtmlModules but we need to overload where the templates are located
@@ -216,6 +220,7 @@ class AboutDescriptor(AboutFields, HtmlDescriptor):
     template_dir_name = "about"
     module_class = AboutModule
 
+
 class StaticTabFields(object):
     """
     The overrides for Static Tabs
@@ -241,6 +246,7 @@ class StaticTabModule(StaticTabFields, HtmlModule):
     """
     pass
 
+
 class StaticTabDescriptor(StaticTabFields, HtmlDescriptor):
     """
     These pieces of course content are treated as HtmlModules but we need to overload where the templates are located

common/lib/xmodule/xmodule/js/spec/helper.coffee

@@ -90,7 +90,13 @@ jasmine.stubbedHtml5Speeds = ['0.75', '1.0', '1.25', '1.50']
 jasmine.stubRequests = ->
   spyOn($, 'ajax').andCallFake (settings) ->
     if match = settings.url.match /youtube\.com\/.+\/videos\/(.+)\?v=2&alt=jsonc/
-      settings.success data: jasmine.stubbedMetadata[match[1]]
+      if settings.success
+        # match[1] - it's video ID
+        settings.success data: jasmine.stubbedMetadata[match[1]]
+      else {
+          always: (callback) ->
+            callback.call(window, {}, 'success');
+        }
     else if match = settings.url.match /static(\/.*)?\/subs\/(.+)\.srt\.sjson/
       settings.success jasmine.stubbedCaption
     else if settings.url.match /.+\/problem_get$/

common/lib/xmodule/xmodule/js/spec/video/general_spec.js

@@ -4,8 +4,6 @@
 
         beforeEach(function () {
             jasmine.stubRequests();
-            oldOTBD = window.onTouchBasedDevice;
-            window.onTouchBasedDevice = jasmine.createSpy('onTouchBasedDevice').andReturn(false);
             this.videosDefinition = '0.75:7tqY6eQzVhE,1.0:cogebirgzzM';
             this['7tqY6eQzVhE'] = '7tqY6eQzVhE';
             this['cogebirgzzM'] = 'cogebirgzzM';
@@ -16,7 +14,6 @@
             window.onYouTubePlayerAPIReady = undefined;
             window.onHTML5PlayerAPIReady = undefined;
             $('source').remove();
-            window.onTouchBasedDevice = oldOTBD;
         });
 
         describe('constructor', function () {
@@ -58,6 +55,46 @@
                         expect(this.state.speed).toEqual('0.75');
                     });
                 });
+
+                describe('Check Youtube link existence', function () {
+                    var statusList = {
+                        error: 'html5',
+                        timeout: 'html5',
+                        abort: 'html5',
+                        parsererror: 'html5',
+                        success: 'youtube',
+                        notmodified: 'youtube'
+                    };
+
+                    function stubDeffered(data, status) {
+                        return {
+                            always: function(callback) {
+                                callback.call(window, data, status);
+                            }
+                        }
+                    }
+
+                    function checkPlayer(videoType, data, status) {
+                        this.state = new window.Video('#example');
+                        spyOn(this.state , 'getVideoMetadata')
+                            .andReturn(stubDeffered(data, status));
+                        this.state.initialize('#example');
+
+                        expect(this.state.videoType).toEqual(videoType);
+                    }
+
+                    it('if video id is incorrect', function () {
+                        checkPlayer('html5', { error: {} }, 'success');
+                    });
+
+                    $.each(statusList, function(status, mode){
+                        it('Status:' + status + ', mode:' + mode, function () {
+                            checkPlayer(mode, {}, status);
+                        });
+                    });
+
+                });
+
             });
 
             describe('HTML5', function () {

common/lib/xmodule/xmodule/js/spec/video/video_player_spec.js

@@ -79,6 +79,8 @@
       it('create Youtube player', function() {
         var oldYT = window.YT;
 
+        jasmine.stubRequests();
+
         window.YT = {
             Player: function () { },
             PlayerState: oldYT.PlayerState

common/lib/xmodule/xmodule/js/src/video/01_initialize.js

@@ -30,8 +30,7 @@ function (VideoPlayer) {
      */
     return function (state, element) {
         _makeFunctionsPublic(state);
-        _initialize(state, element);
-        _renderElements(state);
+        state.initialize(element);
     };
 
     // ***************************************************************
@@ -56,59 +55,12 @@ function (VideoPlayer) {
         // Old private functions. Now also public so that can be
         // tested by Jasmine.
 
+        state.initialize = _.bind(initialize, state);
         state.parseSpeed = _.bind(parseSpeed, state);
         state.fetchMetadata = _.bind(fetchMetadata, state);
         state.parseYoutubeStreams = _.bind(parseYoutubeStreams, state);
         state.parseVideoSources = _.bind(parseVideoSources, state);
-    }
-
-    // function _initialize(element)
-    // The function set initial configuration and preparation.
-
-    function _initialize(state, element) {
-        // This is used in places where we instead would have to check if an element has a CSS class 'fullscreen'.
-        state.isFullScreen = false;
-
-        // The parent element of the video, and the ID.
-        state.el = $(element).find('.video');
-        state.id = state.el.attr('id').replace(/video_/, '');
-
-        // We store all settings passed to us by the server in one place. These are "read only", so don't
-        // modify them. All variable content lives in 'state' object.
-        state.config = {
-            element: element,
-
-            start:              state.el.data('start'),
-            end:                state.el.data('end'),
-
-            caption_data_dir:   state.el.data('caption-data-dir'),
-            caption_asset_path: state.el.data('caption-asset-path'),
-            show_captions:      (state.el.data('show-captions').toString().toLowerCase() === 'true'),
-            youtubeStreams:     state.el.data('streams'),
-
-            sub:                state.el.data('sub'),
-            mp4Source:          state.el.data('mp4-source'),
-            webmSource:         state.el.data('webm-source'),
-            oggSource:          state.el.data('ogg-source'),
-
-            fadeOutTimeout:     1400,
-
-            availableQualities: ['hd720', 'hd1080', 'highres']
-        };
-
-        if (!(_parseYouTubeIDs(state))) {
-            // If we do not have YouTube ID's, try parsing HTML5 video sources.
-            _prepareHTML5Video(state);
-        }
-
-        _configureCaptions(state);
-        _setPlayerMode(state);
-
-        // Possible value are: 'visible', 'hiding', and 'invisible'.
-        state.controlState = 'visible';
-        state.controlHideTimeout = null;
-        state.captionState = 'visible';
-        state.captionHideTimeout = null;
+        state.getVideoMetadata = _.bind(getVideoMetadata, state);
     }
 
     // function _renderElements(state)
@@ -228,12 +180,83 @@ function (VideoPlayer) {
         state.setSpeed($.cookie('video_speed'));
     }
 
+    function _setConfigurations(state) {
+        _configureCaptions(state);
+        _setPlayerMode(state);
+
+        // Possible value are: 'visible', 'hiding', and 'invisible'.
+        state.controlState = 'visible';
+        state.controlHideTimeout = null;
+        state.captionState = 'visible';
+        state.captionHideTimeout = null;
+    }
+
     // ***************************************************************
     // Public functions start here.
     // These are available via the 'state' object. Their context ('this' keyword) is the 'state' object.
     // The magic private function that makes them available and sets up their context is makeFunctionsPublic().
     // ***************************************************************
 
+    // function initialize(element)
+    // The function set initial configuration and preparation.
+
+    function initialize(element) {
+        var _this = this;
+        // This is used in places where we instead would have to check if an element has a CSS class 'fullscreen'.
+        this.isFullScreen = false;
+
+        // The parent element of the video, and the ID.
+        this.el = $(element).find('.video');
+        this.id = this.el.attr('id').replace(/video_/, '');
+
+        // We store all settings passed to us by the server in one place. These are "read only", so don't
+        // modify them. All variable content lives in 'state' object.
+        this.config = {
+            element: element,
+
+            start:              this.el.data('start'),
+            end:                this.el.data('end'),
+
+            caption_data_dir:   this.el.data('caption-data-dir'),
+            caption_asset_path: this.el.data('caption-asset-path'),
+            show_captions:      (this.el.data('show-captions').toString().toLowerCase() === 'true'),
+            youtubeStreams:     this.el.data('streams'),
+
+            sub:                this.el.data('sub'),
+            mp4Source:          this.el.data('mp4-source'),
+            webmSource:         this.el.data('webm-source'),
+            oggSource:          this.el.data('ogg-source'),
+
+            fadeOutTimeout:     1400,
+
+            availableQualities: ['hd720', 'hd1080', 'highres']
+        };
+
+        if (!(_parseYouTubeIDs(this))) {
+            // If we do not have YouTube ID's, try parsing HTML5 video sources.
+            _prepareHTML5Video(this);
+            _setConfigurations(this);
+            _renderElements(this);
+        } else {
+            this.getVideoMetadata()
+                .always(function(json, status) {
+                    var err = $.isPlainObject(json.error) ||
+                                (status !== "success" && status !== "notmodified");
+
+                    if (err){
+                        // When the youtube link doesn't work for any reason
+                        // (for example, the great firewall in china) any
+                        // alternate sources should automatically play.
+                        _prepareHTML5Video(_this);
+                        _this.el.find('a.quality_control').hide();
+                    }
+
+                    _setConfigurations(_this);
+                    _renderElements(_this);
+                });
+        }
+    }
+
     // function parseYoutubeStreams(state, youtubeStreams)
     //
     //     Take a string in the form:
@@ -297,9 +320,9 @@ function (VideoPlayer) {
         this.metadata = {};
 
         $.each(this.videos, function (speed, url) {
-            $.get('https://gdata.youtube.com/feeds/api/videos/' + url + '?v=2&alt=jsonc', (function(data) {
+            _this.getVideoMetadata(url, function(data) {
                 _this.metadata[data.data.id] = data.data;
-            }), 'jsonp');
+            });
         });
     }
 
@@ -329,6 +352,24 @@ function (VideoPlayer) {
         }
     }
 
+    function getVideoMetadata(url, callback) {
+        var successHandler, xhr;
+
+        if (typeof url !== 'string') {
+            url = this.videos['1.0'] || '';
+        }
+
+        successHandler = ($.isFunction(callback)) ? callback : null;
+        xhr = $.ajax({
+            url: 'https://gdata.youtube.com/feeds/api/videos/' + url + '?v=2&alt=jsonc',
+            timeout: 500,
+            dataType: 'jsonp',
+            success: successHandler
+        });
+
+        return xhr;
+    }
+
     function stopBuffering() {
         var video;
 

common/lib/xmodule/xmodule/modulestore/inheritance.py

@@ -9,7 +9,8 @@ INHERITABLE_METADATA = (
     # intended to be set per-course, but can be overridden in for specific
     # elements.  Can be a float.
     'days_early_for_beta',
-    'giturl'  # for git edit link
+    'giturl',  # for git edit link
+    'static_asset_path',       # for static assets placed outside xcontent contentstore
 )
 
 

common/lib/xmodule/xmodule/modulestore/tests/test_mongo.py

@@ -10,7 +10,9 @@ from xblock.runtime import KeyValueStore, InvalidScopeError
 from xmodule.tests import DATA_DIR
 from xmodule.modulestore import Location
 from xmodule.modulestore.mongo import MongoModuleStore, MongoKeyValueStore
-from xmodule.modulestore.xml_importer import import_from_xml
+from xmodule.modulestore.draft import DraftModuleStore
+from xmodule.modulestore.xml_importer import import_from_xml, perform_xlint
+from xmodule.contentstore.mongo import MongoContentStore
 
 from xmodule.modulestore.tests.test_modulestore import check_path_to_location
 
@@ -35,7 +37,7 @@ class TestMongoModuleStore(object):
         # is ok only as long as none of the tests modify the db.
         # If (when!) that changes, need to either reload the db, or load
         # once and copy over to a tmp db for each test.
-        cls.store = cls.initdb()
+        cls.store, cls.content_store, cls.draft_store = cls.initdb()
 
     @classmethod
     def teardownClass(cls):
@@ -46,10 +48,28 @@ class TestMongoModuleStore(object):
     def initdb():
         # connect to the db
         store = MongoModuleStore(HOST, DB, COLLECTION, FS_ROOT, RENDER_TEMPLATE, default_class=DEFAULT_CLASS)
+        # since MongoModuleStore and MongoContentStore are basically assumed to be together, create this class
+        # as well
+        content_store = MongoContentStore(HOST, DB)
+        #
+        # Also test draft store imports
+        #
+        draft_store = DraftModuleStore(HOST, DB, COLLECTION, FS_ROOT, RENDER_TEMPLATE, default_class=DEFAULT_CLASS)
         # Explicitly list the courses to load (don't want the big one)
-        courses = ['toy', 'simple']
-        import_from_xml(store, DATA_DIR, courses)
-        return store
+        courses = ['toy', 'simple', 'simple_with_draft']
+        import_from_xml(store, DATA_DIR, courses, draft_store=draft_store, static_content_store=content_store)
+
+        # also test a course with no importing of static content
+        import_from_xml(
+            store,
+            DATA_DIR,
+            ['test_import_course'],
+            static_content_store=content_store,
+            do_import_static=False,
+            verbose=True
+        )
+
+        return store, content_store, draft_store
 
     @staticmethod
     def destroy_db(connection):
@@ -77,10 +97,12 @@ class TestMongoModuleStore(object):
     def test_get_courses(self):
         '''Make sure the course objects loaded properly'''
         courses = self.store.get_courses()
-        assert_equals(len(courses), 2)
+        assert_equals(len(courses), 4)
         courses.sort(key=lambda c: c.id)
         assert_equals(courses[0].id, 'edX/simple/2012_Fall')
-        assert_equals(courses[1].id, 'edX/toy/2012_Fall')
+        assert_equals(courses[1].id, 'edX/simple_with_draft/2012_Fall')
+        assert_equals(courses[2].id, 'edX/test_import_course/2012_Fall')
+        assert_equals(courses[3].id, 'edX/toy/2012_Fall')
 
     def test_loads(self):
         assert_not_equals(
@@ -112,6 +134,13 @@ class TestMongoModuleStore(object):
         '''Make sure that path_to_location works'''
         check_path_to_location(self.store)
 
+    def test_xlinter(self):
+        '''
+        Run through the xlinter, we know the 'toy' course has violations, but the
+        number will continue to grow over time, so just check > 0
+        '''
+        assert_not_equals(perform_xlint(DATA_DIR, ['toy']), 0)
+
     def test_get_courses_has_no_templates(self):
         courses = self.store.get_courses()
         for course in courses:
@@ -129,7 +158,7 @@ class TestMongoModuleStore(object):
 
             Assumes the information is desired for courses[1] ('toy' course).
             """
-            return courses[1].tabs[index]['name']
+            return courses[2].tabs[index]['name']
 
         # There was a bug where model.save was not getting called after the static tab name
         # was set set for tabs that have a URL slug. 'Syllabus' and 'Resources' fall into that

common/lib/xmodule/xmodule/modulestore/xml_importer.py

@@ -51,7 +51,10 @@ def import_static_content(modules, course_loc, course_data_path, static_content_
                     content.thumbnail_location = thumbnail_location
 
                 #then commit the content
-                static_content_store.save(content)
+                try:
+                    static_content_store.save(content)
+                except Exception as err:
+                    log.exception('Error importing {0}, error={1}'.format(fullname_with_subpath, err))
 
                 #store the remapping information which will be needed to subsitute in the module data
                 remap_dict[fullname_with_subpath] = content_loc.name
@@ -64,7 +67,8 @@ def import_static_content(modules, course_loc, course_data_path, static_content_
 def import_from_xml(store, data_dir, course_dirs=None,
                     default_class='xmodule.raw_module.RawDescriptor',
                     load_error_modules=True, static_content_store=None, target_location_namespace=None,
-                    verbose=False, draft_store=None):
+                    verbose=False, draft_store=None,
+                    do_import_static=True):
     """
     Import the specified xml data_dir into the "store" modulestore,
     using org and course as the location org and course.
@@ -77,6 +81,10 @@ def import_from_xml(store, data_dir, course_dirs=None,
     expects a 'url_name' as an identifier to where things are on disk e.g. ../policies/<url_name>/policy.json as well as metadata keys in
     the policy.json. so we need to keep the original url_name during import
 
+    do_import_static: if False, then static files are not imported into the static content store.  This can be employed for courses which
+                      have substantial unchanging static content, which is to inefficient to import every time the course is loaded.
+                      Static content for some courses may also be served directly by nginx, instead of going through django.
+
     """
 
     xml_module_store = XMLModuleStore(
@@ -116,8 +124,17 @@ def import_from_xml(store, data_dir, course_dirs=None,
                     course_data_path = path(data_dir) / module.data_dir
                     course_location = module.location
 
+                    log.debug('======> IMPORTING course to location {0}'.format(course_location))
+
                     module = remap_namespace(module, target_location_namespace)
 
+                    if not do_import_static:
+                        module.lms.static_asset_path = module.data_dir			# for old-style xblock where this was actually linked to kvs
+                        module._model_data['static_asset_path'] = module.data_dir
+                        log.debug('course static_asset_path={0}'.format(module.lms.static_asset_path))
+
+                    log.debug('course data_dir={0}'.format(module.data_dir))
+
                     # cdodge: more hacks (what else). Seems like we have a problem when importing a course (like 6.002) which
                     # does not have any tabs defined in the policy file. The import goes fine and then displays fine in LMS,
                     # but if someone tries to add a new tab in the CMS, then the LMS barfs because it expects that -
@@ -129,18 +146,35 @@ def import_from_xml(store, data_dir, course_dirs=None,
                                        {"type": "wiki", "name": "Wiki"}]  # note, add 'progress' when we can support it on Edge
 
                     import_module(module, store, course_data_path, static_content_store, course_location,
-                                  target_location_namespace or course_location)
+                                  target_location_namespace or course_location, do_import_static=do_import_static)
 
                     course_items.append(module)
 
             # then import all the static content
-            if static_content_store is not None:
+            if static_content_store is not None and do_import_static:
                 _namespace_rename = target_location_namespace if target_location_namespace is not None else course_location
 
                 # first pass to find everything in /static/
                 import_static_content(xml_module_store.modules[course_id], course_location, course_data_path, static_content_store,
                                       _namespace_rename, subpath='static', verbose=verbose)
 
+            elif verbose and not do_import_static:
+                log.debug('Skipping import of static content, since do_import_static={0}'.format(do_import_static))
+
+            # no matter what do_import_static is, import "static_import" directory
+
+            # This is needed because the "about" pages (eg "overview") are loaded via load_extra_content, and
+            # do not inherit the lms metadata from the course module, and thus do not get "static_content_store"
+            # properly defined.   Static content referenced in those extra pages thus need to come through the
+            # c4x:// contentstore, unfortunately.  Tell users to copy that content into the "static_import" subdir.
+
+            simport = 'static_import'
+            if os.path.exists(course_data_path / simport):
+                _namespace_rename = target_location_namespace if target_location_namespace is not None else course_location
+
+                import_static_content(xml_module_store.modules[course_id], course_location, course_data_path, static_content_store,
+                                      _namespace_rename, subpath=simport, verbose=verbose)
+
             # finally loop through all the modules
             for module in xml_module_store.modules[course_id].itervalues():
                 if module.category == 'course':
@@ -156,7 +190,8 @@ def import_from_xml(store, data_dir, course_dirs=None,
                     log.debug('importing module location {0}'.format(module.location))
 
                 import_module(module, store, course_data_path, static_content_store, course_location,
-                              target_location_namespace if target_location_namespace else course_location)
+                              target_location_namespace if target_location_namespace else course_location,
+                              do_import_static=do_import_static)
 
             # now import any 'draft' items
             if draft_store is not None:
@@ -176,7 +211,8 @@ def import_from_xml(store, data_dir, course_dirs=None,
 
 
 def import_module(module, store, course_data_path, static_content_store,
-                  source_course_location, dest_course_location, allow_not_found=False):
+                  source_course_location, dest_course_location, allow_not_found=False,
+                  do_import_static=True):
 
     logging.debug('processing import of module {0}...'.format(module.location.url()))
 
@@ -196,7 +232,7 @@ def import_module(module, store, course_data_path, static_content_store,
     else:
         module_data = content
 
-    if isinstance(module_data, basestring):
+    if isinstance(module_data, basestring) and do_import_static:
         # we want to convert all 'non-portable' links in the module_data (if it is a string) to
         # portable strings (e.g. /static/)
         module_data = rewrite_nonportable_content_links(
@@ -212,6 +248,15 @@ def import_module(module, store, course_data_path, static_content_store,
 
     # NOTE: It's important to use own_metadata here to avoid writing
     # inherited metadata everywhere.
+
+    # remove any export/import only xml_attributes which are used to wire together draft imports
+    if 'parent_sequential_url' in module.xml_attributes:
+        del module.xml_attributes['parent_sequential_url']
+
+    if 'index_in_children_list' in module.xml_attributes:
+        del module.xml_attributes['index_in_children_list']
+    module.save()
+
     store.update_metadata(module.location, dict(own_metadata(module)))
 
 
@@ -281,7 +326,7 @@ def import_course_draft(xml_module_store, store, draft_store, course_data_path,
                         # this is to make sure private only verticals show up in the list of children since
                         # they would have been filtered out from the non-draft store export
                         if module.location.category == 'vertical':
-                            module.location = module.location._replace(revision=None)
+                            non_draft_location = module.location._replace(revision=None)
                             sequential_url = module.xml_attributes['parent_sequential_url']
                             index = int(module.xml_attributes['index_in_children_list'])
 
@@ -291,15 +336,12 @@ def import_course_draft(xml_module_store, store, draft_store, course_data_path,
                             seq_location = seq_location._replace(org=target_location_namespace.org,
                                                                  course=target_location_namespace.course
                                                                  )
-                            sequential = store.get_item(seq_location)
+                            sequential = store.get_item(seq_location, depth=0)
 
-                            if module.location.url() not in sequential.children:
-                                sequential.children.insert(index, module.location.url())
+                            if non_draft_location.url() not in sequential.children:
+                                sequential.children.insert(index, non_draft_location.url())
                                 store.update_children(sequential.location, sequential.children)
 
-                            del module.xml_attributes['parent_sequential_url']
-                            del module.xml_attributes['index_in_children_list']
-
                         import_module(module, draft_store, course_data_path, static_content_store,
                                       source_location_namespace, target_location_namespace, allow_not_found=True)
                         for child in module.get_children():

common/lib/xmodule/xmodule/tests/test_export.py

@@ -89,12 +89,6 @@ class RoundTripTestCase(unittest.TestCase):
         print("Checking module equality")
         for location in initial_import.modules[course_id].keys():
             print("Checking", location)
-            if location.category == 'html':
-                print(
-                    "Skipping html modules--they can't import in"
-                    " final form without writing files..."
-                )
-                continue
             self.assertEquals(initial_import.modules[course_id][location],
                               second_import.modules[course_id][location])
 

common/test/data/simple_with_draft/README.md

+This is a simple, but non-trivial, course using multiple module types and some nested structure.
+ 

common/test/data/simple_with_draft/course.xml

+<course name="A Simple Course" org="edX" course="simple_with_draft" graceperiod="1 day 5 hours 59 minutes 59 seconds" slug="2012_Fall">
+  <chapter name="Overview">
+    <video name="Welcome" youtube_id_0_75="izygArpw-Qo" youtube_id_1_0="p2Q6BrNhdh8" youtube_id_1_25="1EeWXzPdhSA" youtube_id_1_5="rABDYkeK0x8"/>
+    <videosequence format="Lecture Sequence" name="A simple sequence">
+      <html name="toylab" filename="toylab"/>
+      <video name="S0V1: Video Resources" youtube_id_0_75="EuzkdzfR0i8" youtube_id_1_0="1bK-WdDi6Qw" youtube_id_1_25="0v1VzoDVUTM" youtube_id_1_5="Bxk_-ZJb240"/>
+    </videosequence>
+    <section name="Lecture 2">
+    <sequential>
+        <video youtube_id_1_0="TBvX7HzxexQ"/>
+        <problem name="L1 Problem 1" points="1" type="lecture" showanswer="attempted" filename="L1_Problem_1" rerandomize="never"/>
+      </sequential>
+    </section>
+  </chapter>
+  <chapter name="Chapter 2" url_name='chapter_2'>
+    <section name="Problem Set 1">
+      <sequential>
+        <problem type="lecture" showanswer="attempted" rerandomize="true" display_name="A simple coding problem" name="Simple coding problem" filename="ps01-simple" url_name="ps01-simple"/>
+      </sequential>
+    </section>
+    <video name="Lost Video" youtube_id_1_0="TBvX7HzxexQ"/>
+    <sequential format="Lecture Sequence" url_name='test_sequence'>
+        <vertical url_name='test_vertical'>
+            <html url_name='test_html'>
+                Foobar
+            </html>
+        </vertical>
+    </sequential>
+  </chapter>
+
+</course>

common/test/data/simple_with_draft/drafts/vertical/test_vertical.xml

+<vertical url_name='test_vertical' parent_sequential_url='i4x://edX/simple_with_draft/sequential/test_sequence' index_in_children_list="0">
+    <html url_name='test_html'>
+        Foobar - edit in draft
+    </html>
+</vertical>
\ No newline at end of file

common/test/data/simple_with_draft/html/toylab.html

+<b>Lab 2A: Superposition Experiment</b>
+
+<p>Isn't the toy course great?</p>

common/test/data/simple_with_draft/problem/L1_Problem_1.xml

+<?xml version="1.0"?>
+<problem>
+  <p>
+    <h1>Finger Exercise 1</h1>
+  </p>
+  <p>
+Here are two definitions: </p>
+  <ol class="enumerate">
+    <li>
+      <p>
+Declarative knowledge refers to statements of fact. </p>
+    </li>
+    <li>
+      <p>
+Imperative knowledge refers to 'how to' methods. </p>
+    </li>
+  </ol>
+  <p>
+Which of the following choices is correct? </p>
+  <ol class="enumerate">
+    <li>
+      <p>
+Statement 1 is true, Statement 2 is false </p>
+    </li>
+    <li>
+      <p>
+Statement 1 is false, Statement 2 is true </p>
+    </li>
+    <li>
+      <p>
+Statement 1 and Statement 2 are both false </p>
+    </li>
+    <li>
+      <p>
+Statement 1 and Statement 2 are both true </p>
+    </li>
+  </ol>
+  <p>
+    <symbolicresponse answer="4">
+      <textline size="90" math="1"/>
+    </symbolicresponse>
+  </p>
+</problem>

common/test/data/simple_with_draft/problem/ps01-simple.xml

+<problem><style media="all" type="text/css"/>
+<text><h2>Paying Off Credit Card Debt</h2>
+<p> Each month, a credit
+      card statement will come with the option for you to pay a
+      minimum amount of your charge, usually 2% of the balance due.
+      However, the credit card company earns money by charging
+      interest on the balance that you don't pay. So even if you
+      pay credit card payments on time, interest is still accruing
+      on the outstanding balance.</p>
+<p >Say you've made a
+      $5,000 purchase on a credit card with 18% annual interest
+      rate and 2% minimum monthly payment rate. After a year, how
+      much is the remaining balance? Use the following
+      equations.</p>
+<blockquote>
+<p><strong>Minimum monthly payment</strong>
+= (Minimum monthly payment rate) x (Balance)<br/>
+        (Minimum monthly payment gets split into interest paid and
+        principal paid)<br/>
+<strong>Interest Paid</strong> = (Annual interest rate) / (12
+        months) x (Balance)<br/>
+<strong>Principal paid</strong> = (Minimum monthly payment) -
+        (Interest paid)<br/>
+<strong>Remaining balance</strong> = Balance - (Principal
+        paid)</p>
+</blockquote>
+<p >For month 1, compute the minimum monthly payment by taking 2% of the balance.</p>
+<blockquote xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:v="urn:schemas-microsoft-com:vml" xmlns:x="urn:schemas-microsoft-com:office:excel">
+<p><strong>Minimum monthly payment</strong>
+= .02 x $5000 = $100</p>
+<p>We can't simply deduct this from the balance because
+        there is compounding interest. Of this $100 monthly
+        payment, compute how much will go to paying off interest
+        and how much will go to paying off the principal. Remember
+        that it's the annual interest rate that is given, so we
+        need to divide it by 12 to get the monthly interest
+        rate.</p>
+<p><strong>Interest paid</strong> = .18/12 x $5000 =
+        $75<br/>
+<strong>Principal paid</strong> = $100 - $75 = $25</p>
+<p>The remaining balance at the end of the first month will
+        be the principal paid this month subtracted from the
+        balance at the start of the month.</p>
+<p><strong>Remaining balance</strong> = $5000 - $25 =
+        $4975</p>
+</blockquote>
+<p xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:v="urn:schemas-microsoft-com:vml" xmlns:x="urn:schemas-microsoft-com:office:excel">For month 2, we
+      repeat the same steps.</p>
+<blockquote xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:v="urn:schemas-microsoft-com:vml" xmlns:x="urn:schemas-microsoft-com:office:excel">
+<p><strong>Minimum monthly payment</strong>
+= .02 x $4975 = $99.50<br/>
+<strong>Interest Paid</strong> = .18/12 x $4975 =
+        $74.63<br/>
+<strong>Principal Paid</strong> = $99.50 - $74.63 =
+        $24.87<br/>
+<strong>Remaining Balance</strong> = $4975 - $24.87 =
+        $4950.13</p>
+</blockquote>
+<p xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:v="urn:schemas-microsoft-com:vml" xmlns:x="urn:schemas-microsoft-com:office:excel">After 12 months, the
+      total amount paid is $1167.55, leaving an outstanding balance
+      of $4708.10. Pretty depressing!</p>
+</text></problem>

common/test/data/test_import_course/about/end_date.html

+TBD

common/test/data/test_import_course/chapter/vertical_container.xml

+<sequential>
+    <sequential filename='vertical_sequential' slug='vertical_sequential' />
+</sequential>
\ No newline at end of file

common/test/data/test_import_course/course.xml

+<course org="edX" course="test_import_course" url_name="2012_Fall"/>
\ No newline at end of file

common/test/data/test_import_course/course/2012_Fall.xml

+<course>
+    <textbook title="Textbook" book_url="https://s3.amazonaws.com/edx-textbooks/guttag_computation_v3/"/>
+  <chapter url_name="Overview">
+    <videosequence url_name="Toy_Videos">
+      <html url_name="secret:toylab"/>
+      <html url_name="toyjumpto"/>
+      <html url_name="toyhtml"/>
+      <html url_name="nonportable"/>
+      <html url_name="nonportable_link"/>
+      <video url_name="Video_Resources" youtube_id_1_0="1bK-WdDi6Qw" display_name="Video Resources"/>
+    </videosequence>
+    <video url_name="Welcome" youtube_id_1_0="p2Q6BrNhdh8" display_name="Welcome"/>
+    <video url_name="video_123456789012" youtube_id_1_0="p2Q6BrNhdh8" display_name='Test Video'/>
+    <video url_name="video_4f66f493ac8f" youtube_id_1_0="p2Q6BrNhdh8"/>
+  </chapter>
+  <chapter url_name="secret:magic"/>
+  <chapter url_name="poll_test"/>
+  <chapter url_name="vertical_container"/>
+  <chapter url_name="handout_container"/>
+</course>

common/test/data/test_import_course/info/handouts.html

+<a href='/static/handouts/sample_handout.txt'>Sample</a>
\ No newline at end of file

common/test/data/test_import_course/policies/2012_Fall.json

+{
+    "course/2012_Fall": {
+        "graceperiod": "2 days 5 hours 59 minutes 59 seconds",
+        "start": "2015-07-17T12:00",
+        "display_name": "Toy Course",
+        "graded": "true",
+	"tabs": [
+          {"type": "courseware"},
+          {"type": "course_info", "name": "Course Info"},
+          {"type": "static_tab", "url_slug": "syllabus", "name": "Syllabus"},
+          {"type": "static_tab", "url_slug": "resources", "name": "Resources"},
+          {"type": "discussion", "name": "Discussion"},
+          {"type": "wiki", "name": "Wiki"},
+          {"type": "progress", "name": "Progress"}
+        ]
+    },
+    "chapter/Overview": {
+        "display_name": "Overview"
+    },
+    "videosequence/Toy_Videos": {
+        "display_name": "Toy Videos",
+        "format": "Lecture Sequence"
+    },
+    "html/secret:toylab": {
+        "display_name": "Toy lab"
+    },
+    "video/Video_Resources": {
+        "display_name": "Video Resources"
+    },
+    "video/Welcome": {
+        "display_name": "Welcome"
+    }
+}

common/test/data/test_import_course/sequential/vertical_sequential.xml

+<sequential>
+    <vertical filename="vertical_test" slug="vertical_test" />
+    <html slug="unicode">…</html>
+</sequential>
\ No newline at end of file

common/test/data/test_import_course/static_import/should_be_imported.html

+<p>this file should be in the contentstore</p>

common/test/data/test_import_course/vertical/vertical_test.xml

+<sequential>
+        <video display_name="default" youtube_id_0_75="JMD_ifUUfsU" youtube_id_1_0="OEoXaMPEzfM" youtube_id_1_25="AKqURZnYqpk" youtube_id_1_5="DYpADpL7jAY" name="sample_video"/>
+        <video url_name="separate_file_video"/>
+        <poll_question name="T1_changemind_poll_foo_2" display_name="Change your answer" reset="false">
+        <p>Have you changed your mind?</p>
+        <answer id="yes">Yes</answer>
+        <answer id="no">No</answer>
+    </poll_question>
+</sequential>

common/test/data/test_import_course/video/separate_file_video.xml

+<video display_name="default" youtube_id_0_75="JMD_ifUUfsU" youtube_id_1_0="OEoXaMPEzfM" youtube_id_1_25="AKqURZnYqpk" youtube_id_1_5="DYpADpL7jAY" name="sample_video"/>

common/test/data/toy/course/2012_Fall.xml

@@ -8,6 +8,8 @@
       <html url_name="nonportable"/>
       <html url_name="nonportable_link"/>
       <html url_name="badlink"/>
+      <html url_name="with_styling"/>
+      <html url_name="just_img"/>
       <video url_name="Video_Resources" youtube_id_1_0="1bK-WdDi6Qw" display_name="Video Resources"/>
     </videosequence>
     <video url_name="Welcome" youtube_id_1_0="p2Q6BrNhdh8" display_name="Welcome"/>

common/test/data/toy/html/just_img.html

+<img src="/static/foo_bar.jpg" />
\ No newline at end of file

common/test/data/toy/html/just_img.xml

+<html filename="just_img.html"/>
\ No newline at end of file

common/test/data/toy/html/with_styling.html

+<p style="font:italic bold 72px/30px Georgia, serif; color: red; ">Red text here</p>
\ No newline at end of file

common/test/data/toy/html/with_styling.xml

+<html filename="with_styling.html"/>
\ No newline at end of file

docs/internal/testing.md

@@ -136,6 +136,25 @@ To run a single nose test:
 
     nosetests common/lib/xmodule/xmodule/tests/test_stringify.py:test_stringify
 
+To run a single test and get stdout, with proper env config:
+
+    python manage.py cms --settings test test contentstore.tests.test_import_nostatic -s
+
+To run a single test and get stdout and get coverage:
+
+    python -m coverage run --rcfile=./common/lib/xmodule/.coveragerc which ./manage.py cms --settings test test --traceback --logging-clear-handlers --liveserver=localhost:8000-9000 contentstore.tests.test_import_nostatic -s # cms example 
+    python -m coverage run --rcfile=./lms/.coveragerc which ./manage.py lms --settings test test --traceback --logging-clear-handlers --liveserver=localhost:8000-9000  courseware.tests.test_module_render -s # lms example
+
+generate coverage report:
+
+    coverage report --rcfile=./common/lib/xmodule/.coveragerc 
+
+or to get html report:
+
+    coverage html --rcfile=./common/lib/xmodule/.coveragerc
+    
+then browse reports/common/lib/xmodule/cover/index.html    
+
 
 Very handy: if you uncomment the `pdb=1` line in `setup.cfg`, it will drop you into pdb on error.  This lets you go up and down the stack and see what the values of the variables are.  Check out [the pdb documentation](http://docs.python.org/library/pdb.html)
 

jenkins/test_acceptance.sh

@@ -30,10 +30,22 @@ TESTS_FAILED=0
 # /usr/bin/Xvfb :1 -screen 0 1024x268x24
 # This allows us to run Chrome without a display
 export DISPLAY=:1
+SKIP_TESTS=""
+
+# Testing for the existance of these environment variables
+if [ ! -z ${LETTUCE_BROWSER+x} ]; then
+	SKIP_TESTS="--tag -skip_$LETTUCE_BROWSER"
+fi
+if [ ! -z ${SAUCE_ENABLED+x} ]; then
+	# SAUCE_INFO is a - seperated string PLATFORM-BROWSER-VERSION-DEVICE
+	# Error checking is done in the setting up of the browser
+	IFS='-' read -a SAUCE <<< "${SAUCE_INFO}"
+	SKIP_TESTS="--tag -skip_sauce --tag -skip_${SAUCE[1]}"
+fi
 
 # Run the lms and cms acceptance tests
 # (the -v flag turns off color in the output)
-rake test_acceptance_lms["-v 3"] || TESTS_FAILED=1
-rake test_acceptance_cms["-v 3"] || TESTS_FAILED=1
+rake test_acceptance_lms["-v 3 $SKIP_TESTS"] || TESTS_FAILED=1
+rake test_acceptance_cms["-v 3 $SKIP_TESTS"] || TESTS_FAILED=1
 
 [ $TESTS_FAILED == '0' ]

lms/djangoapps/courseware/courses.py

@@ -81,8 +81,8 @@ def get_opt_course_with_access(user, course_id, action):
 def course_image_url(course):
     """Try to look up the image url for the course.  If it's not found,
     log an error and return the dead link"""
-    if modulestore().get_modulestore_type(course.location.course_id) == XML_MODULESTORE_TYPE:
-        return '/static/' + course.data_dir + "/images/course_image.jpg"
+    if course.lms.static_asset_path or modulestore().get_modulestore_type(course.location.course_id) == XML_MODULESTORE_TYPE:
+        return '/static/' + (course.lms.static_asset_path or getattr(course, 'data_dir', '')) + "/images/course_image.jpg"
     else:
         loc = course.location._replace(tag='c4x', category='asset', name=course.course_image)
         _path = StaticContent.get_url_path_from_location(loc)
@@ -156,7 +156,8 @@ def get_course_about_section(course, section_key):
                 model_data_cache,
                 course.id,
                 not_found_ok=True,
-                wrap_xmodule_display=False
+                wrap_xmodule_display=False,
+                static_asset_path=course.lms.static_asset_path
             )
 
             html = ''
@@ -204,7 +205,8 @@ def get_course_info_section(request, course, section_key):
         loc,
         model_data_cache,
         course.id,
-        wrap_xmodule_display=False
+        wrap_xmodule_display=False,
+        static_asset_path=course.lms.static_asset_path
     )
 
     html = ''
@@ -242,7 +244,8 @@ def get_course_syllabus_section(course, section_key):
                 return replace_static_urls(
                     htmlFile.read().decode('utf-8'),
                     getattr(course, 'data_dir', None),
-                    course_id=course.location.course_id
+                    course_id=course.location.course_id,
+                    static_asset_path=course.lms.static_asset_path,
                 )
         except ResourceNotFoundError:
             log.exception("Missing syllabus section {key} in course {url}".format(

lms/djangoapps/courseware/features/login.feature

@@ -11,7 +11,8 @@ Feature: Login in as a registered user
     And I submit my credentials on the login form
     Then I should see the login error message "This account has not been activated"
 
-    # CHROME ONLY, firefox will not redirect properly
+    # firefox will not redirect properly when the whole suite is run
+    @skip_firefox
     Scenario: Login to an activated account
     Given I am an edX user
     And I am an activated user

lms/djangoapps/courseware/features/signup.feature

@@ -3,7 +3,8 @@ Feature: Sign in
   As a new user
   I want to signup for a student account
 
-  # CHROME ONLY, firefox will not redirect properly
+  # firefox will not redirect properly
+  @skip_firefox
   Scenario: Sign up from the homepage
     Given I visit the homepage
     When I click the link with the text "Register Now"

lms/djangoapps/courseware/features/video.feature

@@ -11,6 +11,8 @@ Feature: Video component
   Given the course has a Video component in Youtube mode
   Then when I view the video it has rendered in Youtube mode
 
+  # Firefox doesn't have HTML5
+  @skip_firefox
   Scenario: Autoplay is enabled in LMS for a Video component
   Given the course has a Video component in HTML5 mode
-  Then when I view the video it has autoplay enabled
+  Then when I view the video it has autoplay enabled
\ No newline at end of file

lms/djangoapps/courseware/module_render.py

@@ -124,7 +124,8 @@ def toc_for_course(user, request, course, active_chapter, active_section, model_
 
 def get_module(user, request, location, model_data_cache, course_id,
                position=None, not_found_ok=False, wrap_xmodule_display=True,
-               grade_bucket_type=None, depth=0):
+               grade_bucket_type=None, depth=0,
+               static_asset_path=''):
     """
     Get an instance of the xmodule class identified by location,
     setting the state based on an existing StudentModule, or creating one if none
@@ -141,6 +142,10 @@ def get_module(user, request, location, model_data_cache, course_id,
                                 position within module
       - depth                 : number of levels of descendents to cache when loading this module.
                                 None means cache all descendents
+      - static_asset_path     : static asset path to use (overrides descriptor's value); needed
+                                by get_course_info_section, because info section modules
+                                do not have a course as the parent module, and thus do not
+                                inherit this lms key value.
 
     Returns: xmodule instance, or None if the user does not have access to the
     module.  If there's an error, will try to return an instance of ErrorModule
@@ -152,7 +157,8 @@ def get_module(user, request, location, model_data_cache, course_id,
         return get_module_for_descriptor(user, request, descriptor, model_data_cache, course_id,
                                          position=position,
                                          wrap_xmodule_display=wrap_xmodule_display,
-                                         grade_bucket_type=grade_bucket_type)
+                                         grade_bucket_type=grade_bucket_type,
+                                         static_asset_path=static_asset_path)
     except ItemNotFoundError:
         if not not_found_ok:
             log.exception("Error in get_module")
@@ -179,7 +185,8 @@ def get_xqueue_callback_url_prefix(request):
 
 
 def get_module_for_descriptor(user, request, descriptor, model_data_cache, course_id,
-                              position=None, wrap_xmodule_display=True, grade_bucket_type=None):
+                              position=None, wrap_xmodule_display=True, grade_bucket_type=None,
+                              static_asset_path=''):
     """
     Implements get_module, extracting out the request-specific functionality.
 
@@ -194,12 +201,14 @@ def get_module_for_descriptor(user, request, descriptor, model_data_cache, cours
 
     return get_module_for_descriptor_internal(user, descriptor, model_data_cache, course_id,
                                               track_function, xqueue_callback_url_prefix,
-                                              position, wrap_xmodule_display, grade_bucket_type)
+                                              position, wrap_xmodule_display, grade_bucket_type,
+                                              static_asset_path)
 
 
 def get_module_for_descriptor_internal(user, descriptor, model_data_cache, course_id,
                                        track_function, xqueue_callback_url_prefix,
-                                       position=None, wrap_xmodule_display=True, grade_bucket_type=None):
+                                       position=None, wrap_xmodule_display=True, grade_bucket_type=None,
+                                       static_asset_path=''):
     """
     Actually implement get_module, without requiring a request.
 
@@ -282,7 +291,8 @@ def get_module_for_descriptor_internal(user, descriptor, model_data_cache, cours
         # inner_get_module, not the parent's callback.  Add it as an argument....
         return get_module_for_descriptor_internal(user, descriptor, model_data_cache, course_id,
                                                   track_function, make_xqueue_callback,
-                                                  position, wrap_xmodule_display, grade_bucket_type)
+                                                  position, wrap_xmodule_display, grade_bucket_type,
+                                                  static_asset_path)
 
     def xblock_model_data(descriptor):
         return DbModel(
@@ -349,6 +359,7 @@ def get_module_for_descriptor_internal(user, descriptor, model_data_cache, cours
             static_replace.replace_static_urls,
             data_directory=getattr(descriptor, 'data_dir', None),
             course_id=course_id,
+            static_asset_path=static_asset_path or descriptor.lms.static_asset_path,
         ),
         replace_course_urls=partial(
             static_replace.replace_course_urls,
@@ -407,7 +418,8 @@ def get_module_for_descriptor_internal(user, descriptor, model_data_cache, cours
     module.get_html = replace_static_urls(
         _get_html,
         getattr(descriptor, 'data_dir', None),
-        course_id=course_id
+        course_id=course_id,
+        static_asset_path=static_asset_path or descriptor.lms.static_asset_path
     )
 
     # Allow URLs of the form '/course/' refer to the root of multicourse directory

lms/djangoapps/courseware/tabs.py

@@ -380,7 +380,8 @@ def get_static_tab_contents(request, course, tab):
     loc = Location(course.location.tag, course.location.org, course.location.course, 'static_tab', tab['url_slug'])
     model_data_cache = ModelDataCache.cache_for_descriptor_descendents(course.id,
         request.user, modulestore().get_instance(course.id, loc), depth=0)
-    tab_module = get_module(request.user, request, loc, model_data_cache, course.id)
+    tab_module = get_module(request.user, request, loc, model_data_cache, course.id,
+                            static_asset_path=course.lms.static_asset_path)
 
     logging.debug('course_module = {0}'.format(tab_module))
 

lms/djangoapps/courseware/tests/test_module_render.py

@@ -19,7 +19,7 @@ from courseware.tests.tests import LoginEnrollmentTestCase, TEST_DATA_MONGO_MODU
 from courseware.model_data import ModelDataCache
 from modulestore_config import TEST_DATA_XML_MODULESTORE
 
-from courseware.courses import get_course_with_access
+from courseware.courses import get_course_with_access, course_image_url, get_course_info_section
 
 from .factories import UserFactory
 
@@ -83,7 +83,7 @@ class ModuleRenderTestCase(LoginEnrollmentTestCase):
 
         # See if the url got rewritten to the target link
         # note if the URL mapping changes then this assertion will break
-        self.assertIn('/courses/'+self.course_id+'/jump_to_id/vertical_test', html)
+        self.assertIn('/courses/' + self.course_id + '/jump_to_id/vertical_test', html)
 
     def test_modx_dispatch(self):
         self.assertRaises(Http404, render.modx_dispatch, 'dummy', 'dummy',
@@ -355,6 +355,38 @@ class TestHtmlModifiers(ModuleStoreTestCase):
             result_fragment.content
         )
 
+    def test_static_asset_path_use(self):
+        '''
+        when a course is loaded with do_import_static=False (see xml_importer.py), then
+        static_asset_path is set as an lms kv in course.  That should make static paths
+        not be mangled (ie not changed to c4x://).
+        '''
+        module = render.get_module(
+            self.user,
+            self.request,
+            self.location,
+            self.model_data_cache,
+            self.course.id,
+            static_asset_path="toy_course_dir",
+        )
+        result_fragment = module.runtime.render(module, None, 'student_view')
+        self.assertIn('href="/static/toy_course_dir', result_fragment.content)
+
+    def test_course_image(self):
+        url = course_image_url(self.course)
+        self.assertTrue(url.startswith('/c4x/'))
+
+        self.course.lms.static_asset_path = "toy_course_dir"
+        url = course_image_url(self.course)
+        self.assertTrue(url.startswith('/static/toy_course_dir/'))
+        self.course.lms.static_asset_path = ""
+
+    def test_get_course_info_section(self):
+        self.course.lms.static_asset_path = "toy_course_dir"
+        get_course_info_section(self.request, self.course, "handouts")
+        # NOTE: check handouts output...right now test course seems to have no such content
+        # at least this makes sure get_course_info_section returns without exception
+
     def test_course_link_rewrite(self):
         module = render.get_module(
             self.user,

lms/envs/acceptance.py

@@ -8,6 +8,7 @@ so that we can run the lettuce acceptance tests.
 # pylint: disable=W0401, W0614
 
 from .test import *
+from .sauce import *
 
 # You need to start the server in debug mode,
 # otherwise the browser will not render the pages correctly
@@ -17,7 +18,7 @@ DEBUG = True
 import logging
 logging.disable(logging.ERROR)
 import os
-import random
+from random import choice, randint
 
 
 def seed():
@@ -65,7 +66,7 @@ DATABASES = {
 
 # Set up XQueue information so that the lms will send
 # requests to a mock XQueue server running locally
-XQUEUE_PORT = random.randint(1024, 65535)
+XQUEUE_PORT = choice(PORTS) if SAUCE.get('SAUCE_ENABLED') else randint(1024, 65535)
 XQUEUE_INTERFACE = {
     "url": "http://127.0.0.1:%d" % XQUEUE_PORT,
     "django_auth": {
@@ -93,5 +94,5 @@ FEEDBACK_SUBMISSION_EMAIL = 'dummy@example.com'
 # Include the lettuce app for acceptance testing, including the 'harvest' django-admin command
 INSTALLED_APPS += ('lettuce.django',)
 LETTUCE_APPS = ('courseware',)
-LETTUCE_SERVER_PORT = random.randint(1024, 65535)
-LETTUCE_BROWSER = 'chrome'
+LETTUCE_SERVER_PORT = choice(PORTS) if SAUCE.get('SAUCE_ENABLED') else randint(1024, 65535)
+LETTUCE_BROWSER = os.environ.get('LETTUCE_BROWSER', 'chrome')

lms/envs/sauce.py

+"""
+This config file extends the test environment configuration
+so that we can run the lettuce acceptance tests on SauceLabs.
+"""
+
+# We intentionally define lots of variables that aren't used, and
+# want to import all variables from base settings files
+# pylint: disable=W0401, W0614
+
+from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
+import os
+
+PORTS = [2000, 2001, 2020, 2109, 2222, 2310, 3000, 3001,
+        3030, 3210, 3333, 4000, 4001, 4040, 4321, 4502, 4503,
+        5050, 5555, 5432, 6060, 6666, 6543, 7000, 7070, 7774,
+        7777, 8003, 8031, 8080, 8081, 8765, 8888,
+        9080, 9090, 9876, 9999, 49221, 55001]
+
+DESIRED_CAPABILITIES = {
+    'chrome': DesiredCapabilities.CHROME,
+    'internetexplorer': DesiredCapabilities.INTERNETEXPLORER,
+    'firefox': DesiredCapabilities.FIREFOX,
+    'opera': DesiredCapabilities.OPERA,
+    'iphone': DesiredCapabilities.IPHONE,
+    'ipad': DesiredCapabilities.IPAD,
+    'safari': DesiredCapabilities.SAFARI,
+    'android': DesiredCapabilities.ANDROID
+}
+
+# All keys must be URL and JSON encodable
+# PLATFORM-BROWSER-VERSION_NUM-DEVICE
+ALL_CONFIG = {
+    'Linux-chrome--': ['Linux', 'chrome', '', ''],
+    'Windows 8-chrome--': ['Windows 8', 'chrome', '', ''],
+    'Windows 7-chrome--': ['Windows 7', 'chrome', '', ''],
+    'Windows XP-chrome--': ['Windows XP', 'chrome', '', ''],
+    'OS X 10.8-chrome--': ['OS X 10.8', 'chrome', '', ''],
+    'OS X 10.6-chrome--': ['OS X 10.6', 'chrome', '', ''],
+
+    'Linux-firefox-23-': ['Linux', 'firefox', '23', ''],
+    'Windows 8-firefox-23-': ['Windows 8', 'firefox', '23', ''],
+    'Windows 7-firefox-23-': ['Windows 7', 'firefox', '23', ''],
+    'Windows XP-firefox-23-': ['Windows XP', 'firefox', '23', ''],
+
+    'OS X 10.8-safari-6-': ['OS X 10.8', 'safari', '6', ''],
+
+    'Windows 8-internetexplorer-10-': ['Windows 8', 'internetexplorer', '10', ''],
+}
+
+SAUCE_INFO = ALL_CONFIG.get(os.environ.get('SAUCE_INFO', 'Linux-chrome--'))
+
+# Information needed to utilize Sauce Labs.
+SAUCE = {
+    'SAUCE_ENABLED': os.environ.get('SAUCE_ENABLED'),
+    'USERNAME': os.environ.get('SAUCE_USER_NAME'),
+    'ACCESS_ID': os.environ.get('SAUCE_API_KEY'),
+    'PLATFORM': SAUCE_INFO[0],
+    'BROWSER': DESIRED_CAPABILITIES.get(SAUCE_INFO[1]),
+    'VERSION': SAUCE_INFO[2],
+    'DEVICE': SAUCE_INFO[3],
+    'SESSION': 'Jenkins Acceptance Tests',
+    'BUILD': os.environ.get('BUILD_DISPLAY_NAME', 'LETTUCE TESTS'),
+}

lms/xmodule_namespace.py

@@ -56,3 +56,4 @@ class LmsNamespace(Namespace):
         default=None,
         scope=Scope.settings
     )
+    static_asset_path = String(help="Path to use for static assets - overrides Studio c4x://", scope=Scope.settings, default='')