Merge pull request #678 from edx/zoldak/acceptance-sauce

Zoldak/acceptance sauce

cms/djangoapps/contentstore/features/advanced-settings.feature

@@ -2,6 +2,7 @@ Feature: Advanced (manual) course policy
   In order to specify course policy settings for which no custom user interface exists
   I want to be able to manually enter JSON key /value pairs
 
+
   Scenario: A course author sees default advanced settings
     Given I have opened a new course in Studio
     When I select the Advanced Settings
@@ -11,6 +12,8 @@ Feature: Advanced (manual) course policy
     Given I am on the Advanced Course Settings page in Studio
     Then the settings are alphabetized
 
+  # Sauce labs does not play nicely with CodeMirror
+  @skip_sauce
   Scenario: Test cancel editing key value
     Given I am on the Advanced Course Settings page in Studio
     When I edit the value of a policy key
@@ -19,6 +22,8 @@ Feature: Advanced (manual) course policy
     And I reload the page
     Then the policy key value is unchanged
 
+  # Sauce labs does not play nicely with CodeMirror
+  @skip_sauce
   Scenario: Test editing key value
     Given I am on the Advanced Course Settings page in Studio
     When I edit the value of a policy key and save
@@ -26,6 +31,8 @@ Feature: Advanced (manual) course policy
     And I reload the page
     Then the policy key value is changed
 
+  # Sauce labs does not play nicely with CodeMirror
+  @skip_sauce
   Scenario: Test how multi-line input appears
     Given I am on the Advanced Course Settings page in Studio
     When I create a JSON object as a value for "discussion_topics"
@@ -33,6 +40,8 @@ Feature: Advanced (manual) course policy
     And I reload the page
     Then it is displayed as formatted
 
+  # Sauce labs does not play nicely with CodeMirror
+  @skip_sauce
   Scenario: Test error if value supplied is of the wrong type
     Given I am on the Advanced Course Settings page in Studio
     When I create a JSON object as a value for "display_name"
@@ -41,6 +50,8 @@ Feature: Advanced (manual) course policy
     Then the policy key value is unchanged
 
   # This feature will work in Firefox only when Firefox is the active window
+  # Sauce labs does not play nicely with CodeMirror
+  @skip_sauce
   Scenario: Test automatic quoting of non-JSON values
     Given I am on the Advanced Course Settings page in Studio
     When I create a non-JSON value not in quotes
@@ -48,6 +59,8 @@ Feature: Advanced (manual) course policy
     And I reload the page
     Then it is displayed as a string
 
+  # Sauce labs does not play nicely with CodeMirror
+  @skip_sauce
   Scenario: Confirmation is shown on save
     Given I am on the Advanced Course Settings page in Studio
     When I edit the value of a policy key

cms/djangoapps/contentstore/features/checklists.feature

@@ -10,7 +10,10 @@ Feature: Course checklists
     Then I can check and uncheck tasks in a checklist
     And They are correctly selected after reloading the page
 
-  # CHROME ONLY, due to issues getting link to be active in firefox
+  # There are issues getting link to be active in browsers other than chrome
+  @skip_firefox
+  @skip_internetexplorer
+  @skip_safari
   Scenario: A task can link to a location within Studio
     Given I have opened Checklists
     When I select a link to the course outline
@@ -18,7 +21,10 @@ Feature: Course checklists
     And I press the browser back button
     Then I am brought back to the course outline in the correct state
 
-  # CHROME ONLY, due to issues getting link to be active in firefox
+  # There are issues getting link to be active in browsers other than chrome
+  @skip_firefox
+  @skip_internetexplorer
+  @skip_safari
   Scenario: A task can link to a location outside Studio
     Given I have opened Checklists
     When I select a link to help page

cms/djangoapps/contentstore/features/course-overview.feature

@@ -64,6 +64,10 @@ Feature: Course Overview
         And I change an assignment's grading status
         Then I am shown a notification
 
+    # Notification is not shown on reorder for IE
+    # Safari does not have moveMouseTo implemented
+    @skip_internetexplorer
+    @skip_safari
    Scenario: Notification is shown on subsection reorder
         Given I have opened a new course section in Studio
         And I have added a new subsection

cms/djangoapps/contentstore/features/course-settings.feature

 Feature: Course Settings
   As a course author, I want to be able to configure my course settings.
 
+  # Safari has trouble keeps dates on refresh
+  @skip_safari
   Scenario: User can set course dates
     Given I have opened a new course in Studio
     When I select Schedule and Details
@@ -8,12 +10,16 @@ Feature: Course Settings
     And I press the "Save" notification button
     Then I see the set dates on refresh
 
+    # IE has trouble with saving information
+    @skip_internetexplorer
   Scenario: User can clear previously set course dates (except start date)
     Given I have set course dates
     And I clear all the dates except start
     And I press the "Save" notification button
     Then I see cleared dates on refresh
 
+    # IE has trouble with saving information
+    @skip_internetexplorer
   Scenario: User cannot clear the course start date
     Given I have set course dates
     And I press the "Save" notification button
@@ -21,6 +27,10 @@ Feature: Course Settings
     Then I receive a warning about course start date
     And The previously set start date is shown on refresh
 
+    # IE has trouble with saving information
+    # Safari gets CSRF token errors
+    @skip_internetexplorer
+    @skip_safari
   Scenario: User can correct the course start date warning
     Given I have tried to clear the course start
     And I have entered a new course start date
@@ -28,12 +38,16 @@ Feature: Course Settings
     Then The warning about course start date goes away
     And My new course start date is shown on refresh
 
+  # Safari does not save + refresh properly through sauce labs
+  @skip_safari
   Scenario: Settings are only persisted when saved
     Given I have set course dates
     And I press the "Save" notification button
     When I change fields
     Then I do not see the new changes persisted on refresh
 
+  # Safari does not save + refresh properly through sauce labs
+  @skip_safari
   Scenario: Settings are reset on cancel
     Given I have set course dates
     And I press the "Save" notification button
@@ -41,6 +55,8 @@ Feature: Course Settings
     And I press the "Cancel" notification button
     Then I do not see the changes
 
+  # Safari gets CSRF token errors
+  @skip_safari
   Scenario: Confirmation is shown on save
     Given I have opened a new course in Studio
     When I select Schedule and Details

cms/djangoapps/contentstore/features/course-team.py

@@ -91,7 +91,7 @@ def remove_course_team_admin(_step, outer_capture, name):
 
 @step(u'"([^"]*)" logs in$')
 def other_user_login(_step, name):
-    world.browser.cookies.delete()
+    world.visit('logout')
     world.visit('/')
 
     signin_css = 'a.action-signin'

cms/djangoapps/contentstore/features/course-updates.feature

 Feature: Course updates
     As a course author, I want to be able to provide updates to my students
 
+    # Internet explorer can't select all so the update appears weirdly
+    @skip_internetexplorer
     Scenario: Users can add updates
         Given I have opened a new course in Studio
         And I go to the course updates page
@@ -8,6 +10,8 @@ Feature: Course updates
         Then I should see the update "Hello"
         And I see a "saving" notification
 
+    # Internet explorer can't select all so the update appears weirdly
+    @skip_internetexplorer
     Scenario: Users can edit updates
         Given I have opened a new course in Studio
         And I go to the course updates page
@@ -33,6 +37,8 @@ Feature: Course updates
         Then I should see the date "June 1, 2013"
         And I see a "saving" notification
 
+    # Internet explorer can't select all so the update appears weirdly
+    @skip_internetexplorer
     Scenario: Users can change handouts
         Given I have opened a new course in Studio
         And I go to the course updates page

cms/djangoapps/contentstore/features/discussion-editor.feature

@@ -6,6 +6,8 @@ Feature: Discussion Component Editor
     And I edit and select Settings
     Then I see three alphabetized settings and their expected values
 
+  # Safari doesn't save the name properly
+  @skip_safari
   Scenario: User can modify display name
     Given I have created a Discussion Tag
     And I edit and select Settings

cms/djangoapps/contentstore/features/grading.feature

@@ -13,7 +13,7 @@ Feature: Course Grading
         When I add "6" new grades
         Then I see I now have "5" grades
 
-    #Cannot reliably make the delete button appear so using javascript instead
+    # Cannot reliably make the delete button appear so using javascript instead
     Scenario: Users can delete grading ranges
         Given I have opened a new course in Studio
         And I am viewing the grading settings
@@ -21,6 +21,9 @@ Feature: Course Grading
         And I delete a grade
         Then I see I now have "2" grades
 
+    # IE and Safari cannot reliably drag and drop through selenium
+    @skip_internetexplorer
+    @skip_safari
     Scenario: Users can move grading ranges
         Given I have opened a new course in Studio
         And I am viewing the grading settings
@@ -85,6 +88,9 @@ Feature: Course Grading
         When I change assignment type "Homework" to ""
         Then the save button is disabled
 
+    # IE and Safari cannot type in grade range name
+    @skip_internetexplorer
+    @skip_safari
     Scenario: User can edit grading range names
         Given I have opened a new course in Studio
         And I have populated the course

cms/djangoapps/contentstore/features/grading.py

@@ -112,10 +112,10 @@ def changes_not_persisted(step):
 
 @step(u'I see the assignment type "(.*)"$')
 def i_see_the_assignment_type(_step, name):
-      assignment_css = '#course-grading-assignment-name'
-      assignments = world.css_find(assignment_css)
-      types = [ele['value'] for ele in assignments]
-      assert name in types
+    assignment_css = '#course-grading-assignment-name'
+    assignments = world.css_find(assignment_css)
+    types = [ele['value'] for ele in assignments]
+    assert name in types
 
 
 @step(u'I change the highest grade range to "(.*)"$')
@@ -144,6 +144,7 @@ def cannot_edit_fail(_step):
         pass  # We should get this exception on failing to edit the element
 
 
+
 @step(u'I change the grace period to "(.*)"$')
 def i_change_grace_period(_step, grace_period):
     grace_period_css = '#course-grading-graceperiod'

cms/djangoapps/contentstore/features/html-editor.feature

@@ -6,6 +6,8 @@ Feature: HTML Editor
     And I edit and select Settings
     Then I see only the HTML display name setting
 
+  # Safari doesn't save the name properly
+  @skip_safari
   Scenario: User can modify display name
     Given I have created a Blank HTML Page
     And I edit and select Settings

cms/djangoapps/contentstore/features/problem-editor.feature

@@ -7,12 +7,16 @@ Feature: Problem Editor
     Then I see five alphabetized settings and their expected values
     And Edit High Level Source is not visible
 
+  # Safari is having trouble saving the values on sauce
+  @skip_safari
   Scenario: User can modify String values
     Given I have created a Blank Common Problem
     When I edit and select Settings
     Then I can modify the display name
     And my display name change is persisted on save
 
+  # Safari is having trouble saving the values on sauce
+  @skip_safari
   Scenario: User can specify special characters in String values
     Given I have created a Blank Common Problem
     When I edit and select Settings
@@ -25,6 +29,8 @@ Feature: Problem Editor
     Then I can revert the display name to unset
     And my display name is unset on save
 
+  # IE will not click the revert button properly
+  @skip_internetexplorer
   Scenario: User can select values in a Select
     Given I have created a Blank Common Problem
     When I edit and select Settings
@@ -32,6 +38,8 @@ Feature: Problem Editor
     And my change to randomization is persisted
     And I can revert to the default value for randomization
 
+  # Safari will input it as 35.
+  @skip_safari
   Scenario: User can modify float input values
     Given I have created a Blank Common Problem
     When I edit and select Settings
@@ -44,16 +52,22 @@ Feature: Problem Editor
     When I edit and select Settings
     Then if I set the weight to "abc", it remains unset
 
+  # Safari will input it as 234.
+  @skip_safari
   Scenario: User cannot type decimal values integer number field
     Given I have created a Blank Common Problem
     When I edit and select Settings
     Then if I set the max attempts to "2.34", it will persist as a valid integer
 
+  # Safari will input it incorrectly
+  @skip_safari
   Scenario: User cannot type out of range values in an integer number field
     Given I have created a Blank Common Problem
     When I edit and select Settings
     Then if I set the max attempts to "-3", it will persist as a valid integer
 
+  # Safari will input it as 35.
+  @skip_safari
   Scenario: Settings changes are not saved on Cancel
     Given I have created a Blank Common Problem
     When I edit and select Settings
@@ -67,6 +81,8 @@ Feature: Problem Editor
     Then Edit High Level Source is visible
 
   # This feature will work in Firefox only when Firefox is the active window
+  # IE will not interact with the high level source in sauce labs
+  @skip_internetexplorer
   Scenario: High Level source is persisted for LaTeX problem (bug STUD-280)
     Given I have created a LaTeX Problem
     When I edit and compile the High Level Source

cms/djangoapps/contentstore/features/static-pages.feature

@@ -15,6 +15,8 @@ Feature: Static Pages
         And I "delete" the "Empty" page
         Then I should not see a "Empty" static page
 
+    # Safari won't update the name properly
+    @skip_safari
     Scenario: Users can edit static pages
         Given I have opened a new course in Studio
         And I go to the static pages page

cms/djangoapps/contentstore/features/subsection.feature

@@ -25,6 +25,8 @@ Feature: Create Subsection
     And I reload the page
     Then I see it marked as Homework
 
+  # Safari has trouble saving the date in Sauce
+  @skip_safari
   Scenario: Set a due date in a different year (bug #256)
     Given I have opened a new subsection in Studio
     And I set the subsection release date to 12/25/2011 03:00

cms/djangoapps/contentstore/features/textbooks.feature

@@ -5,6 +5,9 @@ Feature: Textbooks
     When I go to the textbooks page
     Then I should see a message telling me to create a new textbook
 
+  # IE and Safari on sauce labs will not upload the textbook correctly resulting in an error
+  @skip_internetexplorer
+  @skip_safari
   Scenario: Create a textbook
     Given I have opened a new course in Studio
     And I go to the textbooks page

cms/djangoapps/contentstore/features/upload.feature

 Feature: Upload Files
     As a course author, I want to be able to upload files for my students
 
+    # Uploading isn't working on safari with sauce labs
+    @skip_safari
     Scenario: Users can upload files
         Given I have opened a new course in Studio
         And I go to the files and uploads page
@@ -8,6 +10,8 @@ Feature: Upload Files
         Then I should see the file "test" was uploaded
         And The url for the file "test" is valid
 
+    # Uploading isn't working on safari with sauce labs
+    @skip_safari
     Scenario: Users can update files
         Given I have opened a new course in studio
         And I go to the files and uploads page
@@ -15,6 +19,8 @@ Feature: Upload Files
         And I upload the file "test"
         Then I should see only one "test"
 
+    # Uploading isn't working on safari with sauce labs
+    @skip_safari
     Scenario: Users can delete uploaded files
         Given I have opened a new course in studio
         And I go to the files and uploads page
@@ -23,12 +29,16 @@ Feature: Upload Files
         Then I should not see the file "test" was uploaded
         And I see a confirmation that the file was deleted
 
+    # Uploading isn't working on safari with sauce labs
+    @skip_safari
     Scenario: Users can download files
         Given I have opened a new course in studio
         And I go to the files and uploads page
         When I upload the file "test"
         Then I can download the correct "test" file
 
+    # Uploading isn't working on safari with sauce labs
+    @skip_safari
     Scenario: Users can download updated files
         Given I have opened a new course in studio
         And I go to the files and uploads page

cms/djangoapps/contentstore/features/upload.py

@@ -10,6 +10,7 @@ import os
 
 TEST_ROOT = settings.COMMON_TEST_DATA_ROOT
 
+
 @step(u'I go to the files and uploads page')
 def go_to_uploads(_step):
     menu_css = 'li.nav-course-courseware'
@@ -106,8 +107,8 @@ def get_index(file_name):
 def get_file(file_name):
     index = get_index(file_name)
     assert index != -1
-
     url_css = 'a.filename'
+
     def get_url():
         return world.css_find(url_css)[index]._element.get_attribute('href')
     url = world.retry_on_exception(get_url)

cms/djangoapps/contentstore/features/video-editor.feature

@@ -6,17 +6,23 @@ Feature: Video Component Editor
     And I edit the component
     Then I see the correct video settings and default values
 
+  # Safari has trouble saving values on Sauce
+  @skip_safari
   Scenario: User can modify Video display name
     Given I have created a Video component
     And I edit the component
     Then I can modify the display name
     And my video display name change is persisted on save
 
+  # Sauce Labs cannot delete cookies
+  @skip_sauce
   Scenario: Captions are hidden when "show captions" is false
     Given I have created a Video component
     And I have set "show captions" to False
     Then when I view the video it does not show the captions
 
+  # Sauce Labs cannot delete cookies
+  @skip_sauce
   Scenario: Captions are shown when "show captions" is true
     Given I have created a Video component
     And I have set "show captions" to True

cms/djangoapps/contentstore/features/video.feature

@@ -10,15 +10,21 @@ Feature: Video Component
     Given I have clicked the new unit button
     Then creating a video takes a single click
 
+  # Sauce Labs cannot delete cookies
+  @skip_sauce
   Scenario: Captions are hidden correctly
     Given I have created a Video component
     And I have hidden captions
     Then when I view the video it does not show the captions
 
+  # Sauce Labs cannot delete cookies
+  @skip_sauce
   Scenario: Captions are shown correctly
     Given I have created a Video component
     Then when I view the video it does show the captions
 
+  # Sauce Labs cannot delete cookies
+  @skip_sauce
   Scenario: Captions are toggled correctly
     Given I have created a Video component
     And I have toggled captions

cms/envs/acceptance.py

@@ -8,6 +8,7 @@ so that we can run the lettuce acceptance tests.
 # pylint: disable=W0401, W0614
 
 from .test import *
+from lms.envs.sauce import *
 
 # You need to start the server in debug mode,
 # otherwise the browser will not render the pages correctly
@@ -17,7 +18,7 @@ DEBUG = True
 import logging
 logging.disable(logging.ERROR)
 import os
-import random
+from random import choice, randint
 
 
 def seed():
@@ -75,7 +76,6 @@ DATABASES = {
 # Use the auto_auth workflow for creating users and logging them in
 MITX_FEATURES['AUTOMATIC_AUTH_FOR_TESTING'] = True
 
-
 # HACK
 # Setting this flag to false causes imports to not load correctly in the lettuce python files
 # We do not yet understand why this occurs. Setting this to true is a stopgap measure
@@ -84,5 +84,5 @@ USE_I18N = True
 # Include the lettuce app for acceptance testing, including the 'harvest' django-admin command
 INSTALLED_APPS += ('lettuce.django',)
 LETTUCE_APPS = ('contentstore',)
-LETTUCE_SERVER_PORT = random.randint(1024, 65535)
-LETTUCE_BROWSER = 'chrome'
+LETTUCE_SERVER_PORT = choice(PORTS) if SAUCE.get('SAUCE_ENABLED') else randint(1024, 65535)
+LETTUCE_BROWSER = os.environ.get('LETTUCE_BROWSER', 'chrome')

common/djangoapps/terrain/browser.py

@@ -11,6 +11,10 @@ from logging import getLogger
 from django.core.management import call_command
 from django.conf import settings
 from selenium.common.exceptions import WebDriverException
+from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
+from requests import put
+from base64 import encodestring
+from json import dumps
 
 # Let the LMS and CMS do their one-time setup
 # For example, setting up mongo caches
@@ -42,43 +46,93 @@ LOGGER.info("Loading the lettuce acceptance testing terrain file...")
 MAX_VALID_BROWSER_ATTEMPTS = 20
 
 
+def get_username_and_key():
+    """
+    Returns the Sauce Labs username and access ID as set by environment variables
+    """
+    return {"username": settings.SAUCE.get('USERNAME'), "access-key": settings.SAUCE.get('ACCESS_ID')}
+
+
+def set_job_status(jobid, passed=True):
+    """
+    Sets the job status on sauce labs
+    """
+    body_content = dumps({"passed": passed})
+    config = get_username_and_key()
+    base64string = encodestring('{}:{}'.format(config['username'], config['access-key']))[:-1]
+    result = put('http://saucelabs.com/rest/v1/{}/jobs/{}'.format(config['username'], world.jobid),
+        data=body_content,
+        headers={"Authorization": "Basic {}".format(base64string)})
+    return result.status_code == 200
+
+
+def make_desired_capabilities():
+    """
+    Returns a DesiredCapabilities object corresponding to the environment sauce parameters
+    """
+    desired_capabilities = settings.SAUCE.get('BROWSER', DesiredCapabilities.CHROME)
+    desired_capabilities['platform'] = settings.SAUCE.get('PLATFORM')
+    desired_capabilities['version'] = settings.SAUCE.get('VERSION')
+    desired_capabilities['device-type'] = settings.SAUCE.get('DEVICE')
+    desired_capabilities['name'] = settings.SAUCE.get('SESSION')
+    desired_capabilities['build'] = settings.SAUCE.get('BUILD')
+    desired_capabilities['video-upload-on-pass'] = False
+    desired_capabilities['sauce-advisor'] = False
+    desired_capabilities['record-screenshots'] = False
+    desired_capabilities['selenium-version'] = "2.34.0"
+    desired_capabilities['max-duration'] = 3600
+    desired_capabilities['public'] = 'public restricted'
+    return desired_capabilities
+
+
 @before.harvest
 def initial_setup(server):
     """
     Launch the browser once before executing the tests.
     """
-    browser_driver = getattr(settings, 'LETTUCE_BROWSER', 'chrome')
+    world.absorb(settings.SAUCE.get('SAUCE_ENABLED'), 'SAUCE_ENABLED')
 
-    # There is an issue with ChromeDriver2 r195627 on Ubuntu
-    # in which we sometimes get an invalid browser session.
-    # This is a work-around to ensure that we get a valid session.
-    success = False
-    num_attempts = 0
-    while (not success) and num_attempts < MAX_VALID_BROWSER_ATTEMPTS:
+    if not world.SAUCE_ENABLED:
+        browser_driver = getattr(settings, 'LETTUCE_BROWSER', 'chrome')
 
-        # Get a browser session
-        world.browser = Browser(browser_driver)
+        # There is an issue with ChromeDriver2 r195627 on Ubuntu
+        # in which we sometimes get an invalid browser session.
+        # This is a work-around to ensure that we get a valid session.
+        success = False
+        num_attempts = 0
+        while (not success) and num_attempts < MAX_VALID_BROWSER_ATTEMPTS:
+            world.browser = Browser(browser_driver)
 
-        # Try to visit the main page
-        # If the browser session is invalid, this will
-        # raise a WebDriverException
-        try:
-            world.visit('/')
+            # Try to visit the main page
+            # If the browser session is invalid, this will
+            # raise a WebDriverException
+            try:
+                world.visit('/')
 
-        except WebDriverException:
-            world.browser.quit()
-            num_attempts += 1
+            except WebDriverException:
+                world.browser.quit()
+                num_attempts += 1
 
-        else:
-            success = True
+            else:
+                success = True
 
-    # If we were unable to get a valid session within the limit of attempts,
-    # then we cannot run the tests.
-    if not success:
-        raise IOError("Could not acquire valid {driver} browser session.".format(driver=browser_driver))
+        # If we were unable to get a valid session within the limit of attempts,
+        # then we cannot run the tests.
+        if not success:
+            raise IOError("Could not acquire valid {driver} browser session.".format(driver=browser_driver))
 
-    # Set the browser size to 1280x1024
-    world.browser.driver.set_window_size(1280, 1024)
+        world.browser.driver.set_window_size(1280, 1024)
+
+    else:
+        config = get_username_and_key()
+        world.browser = Browser(
+            'remote',
+            url="http://{}:{}@ondemand.saucelabs.com:80/wd/hub".format(config['username'], config['access-key']),
+            **make_desired_capabilities()
+        )
+        world.browser.driver.implicitly_wait(30)
+
+    world.absorb(world.browser.driver.session_id, 'jobid')
 
 
 @before.each_scenario
@@ -97,7 +151,6 @@ def clear_data(scenario):
     world.spew('scenario_dict')
 
 
-
 @after.each_scenario
 def reset_databases(scenario):
     '''
@@ -128,4 +181,6 @@ def teardown_browser(total):
     """
     Quit the browser after executing the tests.
     """
+    if world.SAUCE_ENABLED:
+        set_job_status(world.jobid, total.scenarios_ran == total.scenarios_passed)
     world.browser.quit()

common/djangoapps/terrain/steps.py

@@ -99,7 +99,7 @@ def i_am_logged_in_user(step):
 
 @step('I am not logged in$')
 def i_am_not_logged_in(step):
-    world.browser.cookies.delete()
+    world.visit('logout')
 
 
 @step('I am staff for course "([^"]*)"$')
@@ -138,10 +138,13 @@ def should_have_link_with_path_and_text(step, path, text):
 
 @step(r'should( not)? see "(.*)" (?:somewhere|anywhere) (?:in|on) (?:the|this) page')
 def should_see_in_the_page(step, doesnt_appear, text):
+    multiplier = 1
+    if world.SAUCE_ENABLED:
+        multiplier = 2
     if doesnt_appear:
-        assert world.browser.is_text_not_present(text, wait_time=5)
+        assert world.browser.is_text_not_present(text, wait_time=5*multiplier)
     else:
-        assert world.browser.is_text_present(text, wait_time=5)
+        assert world.browser.is_text_present(text, wait_time=5*multiplier)
 
 
 @step('I am logged in$')
@@ -150,7 +153,7 @@ def i_am_logged_in(step):
     world.log_in(username='robot', password='test')
     world.browser.visit(django_url('/'))
     # You should not see the login link
-    assert_equals(world.browser.find_by_css('a#login'), [])
+    assert world.is_css_not_present('a#login')
 
 
 @step(u'I am an edX user$')

jenkins/test_acceptance.sh

@@ -30,10 +30,22 @@ TESTS_FAILED=0
 # /usr/bin/Xvfb :1 -screen 0 1024x268x24
 # This allows us to run Chrome without a display
 export DISPLAY=:1
+SKIP_TESTS=""
+
+# Testing for the existance of these environment variables
+if [ ! -z ${LETTUCE_BROWSER+x} ]; then
+	SKIP_TESTS="--tag -skip_$LETTUCE_BROWSER"
+fi
+if [ ! -z ${SAUCE_ENABLED+x} ]; then
+	# SAUCE_INFO is a - seperated string PLATFORM-BROWSER-VERSION-DEVICE
+	# Error checking is done in the setting up of the browser
+	IFS='-' read -a SAUCE <<< "${SAUCE_INFO}"
+	SKIP_TESTS="--tag -skip_sauce --tag -skip_${SAUCE[1]}"
+fi
 
 # Run the lms and cms acceptance tests
 # (the -v flag turns off color in the output)
-rake test_acceptance_lms["-v 3"] || TESTS_FAILED=1
-rake test_acceptance_cms["-v 3"] || TESTS_FAILED=1
+rake test_acceptance_lms["-v 3 $SKIP_TESTS"] || TESTS_FAILED=1
+rake test_acceptance_cms["-v 3 $SKIP_TESTS"] || TESTS_FAILED=1
 
 [ $TESTS_FAILED == '0' ]

lms/djangoapps/courseware/features/login.feature

@@ -11,7 +11,8 @@ Feature: Login in as a registered user
     And I submit my credentials on the login form
     Then I should see the login error message "This account has not been activated"
 
-    # CHROME ONLY, firefox will not redirect properly
+    # firefox will not redirect properly when the whole suite is run
+    @skip_firefox
     Scenario: Login to an activated account
     Given I am an edX user
     And I am an activated user

lms/djangoapps/courseware/features/signup.feature

@@ -3,7 +3,8 @@ Feature: Sign in
   As a new user
   I want to signup for a student account
 
-  # CHROME ONLY, firefox will not redirect properly
+  # firefox will not redirect properly
+  @skip_firefox
   Scenario: Sign up from the homepage
     Given I visit the homepage
     When I click the link with the text "Register Now"

lms/djangoapps/courseware/features/video.feature

@@ -11,6 +11,8 @@ Feature: Video component
   Given the course has a Video component in Youtube mode
   Then when I view the video it has rendered in Youtube mode
 
+  # Firefox doesn't have HTML5
+  @skip_firefox
   Scenario: Autoplay is enabled in LMS for a Video component
   Given the course has a Video component in HTML5 mode
-  Then when I view the video it has autoplay enabled
+  Then when I view the video it has autoplay enabled
\ No newline at end of file

lms/envs/acceptance.py

@@ -8,6 +8,7 @@ so that we can run the lettuce acceptance tests.
 # pylint: disable=W0401, W0614
 
 from .test import *
+from .sauce import *
 
 # You need to start the server in debug mode,
 # otherwise the browser will not render the pages correctly
@@ -17,7 +18,7 @@ DEBUG = True
 import logging
 logging.disable(logging.ERROR)
 import os
-import random
+from random import choice, randint
 
 
 def seed():
@@ -65,7 +66,7 @@ DATABASES = {
 
 # Set up XQueue information so that the lms will send
 # requests to a mock XQueue server running locally
-XQUEUE_PORT = random.randint(1024, 65535)
+XQUEUE_PORT = choice(PORTS) if SAUCE.get('SAUCE_ENABLED') else randint(1024, 65535)
 XQUEUE_INTERFACE = {
     "url": "http://127.0.0.1:%d" % XQUEUE_PORT,
     "django_auth": {
@@ -93,5 +94,5 @@ FEEDBACK_SUBMISSION_EMAIL = 'dummy@example.com'
 # Include the lettuce app for acceptance testing, including the 'harvest' django-admin command
 INSTALLED_APPS += ('lettuce.django',)
 LETTUCE_APPS = ('courseware',)
-LETTUCE_SERVER_PORT = random.randint(1024, 65535)
-LETTUCE_BROWSER = 'chrome'
+LETTUCE_SERVER_PORT = choice(PORTS) if SAUCE.get('SAUCE_ENABLED') else randint(1024, 65535)
+LETTUCE_BROWSER = os.environ.get('LETTUCE_BROWSER', 'chrome')

lms/envs/sauce.py

+"""
+This config file extends the test environment configuration
+so that we can run the lettuce acceptance tests on SauceLabs.
+"""
+
+# We intentionally define lots of variables that aren't used, and
+# want to import all variables from base settings files
+# pylint: disable=W0401, W0614
+
+from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
+import os
+
+PORTS = [2000, 2001, 2020, 2109, 2222, 2310, 3000, 3001,
+        3030, 3210, 3333, 4000, 4001, 4040, 4321, 4502, 4503,
+        5050, 5555, 5432, 6060, 6666, 6543, 7000, 7070, 7774,
+        7777, 8003, 8031, 8080, 8081, 8765, 8888,
+        9080, 9090, 9876, 9999, 49221, 55001]
+
+DESIRED_CAPABILITIES = {
+    'chrome': DesiredCapabilities.CHROME,
+    'internet explorer': DesiredCapabilities.INTERNETEXPLORER,
+    'firefox': DesiredCapabilities.FIREFOX,
+    'opera': DesiredCapabilities.OPERA,
+    'iphone': DesiredCapabilities.IPHONE,
+    'ipad': DesiredCapabilities.IPAD,
+    'safari': DesiredCapabilities.SAFARI,
+    'android': DesiredCapabilities.ANDROID
+}
+
+# All keys must be URL and JSON encodable
+# PLATFORM-BROWSER-VERSION_NUM-DEVICE
+ALL_CONFIG = {
+    'Linux-chrome--': ['Linux', 'chrome', '', ''],
+    'Windows 8-chrome--': ['Windows 8', 'chrome', '', ''],
+    'Windows 7-chrome--': ['Windows 7', 'chrome', '', ''],
+    'Windows XP-chrome--': ['Windows XP', 'chrome', '', ''],
+    'OS X 10.8-chrome--': ['OS X 10.8', 'chrome', '', ''],
+    'OS X 10.6-chrome--': ['OS X 10.6', 'chrome', '', ''],
+
+    'Linux-firefox-23-': ['Linux', 'firefox', '23', ''],
+    'Windows 8-firefox-23-': ['Windows 8', 'firefox', '23', ''],
+    'Windows 7-firefox-23-': ['Windows 7', 'firefox', '23', ''],
+    'Windows XP-firefox-23-': ['Windows XP', 'firefox', '23', ''],
+
+    'OS X 10.8-safari-6-': ['OS X 10.8', 'safari', '6', ''],
+
+    'Windows 8-internetexplorer-10-': ['Windows 8', 'internetexplorer', '10', ''],
+}
+
+SAUCE_INFO = ALL_CONFIG.get(os.environ.get('SAUCE_INFO', 'Linux-chrome--'))
+
+# Information needed to utilize Sauce Labs.
+SAUCE = {
+    'SAUCE_ENABLED': os.environ.get('SAUCE_ENABLED'),
+    'USERNAME': os.environ.get('SAUCE_USER_NAME'),
+    'ACCESS_ID': os.environ.get('SAUCE_API_KEY'),
+    'PLATFORM': SAUCE_INFO[0],
+    'BROWSER': DESIRED_CAPABILITIES.get(SAUCE_INFO[1]),
+    'VERSION': SAUCE_INFO[2],
+    'DEVICE': SAUCE_INFO[3],
+    'SESSION': 'Jenkins Acceptance Tests',
+    'BUILD': os.environ.get('BUILD_DISPLAY_NAME', 'LETTUCE TESTS'),
+}