Enables contacting partner specific URLs for course information.

Added course slot number to d_program_course.

Various fixes for ProblemResponseReportWorkflow and related tasks

Remove unneeded dependence on overwrite-n-days.

Do not use --no-cache-dir for pre.txt.

Ensures that interval* parameters are passed through to all of the downstream tasks.

Updates acceptance tests and fixture data to demonstrate and verify the issue.

Removes the erroneous complete() methods, which were causing the hive partition
tasks to appear complete, when they hadn't loaded the hive table yet.

Updates acceptance tests to do two runs to demostrate table not found issue:
one with raw, JSON data from REST API, and one with processed, hive data

The --no-cache-dir option is not defined on older versions of pip.
The pre.txt requirements specifically upgrade to a newer pip, so
the option may not work on that file, but should work fine after it
has been loaded.

In addition to removing the marker file, also remove the output
files, since HDFS does not support renaming onto an already-existing file.

Fix problem with running ImportEnrollmentsIntoMysql out of box,
which encounters an error in a task because overwrite-n-days is not
provided.  It's not needed in two tasks, so this removes the
parameter from the tasks in question.

Add backquotes for additional Hive columns.

This is needed for upgrading past Hive 1.2.

Add --no-cache-dir option for pip install cmds.

* Allow multiple secure configurations to be loaded.

This change allows a caller to specify multiple configuration files in
the secure config repo to be loaded, which enables the caller to specify
a "base" configuration, containing special values, which can be
interpolated into subsequent configuration files that are loaded.

These configuration paths are now passed on from remote-task itself to
launch-task when invoked on an EMR cluster.

* Weave through paths in a more consistent fashion.

We're now properly including 'uid', which is hard-coded to 'automation'
in production, to the paths.  This should make the code more resilient
to cases where no remote name is specified.

We've also moved the constants back into remote.py, given that they
weren't necessary in local.py and thus did not need to be shared.

* Remove old local wildcard import.

* Remove our extra command-line arguments before invoking Luigi.

Luigi gets angry when it runs a workflow and sees command-line arguments
that Luigi itself didn't ask for or that the workflow didn't ask for..
which is precisely what --additional-config is.

This change modifies sys.argv in place, removing all key/value pairs for
--additional-config, before we invoke luigi.run().  It *seems* to work,
but we're also modifying a global in the sys module, so, you know..

* Weave arguments through run(); don't modify sys.argv.

* Remove extraneous/old comment.

Revert "Add support for templating to Luigi configuration files. (#398)"

This reverts commit 82c99915.

Task to calculate active users this year.

* Allow multiple secure configurations to be loaded.

This change allows a caller to specify multiple configuration files in
the secure config repo to be loaded, which enables the caller to specify
a "base" configuration, containing special values, which can be
interpolated into subsequent configuration files that are loaded.

These configuration paths are now passed on from remote-task itself to
launch-task when invoked on an EMR cluster.

Make acceptance test work on hive 1x & 2x.

Add overwrite flag to LastCountryOfUserTableTask.

Handle problem-checks with bad answer_ids.

When testing ficus.master, it was found that running the location task
was resulting in multiple entries in the resultstore. The
last_country_of_user table is supposed to only have one partition of
data loaded into it, but when run with overwrite flag set, the table's
contents weren't being deleted. This resulted in a left join (against
student_courseenrollment) would have multiple entries per student
enrollment.

Fix is to drop and recreate the table each time, if the overwrite flag
is set.

This was first identified and fixed on Ficus.  See PR #400.

Rather than trying to output non-ascii answer id's, just screen out
those that are not ascii.

Add InsertToMysqlVideo*.allow_empty_insert

Fixed acceptance tests validation.

Remove ancient enrollment report

This was coded in warehouse/enrollment sub-directory.  It was
our first job.  There are better ways....

because some deployments don't use video in their courses, or have quiet
days when no one watches a video.

* boolean parameter (default False)
* configurable from the [videos] section in override.cfg

Document tasks to run to update insights

Make changes necessary for Hive 2.1 upgrade.

* Backquote column names that use reserved keywords.

* Backquote table names as well.

* We use "interval" in the calendar table as the partition,
  so use date_interval for calendar partition instead.
  "Interval" is a reserved SQL word.

* Backquote `date` in explicit Hive queries.

* Change default Hive version from 0.11 to 1.0, in code and config files.

Strip id in video timeline.