Change-Id: I89a9a3882cdc4199de16272c1577374f61b2250f

Includes deleting the output_root directory at task creation.

Change-Id: I0ba51f43e79e5794fd50ddf8000ea6c60d72590f

If a cluster is newly provisioned it is possible that the ansible metadata cache is not updated before we attempt to connect to it. Ansible then fails to connect to the cluster since it doesn't know it exists. Ensure the cache is refreshed before initiating new connections.

Change-Id: I0737e4b6356f5828a31c65509543939fc2c27df0

Provide coverage for the integration of an externally generated courseware_studentmodule in to the research data exports.

Change-Id: I32e3fbf5928581581babb1502fd17a590cbaf960

* Update StudentModulePerCourseTask to use multipart upload.

* Move extra_modules calls into launcher's attach call: add boto,
  cjson, and filechunkio to argparse.

* Fix bug in PathSetTask.generate_file_list() with redundant root in
  returned path.

Change-Id: I8861ccfb02ee10c5188e11c9ceb86eca9f7b3363

We frequently connect to these hosts through an ssh tunnel to the bastion box. This will enable us to do stuff like manage users on these boxes that are buried within the VPC.

Change-Id: I6121585bafc7ae661e66060eeefdd472edcacbb2

This broke when we stopped using /tmp to store logs etc. Pushed the logic into ansible.

Change-Id: Ib32c236159bbb6654af9c69e98b5540050a3df07

The latter task defaults to reading from a Mysql table using direct option and default Mysql delimiters.

Change-Id: I8d8a18f74c65ef021943ddd9630b730d2a4aadf2

In the nearish future we will force login as users that can run commands as the hadoop user using sudo.

Change-Id: If891e176de0a5de4d6d38a01a258a997e14f905d

This way we can give clusters semantic names like "Production" and point the scheduler at those names instead of relying on the job flow ID which is not very human friendly / memorable.

Change-Id: Iebeab022e09047b625f4725300d357905555ef04

Before this patch, changing a problem from anything to a multiple choice question would cause failures since the most recent submissions would indicate that all past submissions should contain monikers, when in fact they won't. The answer distribution output for this scenario will definitely be wierd, but all bets are off when you modify a problem like this.

Change-Id: Ib72439c37fbe178f2320f7c48cc8bbd687ed7f02

Only writing was broken, so don't override the HdfsTarget behavior for reading. It breaks the ability to read a directory from S3 as if it were a single file.

Change-Id: Ibaf6c7f92fdda8c695b3d71a1c19aeb95c338c34

Change-Id: Ic059030e633848fcde13216d5a96b8b03e5c4b55

Also strip leading and trailing whitespace from all values before output,
and normalize whitespace when stripping HTML markup.

Change-Id: Iac33e2776d22f4459091d7031605cf14952f9688

By default, AWS does not apply an ACL to keys that are put into a bucket from another account. Having no ACL at all effectively renders the object useless since it cannot be read or anything. The only workaround we found was to explicitly set the ACL policy when putting the object. Additionally boto will list all of the keys in the bucket if it is passed "validate=True" this requires an additional permission.  We want to minimize the set of required permissions so we get a reference to the bucket without validating that it exists.

Change-Id: I0f4d6919c7d2817acb1eb55314191648599c3a0f

Change-Id: I441c50fc917edae695314bad0e67c6ee336fd62e

Change-Id: I44b132e333e20fb32d032722406b61748a5a71fd

Fix AN-663 and AN-664.

Change-Id: I704e64a7679c8fec35f2e70355b55f6cffb06580

Fixes: AN-653

Change-Id: I16dcc36e32e1918f68c5b087c1574540cbb5e334

* Log unknown answer_id values that should be hidden.
* Fix handling of answers with extended characters.
* Add "output_root" argument to AnswerDistributionOneFilePerCourseTask.
* Modify IgnoredTarget to work in remote tasks.
* Fix some more Pep8 and pylint errors.

Change-Id: I8bccc8f9a7028c54773af288d4ac110110c495c3

Fixes: AN-589
Change-Id: I0156691780f9b9ba0d787d60ae301a32d51b7a26

Also resolve remaining PEP8 warnings.

Change-Id: I1e5e1947ededbba97540e661c3fb6adcac100597

Implemented as two tasks.

Change-Id: I9219e42f7e347242b4cea2515f49480b2390ed16

Change-Id: I37c37c1c95fefae275b1bcede6cb0f55b956295c

Change-Id: I5c1d8f476587362b5d20d028aeb6283c4c3b18b0

Some courses were created accidentally, or their students were moved to another course_id.  We want to be able to filter out these courses and exclude them from our various aggregate counts.

Change-Id: I0828d9bb0dcb90a5d9e14b5a5c496c73ed0c3dac
Fixes: AN-492

Change-Id: Ib73d66d635eda6f7c0d372996a5e136011d97609

Make courses appear (and count) if the courses have offsets, even if
they have no explicit enrollment counts since the offset date.  At
present, such courses are ignored, which results in an undercount for
total enrollment over all courses.

Change-Id: I67c4b6fba671a1bfc7024c566fe51212cb7eda44

Change-Id: Id11f469b3fd8372c6b85280e9a8882595e62b76b

Change-Id: Iccd7c7ec7970375fd8e5151320575277674e949a

Change-Id: I0c1e80771b8b7818096bf16aaf44da50518ad773

Change-Id: I926b8744f56093fabc5cab321b4cd720adc1402b

* Add support for gathering data from MySQL databases.
* Gather user registration data from a database that is expected to conform to the LMS schema.

Fixes: AN-515

Change-Id: Ic0cfd2d608d209f33a29a192dac041a9810805ba

Change-Id: I195446721b99f6612468f94018882f14d173629c