Merge pull request #2939 from open-craft/jill/olive-22-jenkins-sshagent

Various fixes to allows jenkins analytics to run analytics tasks on AWS resources

* Uses JENKINS_HOME=/edx/var/jenkins.  This addresses the pyinstrument
  re-install issue seen when JENKINS_HOME is a symlink.
* Installs libffi-dev and libpq-dev packages with analytics_jenkins playbook
* Adds an SSH Credential, which is passed as the MASTER_SSH_CREDENTIAL_ID
  parameter to the seed job.
* When the seed job is run, ignore unreferenced jobs, instead of deleting them.
  This allows you to run and re-run the seed job from Jenkins to update a
  particular analytics job, and untick the jobs you don't want to update.
* Lets <ANALYTICS_TASK>_EXTRA_VARS be a @path/to/file.yml

playbooks/roles/jenkins_analytics/README.md

@@ -31,7 +31,7 @@ This file needs to contain, at least, the following variables
 
 * `JENKINS_ANALYTICS_USER_PASSWORD_PLAIN`.
   See [Jenkins User Password](#jenkins-user-password) for details.
-* (`JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID` and `JENKINS_ANALYTICS_GITHUB_KEY`)
+* (`JENKINS_ANALYTICS_GITHUB_*` and `ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_*`)
   and/or `JENKINS_ANALYTICS_CREDENTIALS`.
   See [Jenkins Credentials](#jenkins-credentials) for details.
 * `ANALYTICS_SCHEDULE_SECURE_REPO_*` and `ANALYTICS_SCHEDULE_<TASK_NAME>_EXTRA_VARS`.
@@ -51,44 +51,83 @@ You'll need to set a plain password so ansible can reach Jenkins via the command
 #### Jenkins credentials
 
 Jenkins contains its own credential store. To fill it with credentials,
-please use the `JENKINS_ANALYTICS_CREDENTIALS` variable. This variable
-is a list of objects, each object representing a single credential.
-For now passwords and ssh-keys are supported.
+we recommend overriding these variables:
+
+* `JENKINS_ANALYTICS_GITHUB_USER`: github username, with read access to the
+  secure config and job dsl repos.
+* `JENKINS_ANALYTICS_GITHUB_PASSPHRASE`: optional passphrase, if required for
+  `JENKINS_ANALYTICS_GITHUB_USER`.  Default is `null`.
+* `JENKINS_ANALYTICS_GITHUB_KEY`: private key for the `JENKINS_ANALYTICS_GITHUB_USER`, e.g.
+   `"{{ lookup('file', '/home/you/.ssh/id_rsa') }}"`
+* `ANALYTICS_SCHEDULE_SECURE_REPO_MASTER_SSH_CREDENTIAL_FILE`: path to the ssh
+  key file, relative to the `ANALYTICS_SCHEDULE_SECURE_REPO_URL`.
+  This file will be used as the private key to grant ssh access to the EMR instances.
+   See [Jenkins Seed Job Configuration](#jenkins-seed-job-configuration) for details.
 
-If you only need credentials to access github repositories
-you can override `JENKINS_ANALYTICS_GITHUB_KEY`,
-which should contain contents of private key used for
-authentication to checkout github repositories.
+Note that because the `ANALYTICS_SCHEDULE_SECURE_REPO_*` isn't cloned until the
+seed job is built, the `ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_ID` credential uses
+`type: ssh-private-keyfile`, which allows the credential to be created before
+the private key file actually exists on the file system.
 
-Each credential has a unique ID, which is used to match
-the credential to the task(s) for which it is needed
+Alternatively, you may override the `JENKINS_ANALYTICS_CREDENTIALS` variable.
+This variable is a list of objects, each object representing a single
+credential.  For now passwords, ssh-keys, and ssh key files are supported.
+Each credential has a unique ID, which is used to match the credential to the
+task(s) for which it is needed.
 
-Examples of credentials variables:
+Default value for `JENKINS_ANALYTICS_CREDENTIALS`, and the variables it depends on:
 
-    JENKINS_ANALYTICS_GITHUB_KEY: "{{ lookup('file', 'path to keyfile')
+    JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID: 'github-deploy-key'
+    JENKINS_ANALYTICS_GITHUB_USER: 'git'
+    JENKINS_ANALYTICS_GITHUB_PASSPHRASE: null
 
-    JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID: "github-readonly-key"
+    ANALYTICS_SCHEDULE_SECURE_REPO_DEST: "analytics-secure-config"
+    ANALYTICS_SCHEDULE_SECURE_REPO_MASTER_SSH_CREDENTIAL_FILE: "aws.pem"
+    ANALYTICS_SCHEDULE_SEED_JOB_NAME: "AnalyticsSeedJob"
+    ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_ID: "ssh-access-key"
+    ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_USER: "hadoop"
+    ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_PASSPHRASE: null
+    ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_FILE: "{{ jenkins_home }}/workspace/{{ ANALYTICS_SCHEDULE_SEED_JOB_NAME }}/{{ ANALYTICS_SCHEDULE_SECURE_REPO_DEST }}/{{ ANALYTICS_SCHEDULE_SECURE_REPO_MASTER_SSH_CREDENTIAL_FILE }}"
 
     JENKINS_ANALYTICS_CREDENTIALS:
-      # id is a scope-unique credential identifier
-      - id: test-password
-        # Scope must be global. To have other scopes you'll need to modify addCredentials.groovy
+      - id: "{{ JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID }}"
+        scope: GLOBAL
+        username: "{{ JENKINS_ANALYTICS_GITHUB_USER }}"
+        type: ssh-private-key
+        passphrase: "{{ JENKINS_ANALYTICS_GITHUB_PASSPHRASE }}"
+        description: github access key, generated by ansible
+        privatekey: "{{ JENKINS_ANALYTICS_GITHUB_KEY }}"
+      - id: "{{ ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_ID }}"
         scope: GLOBAL
-        # Username associated with this password
-        username: jenkins
-        type: username-password
-        description: Autogenerated by ansible
-        password: 'password'
-      # id is a scope-unique credential identifier
+        username: "{{ ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_USER }}"
+        type: ssh-private-keyfile
+        passphrase: "{{ ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_PASSPHRASE }}"
+        description: ssh access key, generated by ansible
+        privatekey: "{{ ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_FILE }}"
+
+If you wish to use an explicit SSH key instead of reading it from a file, you
+could override `JENKINS_ANALYTICS_CREDENTIALS` like this:
+
+    ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_KEY: |
+        -----BEGIN RSA PRIVATE KEY-----
+        ...
+        -----END RSA PRIVATE KEY-----
+
+    JENKINS_ANALYTICS_CREDENTIALS:
       - id: "{{ JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID }}"
         scope: GLOBAL
-        # Username this ssh-key is attached to
-        username: git
-        # Type of credential, see other entries for example
+        username: "{{ JENKINS_ANALYTICS_GITHUB_USER }}"
         type: ssh-private-key
-        passphrase: null
-        description: Autogenerated by ansible
-        privatekey: "{{ JENKINS_ANALYTICS_GITHUB_KEY }}'
+        passphrase: "{{ JENKINS_ANALYTICS_GITHUB_PASSPHRASE }}"
+        description: github access key, generated by ansible
+        privatekey: "{{ JENKINS_ANALYTICS_GITHUB_KEY }}"
+      - id: "{{ ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_ID }}"
+        scope: GLOBAL
+        username: "{{ ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_USER }}"
+        type: ssh-private-key
+        passphrase: "{{ ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_PASSPHRASE }}"
+        description: ssh access key, generated by ansible
+        privatekey: "{{ ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_KEY }}"
 
 #### Jenkins seed job configuration
 
@@ -179,14 +218,19 @@ The full list of seed job configuration variables is:
   `@daily`, `@weekly`, see [stackoverflow](http://stackoverflow.com/a/12472740)
   for details.  Set to empty string to disable cron.
   Default is different for each analytics task.
-* `ANALYTICS_SCHEDULE_<TASK_NAME>_EXTRA_VARS`: YML config or @file location to
-  override the analytics task parameters.
+* `ANALYTICS_SCHEDULE_<TASK_NAME>_EXTRA_VARS`: YML @file location to
+  override the analytics task parameters.  File locations can be absolute, or
+  relative to the seed job workspace.
+  You may choose to use raw YAML instead of a @file location, but be aware that
+  any changes made in the Jenkins GUI will be overridden if the
+  `jenkins_analytics` ansible role is re-run.
+
   Consult the individual analytics task DSL for details on the options and defaults.
 
 For example:
 
     ANALYTICS_SCHEDULE_ANSWER_DISTRIBUTION: true
-    ANALYTICS_SCHEDULE_ANSWER_DISTRIBUTION_EXTRA_VARS: "@path/to/answer_dist_defaults.yml"
+    ANALYTICS_SCHEDULE_ANSWER_DISTRIBUTION_EXTRA_VARS: "@{{ ANALYTICS_SCHEDULE_SECURE_REPO_DEST }}/analytics-tasks/answer-dist.yml"
 
     ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL: true
     ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS:
@@ -228,7 +272,7 @@ To use a path relative to the analytics task workspace, build an absolute path
 using the `$WORKSPACE` variable provided by Jenkins, e.g.,
 
     ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS:
-      EMR_EXTRA_VARS: '@$WORKSPACE/../AnalyticsSeedTask/analytics-secure-config/emr-vars.yml'
+      EMR_EXTRA_VARS: '@$WORKSPACE/analytics-secure-config/emr-vars.yml'
 
 
 **Raw JSON**

playbooks/roles/jenkins_analytics/defaults/main.yml

 ---
 
+# Packages required to build edx-analytics-pipeline
+JENKINS_ANALYTICS_EXTRA_PKGS:
+  - libpq-dev
+  - libffi-dev
+
 # See README.md for variable descriptions
 
 JENKINS_ANALYTICS_USER_PASSWORD_PLAIN: jenkins
 JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID: 'github-deploy-key'
-
-JENKINS_ANALYTICS_CREDENTIALS:
-  - id: "{{ JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID }}"
-    scope: GLOBAL
-    username: git
-    type: ssh-private-key
-    passphrase: null
-    description: Autogenerated by ansible
-    privatekey: "{{ JENKINS_ANALYTICS_GITHUB_KEY }}"
+JENKINS_ANALYTICS_GITHUB_USER: 'git'
+JENKINS_ANALYTICS_GITHUB_PASSPHRASE: null
 
 JENKINS_ANALYTICS_CONCURRENT_JOBS_COUNT: 2
 
@@ -20,34 +18,47 @@ ANALYTICS_SCHEDULE_SECURE_REPO_URL: null
 ANALYTICS_SCHEDULE_SECURE_REPO_DEST: "analytics-secure-config"
 ANALYTICS_SCHEDULE_SECURE_REPO_VERSION: "master"
 ANALYTICS_SCHEDULE_SECURE_REPO_CREDENTIAL_ID: "{{ JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID }}"
+ANALYTICS_SCHEDULE_SECURE_REPO_MASTER_SSH_CREDENTIAL_FILE: "aws.pem"
 ANALYTICS_SCHEDULE_JOBS_DSL_REPO_URL: "git@github.com:edx-ops/edx-jenkins-job-dsl.git"
 ANALYTICS_SCHEDULE_JOBS_DSL_REPO_VERSION: "master"
 ANALYTICS_SCHEDULE_JOBS_DSL_REPO_CREDENTIAL_ID: "{{ JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID }}"
+
+ANALYTICS_SCHEDULE_SEED_JOB_NAME: "AnalyticsSeedJob"
+ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_ID: "ssh-access-key"
+ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_USER: "hadoop"
+ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_PASSPHRASE: null
+ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_FILE: "{{ jenkins_home }}/workspace/{{ ANALYTICS_SCHEDULE_SEED_JOB_NAME }}/{{ ANALYTICS_SCHEDULE_SECURE_REPO_DEST }}/{{ ANALYTICS_SCHEDULE_SECURE_REPO_MASTER_SSH_CREDENTIAL_FILE }}"
 ANALYTICS_SCHEDULE_JOBS_DSL_CLASSPATH: |
   src/main/groovy
   lib/*.jar
 ANALYTICS_SCHEDULE_JOBS_DSL_TARGET_JOBS: "jobs/analytics-edx-jenkins.edx.org/*Jobs.groovy"
 
+JENKINS_ANALYTICS_CREDENTIALS:
+  - id: "{{ JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID }}"
+    scope: GLOBAL
+    username: "{{ JENKINS_ANALYTICS_GITHUB_USER }}"
+    type: ssh-private-key
+    passphrase: "{{ JENKINS_ANALYTICS_GITHUB_PASSPHRASE }}"
+    description: github access key, generated by ansible
+    privatekey: "{{ JENKINS_ANALYTICS_GITHUB_KEY }}"
+  - id: "{{ ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_ID }}"
+    scope: GLOBAL
+    username: "{{ ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_USER }}"
+    type: ssh-private-keyfile
+    passphrase: "{{ ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_PASSPHRASE }}"
+    description: ssh access key, generated by ansible
+    privatekey: "{{ ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_FILE }}"
+
 ANALYTICS_SCHEDULE_ANSWER_DISTRIBUTION: true
-ANALYTICS_SCHEDULE_ANSWER_DISTRIBUTION_EXTRA_VARS:
-  SECURE_REPO: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_URL }}"
-  SECURE_BRANCH: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_VERSION }}"
+ANALYTICS_SCHEDULE_ANSWER_DISTRIBUTION_EXTRA_VARS: ''
 ANALYTICS_SCHEDULE_COURSE_ACTIVITY_WEEKLY: true
-ANALYTICS_SCHEDULE_COURSE_ACTIVITY_WEEKLY_EXTRA_VARS:
-  SECURE_REPO: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_URL }}"
-  SECURE_BRANCH: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_VERSION }}"
+ANALYTICS_SCHEDULE_COURSE_ACTIVITY_WEEKLY_EXTRA_VARS: ''
 ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL: true
-ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS:
-  SECURE_REPO: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_URL }}"
-  SECURE_BRANCH: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_VERSION }}"
+ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS: ''
 ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_ALL_VIDEO: true
-ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_ALL_VIDEO_EXTRA_VARS:
-  SECURE_REPO: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_URL }}"
-  SECURE_BRANCH: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_VERSION }}"
+ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_ALL_VIDEO_EXTRA_VARS: ''
 ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_COURSE_ENROLL_BY_COUNTRY: true
-ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_COURSE_ENROLL_BY_COUNTRY_EXTRA_VARS:
-  SECURE_REPO: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_URL }}"
-  SECURE_BRANCH: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_VERSION }}"
+ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_COURSE_ENROLL_BY_COUNTRY_EXTRA_VARS: ''
 
 jenkins_credentials_root: '/tmp/credentials'
 jenkins_credentials_file_dest: "{{ jenkins_credentials_root }}/credentials.json"
@@ -64,7 +75,7 @@ jenkins_auth_realm:
 # For now only a single seed job is supported, adding more would require
 # Ansible 2.+ or converting _execute_jenkins_cli to a module
 jenkins_seed_job:
-  name: AnalyticsSeedJob
+  name: "{{ ANALYTICS_SCHEDULE_SEED_JOB_NAME }}"
   multiscm:
     - scm:
       type: git
@@ -81,6 +92,7 @@ jenkins_seed_job:
   dsl:
     gradle_tasks: "clean libs test"
     removed_view_action: IGNORE
-    removed_job_action: DELETE
+    removed_job_action: IGNORE
     additional_classpath: "{{ ANALYTICS_SCHEDULE_JOBS_DSL_CLASSPATH }}"
     target_jobs: "{{ ANALYTICS_SCHEDULE_JOBS_DSL_TARGET_JOBS }}"
+

playbooks/roles/jenkins_analytics/tasks/main.yml

 ---
 
+- name: install jenkins analytics extra system packages
+  apt:
+    pkg={{ item }} state=present update_cache=yes
+  with_items: JENKINS_ANALYTICS_EXTRA_PKGS
+  tags:
+  - jenkins
+
 - fail: msg=included unix realm by accident
   when: jenkins_auth_realm.name != "unix"
   tags:

playbooks/roles/jenkins_analytics/templates/addCredentials.groovy

@@ -41,6 +41,14 @@ boolean addSSHUserPrivateKey(scope, id, username, privateKey, passphrase, descri
     return true
 }
 
+boolean addSSHUserPrivateKeyFile(scope, id, username, privateKey, passphrase, description) {
+    provider = SystemCredentialsProvider.getInstance()
+    source = new BasicSSHUserPrivateKey.FileOnMasterPrivateKeySource(privateKey)
+    provider.getCredentials().add(new BasicSSHUserPrivateKey(scope, id, username, source, passphrase, description))
+    provider.save()
+    return true
+}
+
 def jsonFile = new File("{{ jenkins_credentials_file_dest }}");
 
 if (!jsonFile.exists()){
@@ -84,4 +92,13 @@ credentialList.each { credential ->
 
         addSSHUserPrivateKey(scope, credential.id, credential.username, credential.privatekey, credential.passphrase, credential.description)
     }
+
+    if (credential.type == "ssh-private-keyfile") {
+
+        if (credential.passphrase != null && credential.passphrase.trim().length() == 0){
+            credential.passphrase = null;
+        }
+
+        addSSHUserPrivateKeyFile(scope, credential.id, credential.username, credential.privatekey, credential.passphrase, credential.description)
+    }
 }

playbooks/roles/jenkins_analytics/templates/seed_job_template.xml

@@ -12,66 +12,74 @@
     </jenkins.advancedqueue.AdvancedQueueSorterJobProperty>
     <hudson.model.ParametersDefinitionProperty>
         <parameterDefinitions>
-            <hudson.model.StringParameterDefinition>
+            <com.cloudbees.plugins.credentials.CredentialsParameterDefinition plugin="credentials@1.24">
+                <name>MASTER_SSH_CREDENTIAL_ID</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_MASTER_SSH_CREDENTIAL_ID }}</defaultValue>
+                <credentialType>com.cloudbees.plugins.credentials.common.StandardCredentials</credentialType>
+                <required>true</required>
+                <description>Jenkins Credential ID used for ssh access to EMR resources.</description>
+            </com.cloudbees.plugins.credentials.CredentialsParameterDefinition>
+            <com.cloudbees.plugins.credentials.CredentialsParameterDefinition plugin="credentials@1.24">
                 <name>GIT_CREDENTIAL_ID</name>
                 <defaultValue>{{ ANALYTICS_SCHEDULE_SECURE_REPO_CREDENTIAL_ID | default('') }}</defaultValue>
-                <description>Jenkins Credential ID used for cloning secure git
-                  repos.  Must match a Credential configured in Jenkins.</description>
-            </hudson.model.StringParameterDefinition>
+                <credentialType>com.cloudbees.plugins.credentials.common.StandardCredentials</credentialType>
+                <required>false</required>
+                <description>Jenkins Credential ID used for cloning secure git repos.</description>
+            </com.cloudbees.plugins.credentials.CredentialsParameterDefinition>
             <hudson.model.BooleanParameterDefinition>
                 <name>ANSWER_DISTRIBUTION</name>
                 <defaultValue>{{ ANALYTICS_SCHEDULE_ANSWER_DISTRIBUTION | default(true) }}</defaultValue>
                 <description>Create or update the AnswerDistributionWorkflow analytics task.</description>
             </hudson.model.BooleanParameterDefinition>
-            <hudson.model.TextParameterDefinition>
+            <hudson.model.StringParameterDefinition>
                 <name>ANSWER_DISTRIBUTION_EXTRA_VARS</name>
-                <defaultValue>{{ ANALYTICS_SCHEDULE_ANSWER_DISTRIBUTION_EXTRA_VARS | default('{}') | to_nice_json }}</defaultValue>
-                <description>Set default values for the AnswerDistributionWorkflow job parameters.  Format as YAML.</description>
-            </hudson.model.TextParameterDefinition>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_ANSWER_DISTRIBUTION_EXTRA_VARS | default('') }}</defaultValue>
+                <description>Default values for the AnswerDistributionWorkflow job parameters.  Provide YAML file as @path/to/file.yml, absolute or relative to seed job workpace.</description>
+            </hudson.model.StringParameterDefinition>
 
             <hudson.model.BooleanParameterDefinition>
                 <name>COURSE_ACTIVITY_WEEKLY</name>
                 <defaultValue>{{ ANALYTICS_SCHEDULE_COURSE_ACTIVITY_WEEKLY | default(true) }}</defaultValue>
                 <description>Create or update the CourseActivityWeeklyTask analytics task.</description>
             </hudson.model.BooleanParameterDefinition>
-            <hudson.model.TextParameterDefinition>
+            <hudson.model.StringParameterDefinition>
                 <name>COURSE_ACTIVITY_WEEKLY_EXTRA_VARS</name>
-                <defaultValue>{{ ANALYTICS_SCHEDULE_COURSE_ACTIVITY_WEEKLY_EXTRA_VARS | default('{}') | to_nice_json }}</defaultValue>
-                <description>Set default values for the CourseActivityWeeklyTask job parameters.  Format as YAML.</description>
-            </hudson.model.TextParameterDefinition>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_COURSE_ACTIVITY_WEEKLY_EXTRA_VARS | default('') }}</defaultValue>
+                <description>Default values for the CourseActivityWeeklyTask job parameters.  Provide YAML file as @path/to/file.yml, absolute or relative to seed job workpace.</description>
+            </hudson.model.StringParameterDefinition>
 
             <hudson.model.BooleanParameterDefinition>
                 <name>IMPORT_ENROLLMENTS_INTO_MYSQL</name>
                 <defaultValue>{{ ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL | default(true) }}</defaultValue>
                 <description>Create or update the ImportEnrollmentsIntoMysql analytics task.</description>
             </hudson.model.BooleanParameterDefinition>
-            <hudson.model.TextParameterDefinition>
+            <hudson.model.StringParameterDefinition>
                 <name>IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS</name>
-                <defaultValue>{{ ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS | default('{}') | to_nice_json }}</defaultValue>
-                <description>Set default values for the ImportEnrollmentsIntoMysql job parameters.  Format as YAML.</description>
-            </hudson.model.TextParameterDefinition>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS | default('') }}</defaultValue>
+                <description>Default values for the ImportEnrollmentsIntoMysql job parameters.  Provide YAML file as @path/to/file.yml, absolute or relative to seed job workpace.</description>
+            </hudson.model.StringParameterDefinition>
 
             <hudson.model.BooleanParameterDefinition>
                 <name>INSERT_TO_MYSQL_ALL_VIDEO</name>
                 <defaultValue>{{ ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_ALL_VIDEO | default(true) }}</defaultValue>
                 <description>Create or update the InsertToMysqlAllVideoTask analytics task.</description>
             </hudson.model.BooleanParameterDefinition>
-            <hudson.model.TextParameterDefinition>
+            <hudson.model.StringParameterDefinition>
                 <name>INSERT_TO_MYSQL_ALL_VIDEO_EXTRA_VARS</name>
-                <defaultValue>{{ ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_ALL_VIDEO_EXTRA_VARS | default('{}') | to_nice_json }}</defaultValue>
-                <description>Set default values for the InsertToMysqlAllVideoTask job parameters.  Format as YAML.</description>
-            </hudson.model.TextParameterDefinition>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_ALL_VIDEO_EXTRA_VARS | default('') }}</defaultValue>
+                <description>Default values for the InsertToMysqlAllVideoTask job parameters.  Provide YAML file as @path/to/file.yml, absolute or relative to seed job workpace.</description>
+            </hudson.model.StringParameterDefinition>
 
             <hudson.model.BooleanParameterDefinition>
                 <name>INSERT_TO_MYSQL_COURSE_ENROLL_BY_COUNTRY</name>
                 <defaultValue>{{ ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_COURSE_ENROLL_BY_COUNTRY | default(true) }}</defaultValue>
                 <description>Create or update the InsertToMysqlCourseEnrollByCountryWorkflow analytics task.</description>
             </hudson.model.BooleanParameterDefinition>
-            <hudson.model.TextParameterDefinition>
+            <hudson.model.StringParameterDefinition>
                 <name>INSERT_TO_MYSQL_COURSE_ENROLL_BY_COUNTRY_EXTRA_VARS</name>
-                <defaultValue>{{ ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_COURSE_ENROLL_BY_COUNTRY_EXTRA_VARS | default('{}') | to_nice_json }}</defaultValue>
-                <description>Set default values for the InsertToMysqlCourseEnrollByCountryWorkflowJob job parameters.  Format as YAML.</description>
-            </hudson.model.TextParameterDefinition>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_COURSE_ENROLL_BY_COUNTRY_EXTRA_VARS | default('') }}</defaultValue>
+                <description>Default values for the InsertToMysqlCourseEnrollByCountryWorkflowJob job parameters.  Provide YAML file as @path/to/file.yml, absolute or relative to seed job workpace.</description>
+            </hudson.model.StringParameterDefinition>
         </parameterDefinitions>
     </hudson.model.ParametersDefinitionProperty>
   </properties>

playbooks/roles/jenkins_master/tasks/main.yml

@@ -50,6 +50,18 @@
     - java
     - jenkins
 
+- name: set jenkins home
+  lineinfile:
+    backup: yes
+    dest: /etc/default/jenkins
+    regexp: '^JENKINS_HOME='
+    line: 'JENKINS_HOME=\"{{ jenkins_home }}\"'
+  notify:
+    - restart Jenkins
+  tags:
+    - java
+    - jenkins
+
 # Move /var/lib/jenkins to Jenkins home (on the EBS)
 - name: move /var/lib/jenkins
   command: mv /var/lib/jenkins {{ jenkins_home }}