Create Jenkins jobs for analytics tasks using a job DSL repo.

* defaults/main.yml: adds analytics schedule variables for configuring the AnalyticsSeedJob, with all analytics tasks enabled by default.
* templates/seed_job_template.xml: use updated analytics seed job config and description
* tasks/main.yml: generates a hashed password for the jenkins user, and adds jenkins-auth and create-seed-job tags.
* README.md: documents seed job parameters and jenkins auth change

playbooks/roles/jenkins_analytics/README.md

@@ -11,6 +11,14 @@ This role performs the following steps:
 * Enables the use of Jenkins CLI.
 * Installs a seed job from configured repository, launches it and waits
   for it to finish.
+* The seed job creates the analytics task jobs.
+
+Each analytics task job is created using a task-specific DSL script which
+determines the structure of the Jenkins job, e.g. its scheduled frequency, the
+git repos cloned to run the task, the parameters the job requires, and the
+shell script used to run the analytics task.  These DSL scripts live in a
+separate git repo, configured by `ANALYTICS_SCHEDULE_JOBS_DSL_REPO_*`.
+
 
 ## Configuration
 
@@ -21,10 +29,13 @@ all required variables from this section.
 This file needs to contain, at least, the following variables
 (see the next few sections for more information about them):
 
-* `JENKINS_ANALYTICS_USER_PASSWORD_HASHED` 
-* `JENKINS_ANALYTICS_USER_PASSWORD_PLAIN`
-* `JENKINS_ANALYTICS_GITHUB_KEY` or `JENKINS_ANALYTICS_CREDENTIALS`
-
+* `JENKINS_ANALYTICS_USER_PASSWORD_PLAIN`.
+  See [Jenkins User Password](#jenkins-user-password) for details.
+* (`JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID` and `JENKINS_ANALYTICS_GITHUB_KEY`)
+  and/or `JENKINS_ANALYTICS_CREDENTIALS`.
+  See [Jenkins Credentials](#jenkins-credentials) for details.
+* `ANALYTICS_SCHEDULE_SECURE_REPO_*` and `ANALYTICS_SCHEDULE_<TASK_NAME>_EXTRA_VARS`.
+  See [Jenkins Seed Job Configuration](#jenkins-seed-job-configuration) for details.
 
 ### End-user editable configuration
 
@@ -33,20 +44,10 @@ This file needs to contain, at least, the following variables
 You'll need to override default `jenkins` user password, please do that
 as this sets up the **shell** password for this user.
 
-You'll need to set both a plain password and a hashed one.
-To obtain a hashed password use the `mkpasswd` command, for example: 
-`mkpasswd --method=sha-512`. (Note: a hashed password is required 
-to have clean "changed"/"unchanged" notification for this step 
-in Ansible.) 
+You'll need to set a plain password so ansible can reach Jenkins via the command line tool.
 
-* `JENKINS_ANALYTICS_USER_PASSWORD_HASHED`: hashed password 
 * `JENKINS_ANALYTICS_USER_PASSWORD_PLAIN`: plain password
 
-#### Jenkins seed job configuration 
-
-This will be filled as part of PR[#2830](https://github.com/edx/configuration/pull/2830). 
-For now go with defaults. 
-
 #### Jenkins credentials
 
 Jenkins contains its own credential store. To fill it with credentials,
@@ -64,7 +65,9 @@ the credential to the task(s) for which it is needed
 
 Examples of credentials variables:
 
-    JENKINS_ANALYTICS_GITHUB_KEY: "{{ lookup('file', 'path to keyfile') }}" 
+    JENKINS_ANALYTICS_GITHUB_KEY: "{{ lookup('file', 'path to keyfile')
+
+    JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID: "github-readonly-key"
 
     JENKINS_ANALYTICS_CREDENTIALS:
       # id is a scope-unique credential identifier
@@ -77,21 +80,181 @@ Examples of credentials variables:
         description: Autogenerated by ansible
         password: 'password'
       # id is a scope-unique credential identifier
-      - id: github-deploy-key
+      - id: "{{ JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID }}"
         scope: GLOBAL
         # Username this ssh-key is attached to
         username: git
         # Type of credential, see other entries for example
         type: ssh-private-key
-        passphrase: 'foobar'
-        description: Generated by ansible
-        privatekey: |
-          -----BEGIN RSA PRIVATE KEY-----
-          Proc-Type: 4,ENCRYPTED
-          DEK-Info: AES-128-CBC,....
+        passphrase: null
+        description: Autogenerated by ansible
+        privatekey: "{{ JENKINS_ANALYTICS_GITHUB_KEY }}'
+
+#### Jenkins seed job configuration
 
-          Key contents
-          -----END RSA PRIVATE KEY-----
+The seed job creates the Analytics Jobs that will run the analytics tasks.  By
+default, the seed job creates all the available Analytics Jobs, but you can disable
+these jobs, and set their parameters, using `ANALYTICS_SCHEDULE_<TASK_NAME>_*`.
+
+Currently supported analytics tasks are:
+
+* `ANSWER_DISTRIBUTION`: invokes
+  `edx.analytics.tasks.answer_dist.AnswerDistributionWorkflow` via the
+  `AnswerDistributionWorkflow.groovy` DSL.
+* `IMPORT_ENROLLMENTS_INTO_MYSQL`: invokes
+  `edx.analytics.tasks.enrollments.ImportEnrollmentsIntoMysql` via the
+  `ImportEnrollmentsIntoMysql.groovy` DSL.
+* `COURSE_ACTIVITY_WEEKLY`: invokes
+  `edx.analytics.tasks.user_activity.CourseActivityWeeklyTask` via the
+  `CourseActivityWeeklyTask.groovy` DSL.
+* `INSERT_TO_MYSQL_ALL_VIDEO`: invokes
+  `edx.analytics.tasks.video.InsertToMysqlAllVideoTask` via the
+  `InsertToMysqlAllVideoTask.groovy` DSL.
+* `INSERT_TO_MYSQL_COURSE_ENROLL_BY_COUNTRY:` invokes
+  `edx.analytics.tasks.location_per_course.InsertToMysqlCourseEnrollByCountryWorkflow` via the
+  `InsertToMysqlCourseEnrollByCountryWorkflow.groovy` DSL.
+
+Since running the analytics tasks on EMR requires confidential ssh keys, the
+convention is to store them in a secure repo, which is then cloned when running
+the seed job.  To use a secure repo, override
+`ANALYTICS_SCHEDULE_SECURE_REPO_URL` and
+`ANALYTICS_SCHEDULE_SECURE_REPO_VERSION`.
+
+For example:
+
+    ANALYTICS_SCHEDULE_SECURE_REPO_URL: "git@github.com:open-craft/analytics-sandbox-private.git"
+    ANALYTICS_SCHEDULE_SECURE_REPO_VERSION: "customer-analytics-schedule"
+
+The seed job also clones a second repo, which contains the DSL scripts that
+contain the analytics task DSLs.  That repo is configured using
+`ANALYTICS_SCHEDULE_JOBS_DSL_REPO_*`, and it will be cloned directly into the
+seed job workspace.
+
+**Note:** There are two ways to specify a ssl-based github repo URL.  Note the
+subtle difference in the paths: `github.com:your-org` vs. `github.com/your-org`.
+
+* git@github.com:your-org/private-repo.git ✓
+* ssh://git@github.com/your-org/private-repo.git ✓
+
+*Not like this:*
+
+* git@github.com/your-org/private-repo.git ❌
+* ssh://git@github.com:your-org/private-repo.git ❌
+
+The full list of seed job configuration variables is:
+
+* `ANALYTICS_SCHEDULE_SECURE_REPO_URL`: Optional URL for the git repo that contains the
+  analytics task schedule configuration file.  If set, Jenkins will clone this
+  repo when the seed job is run.  Default is `null`.
+* `ANALYTICS_SCHEDULE_SECURE_REPO_VERSION`: Optional branch/tagname to checkout
+  for the secure repo.  Default is `master`.
+* `ANALYTICS_SCHEDULE_SECURE_REPO_DEST`: Optional target dir for the the secure
+  repo clone, relative to the seed job workspace.  Default is `analytics-secure-config`.
+* `ANALYTICS_SCHEDULE_SECURE_REPO_CREDENTIAL_ID`: Credential id with read
+  access to the secure repo.  Default is `{{ JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID }}`.
+  See [Jenkins Credentials](#jenkins-credentials) below for details.
+* `ANALYTICS_SCHEDULE_JOBS_DSL_REPO_URL`: Optional URL for the git repo that contains the analytics job DSLs.
+  Default is `git@github.com:edx-ops/edx-jenkins-job-dsl.git`.
+  This repo is cloned directly into the seed job workspace.
+* `ANALYTICS_SCHEDULE_JOBS_DSL_REPO_VERSION`: Optional branch/tagname to checkout for the job DSL repo.
+  Default is `master`.
+* `ANALYTICS_SCHEDULE_JOBS_DSL_REPO_CREDENTIAL_ID`: Credential id with read access to the job DSL repo.
+  Default is `{{ JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID }}`.
+  See [Jenkins Credentials](#jenkins-credentials) below for details.
+* `ANALYTICS_SCHEDULE_JOBS_DSL_CLASSPATH`: Optional additional classpath jars
+  and dirs required to run the job DSLs.
+  Each path must be newline-separated, and relative to the seed job workspace.
+  Default is:
+
+        src/main/groovy
+        lib/*.jar
+
+* `ANALYTICS_SCHEDULE_JOBS_DSL_TARGET_JOBS`: DSLs for the top-level seed job to run on build.
+  Default is `jobs/analytics-edx-jenkins.edx.org/*Jobs.groovy`
+
+
+* `ANALYTICS_SCHEDULE_<TASK_NAME>`:  `true`|`false`.  Must be set to `true` to create the analytics task.
+* `ANALYTICS_SCHEDULE_<TASK_NAME>_FREQUENCY`: Optional string representing how
+  often the analytics task should be run.  Uses a modified cron syntax, e.g.
+  `@daily`, `@weekly`, see [stackoverflow](http://stackoverflow.com/a/12472740)
+  for details.  Set to empty string to disable cron.
+  Default is different for each analytics task.
+* `ANALYTICS_SCHEDULE_<TASK_NAME>_EXTRA_VARS`: YML config or @file location to
+  override the analytics task parameters.
+  Consult the individual analytics task DSL for details on the options and defaults.
+
+For example:
+
+    ANALYTICS_SCHEDULE_ANSWER_DISTRIBUTION: true
+    ANALYTICS_SCHEDULE_ANSWER_DISTRIBUTION_EXTRA_VARS: "@path/to/answer_dist_defaults.yml"
+
+    ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL: true
+    ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS:
+      TASKS_REPO: "https://github.com/open-craft/edx-analytics-pipeline.git"
+      TASKS_BRANCH: "analytics-sandbox"
+      CONFIG_REPO: "https://github.com/open-craft/edx-analytics-configuration.git"
+      CONFIG_BRANCH: "analytics-sandbox"
+      JOB_NAME: "ImportEnrollmentsIntoMysql"
+      JOB_FREQUENCY: "@monthly"
+      CLUSTER_NAME: "AnswerDistribution"
+      EMR_EXTRA_VARS: "@/home/jenkins/emr-vars.yml"  # see [EMR Configuration](#emr-configuration)
+      FROM_DATE: "2016-01-01"
+      TASK_USER: "hadoop"
+      NOTIFY_EMAIL_ADDRESSES: "staff@example.com
+
+##### EMR Configuration
+
+The `EMR_EXTRA_VARS` parameter for each analytics task is passed by the analytics
+task shell command to the ansible playbook for provisioning and terminating the
+EMR cluster.
+
+Because `EMR_EXTRA_VARS` passes via the shell, it may reference other analytics
+task parameters as shell variables, e.g. `$S3_PACKAGE_BUCKET`.
+
+**File path**
+
+The easiest way to modify this parameter is to provide a `@/path/to/file.yml`
+or `@/path/to/file.json`.  The file path must be absolute, e.g.,
+
+    ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS:
+      EMR_EXTRA_VARS: '@/home/jenkins/emr-vars.yml'
+
+Or relative to the analytics-configuration repo cloned by the analytics task, e.g.,
+
+    ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS:
+      EMR_EXTRA_VARS: '@./config/emr-vars.yml'
+
+To use a path relative to the analytics task workspace, build an absolute path
+using the `$WORKSPACE` variable provided by Jenkins, e.g.,
+
+    ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS:
+      EMR_EXTRA_VARS: '@$WORKSPACE/../AnalyticsSeedTask/analytics-secure-config/emr-vars.yml'
+
+
+**Raw JSON**
+
+The other option, utilised by the DSL `EMR_EXTRA_VARS` default value, is to use a
+JSON string.  Take care to use a *JSON string*, not raw JSON itself, as YAML is
+a JSON superset, and we don't want the JSON to be parsed by ansible.
+
+Also, because formatting valid JSON is difficult, be sure to run the text
+through a JSON validator before deploying.
+
+As with file paths, the JSON text can use analytics task parameters as shell
+variables, e.g.,
+
+    ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS:
+      AUTOMATION_KEYPAIR_NAME: 'analytics-sandbox'
+      VPC_SUBNET_ID: 'subnet-cd1b9c94'
+      EMR_LOG_BUCKET: 's3://analytics-sandbox-emr-logs'
+      CLUSTER_NAME: 'Analytics EMR Cluster'
+      EMR_EXTRA_VARS: |
+        {
+          "name": "$CLUSTER_NAME",
+          "keypair_name": "$AUTOMATION_KEYPAIR_NAME",
+          "vpc_subnet_id": "$VPC_SUBNET_ID",
+          "log_uri": "$EMR_LOG_BUCKET"
+        }
 
 #### Other useful variables
 
@@ -131,8 +294,7 @@ Unix realm requires the following settings:
 * `service`: Jenkins uses PAM configuration for this service. `su` is
 a safe choice as it doesn't require a user to have the ability to login
 remotely.
-* `plain_password`:  plaintext password, **you should change** default values.
-* `hashed_password`: hashed password
+* `plain_password`:  plaintext password, **you must change** default values.
 
 Example realm configuration:
 
@@ -140,7 +302,6 @@ Example realm configuration:
       name: unix
       service: su
       plain_password: jenkins
-      hashed_password: $6$rAVyI.p2wXVDKk5w$y0G1MQehmHtvaPgdtbrnvAsBqYQ99g939vxrdLXtPQCh/e7GJVwbnqIKZpve8EcMLTtq.7sZwTBYV9Tdjgf1k.
 
 
 #### Seed job configuration

playbooks/roles/jenkins_analytics/defaults/main.yml

@@ -2,11 +2,11 @@
 
 # See README.md for variable descriptions
 
-JENKINS_ANALYTICS_USER_PASSWORD_HASHED: $6$rAVyI.p2wXVDKk5w$y0G1MQehmHtvaPgdtbrnvAsBqYQ99g939vxrdLXtPQCh/e7GJVwbnqIKZpve8EcMLTtq.7sZwTBYV9Tdjgf1k.
 JENKINS_ANALYTICS_USER_PASSWORD_PLAIN: jenkins
+JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID: 'github-deploy-key'
 
 JENKINS_ANALYTICS_CREDENTIALS:
-  - id: github-deploy-key
+  - id: "{{ JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID }}"
     scope: GLOBAL
     username: git
     type: ssh-private-key
@@ -16,6 +16,39 @@ JENKINS_ANALYTICS_CREDENTIALS:
 
 JENKINS_ANALYTICS_CONCURRENT_JOBS_COUNT: 2
 
+ANALYTICS_SCHEDULE_SECURE_REPO_URL: null
+ANALYTICS_SCHEDULE_SECURE_REPO_DEST: "analytics-secure-config"
+ANALYTICS_SCHEDULE_SECURE_REPO_VERSION: "master"
+ANALYTICS_SCHEDULE_SECURE_REPO_CREDENTIAL_ID: "{{ JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID }}"
+ANALYTICS_SCHEDULE_JOBS_DSL_REPO_URL: "git@github.com:edx-ops/edx-jenkins-job-dsl.git"
+ANALYTICS_SCHEDULE_JOBS_DSL_REPO_VERSION: "master"
+ANALYTICS_SCHEDULE_JOBS_DSL_REPO_CREDENTIAL_ID: "{{ JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID }}"
+ANALYTICS_SCHEDULE_JOBS_DSL_CLASSPATH: |
+  src/main/groovy
+  lib/*.jar
+ANALYTICS_SCHEDULE_JOBS_DSL_TARGET_JOBS: "jobs/analytics-edx-jenkins.edx.org/*Jobs.groovy"
+
+ANALYTICS_SCHEDULE_ANSWER_DISTRIBUTION: true
+ANALYTICS_SCHEDULE_ANSWER_DISTRIBUTION_EXTRA_VARS:
+  SECURE_REPO: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_URL }}"
+  SECURE_BRANCH: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_VERSION }}"
+ANALYTICS_SCHEDULE_COURSE_ACTIVITY_WEEKLY: true
+ANALYTICS_SCHEDULE_COURSE_ACTIVITY_WEEKLY_EXTRA_VARS:
+  SECURE_REPO: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_URL }}"
+  SECURE_BRANCH: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_VERSION }}"
+ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL: true
+ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS:
+  SECURE_REPO: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_URL }}"
+  SECURE_BRANCH: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_VERSION }}"
+ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_ALL_VIDEO: true
+ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_ALL_VIDEO_EXTRA_VARS:
+  SECURE_REPO: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_URL }}"
+  SECURE_BRANCH: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_VERSION }}"
+ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_COURSE_ENROLL_BY_COUNTRY: true
+ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_COURSE_ENROLL_BY_COUNTRY_EXTRA_VARS:
+  SECURE_REPO: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_URL }}"
+  SECURE_BRANCH: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_VERSION }}"
+
 jenkins_credentials_root: '/tmp/credentials'
 jenkins_credentials_file_dest: "{{ jenkins_credentials_root }}/credentials.json"
 jenkins_credentials_script: "{{ jenkins_credentials_root }}/addCredentials.groovy"
@@ -27,16 +60,27 @@ jenkins_auth_realm:
   service: su
   # Change this default password: (see README.md to see how you can do it)
   plain_password: "{{ JENKINS_ANALYTICS_USER_PASSWORD_PLAIN }}"
-  hashed_password: "{{ JENKINS_ANALYTICS_USER_PASSWORD_HASHED }}"
 
+# For now only a single seed job is supported, adding more would require
+# Ansible 2.+ or converting _execute_jenkins_cli to a module
 jenkins_seed_job:
-  name: analytics-seed-job
-  time_trigger: "H * * * *"
-  removed_job_action: "DELETE"
-  removed_view_action: "IGNORE"
-  scm:
+  name: AnalyticsSeedJob
+  multiscm:
+    - scm:
+      type: git
+      url: "{{ ANALYTICS_SCHEDULE_JOBS_DSL_REPO_URL }}"
+      dest: ""
+      branch: "{{ ANALYTICS_SCHEDULE_JOBS_DSL_REPO_VERSION | default('master') }}"
+      credential_id: "{{ ANALYTICS_SCHEDULE_JOBS_DSL_REPO_CREDENTIAL_ID | default('') }}"
+    - scm:
       type: git
-    url: "git@github.com:edx-ops/edx-jenkins-job-dsl.git"
-    credential_id: "github-deploy-key"
-    target_jobs: "jobs/analytics-edx-jenkins.edx.org/*Jobs.groovy"
-    additional_classpath: "src/main/groovy"
+      url: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_URL | default('') }}"
+      dest: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_DEST | default('') }}"
+      branch: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_VERSION | default('master') }}"
+      credential_id: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_CREDENTIAL_ID | default('') }}"
+  dsl:
+    gradle_tasks: "clean libs test"
+    removed_view_action: IGNORE
+    removed_job_action: DELETE
+    additional_classpath: "{{ ANALYTICS_SCHEDULE_JOBS_DSL_CLASSPATH }}"
+    target_jobs: "{{ ANALYTICS_SCHEDULE_JOBS_DSL_TARGET_JOBS }}"

playbooks/roles/jenkins_analytics/tasks/main.yml

@@ -2,16 +2,29 @@
 
 - fail: msg=included unix realm by accident
   when: jenkins_auth_realm.name != "unix"
+  tags: 
+    - jenkins-auth
 
 - fail: msg=Please change default password for jenkins user
   when: jenkins_auth_realm.plain_password == 'jenkins'
+  tags: 
+    - jenkins-auth
+
+- name: generate jenkins user password
+  shell: "openssl passwd -1 '{{ jenkins_auth_realm.plain_password | quote }}'"
+  register: jenkins_user_password_hash
+  no_log: True
+  tags: 
+    - jenkins-auth
 
 - user:
     name: "{{ jenkins_user }}"
+    state: present
     groups: shadow
     append: yes
-    password: "{{ jenkins_auth_realm.hashed_password }}"
-    update_password: always
+    password: "{{ jenkins_user_password_hash.stdout }}"
+  tags: 
+    - jenkins-auth
 
 - name: template config.xml
   template:
@@ -19,6 +32,8 @@
     dest: "{{ jenkins_home }}/config.xml"
     owner: "{{ jenkins_user }}"
     group: "{{ jenkins_group }}"
+  tags: 
+    - jenkins-auth
 
 # Unconditionally restart Jenkins, this has two side-effects:
 # 1. Jenkins uses new auth realm
@@ -28,17 +43,23 @@
 
 - name: restart Jenkins
   service: name=jenkins state=restarted
+  tags: 
+    - jenkins-auth
 
 # Upload Jenkins credentials
 
 - name: create credentials dir
   file: name={{ jenkins_credentials_root }} state=directory
+  tags: 
+    - jenkins-auth
 
 - name: upload groovy script
   template:
     src: addCredentials.groovy
     dest: "{{ jenkins_credentials_script }}"
     mode: "600"
+  tags: 
+    - jenkins-auth
 
 - name: upload credentials file
   template:
@@ -46,28 +67,40 @@
     dest: "{{ jenkins_credentials_file_dest }}"
     mode: "600"
     owner: "{{ jenkins_user }}"
+  tags: 
+    - jenkins-auth
 
 - name: add credentials
   include: execute_jenkins_cli.yaml
   vars:
     jenkins_command_string: "groovy {{ jenkins_credentials_script }}"
+  tags: 
+    - jenkins-auth
 
 - name: clean up
   file: name={{ jenkins_credentials_root }} state=absent
+  tags: 
+    - jenkins-auth
 
 # Upload seed job
 
 - name: upload job file
   template: src=seed_job_template.xml dest=/tmp/{{ jenkins_seed_job.name }} mode="600"
+  tags:
+    - create-seed-job
 
 - name: check if job is present
   include: execute_jenkins_cli.yaml
   vars:
     jenkins_command_string: "get-job {{ jenkins_seed_job.name }}"
     jenkins_ignore_cli_errors: yes
+  tags:
+    - create-seed-job
 
 - set_fact:
     get_job_output: "{{ jenkins_command_output }}"
+  tags:
+    - create-seed-job
 
 # Upload seed job to Jenkins
 
@@ -77,6 +110,8 @@
     jenkins_command_string: "create-job {{ jenkins_seed_job.name }}"
     jenkins_command_prefix: "cat /tmp/{{ jenkins_seed_job.name }} | "
   when: get_job_output.rc != 0
+  tags:
+    - create-seed-job
 
 - name: update seed job
   include: execute_jenkins_cli.yaml
@@ -84,6 +119,8 @@
     jenkins_command_string: "update-job {{ jenkins_seed_job.name }}"
     jenkins_command_prefix: "cat /tmp/{{ jenkins_seed_job.name }} | "
   when: get_job_output.rc == 0
+  tags:
+    - create-seed-job
 
 # Build the seed job
 
@@ -91,3 +128,5 @@
   include: execute_jenkins_cli.yaml
   vars:
     jenkins_command_string: "build {{ jenkins_seed_job.name }} -s"
+  tags:
+    - create-seed-job

playbooks/roles/jenkins_analytics/templates/seed_job_template.xml

@@ -2,7 +2,7 @@
 <project>
   <actions/>
   <description>
-    Seed job autogenerated by ansible, it will be overridden.
+    Creates and configures the analytics task jobs.
   </description>
   <keepDependencies>false</keepDependencies>
   <properties>
@@ -10,22 +10,93 @@
       <useJobPriority>false</useJobPriority>
       <priority>-1</priority>
     </jenkins.advancedqueue.AdvancedQueueSorterJobProperty>
+    <hudson.model.ParametersDefinitionProperty>
+        <parameterDefinitions>
+            <hudson.model.StringParameterDefinition>
+                <name>GIT_CREDENTIAL_ID</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_SECURE_REPO_CREDENTIAL_ID | default('') }}</defaultValue>
+                <description>Jenkins Credential ID used for cloning secure git
+                  repos.  Must match a Credential configured in Jenkins.</description>
+            </hudson.model.StringParameterDefinition>
+            <hudson.model.BooleanParameterDefinition>
+                <name>ANSWER_DISTRIBUTION</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_ANSWER_DISTRIBUTION | default(true) }}</defaultValue>
+                <description>Create or update the AnswerDistributionWorkflow analytics task.</description>
+            </hudson.model.BooleanParameterDefinition>
+            <hudson.model.TextParameterDefinition>
+                <name>ANSWER_DISTRIBUTION_EXTRA_VARS</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_ANSWER_DISTRIBUTION_EXTRA_VARS | default('{}') | to_nice_json }}</defaultValue>
+                <description>Set default values for the AnswerDistributionWorkflow job parameters.  Format as YAML.</description>
+            </hudson.model.TextParameterDefinition>
+
+            <hudson.model.BooleanParameterDefinition>
+                <name>COURSE_ACTIVITY_WEEKLY</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_COURSE_ACTIVITY_WEEKLY | default(true) }}</defaultValue>
+                <description>Create or update the CourseActivityWeeklyTask analytics task.</description>
+            </hudson.model.BooleanParameterDefinition>
+            <hudson.model.TextParameterDefinition>
+                <name>COURSE_ACTIVITY_WEEKLY_EXTRA_VARS</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_COURSE_ACTIVITY_WEEKLY_EXTRA_VARS | default('{}') | to_nice_json }}</defaultValue>
+                <description>Set default values for the CourseActivityWeeklyTask job parameters.  Format as YAML.</description>
+            </hudson.model.TextParameterDefinition>
+
+            <hudson.model.BooleanParameterDefinition>
+                <name>IMPORT_ENROLLMENTS_INTO_MYSQL</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL | default(true) }}</defaultValue>
+                <description>Create or update the ImportEnrollmentsIntoMysql analytics task.</description>
+            </hudson.model.BooleanParameterDefinition>
+            <hudson.model.TextParameterDefinition>
+                <name>IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS | default('{}') | to_nice_json }}</defaultValue>
+                <description>Set default values for the ImportEnrollmentsIntoMysql job parameters.  Format as YAML.</description>
+            </hudson.model.TextParameterDefinition>
+
+            <hudson.model.BooleanParameterDefinition>
+                <name>INSERT_TO_MYSQL_ALL_VIDEO</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_ALL_VIDEO | default(true) }}</defaultValue>
+                <description>Create or update the InsertToMysqlAllVideoTask analytics task.</description>
+            </hudson.model.BooleanParameterDefinition>
+            <hudson.model.TextParameterDefinition>
+                <name>INSERT_TO_MYSQL_ALL_VIDEO_EXTRA_VARS</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_ALL_VIDEO_EXTRA_VARS | default('{}') | to_nice_json }}</defaultValue>
+                <description>Set default values for the InsertToMysqlAllVideoTask job parameters.  Format as YAML.</description>
+            </hudson.model.TextParameterDefinition>
+
+            <hudson.model.BooleanParameterDefinition>
+                <name>INSERT_TO_MYSQL_COURSE_ENROLL_BY_COUNTRY</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_COURSE_ENROLL_BY_COUNTRY | default(true) }}</defaultValue>
+                <description>Create or update the InsertToMysqlCourseEnrollByCountryWorkflow analytics task.</description>
+            </hudson.model.BooleanParameterDefinition>
+            <hudson.model.TextParameterDefinition>
+                <name>INSERT_TO_MYSQL_COURSE_ENROLL_BY_COUNTRY_EXTRA_VARS</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_COURSE_ENROLL_BY_COUNTRY_EXTRA_VARS | default('{}') | to_nice_json }}</defaultValue>
+                <description>Set default values for the InsertToMysqlCourseEnrollByCountryWorkflowJob job parameters.  Format as YAML.</description>
+            </hudson.model.TextParameterDefinition>
+        </parameterDefinitions>
+    </hudson.model.ParametersDefinitionProperty>
   </properties>
+  <scm class='org.jenkinsci.plugins.multiplescms.MultiSCM'>
+    <scms>
+    {% for scm in jenkins_seed_job.multiscm %}
+      {% if scm.url %}
       <scm class="hudson.plugins.git.GitSCM" plugin="git@2.4.0">
         <configVersion>2</configVersion>
         <userRemoteConfigs>
           <hudson.plugins.git.UserRemoteConfig>
-        <url>{{ jenkins_seed_job.scm.url}}</url>
-        {% if jenkins_seed_job.scm.credential_id is defined and jenkins_seed_job.scm.credential_id %}
-        <credentialsId>{{ jenkins_seed_job.scm.credential_id }}</credentialsId>
+            <url>{{ scm.url }}</url>
+            {% if scm.credential_id %}
+            <credentialsId>{{ scm.credential_id }}</credentialsId>
             {% endif %}
           </hudson.plugins.git.UserRemoteConfig>
         </userRemoteConfigs>
         <branches>
           <hudson.plugins.git.BranchSpec>
-        <name>master</name>
+            <name>{{ scm.branch }}</name>
           </hudson.plugins.git.BranchSpec>
         </branches>
+        {% if scm.dest %}
+        <relativeTargetDir>{{ scm.dest }}</relativeTargetDir>
+        {% endif %}
         <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>
         <browser class="hudson.plugins.git.browser.AssemblaWeb">
           <url></url>
@@ -33,21 +104,20 @@
         <submoduleCfg class="list"/>
         <extensions/>
       </scm>
+      {% endif %}
+    {% endfor %}
+    </scms>
+  </scm>
   <canRoam>true</canRoam>
   <disabled>false</disabled>
   <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
   <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
-  <triggers>
-    <hudson.triggers.TimerTrigger>
-      <spec>{{ jenkins_seed_job.time_trigger }}</spec>
-    </hudson.triggers.TimerTrigger>
-  </triggers>
   <concurrentBuild>false</concurrentBuild>
   <builders>
     <hudson.plugins.gradle.Gradle plugin="gradle@1.24">
       <description></description>
       <switches></switches>
-      <tasks>clean test</tasks>
+      <tasks>{{ jenkins_seed_job.dsl.gradle_tasks }}</tasks>
       <rootBuildScriptDir></rootBuildScriptDir>
       <buildFile></buildFile>
       <gradleName>(x)</gradleName>
@@ -57,13 +127,13 @@
       <useWorkspaceAsHome>false</useWorkspaceAsHome>
     </hudson.plugins.gradle.Gradle>
     <javaposse.jobdsl.plugin.ExecuteDslScripts plugin="job-dsl@1.43">
-      <targets>{{ jenkins_seed_job.scm.target_jobs }}</targets>
+      <targets>{{ jenkins_seed_job.dsl.target_jobs }}</targets>
       <usingScriptText>false</usingScriptText>
       <ignoreExisting>false</ignoreExisting>
-      <removedJobAction>{{ jenkins_seed_job.removed_job_action }}</removedJobAction>
-      <removedViewAction>{{ jenkins_seed_job.removed_view_action }}</removedViewAction>
-      <lookupStrategy>JENKINS_ROOT</lookupStrategy>
-      <additionalClasspath>{{ jenkins_seed_job.scm.additional_classpath }}</additionalClasspath>
+      <removedJobAction>{{ jenkins_seed_job.dsl.removed_job_action }}</removedJobAction>
+      <removedViewAction>{{ jenkins_seed_job.dsl.removed_view_action }}</removedViewAction>
+      <lookupStrategy>SEED_JOB</lookupStrategy>
+      <additionalClasspath>{{ jenkins_seed_job.dsl.additional_classpath }}</additionalClasspath>
     </javaposse.jobdsl.plugin.ExecuteDslScripts>
   </builders>
   <publishers/>