Create Jenkins jobs for analytics tasks using a job DSL repo.

* defaults/main.yml: adds analytics schedule variables for configuring the AnalyticsSeedJob, with all analytics tasks enabled by default.
* templates/seed_job_template.xml: use updated analytics seed job config and description
* tasks/main.yml: generates a hashed password for the jenkins user, and adds jenkins-auth and create-seed-job tags.
* README.md: documents seed job parameters and jenkins auth change

playbooks/roles/jenkins_analytics/README.md

 # Jenkins Analytics
 
-A role that sets up Jenkins for scheduling analytics tasks. 
+A role that sets up Jenkins for scheduling analytics tasks.
 
 This role performs the following steps:
 
 * Installs Jenkins using `jenkins_master`.
 * Configures `config.xml` to enable security and use
   Linux Auth Domain.
-* Creates Jenkins credentials. 
-* Enables the use of Jenkins CLI. 
+* Creates Jenkins credentials.
+* Enables the use of Jenkins CLI.
 * Installs a seed job from configured repository, launches it and waits
   for it to finish.
+* The seed job creates the analytics task jobs.
+
+Each analytics task job is created using a task-specific DSL script which
+determines the structure of the Jenkins job, e.g. its scheduled frequency, the
+git repos cloned to run the task, the parameters the job requires, and the
+shell script used to run the analytics task.  These DSL scripts live in a
+separate git repo, configured by `ANALYTICS_SCHEDULE_JOBS_DSL_REPO_*`.
+
 
 ## Configuration
 
 When you are using vagrant you **need** to set `VAGRANT_JENKINS_LOCAL_VARS_FILE`
-environment variable. This variable must point to a file containing 
+environment variable. This variable must point to a file containing
 all required variables from this section.
 
-This file needs to contain, at least, the following variables 
-(see the next few sections for more information about them): 
+This file needs to contain, at least, the following variables
+(see the next few sections for more information about them):
 
-* `JENKINS_ANALYTICS_USER_PASSWORD_HASHED` 
-* `JENKINS_ANALYTICS_USER_PASSWORD_PLAIN`
-* `JENKINS_ANALYTICS_GITHUB_KEY` or `JENKINS_ANALYTICS_CREDENTIALS`
+* `JENKINS_ANALYTICS_USER_PASSWORD_PLAIN`.
+  See [Jenkins User Password](#jenkins-user-password) for details.
+* (`JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID` and `JENKINS_ANALYTICS_GITHUB_KEY`)
+  and/or `JENKINS_ANALYTICS_CREDENTIALS`.
+  See [Jenkins Credentials](#jenkins-credentials) for details.
+* `ANALYTICS_SCHEDULE_SECURE_REPO_*` and `ANALYTICS_SCHEDULE_<TASK_NAME>_EXTRA_VARS`.
+  See [Jenkins Seed Job Configuration](#jenkins-seed-job-configuration) for details.
 
- 
-### End-user editable configuration 
+### End-user editable configuration
 
 #### Jenkins user password
 
 You'll need to override default `jenkins` user password, please do that
-as this sets up the **shell** password for this user. 
+as this sets up the **shell** password for this user.
 
-You'll need to set both a plain password and a hashed one.
-To obtain a hashed password use the `mkpasswd` command, for example: 
-`mkpasswd --method=sha-512`. (Note: a hashed password is required 
-to have clean "changed"/"unchanged" notification for this step 
-in Ansible.) 
+You'll need to set a plain password so ansible can reach Jenkins via the command line tool.
 
-* `JENKINS_ANALYTICS_USER_PASSWORD_HASHED`: hashed password 
 * `JENKINS_ANALYTICS_USER_PASSWORD_PLAIN`: plain password
 
-#### Jenkins seed job configuration 
-
-This will be filled as part of PR[#2830](https://github.com/edx/configuration/pull/2830). 
-For now go with defaults. 
-
 #### Jenkins credentials
 
-Jenkins contains its own credential store. To fill it with credentials, 
-please use the `JENKINS_ANALYTICS_CREDENTIALS` variable. This variable 
+Jenkins contains its own credential store. To fill it with credentials,
+please use the `JENKINS_ANALYTICS_CREDENTIALS` variable. This variable
 is a list of objects, each object representing a single credential.
-For now passwords and ssh-keys are supported. 
+For now passwords and ssh-keys are supported.
 
 If you only need credentials to access github repositories
 you can override `JENKINS_ANALYTICS_GITHUB_KEY`,
-which should contain contents of private key used for 
-authentication to checkout github repositories.  
+which should contain contents of private key used for
+authentication to checkout github repositories.
 
-Each credential has a unique ID, which is used to match 
+Each credential has a unique ID, which is used to match
 the credential to the task(s) for which it is needed
 
 Examples of credentials variables:
- 
-    JENKINS_ANALYTICS_GITHUB_KEY: "{{ lookup('file', 'path to keyfile') }}" 
-        
+
+    JENKINS_ANALYTICS_GITHUB_KEY: "{{ lookup('file', 'path to keyfile')
+
+    JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID: "github-readonly-key"
+
     JENKINS_ANALYTICS_CREDENTIALS:
       # id is a scope-unique credential identifier
       - id: test-password
@@ -77,29 +80,189 @@ Examples of credentials variables:
         description: Autogenerated by ansible
         password: 'password'
       # id is a scope-unique credential identifier
-      - id: github-deploy-key
+      - id: "{{ JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID }}"
         scope: GLOBAL
         # Username this ssh-key is attached to
         username: git
         # Type of credential, see other entries for example
-        type: ssh-private-key        
-        passphrase: 'foobar'
-        description: Generated by ansible
-        privatekey: |
-          -----BEGIN RSA PRIVATE KEY-----
-          Proc-Type: 4,ENCRYPTED
-          DEK-Info: AES-128-CBC,....
-
-          Key contents
-          -----END RSA PRIVATE KEY-----
+        type: ssh-private-key
+        passphrase: null
+        description: Autogenerated by ansible
+        privatekey: "{{ JENKINS_ANALYTICS_GITHUB_KEY }}'
+
+#### Jenkins seed job configuration
+
+The seed job creates the Analytics Jobs that will run the analytics tasks.  By
+default, the seed job creates all the available Analytics Jobs, but you can disable
+these jobs, and set their parameters, using `ANALYTICS_SCHEDULE_<TASK_NAME>_*`.
+
+Currently supported analytics tasks are:
+
+* `ANSWER_DISTRIBUTION`: invokes
+  `edx.analytics.tasks.answer_dist.AnswerDistributionWorkflow` via the
+  `AnswerDistributionWorkflow.groovy` DSL.
+* `IMPORT_ENROLLMENTS_INTO_MYSQL`: invokes
+  `edx.analytics.tasks.enrollments.ImportEnrollmentsIntoMysql` via the
+  `ImportEnrollmentsIntoMysql.groovy` DSL.
+* `COURSE_ACTIVITY_WEEKLY`: invokes
+  `edx.analytics.tasks.user_activity.CourseActivityWeeklyTask` via the
+  `CourseActivityWeeklyTask.groovy` DSL.
+* `INSERT_TO_MYSQL_ALL_VIDEO`: invokes
+  `edx.analytics.tasks.video.InsertToMysqlAllVideoTask` via the
+  `InsertToMysqlAllVideoTask.groovy` DSL.
+* `INSERT_TO_MYSQL_COURSE_ENROLL_BY_COUNTRY:` invokes
+  `edx.analytics.tasks.location_per_course.InsertToMysqlCourseEnrollByCountryWorkflow` via the
+  `InsertToMysqlCourseEnrollByCountryWorkflow.groovy` DSL.
+
+Since running the analytics tasks on EMR requires confidential ssh keys, the
+convention is to store them in a secure repo, which is then cloned when running
+the seed job.  To use a secure repo, override
+`ANALYTICS_SCHEDULE_SECURE_REPO_URL` and
+`ANALYTICS_SCHEDULE_SECURE_REPO_VERSION`.
+
+For example:
+
+    ANALYTICS_SCHEDULE_SECURE_REPO_URL: "git@github.com:open-craft/analytics-sandbox-private.git"
+    ANALYTICS_SCHEDULE_SECURE_REPO_VERSION: "customer-analytics-schedule"
+
+The seed job also clones a second repo, which contains the DSL scripts that
+contain the analytics task DSLs.  That repo is configured using
+`ANALYTICS_SCHEDULE_JOBS_DSL_REPO_*`, and it will be cloned directly into the
+seed job workspace.
+
+**Note:** There are two ways to specify a ssl-based github repo URL.  Note the
+subtle difference in the paths: `github.com:your-org` vs. `github.com/your-org`.
+
+* git@github.com:your-org/private-repo.git ✓
+* ssh://git@github.com/your-org/private-repo.git ✓
+
+*Not like this:*
+
+* git@github.com/your-org/private-repo.git ❌
+* ssh://git@github.com:your-org/private-repo.git ❌
+
+The full list of seed job configuration variables is:
+
+* `ANALYTICS_SCHEDULE_SECURE_REPO_URL`: Optional URL for the git repo that contains the
+  analytics task schedule configuration file.  If set, Jenkins will clone this
+  repo when the seed job is run.  Default is `null`.
+* `ANALYTICS_SCHEDULE_SECURE_REPO_VERSION`: Optional branch/tagname to checkout
+  for the secure repo.  Default is `master`.
+* `ANALYTICS_SCHEDULE_SECURE_REPO_DEST`: Optional target dir for the the secure
+  repo clone, relative to the seed job workspace.  Default is `analytics-secure-config`.
+* `ANALYTICS_SCHEDULE_SECURE_REPO_CREDENTIAL_ID`: Credential id with read
+  access to the secure repo.  Default is `{{ JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID }}`.
+  See [Jenkins Credentials](#jenkins-credentials) below for details.
+* `ANALYTICS_SCHEDULE_JOBS_DSL_REPO_URL`: Optional URL for the git repo that contains the analytics job DSLs.
+  Default is `git@github.com:edx-ops/edx-jenkins-job-dsl.git`.
+  This repo is cloned directly into the seed job workspace.
+* `ANALYTICS_SCHEDULE_JOBS_DSL_REPO_VERSION`: Optional branch/tagname to checkout for the job DSL repo.
+  Default is `master`.
+* `ANALYTICS_SCHEDULE_JOBS_DSL_REPO_CREDENTIAL_ID`: Credential id with read access to the job DSL repo.
+  Default is `{{ JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID }}`.
+  See [Jenkins Credentials](#jenkins-credentials) below for details.
+* `ANALYTICS_SCHEDULE_JOBS_DSL_CLASSPATH`: Optional additional classpath jars
+  and dirs required to run the job DSLs.
+  Each path must be newline-separated, and relative to the seed job workspace.
+  Default is:
+
+        src/main/groovy
+        lib/*.jar
+
+* `ANALYTICS_SCHEDULE_JOBS_DSL_TARGET_JOBS`: DSLs for the top-level seed job to run on build.
+  Default is `jobs/analytics-edx-jenkins.edx.org/*Jobs.groovy`
+
+
+* `ANALYTICS_SCHEDULE_<TASK_NAME>`:  `true`|`false`.  Must be set to `true` to create the analytics task.
+* `ANALYTICS_SCHEDULE_<TASK_NAME>_FREQUENCY`: Optional string representing how
+  often the analytics task should be run.  Uses a modified cron syntax, e.g.
+  `@daily`, `@weekly`, see [stackoverflow](http://stackoverflow.com/a/12472740)
+  for details.  Set to empty string to disable cron.
+  Default is different for each analytics task.
+* `ANALYTICS_SCHEDULE_<TASK_NAME>_EXTRA_VARS`: YML config or @file location to
+  override the analytics task parameters.
+  Consult the individual analytics task DSL for details on the options and defaults.
+
+For example:
+
+    ANALYTICS_SCHEDULE_ANSWER_DISTRIBUTION: true
+    ANALYTICS_SCHEDULE_ANSWER_DISTRIBUTION_EXTRA_VARS: "@path/to/answer_dist_defaults.yml"
+
+    ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL: true
+    ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS:
+      TASKS_REPO: "https://github.com/open-craft/edx-analytics-pipeline.git"
+      TASKS_BRANCH: "analytics-sandbox"
+      CONFIG_REPO: "https://github.com/open-craft/edx-analytics-configuration.git"
+      CONFIG_BRANCH: "analytics-sandbox"
+      JOB_NAME: "ImportEnrollmentsIntoMysql"
+      JOB_FREQUENCY: "@monthly"
+      CLUSTER_NAME: "AnswerDistribution"
+      EMR_EXTRA_VARS: "@/home/jenkins/emr-vars.yml"  # see [EMR Configuration](#emr-configuration)
+      FROM_DATE: "2016-01-01"
+      TASK_USER: "hadoop"
+      NOTIFY_EMAIL_ADDRESSES: "staff@example.com
+
+##### EMR Configuration
+
+The `EMR_EXTRA_VARS` parameter for each analytics task is passed by the analytics
+task shell command to the ansible playbook for provisioning and terminating the
+EMR cluster.
+
+Because `EMR_EXTRA_VARS` passes via the shell, it may reference other analytics
+task parameters as shell variables, e.g. `$S3_PACKAGE_BUCKET`.
+
+**File path**
+
+The easiest way to modify this parameter is to provide a `@/path/to/file.yml`
+or `@/path/to/file.json`.  The file path must be absolute, e.g.,
+
+    ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS:
+      EMR_EXTRA_VARS: '@/home/jenkins/emr-vars.yml'
+
+Or relative to the analytics-configuration repo cloned by the analytics task, e.g.,
+
+    ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS:
+      EMR_EXTRA_VARS: '@./config/emr-vars.yml'
+
+To use a path relative to the analytics task workspace, build an absolute path
+using the `$WORKSPACE` variable provided by Jenkins, e.g.,
+
+    ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS:
+      EMR_EXTRA_VARS: '@$WORKSPACE/../AnalyticsSeedTask/analytics-secure-config/emr-vars.yml'
+
+
+**Raw JSON**
+
+The other option, utilised by the DSL `EMR_EXTRA_VARS` default value, is to use a
+JSON string.  Take care to use a *JSON string*, not raw JSON itself, as YAML is
+a JSON superset, and we don't want the JSON to be parsed by ansible.
+
+Also, because formatting valid JSON is difficult, be sure to run the text
+through a JSON validator before deploying.
+
+As with file paths, the JSON text can use analytics task parameters as shell
+variables, e.g.,
+
+    ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS:
+      AUTOMATION_KEYPAIR_NAME: 'analytics-sandbox'
+      VPC_SUBNET_ID: 'subnet-cd1b9c94'
+      EMR_LOG_BUCKET: 's3://analytics-sandbox-emr-logs'
+      CLUSTER_NAME: 'Analytics EMR Cluster'
+      EMR_EXTRA_VARS: |
+        {
+          "name": "$CLUSTER_NAME",
+          "keypair_name": "$AUTOMATION_KEYPAIR_NAME",
+          "vpc_subnet_id": "$VPC_SUBNET_ID",
+          "log_uri": "$EMR_LOG_BUCKET"
+        }
 
 #### Other useful variables
 
-* `JENKINS_ANALYTICS_CONCURRENT_JOBS_COUNT`: Configures number of 
-  executors (or concurrent jobs this Jenkins instance can 
-  execute). Defaults to `2`. 
+* `JENKINS_ANALYTICS_CONCURRENT_JOBS_COUNT`: Configures number of
+  executors (or concurrent jobs this Jenkins instance can
+  execute). Defaults to `2`.
 
-### General configuration 
+### General configuration
 
 Following variables are used by this role:
 
@@ -131,8 +294,7 @@ Unix realm requires the following settings:
 * `service`: Jenkins uses PAM configuration for this service. `su` is
 a safe choice as it doesn't require a user to have the ability to login
 remotely.
-* `plain_password`:  plaintext password, **you should change** default values.
-* `hashed_password`: hashed password
+* `plain_password`:  plaintext password, **you must change** default values.
 
 Example realm configuration:
 
@@ -140,7 +302,6 @@ Example realm configuration:
       name: unix
       service: su
       plain_password: jenkins
-      hashed_password: $6$rAVyI.p2wXVDKk5w$y0G1MQehmHtvaPgdtbrnvAsBqYQ99g939vxrdLXtPQCh/e7GJVwbnqIKZpve8EcMLTtq.7sZwTBYV9Tdjgf1k.
 
 
 #### Seed job configuration
@@ -159,11 +320,11 @@ attributes:
   * `scm.type`: It must have value of `git`.
   * `scm.url`: URL for the repository.
   * `scm.credential_id`: Id of a credential to use when authenticating to the
-    repository.     
-    This setting is optional. If it is missing or falsy, credentials will be omitted. 
-    Please note that when you use ssh repository url, you'll need to set up a key regardless 
+    repository.
+    This setting is optional. If it is missing or falsy, credentials will be omitted.
+    Please note that when you use ssh repository url, you'll need to set up a key regardless
     of whether the repository is public or private (to establish an ssh connection
-    you need a valid public key). 
+    you need a valid public key).
   * `scm.target_jobs`: A shell glob expression relative to repo root selecting
     jobs to import.
   * `scm.additional_classpath`: A path relative to repo root, pointing to a

playbooks/roles/jenkins_analytics/defaults/main.yml

@@ -2,11 +2,11 @@
 
 # See README.md for variable descriptions
 
-JENKINS_ANALYTICS_USER_PASSWORD_HASHED: $6$rAVyI.p2wXVDKk5w$y0G1MQehmHtvaPgdtbrnvAsBqYQ99g939vxrdLXtPQCh/e7GJVwbnqIKZpve8EcMLTtq.7sZwTBYV9Tdjgf1k.
 JENKINS_ANALYTICS_USER_PASSWORD_PLAIN: jenkins
+JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID: 'github-deploy-key'
 
 JENKINS_ANALYTICS_CREDENTIALS:
-  - id: github-deploy-key
+  - id: "{{ JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID }}"
     scope: GLOBAL
     username: git
     type: ssh-private-key
@@ -16,6 +16,39 @@ JENKINS_ANALYTICS_CREDENTIALS:
 
 JENKINS_ANALYTICS_CONCURRENT_JOBS_COUNT: 2
 
+ANALYTICS_SCHEDULE_SECURE_REPO_URL: null
+ANALYTICS_SCHEDULE_SECURE_REPO_DEST: "analytics-secure-config"
+ANALYTICS_SCHEDULE_SECURE_REPO_VERSION: "master"
+ANALYTICS_SCHEDULE_SECURE_REPO_CREDENTIAL_ID: "{{ JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID }}"
+ANALYTICS_SCHEDULE_JOBS_DSL_REPO_URL: "git@github.com:edx-ops/edx-jenkins-job-dsl.git"
+ANALYTICS_SCHEDULE_JOBS_DSL_REPO_VERSION: "master"
+ANALYTICS_SCHEDULE_JOBS_DSL_REPO_CREDENTIAL_ID: "{{ JENKINS_ANALYTICS_GITHUB_CREDENTIAL_ID }}"
+ANALYTICS_SCHEDULE_JOBS_DSL_CLASSPATH: |
+  src/main/groovy
+  lib/*.jar
+ANALYTICS_SCHEDULE_JOBS_DSL_TARGET_JOBS: "jobs/analytics-edx-jenkins.edx.org/*Jobs.groovy"
+
+ANALYTICS_SCHEDULE_ANSWER_DISTRIBUTION: true
+ANALYTICS_SCHEDULE_ANSWER_DISTRIBUTION_EXTRA_VARS:
+  SECURE_REPO: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_URL }}"
+  SECURE_BRANCH: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_VERSION }}"
+ANALYTICS_SCHEDULE_COURSE_ACTIVITY_WEEKLY: true
+ANALYTICS_SCHEDULE_COURSE_ACTIVITY_WEEKLY_EXTRA_VARS:
+  SECURE_REPO: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_URL }}"
+  SECURE_BRANCH: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_VERSION }}"
+ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL: true
+ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS:
+  SECURE_REPO: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_URL }}"
+  SECURE_BRANCH: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_VERSION }}"
+ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_ALL_VIDEO: true
+ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_ALL_VIDEO_EXTRA_VARS:
+  SECURE_REPO: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_URL }}"
+  SECURE_BRANCH: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_VERSION }}"
+ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_COURSE_ENROLL_BY_COUNTRY: true
+ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_COURSE_ENROLL_BY_COUNTRY_EXTRA_VARS:
+  SECURE_REPO: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_URL }}"
+  SECURE_BRANCH: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_VERSION }}"
+
 jenkins_credentials_root: '/tmp/credentials'
 jenkins_credentials_file_dest: "{{ jenkins_credentials_root }}/credentials.json"
 jenkins_credentials_script: "{{ jenkins_credentials_root }}/addCredentials.groovy"
@@ -27,16 +60,27 @@ jenkins_auth_realm:
   service: su
   # Change this default password: (see README.md to see how you can do it)
   plain_password: "{{ JENKINS_ANALYTICS_USER_PASSWORD_PLAIN }}"
-  hashed_password: "{{ JENKINS_ANALYTICS_USER_PASSWORD_HASHED }}"
 
+# For now only a single seed job is supported, adding more would require
+# Ansible 2.+ or converting _execute_jenkins_cli to a module
 jenkins_seed_job:
-  name: analytics-seed-job
-  time_trigger: "H * * * *"
-  removed_job_action: "DELETE"
-  removed_view_action: "IGNORE"
-  scm:
-    type: git
-    url: "git@github.com:edx-ops/edx-jenkins-job-dsl.git"
-    credential_id: "github-deploy-key"
-    target_jobs: "jobs/analytics-edx-jenkins.edx.org/*Jobs.groovy"
-    additional_classpath: "src/main/groovy"
+  name: AnalyticsSeedJob
+  multiscm:
+    - scm:
+      type: git
+      url: "{{ ANALYTICS_SCHEDULE_JOBS_DSL_REPO_URL }}"
+      dest: ""
+      branch: "{{ ANALYTICS_SCHEDULE_JOBS_DSL_REPO_VERSION | default('master') }}"
+      credential_id: "{{ ANALYTICS_SCHEDULE_JOBS_DSL_REPO_CREDENTIAL_ID | default('') }}"
+    - scm:
+      type: git
+      url: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_URL | default('') }}"
+      dest: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_DEST | default('') }}"
+      branch: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_VERSION | default('master') }}"
+      credential_id: "{{ ANALYTICS_SCHEDULE_SECURE_REPO_CREDENTIAL_ID | default('') }}"
+  dsl:
+    gradle_tasks: "clean libs test"
+    removed_view_action: IGNORE
+    removed_job_action: DELETE
+    additional_classpath: "{{ ANALYTICS_SCHEDULE_JOBS_DSL_CLASSPATH }}"
+    target_jobs: "{{ ANALYTICS_SCHEDULE_JOBS_DSL_TARGET_JOBS }}"

playbooks/roles/jenkins_analytics/tasks/main.yml

@@ -2,16 +2,29 @@
 
 - fail: msg=included unix realm by accident
   when: jenkins_auth_realm.name != "unix"
+  tags: 
+    - jenkins-auth
 
 - fail: msg=Please change default password for jenkins user
   when: jenkins_auth_realm.plain_password == 'jenkins'
+  tags: 
+    - jenkins-auth
+
+- name: generate jenkins user password
+  shell: "openssl passwd -1 '{{ jenkins_auth_realm.plain_password | quote }}'"
+  register: jenkins_user_password_hash
+  no_log: True
+  tags: 
+    - jenkins-auth
 
 - user:
     name: "{{ jenkins_user }}"
+    state: present
     groups: shadow
     append: yes
-    password: "{{ jenkins_auth_realm.hashed_password }}"
-    update_password: always
+    password: "{{ jenkins_user_password_hash.stdout }}"
+  tags: 
+    - jenkins-auth
 
 - name: template config.xml
   template:
@@ -19,6 +32,8 @@
     dest: "{{ jenkins_home }}/config.xml"
     owner: "{{ jenkins_user }}"
     group: "{{ jenkins_group }}"
+  tags: 
+    - jenkins-auth
 
 # Unconditionally restart Jenkins, this has two side-effects:
 # 1. Jenkins uses new auth realm
@@ -28,17 +43,23 @@
 
 - name: restart Jenkins
   service: name=jenkins state=restarted
+  tags: 
+    - jenkins-auth
 
 # Upload Jenkins credentials
 
 - name: create credentials dir
   file: name={{ jenkins_credentials_root }} state=directory
+  tags: 
+    - jenkins-auth
 
 - name: upload groovy script
   template:
     src: addCredentials.groovy
     dest: "{{ jenkins_credentials_script }}"
     mode: "600"
+  tags: 
+    - jenkins-auth
 
 - name: upload credentials file
   template:
@@ -46,48 +67,66 @@
     dest: "{{ jenkins_credentials_file_dest }}"
     mode: "600"
     owner: "{{ jenkins_user }}"
+  tags: 
+    - jenkins-auth
 
 - name: add credentials
   include: execute_jenkins_cli.yaml
   vars:
     jenkins_command_string: "groovy {{ jenkins_credentials_script }}"
+  tags: 
+    - jenkins-auth
 
 - name: clean up
   file: name={{ jenkins_credentials_root }} state=absent
+  tags: 
+    - jenkins-auth
 
 # Upload seed job
 
 - name: upload job file
-  template: src=seed_job_template.xml dest=/tmp/{{  jenkins_seed_job.name }} mode="600"
+  template: src=seed_job_template.xml dest=/tmp/{{ jenkins_seed_job.name }} mode="600"
+  tags:
+    - create-seed-job
 
 - name: check if job is present
   include: execute_jenkins_cli.yaml
   vars:
-    jenkins_command_string: "get-job {{  jenkins_seed_job.name }}"
+    jenkins_command_string: "get-job {{ jenkins_seed_job.name }}"
     jenkins_ignore_cli_errors: yes
+  tags:
+    - create-seed-job
 
 - set_fact:
     get_job_output: "{{ jenkins_command_output }}"
+  tags:
+    - create-seed-job
 
 # Upload seed job to Jenkins
 
 - name: Create seed job if absent
   include: execute_jenkins_cli.yaml
   vars:
-    jenkins_command_string: "create-job {{  jenkins_seed_job.name }}"
-    jenkins_command_prefix: "cat /tmp/{{  jenkins_seed_job.name }} | "
+    jenkins_command_string: "create-job {{ jenkins_seed_job.name }}"
+    jenkins_command_prefix: "cat /tmp/{{ jenkins_seed_job.name }} | "
   when: get_job_output.rc != 0
+  tags:
+    - create-seed-job
 
 - name: update seed job
   include: execute_jenkins_cli.yaml
   vars:
-    jenkins_command_string: "update-job {{  jenkins_seed_job.name }}"
-    jenkins_command_prefix: "cat /tmp/{{  jenkins_seed_job.name }} | "
+    jenkins_command_string: "update-job {{ jenkins_seed_job.name }}"
+    jenkins_command_prefix: "cat /tmp/{{ jenkins_seed_job.name }} | "
   when: get_job_output.rc == 0
+  tags:
+    - create-seed-job
 
 # Build the seed job
 
 - name: Build the seed job
   include: execute_jenkins_cli.yaml
   vars:
-    jenkins_command_string: "build {{  jenkins_seed_job.name }} -s"
+    jenkins_command_string: "build {{ jenkins_seed_job.name }} -s"
+  tags:
+    - create-seed-job

playbooks/roles/jenkins_analytics/templates/seed_job_template.xml

@@ -2,7 +2,7 @@
 <project>
   <actions/>
   <description>
-    Seed job autogenerated by ansible, it will be overridden.
+    Creates and configures the analytics task jobs.
   </description>
   <keepDependencies>false</keepDependencies>
   <properties>
@@ -10,44 +10,114 @@
       <useJobPriority>false</useJobPriority>
       <priority>-1</priority>
     </jenkins.advancedqueue.AdvancedQueueSorterJobProperty>
+    <hudson.model.ParametersDefinitionProperty>
+        <parameterDefinitions>
+            <hudson.model.StringParameterDefinition>
+                <name>GIT_CREDENTIAL_ID</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_SECURE_REPO_CREDENTIAL_ID | default('') }}</defaultValue>
+                <description>Jenkins Credential ID used for cloning secure git
+                  repos.  Must match a Credential configured in Jenkins.</description>
+            </hudson.model.StringParameterDefinition>
+            <hudson.model.BooleanParameterDefinition>
+                <name>ANSWER_DISTRIBUTION</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_ANSWER_DISTRIBUTION | default(true) }}</defaultValue>
+                <description>Create or update the AnswerDistributionWorkflow analytics task.</description>
+            </hudson.model.BooleanParameterDefinition>
+            <hudson.model.TextParameterDefinition>
+                <name>ANSWER_DISTRIBUTION_EXTRA_VARS</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_ANSWER_DISTRIBUTION_EXTRA_VARS | default('{}') | to_nice_json }}</defaultValue>
+                <description>Set default values for the AnswerDistributionWorkflow job parameters.  Format as YAML.</description>
+            </hudson.model.TextParameterDefinition>
+
+            <hudson.model.BooleanParameterDefinition>
+                <name>COURSE_ACTIVITY_WEEKLY</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_COURSE_ACTIVITY_WEEKLY | default(true) }}</defaultValue>
+                <description>Create or update the CourseActivityWeeklyTask analytics task.</description>
+            </hudson.model.BooleanParameterDefinition>
+            <hudson.model.TextParameterDefinition>
+                <name>COURSE_ACTIVITY_WEEKLY_EXTRA_VARS</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_COURSE_ACTIVITY_WEEKLY_EXTRA_VARS | default('{}') | to_nice_json }}</defaultValue>
+                <description>Set default values for the CourseActivityWeeklyTask job parameters.  Format as YAML.</description>
+            </hudson.model.TextParameterDefinition>
+
+            <hudson.model.BooleanParameterDefinition>
+                <name>IMPORT_ENROLLMENTS_INTO_MYSQL</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL | default(true) }}</defaultValue>
+                <description>Create or update the ImportEnrollmentsIntoMysql analytics task.</description>
+            </hudson.model.BooleanParameterDefinition>
+            <hudson.model.TextParameterDefinition>
+                <name>IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_IMPORT_ENROLLMENTS_INTO_MYSQL_EXTRA_VARS | default('{}') | to_nice_json }}</defaultValue>
+                <description>Set default values for the ImportEnrollmentsIntoMysql job parameters.  Format as YAML.</description>
+            </hudson.model.TextParameterDefinition>
+
+            <hudson.model.BooleanParameterDefinition>
+                <name>INSERT_TO_MYSQL_ALL_VIDEO</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_ALL_VIDEO | default(true) }}</defaultValue>
+                <description>Create or update the InsertToMysqlAllVideoTask analytics task.</description>
+            </hudson.model.BooleanParameterDefinition>
+            <hudson.model.TextParameterDefinition>
+                <name>INSERT_TO_MYSQL_ALL_VIDEO_EXTRA_VARS</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_ALL_VIDEO_EXTRA_VARS | default('{}') | to_nice_json }}</defaultValue>
+                <description>Set default values for the InsertToMysqlAllVideoTask job parameters.  Format as YAML.</description>
+            </hudson.model.TextParameterDefinition>
+
+            <hudson.model.BooleanParameterDefinition>
+                <name>INSERT_TO_MYSQL_COURSE_ENROLL_BY_COUNTRY</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_COURSE_ENROLL_BY_COUNTRY | default(true) }}</defaultValue>
+                <description>Create or update the InsertToMysqlCourseEnrollByCountryWorkflow analytics task.</description>
+            </hudson.model.BooleanParameterDefinition>
+            <hudson.model.TextParameterDefinition>
+                <name>INSERT_TO_MYSQL_COURSE_ENROLL_BY_COUNTRY_EXTRA_VARS</name>
+                <defaultValue>{{ ANALYTICS_SCHEDULE_INSERT_TO_MYSQL_COURSE_ENROLL_BY_COUNTRY_EXTRA_VARS | default('{}') | to_nice_json }}</defaultValue>
+                <description>Set default values for the InsertToMysqlCourseEnrollByCountryWorkflowJob job parameters.  Format as YAML.</description>
+            </hudson.model.TextParameterDefinition>
+        </parameterDefinitions>
+    </hudson.model.ParametersDefinitionProperty>
   </properties>
-  <scm class="hudson.plugins.git.GitSCM" plugin="git@2.4.0">
-    <configVersion>2</configVersion>
-    <userRemoteConfigs>
-      <hudson.plugins.git.UserRemoteConfig>
-        <url>{{ jenkins_seed_job.scm.url}}</url>
-        {% if jenkins_seed_job.scm.credential_id is defined and jenkins_seed_job.scm.credential_id %}
-        <credentialsId>{{ jenkins_seed_job.scm.credential_id }}</credentialsId>
+  <scm class='org.jenkinsci.plugins.multiplescms.MultiSCM'>
+    <scms>
+    {% for scm in jenkins_seed_job.multiscm %}
+      {% if scm.url %}
+      <scm class="hudson.plugins.git.GitSCM" plugin="git@2.4.0">
+        <configVersion>2</configVersion>
+        <userRemoteConfigs>
+          <hudson.plugins.git.UserRemoteConfig>
+            <url>{{ scm.url }}</url>
+            {% if scm.credential_id %}
+            <credentialsId>{{ scm.credential_id }}</credentialsId>
+            {% endif %}
+          </hudson.plugins.git.UserRemoteConfig>
+        </userRemoteConfigs>
+        <branches>
+          <hudson.plugins.git.BranchSpec>
+            <name>{{ scm.branch }}</name>
+          </hudson.plugins.git.BranchSpec>
+        </branches>
+        {% if scm.dest %}
+        <relativeTargetDir>{{ scm.dest }}</relativeTargetDir>
         {% endif %}
-      </hudson.plugins.git.UserRemoteConfig>
-    </userRemoteConfigs>
-    <branches>
-      <hudson.plugins.git.BranchSpec>
-        <name>master</name>
-      </hudson.plugins.git.BranchSpec>
-    </branches>
-    <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>
-    <browser class="hudson.plugins.git.browser.AssemblaWeb">
-      <url></url>
-    </browser>
-    <submoduleCfg class="list"/>
-    <extensions/>
+        <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>
+        <browser class="hudson.plugins.git.browser.AssemblaWeb">
+          <url></url>
+        </browser>
+        <submoduleCfg class="list"/>
+        <extensions/>
+      </scm>
+      {% endif %}
+    {% endfor %}
+    </scms>
   </scm>
   <canRoam>true</canRoam>
   <disabled>false</disabled>
   <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
   <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
-  <triggers>
-    <hudson.triggers.TimerTrigger>
-      <spec>{{ jenkins_seed_job.time_trigger }}</spec>
-    </hudson.triggers.TimerTrigger>
-  </triggers>
   <concurrentBuild>false</concurrentBuild>
   <builders>
     <hudson.plugins.gradle.Gradle plugin="gradle@1.24">
       <description></description>
       <switches></switches>
-      <tasks>clean test</tasks>
+      <tasks>{{ jenkins_seed_job.dsl.gradle_tasks }}</tasks>
       <rootBuildScriptDir></rootBuildScriptDir>
       <buildFile></buildFile>
       <gradleName>(x)</gradleName>
@@ -57,13 +127,13 @@
       <useWorkspaceAsHome>false</useWorkspaceAsHome>
     </hudson.plugins.gradle.Gradle>
     <javaposse.jobdsl.plugin.ExecuteDslScripts plugin="job-dsl@1.43">
-      <targets>{{ jenkins_seed_job.scm.target_jobs }}</targets>
+      <targets>{{ jenkins_seed_job.dsl.target_jobs }}</targets>
       <usingScriptText>false</usingScriptText>
       <ignoreExisting>false</ignoreExisting>
-      <removedJobAction>{{ jenkins_seed_job.removed_job_action }}</removedJobAction>
-      <removedViewAction>{{ jenkins_seed_job.removed_view_action }}</removedViewAction>
-      <lookupStrategy>JENKINS_ROOT</lookupStrategy>
-      <additionalClasspath>{{ jenkins_seed_job.scm.additional_classpath }}</additionalClasspath>
+      <removedJobAction>{{ jenkins_seed_job.dsl.removed_job_action }}</removedJobAction>
+      <removedViewAction>{{ jenkins_seed_job.dsl.removed_view_action }}</removedViewAction>
+      <lookupStrategy>SEED_JOB</lookupStrategy>
+      <additionalClasspath>{{ jenkins_seed_job.dsl.additional_classpath }}</additionalClasspath>
     </javaposse.jobdsl.plugin.ExecuteDslScripts>
   </builders>
   <publishers/>