Merge pull request #2810 from edx/jbzdak/olive-19-auto-provision-analytics-devstack

OLIVE-19 Enable automatic provisioning of jenkins instance dedicated to running analytics tasks. 

.gitignore

@@ -24,3 +24,6 @@ vagrant_ansible_inventory_default
 ## Make artifacts
 .build
 playbooks/edx-east/travis-test.yml
+
+## Local virtualenv
+/venv

playbooks/analytics-jenkins.yml

+- name: Configure instance(s)
+  hosts: all
+  sudo: True
+  roles:
+  - jenkins_analytics

playbooks/roles/jenkins_analytics/README.md

+# Jenkins Analytics
+
+A role that sets up Jenkins for scheduling analytics tasks. 
+
+This role performs the following steps:
+
+* Installs Jenkins using `jenkins_master`.
+* Configures `config.xml` to enable security and use
+  Linux Auth Domain.
+* Creates Jenkins credentials. 
+* Enables the use of Jenkins CLI. 
+* Installs a seed job from configured repository, launches it and waits
+  for it to finish.
+
+## Configuration
+
+When you are using vagrant you **need** to set `VAGRANT_JENKINS_LOCAL_VARS_FILE`
+environment variable. This variable must point to a file containing 
+all required variables from this section.
+
+This file needs to contain, at least, the following variables 
+(see the next few sections for more information about them): 
+
+* `JENKINS_ANALYTICS_USER_PASSWORD_HASHED` 
+* `JENKINS_ANALYTICS_USER_PASSWORD_PLAIN`
+* `JENKINS_ANALYTICS_GITHUB_KEY` or `JENKINS_ANALYTICS_CREDENTIALS`
+
+ 
+### End-user editable configuration 
+
+#### Jenkins user password
+
+You'll need to override default `jenkins` user password, please do that
+as this sets up the **shell** password for this user. 
+
+You'll need to set both a plain password and a hashed one.
+To obtain a hashed password use the `mkpasswd` command, for example: 
+`mkpasswd --method=sha-512`. (Note: a hashed password is required 
+to have clean "changed"/"unchanged" notification for this step 
+in Ansible.) 
+
+* `JENKINS_ANALYTICS_USER_PASSWORD_HASHED`: hashed password 
+* `JENKINS_ANALYTICS_USER_PASSWORD_PLAIN`: plain password
+
+#### Jenkins seed job configuration 
+
+This will be filled as part of PR[#2830](https://github.com/edx/configuration/pull/2830). 
+For now go with defaults. 
+
+#### Jenkins credentials
+
+Jenkins contains its own credential store. To fill it with credentials, 
+please use the `JENKINS_ANALYTICS_CREDENTIALS` variable. This variable 
+is a list of objects, each object representing a single credential.
+For now passwords and ssh-keys are supported. 
+
+If you only need credentials to access github repositories
+you can override `JENKINS_ANALYTICS_GITHUB_KEY`,
+which should contain contents of private key used for 
+authentication to checkout github repositories.  
+
+Each credential has a unique ID, which is used to match 
+the credential to the task(s) for which it is needed
+
+Examples of credentials variables:
+ 
+    JENKINS_ANALYTICS_GITHUB_KEY: "{{ lookup('file', 'path to keyfile') }}" 
+        
+    JENKINS_ANALYTICS_CREDENTIALS:
+      # id is a scope-unique credential identifier
+      - id: test-password
+        # Scope must be global. To have other scopes you'll need to modify addCredentials.groovy
+        scope: GLOBAL
+        # Username associated with this password
+        username: jenkins
+        type: username-password
+        description: Autogenerated by ansible
+        password: 'password'
+      # id is a scope-unique credential identifier
+      - id: github-deploy-key
+        scope: GLOBAL
+        # Username this ssh-key is attached to
+        username: git
+        # Type of credential, see other entries for example
+        type: ssh-private-key        
+        passphrase: 'foobar'
+        description: Generated by ansible
+        privatekey: |
+          -----BEGIN RSA PRIVATE KEY-----
+          Proc-Type: 4,ENCRYPTED
+          DEK-Info: AES-128-CBC,....
+
+          Key contents
+          -----END RSA PRIVATE KEY-----
+
+#### Other useful variables
+
+* `JENKINS_ANALYTICS_CONCURRENT_JOBS_COUNT`: Configures number of 
+  executors (or concurrent jobs this Jenkins instance can 
+  execute). Defaults to `2`. 
+
+### General configuration 
+
+Following variables are used by this role:
+
+Variables used by command waiting on Jenkins start-up after running
+`jenkins_master` role:
+
+    jenkins_connection_retries: 60
+    jenkins_connection_delay: 0.5
+
+#### Auth realm
+
+Jenkins auth realm encapsulates user management in Jenkins, that is:
+
+* What users can log in
+* What credentials they use to log in
+
+Realm type stored in `jenkins_auth_realm.name` variable.
+
+In future we will try to enable other auth domains, while
+preserving the ability to run cli.
+
+##### Unix Realm
+
+For now only `unix` realm supported -- which requires every Jenkins
+user to have a shell account on the server.
+
+Unix realm requires the following settings:
+
+* `service`: Jenkins uses PAM configuration for this service. `su` is
+a safe choice as it doesn't require a user to have the ability to login
+remotely.
+* `plain_password`:  plaintext password, **you should change** default values.
+* `hashed_password`: hashed password
+
+Example realm configuration:
+
+    jenkins_auth_realm:
+      name: unix
+      service: su
+      plain_password: jenkins
+      hashed_password: $6$rAVyI.p2wXVDKk5w$y0G1MQehmHtvaPgdtbrnvAsBqYQ99g939vxrdLXtPQCh/e7GJVwbnqIKZpve8EcMLTtq.7sZwTBYV9Tdjgf1k.
+
+
+#### Seed job configuration
+
+Seed job is configured in `jenkins_seed_job` variable, which has the following
+attributes:
+
+* `name`:  Name of the job in Jenkins.
+* `time_trigger`: A Jenkins cron entry defining how often this job should run.
+* `removed_job_action`: what to do when a job created by a previous run of seed job
+  is missing from current run. This can be either `DELETE` or`IGNORE`.
+* `removed_view_action`: what to do when a view created by a previous run of seed job
+  is missing from current run. This can be either `DELETE` or`IGNORE`.
+* `scm`: Scm object is used to define seed job repository and related settings.
+  It has the following properties:
+  * `scm.type`: It must have value of `git`.
+  * `scm.url`: URL for the repository.
+  * `scm.credential_id`: Id of a credential to use when authenticating to the
+    repository.     
+    This setting is optional. If it is missing or falsy, credentials will be omitted. 
+    Please note that when you use ssh repository url, you'll need to set up a key regardless 
+    of whether the repository is public or private (to establish an ssh connection
+    you need a valid public key). 
+  * `scm.target_jobs`: A shell glob expression relative to repo root selecting
+    jobs to import.
+  * `scm.additional_classpath`: A path relative to repo root, pointing to a
+     directory that contains additional groovy scripts used by the seed jobs.
+
+Example scm configuration:
+
+    jenkins_seed_job:
+      name: seed
+      time_trigger: "H * * * *"
+      removed_job_action: "DELETE"
+      removed_view_action: "IGNORE"
+      scm:
+        type: git
+        url: "git@github.com:edx-ops/edx-jenkins-job-dsl.git"
+        credential_id: "github-deploy-key"
+        target_jobs: "jobs/analytics-edx-jenkins.edx.org/*Jobs.groovy"
+        additional_classpath: "src/main/groovy"
+
+Known issues
+------------
+
+1. Playbook named `execute_ansible_cli.yaml`, should be converted to an
+   Ansible module (it is already used in a module-ish way).
+2. Anonymous user has discover and get job permission, as without it
+   `get-job`, `build <<job>>` commands wouldn't work.
+   Giving anonymous these permissions is a workaround for
+   transient Jenkins issue (reported [couple][1] [of][2] [times][3]).
+3. We force unix authentication method -- that is, every user that can login
+   to Jenkins also needs to have a shell account on master.
+
+
+Dependencies
+------------
+
+- `jenkins_master`
+
+[1]: https://issues.jenkins-ci.org/browse/JENKINS-12543
+[2]: https://issues.jenkins-ci.org/browse/JENKINS-11024
+[3]: https://issues.jenkins-ci.org/browse/JENKINS-22143

playbooks/roles/jenkins_analytics/defaults/main.yml

+---
+
+# See README.md for variable descriptions
+
+JENKINS_ANALYTICS_USER_PASSWORD_HASHED: $6$rAVyI.p2wXVDKk5w$y0G1MQehmHtvaPgdtbrnvAsBqYQ99g939vxrdLXtPQCh/e7GJVwbnqIKZpve8EcMLTtq.7sZwTBYV9Tdjgf1k.
+JENKINS_ANALYTICS_USER_PASSWORD_PLAIN: jenkins
+
+JENKINS_ANALYTICS_CREDENTIALS:
+  - id: github-deploy-key
+    scope: GLOBAL
+    username: git
+    type: ssh-private-key
+    passphrase: null
+    description: Autogenerated by ansible
+    privatekey: "{{ JENKINS_ANALYTICS_GITHUB_KEY }}"
+
+JENKINS_ANALYTICS_CONCURRENT_JOBS_COUNT: 2
+
+jenkins_credentials_root: '/tmp/credentials'
+jenkins_credentials_file_dest: "{{ jenkins_credentials_root }}/credentials.json"
+jenkins_credentials_script: "{{ jenkins_credentials_root }}/addCredentials.groovy"
+
+jenkins_connection_retries: 240
+jenkins_connection_delay: 1
+jenkins_auth_realm:
+  name: unix
+  service: su
+  # Change this default password: (see README.md to see how you can do it)
+  plain_password: "{{ JENKINS_ANALYTICS_USER_PASSWORD_PLAIN }}"
+  hashed_password: "{{ JENKINS_ANALYTICS_USER_PASSWORD_HASHED }}"
+
+jenkins_seed_job:
+  name: analytics-seed-job
+  time_trigger: "H * * * *"
+  removed_job_action: "DELETE"
+  removed_view_action: "IGNORE"
+  scm:
+    type: git
+    url: "git@github.com:edx-ops/edx-jenkins-job-dsl.git"
+    credential_id: "github-deploy-key"
+    target_jobs: "jobs/analytics-edx-jenkins.edx.org/*Jobs.groovy"
+    additional_classpath: "src/main/groovy"

playbooks/roles/jenkins_analytics/meta/main.yml

+dependencies:
+  - jenkins_master

playbooks/roles/jenkins_analytics/tasks/execute_jenkins_cli.yaml

+---
+
+- fail: msg=for now we can execute commands iff jenkins auth realm is unix
+  when: jenkins_auth_realm.name != "unix"
+
+- set_fact:
+    jenkins_cli_root: "/tmp/jenkins-cli/{{ ansible_ssh_user }}"
+- set_fact:
+    jenkins_cli_jar: "{{ jenkins_cli_root }}/jenkins_cli.jar"
+    jenkins_cli_pass: "{{ jenkins_cli_root }}/jenkins_cli_pass"
+
+- name: create cli dir
+  file: name={{ jenkins_cli_root }} state=directory mode="700"
+
+- name: create pass file
+  template: src=jenkins-pass-file.j2 dest={{ jenkins_cli_pass }} mode="600"
+
+- name: Wait for Jenkins CLI
+  uri:
+    url: "http://localhost:{{ jenkins_port }}/cli/"
+    method: GET
+    return_content: yes
+    status_code: 200,403
+  register: result
+  until: (result.status is defined) and ((result.status == 403) or (results.status == 200))
+  retries: "{{ jenkins_connection_retries }}"
+  delay: "{{ jenkins_connection_delay }}"
+  changed_when: false
+
+- name: get cli
+  get_url:
+    url: "http://localhost:{{ jenkins_port }}/jnlpJars/jenkins-cli.jar"
+    dest: "{{ jenkins_cli_jar }}"
+
+- name: login
+  command: java -jar {{ jenkins_cli_jar }} -s http://localhost:{{ jenkins_port }}
+           login --username={{ jenkins_user }}
+           --password-file={{ jenkins_cli_pass }}
+
+- name: execute command
+  shell: >
+    {{ jenkins_command_prefix|default('') }} java -jar {{ jenkins_cli_jar }} -s http://localhost:{{ jenkins_port }}
+    {{ jenkins_command_string }}
+  register: jenkins_command_output
+  ignore_errors: "{{ jenkins_ignore_cli_errors|default (False) }}"
+
+- name: "clean up --- remove the credentials dir"
+  file: name=jenkins_cli_root state=absent
+
+- name: "clean up --- remove cached Jenkins credentials"
+  command: rm -rf $HOME/.jenkins

playbooks/roles/jenkins_analytics/tasks/main.yml

+---
+
+- fail: msg=included unix realm by accident
+  when: jenkins_auth_realm.name != "unix"
+
+- fail: msg=Please change default password for jenkins user
+  when: jenkins_auth_realm.plain_password == 'jenkins'
+
+- user:
+    name: "{{ jenkins_user }}"
+    groups: shadow
+    append: yes
+    password: "{{ jenkins_auth_realm.hashed_password }}"
+    update_password: always
+
+- name: template config.xml
+  template:
+    src: jenkins.config.main.xml
+    dest: "{{ jenkins_home }}/config.xml"
+    owner: "{{ jenkins_user }}"
+    group: "{{ jenkins_group }}"
+
+# Unconditionally restart Jenkins, this has two side-effects:
+# 1. Jenkins uses new auth realm
+# 2. We guarantee that jenkins is started (this is not certain
+#    as Jenkins is started by handlers from jenkins_master,
+#    these handlers are launched after this role).
+
+- name: restart Jenkins
+  service: name=jenkins state=restarted
+
+# Upload Jenkins credentials
+
+- name: create credentials dir
+  file: name={{ jenkins_credentials_root }} state=directory
+
+- name: upload groovy script
+  template:
+    src: addCredentials.groovy
+    dest: "{{ jenkins_credentials_script }}"
+    mode: "600"
+
+- name: upload credentials file
+  template:
+    src: credentials_file.json.j2
+    dest: "{{ jenkins_credentials_file_dest }}"
+    mode: "600"
+    owner: "{{ jenkins_user }}"
+
+- name: add credentials
+  include: execute_jenkins_cli.yaml
+  vars:
+    jenkins_command_string: "groovy {{ jenkins_credentials_script }}"
+
+- name: clean up
+  file: name={{ jenkins_credentials_root }} state=absent
+
+# Upload seed job
+
+- name: upload job file
+  template: src=seed_job_template.xml dest=/tmp/{{  jenkins_seed_job.name }} mode="600"
+
+- name: check if job is present
+  include: execute_jenkins_cli.yaml
+  vars:
+    jenkins_command_string: "get-job {{  jenkins_seed_job.name }}"
+    jenkins_ignore_cli_errors: yes
+
+- set_fact:
+    get_job_output: "{{ jenkins_command_output }}"
+
+# Upload seed job to Jenkins
+
+- name: Create seed job if absent
+  include: execute_jenkins_cli.yaml
+  vars:
+    jenkins_command_string: "create-job {{  jenkins_seed_job.name }}"
+    jenkins_command_prefix: "cat /tmp/{{  jenkins_seed_job.name }} | "
+  when: get_job_output.rc != 0
+
+- name: update seed job
+  include: execute_jenkins_cli.yaml
+  vars:
+    jenkins_command_string: "update-job {{  jenkins_seed_job.name }}"
+    jenkins_command_prefix: "cat /tmp/{{  jenkins_seed_job.name }} | "
+  when: get_job_output.rc == 0
+
+# Build the seed job
+
+- name: Build the seed job
+  include: execute_jenkins_cli.yaml
+  vars:
+    jenkins_command_string: "build {{  jenkins_seed_job.name }} -s"

playbooks/roles/jenkins_analytics/templates/addCredentials.groovy

+/**
+ * This script can be run via the Jenkins CLI as follows:
+ *
+ * java -jar /var/jenkins/war/WEB-INF/jenkins-cli.jar -s http://localhost:8080 groovy addCredentials.groovy
+ *
+ * For a given json file, this script will create a set of credentials.
+ * The script can be run safely multiple times and it will update each changed credential
+ * (deleting credentials is not currently supported).
+ *
+ * This is useful in conjunction with the job-dsl to bootstrap a barebone Jenkins instance.
+ *
+ * This script will currently fail if the plugins it requires have not been installed:
+ *
+ * credentials-plugin
+ * credentials-ssh-plugin
+ */
+
+
+import com.cloudbees.plugins.credentials.Credentials
+import com.cloudbees.plugins.credentials.CredentialsScope
+import com.cloudbees.plugins.credentials.common.IdCredentials
+import com.cloudbees.plugins.credentials.domains.Domain
+import hudson.model.*
+import com.cloudbees.plugins.credentials.SystemCredentialsProvider
+import com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl
+import com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey
+import groovy.json.JsonSlurper;
+
+boolean addUsernamePassword(scope, id, username, password, description) {
+    provider = SystemCredentialsProvider.getInstance()
+    provider.getCredentials().add(new UsernamePasswordCredentialsImpl(scope, id, description, username, password))
+    provider.save()
+    return true
+}
+
+boolean addSSHUserPrivateKey(scope, id, username, privateKey, passphrase, description) {
+    provider = SystemCredentialsProvider.getInstance()
+    source = new BasicSSHUserPrivateKey.DirectEntryPrivateKeySource(privateKey)
+    provider.getCredentials().add(new BasicSSHUserPrivateKey(scope, id, username, source, passphrase, description))
+    provider.save()
+    return true
+}
+
+def jsonFile = new File("{{ jenkins_credentials_file_dest }}");
+
+if (!jsonFile.exists()){
+    throw RuntimeException("Credentials file does not exist on remote host");
+}
+
+def jsonSlurper = new JsonSlurper()
+def credentialList = jsonSlurper.parse(new FileReader(jsonFile))
+
+credentialList.each { credential ->
+
+    if (credential.scope != "GLOBAL"){
+        throw new RuntimeException("Sorry for now only global scope is supported");
+    }
+
+    scope = CredentialsScope.valueOf(credential.scope)
+
+    def provider = SystemCredentialsProvider.getInstance();
+
+    def toRemove = [];
+
+    for (Credentials current_credentials: provider.getCredentials()){
+        if (current_credentials instanceof IdCredentials){
+            if (current_credentials.getId() == credential.id){
+                toRemove.add(current_credentials);
+            }
+        }
+    }
+
+    toRemove.each {curr ->provider.getCredentials().remove(curr)};
+
+    if (credential.type == "username-password") {
+        addUsernamePassword(scope, credential.id, credential.username, credential.password, credential.description)
+    }
+
+    if (credential.type == "ssh-private-key") {
+
+        if (credential.passphrase != null && credential.passphrase.trim().length() == 0){
+            credential.passphrase = null;
+        }
+
+        addSSHUserPrivateKey(scope, credential.id, credential.username, credential.privatekey, credential.passphrase, credential.description)
+    }
+}

playbooks/roles/jenkins_analytics/templates/credentials_file.json.j2

+{{ JENKINS_ANALYTICS_CREDENTIALS|to_json }}
\ No newline at end of file

playbooks/roles/jenkins_analytics/templates/jenkins-pass-file.j2

+{{ jenkins_auth_realm.plain_password }}
\ No newline at end of file

playbooks/roles/jenkins_analytics/templates/jenkins.config.main.xml

+<?xml version='1.0' encoding='UTF-8'?>
+<hudson>
+  <disabledAdministrativeMonitors/>
+  <version>1.638</version>
+  <numExecutors>{{ JENKINS_ANALYTICS_CONCURRENT_JOBS_COUNT }}</numExecutors>
+  <mode>NORMAL</mode>
+  <useSecurity>true</useSecurity>
+  {% if jenkins_auth_realm.name == "unix" %}
+   <authorizationStrategy class="hudson.security.GlobalMatrixAuthorizationStrategy">
+    <permission>com.cloudbees.plugins.credentials.CredentialsProvider.Create:jenkins</permission>
+    <permission>com.cloudbees.plugins.credentials.CredentialsProvider.Delete:jenkins</permission>
+    <permission>com.cloudbees.plugins.credentials.CredentialsProvider.ManageDomains:jenkins</permission>
+    <permission>com.cloudbees.plugins.credentials.CredentialsProvider.Update:jenkins</permission>
+    <permission>com.cloudbees.plugins.credentials.CredentialsProvider.View:jenkins</permission>
+    <permission>hudson.model.Computer.Build:jenkins</permission>
+    <permission>hudson.model.Computer.Configure:jenkins</permission>
+    <permission>hudson.model.Computer.Connect:jenkins</permission>
+    <permission>hudson.model.Computer.Create:jenkins</permission>
+    <permission>hudson.model.Computer.Delete:jenkins</permission>
+    <permission>hudson.model.Computer.Disconnect:jenkins</permission>
+    <permission>hudson.model.Hudson.Administer:jenkins</permission>
+    <permission>hudson.model.Hudson.ConfigureUpdateCenter:jenkins</permission>
+    <permission>hudson.model.Hudson.Read:jenkins</permission>
+    <permission>hudson.model.Hudson.RunScripts:jenkins</permission>
+    <permission>hudson.model.Hudson.UploadPlugins:jenkins</permission>
+    <permission>hudson.model.Item.Build:jenkins</permission>
+    <permission>hudson.model.Item.Cancel:jenkins</permission>
+    <permission>hudson.model.Item.Configure:jenkins</permission>
+    <permission>hudson.model.Item.Create:jenkins</permission>
+    <permission>hudson.model.Item.Delete:jenkins</permission>
+    <permission>hudson.model.Item.Discover:anonymous</permission>
+    <permission>hudson.model.Item.Discover:jenkins</permission>
+    <permission>hudson.model.Item.Move:jenkins</permission>
+    <permission>hudson.model.Item.Read:anonymous</permission>
+    <permission>hudson.model.Item.Read:jenkins</permission>
+    <permission>hudson.model.Item.Workspace:jenkins</permission>
+    <permission>hudson.model.Run.Delete:jenkins</permission>
+    <permission>hudson.model.Run.Update:jenkins</permission>
+    <permission>hudson.model.View.Configure:jenkins</permission>
+    <permission>hudson.model.View.Create:jenkins</permission>
+    <permission>hudson.model.View.Delete:jenkins</permission>
+    <permission>hudson.model.View.Read:jenkins</permission>
+    <permission>hudson.scm.SCM.Tag:jenkins</permission>
+  </authorizationStrategy>
+  <securityRealm class="hudson.security.PAMSecurityRealm" plugin="pam-auth@1.2">
+    <serviceName>{{ jenkins_auth_realm.service }}</serviceName>
+  </securityRealm>
+  {% endif %}
+  <disableRememberMe>false</disableRememberMe>
+  <projectNamingStrategy class="jenkins.model.ProjectNamingStrategy$DefaultProjectNamingStrategy"/>
+  <workspaceDir>${JENKINS_HOME}/workspace/${ITEM_FULLNAME}</workspaceDir>
+  <buildsDir>${ITEM_ROOTDIR}/builds</buildsDir>
+  <markupFormatter class="hudson.markup.EscapedMarkupFormatter"/>
+  <jdks/>
+  <viewsTabBar class="hudson.views.DefaultViewsTabBar"/>
+  <myViewsTabBar class="hudson.views.DefaultMyViewsTabBar"/>
+  <clouds/>
+  <quietPeriod>5</quietPeriod>
+  <scmCheckoutRetryCount>0</scmCheckoutRetryCount>
+  <views>
+    <hudson.model.AllView>
+      <owner class="hudson" reference="../../.."/>
+      <name>All</name>
+      <filterExecutors>false</filterExecutors>
+      <filterQueue>false</filterQueue>
+      <properties class="hudson.model.View$PropertyList"/>
+    </hudson.model.AllView>
+  </views>
+  <primaryView>All</primaryView>
+  <slaveAgentPort>0</slaveAgentPort>
+  <label>312312321</label>
+  <nodeProperties/>
+  <globalNodeProperties/>
+</hudson>

playbooks/roles/jenkins_analytics/templates/seed_job_template.xml

+<?xml version='1.0' encoding='UTF-8'?>
+<project>
+  <actions/>
+  <description>
+    Seed job autogenerated by ansible, it will be overridden.
+  </description>
+  <keepDependencies>false</keepDependencies>
+  <properties>
+    <jenkins.advancedqueue.AdvancedQueueSorterJobProperty plugin="PrioritySorter@2.9">
+      <useJobPriority>false</useJobPriority>
+      <priority>-1</priority>
+    </jenkins.advancedqueue.AdvancedQueueSorterJobProperty>
+  </properties>
+  <scm class="hudson.plugins.git.GitSCM" plugin="git@2.4.0">
+    <configVersion>2</configVersion>
+    <userRemoteConfigs>
+      <hudson.plugins.git.UserRemoteConfig>
+        <url>{{ jenkins_seed_job.scm.url}}</url>
+        {% if jenkins_seed_job.scm.credential_id is defined and jenkins_seed_job.scm.credential_id %}
+        <credentialsId>{{ jenkins_seed_job.scm.credential_id }}</credentialsId>
+        {% endif %}
+      </hudson.plugins.git.UserRemoteConfig>
+    </userRemoteConfigs>
+    <branches>
+      <hudson.plugins.git.BranchSpec>
+        <name>master</name>
+      </hudson.plugins.git.BranchSpec>
+    </branches>
+    <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>
+    <browser class="hudson.plugins.git.browser.AssemblaWeb">
+      <url></url>
+    </browser>
+    <submoduleCfg class="list"/>
+    <extensions/>
+  </scm>
+  <canRoam>true</canRoam>
+  <disabled>false</disabled>
+  <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
+  <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
+  <triggers>
+    <hudson.triggers.TimerTrigger>
+      <spec>{{ jenkins_seed_job.time_trigger }}</spec>
+    </hudson.triggers.TimerTrigger>
+  </triggers>
+  <concurrentBuild>false</concurrentBuild>
+  <builders>
+    <hudson.plugins.gradle.Gradle plugin="gradle@1.24">
+      <description></description>
+      <switches></switches>
+      <tasks>clean test</tasks>
+      <rootBuildScriptDir></rootBuildScriptDir>
+      <buildFile></buildFile>
+      <gradleName>(x)</gradleName>
+      <useWrapper>true</useWrapper>
+      <makeExecutable>false</makeExecutable>
+      <fromRootBuildScriptDir>true</fromRootBuildScriptDir>
+      <useWorkspaceAsHome>false</useWorkspaceAsHome>
+    </hudson.plugins.gradle.Gradle>
+    <javaposse.jobdsl.plugin.ExecuteDslScripts plugin="job-dsl@1.43">
+      <targets>{{ jenkins_seed_job.scm.target_jobs }}</targets>
+      <usingScriptText>false</usingScriptText>
+      <ignoreExisting>false</ignoreExisting>
+      <removedJobAction>{{ jenkins_seed_job.removed_job_action }}</removedJobAction>
+      <removedViewAction>{{ jenkins_seed_job.removed_view_action }}</removedViewAction>
+      <lookupStrategy>JENKINS_ROOT</lookupStrategy>
+      <additionalClasspath>{{ jenkins_seed_job.scm.additional_classpath }}</additionalClasspath>
+    </javaposse.jobdsl.plugin.ExecuteDslScripts>
+  </builders>
+  <publishers/>
+  <buildWrappers/>
+</project>

playbooks/roles/jenkins_master/defaults/main.yml

@@ -19,7 +19,9 @@ jenkins_plugins:
     - { name: "build-name-setter", version: "1.3" }
     - { name: "build-pipeline-plugin", version: "1.4" }
     - { name: "build-timeout", version: "1.14.1" }
+    - { name: "build-user-vars-plugin", version: "1.5" }
     - { name: "buildgraph-view", version: "1.1.1" }
+    - { name: "cloudbees-folder", version: "5.2.1" }
     - { name: "cobertura", version: "1.9.6" }
     - { name: "copyartifact", version: "1.32.1" }
     - { name: "copy-to-slave", version: "1.4.3" }
@@ -34,15 +36,19 @@ jenkins_plugins:
     - { name: "github", version: "1.14.0" }
     - { name: "github-api", version: "1.69" }
     - { name: "github-oauth", version: "0.20" }
-    - { name: "github-sqs-plugin", version: "1.6" }
+    - { name: "github-sqs-plugin", version: "1.5" }
+    - { name: "gradle", version: "1.24" }
+    - { name: "grails", version: "1.7" }
     - { name: "groovy-postbuild", version: "2.2" }
     - { name: "htmlpublisher", version: "1.3" }
     - { name: "javadoc", version: "1.3" }
     - { name: "jobConfigHistory", version: "2.10" }
+    - { name: "job-dsl", version: "1.43" }
     - { name: "junit", version: "1.3" }
     - { name: "ldap", version: "1.11" }
     - { name: "mailer", version: "1.16" }
     - { name: "mapdb-api", version: "1.0.6.0" }
+    - { name: "mask-passwords", version: "2.8" }
     - { name: "matrix-auth", version: "1.2" }
     - { name: "matrix-project", version: "1.4" }
     - { name: "monitoring", version: "1.56.0" }

playbooks/roles/jenkins_master/tasks/main.yml

@@ -99,7 +99,7 @@
     path: "{{ jenkins_home }}/plugins/{{ item.item.name }}.hpi"
     owner: "{{ jenkins_user }}"
     group: "{{ jenkins_group }}"
-    mode: 644
+    mode: "644"
   with_items: jenkins_plugin_downloads.results
   when: item.changed
   notify:
@@ -131,7 +131,7 @@
 
 - name: set custom plugin permissions
   file: path={{ jenkins_home }}/plugins/{{ item.item.package }}
-        owner={{ jenkins_user }} group={{ jenkins_group }} mode=700
+        owner={{ jenkins_user }} group={{ jenkins_group }} mode="700"
   with_items: jenkins_custom_plugins_checkout.results
   when: item.changed
 

vagrant/base/analytics_jenkins/Vagrantfile

+# -*- mode: ruby -*-
+
+VAGRANTFILE_API_VERSION = '2'
+MEMORY = 2048
+PRIVATE_IP = ENV['VAGRANT_PRIVATE_IP'] || '192.168.33.15'
+
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+  config.vm.box = 'ubuntu/trusty64'
+
+  config.vm.network 'private_network', ip: PRIVATE_IP
+  config.vm.synced_folder  '.', '/vagrant', disabled: true
+
+  config.vm.provider 'virtualbox' do |vb|
+    vb.memory = MEMORY
+  end
+
+  unless ENV['VAGRANT_NO_PORTS']
+    config.vm.network :forwarded_port, guest: 8080, host: 8080  # Jenkins
+  end
+
+  unless ENV['VAGRANT_JENKINS_LOCAL_VARS_FILE']
+    raise 'Please set VAGRANT_JENKINS_LOCAL_VARS_FILE environment variable. '\
+          'That variable should point to a file containing variable '\
+          'overrides for analytics_jenkins role. For required overrides '\
+          'see README.md in the analytics_jenkins role folder.'
+  end
+
+  config.vm.provision :ansible do |ansible|
+    ansible.playbook = '../../../playbooks/analytics-jenkins.yml'
+    ansible.verbose = 'vvvv'
+
+    ansible.extra_vars = ENV['VAGRANT_JENKINS_LOCAL_VARS_FILE']
+  end
+
+end