Adding in a new go-agent to use for marketing pipelines. Contains necessary…

Adding in a new go-agent to use for marketing pipelines. Contains necessary modules to make requirements in edx-mktg

docker/build/go-agent-marketing/Dockerfile

+# !!!!NOTICE!!!! ---- Runner of this pipeline take heed!! You must replace the following URL with the REAL URL
+FROM ############.dkr.ecr.us-east-1.amazonaws.com/edx/release-pipeline/prod-tools-goagent:latest
+
+# Install necessary modules for running make requirements in edx-mktg
+# Using rvm so we can control the ruby version installed. This also installs gem 2.6.12
+RUN apt-add-repository -y ppa:rael-gc/rvm && apt-get update && \
+    apt-get install -y rvm && source /usr/share/rvm/scripts/rvm && \
+    rvm install ruby-2.4.1 && gem install bundler -v 1.16.0
+# Installs node 8.9.3 and npm 5.5.1 as of 12/13/17. Unlikely to change much since node 9 is a stable version for other OS
+RUN curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash - && \
+    apt-get update && apt-get install -y nodejs
+
+# Install php
+RUN apt-get update && apt-get install -y \
+    php5-common \
+    php5-cli
+
+# Install drush (drupal shell) for access to Drupal commands/Acquia
+RUN php -r "readfile('http://files.drush.org/drush.phar');" > drush && \
+    chmod +x drush && \
+    sudo mv drush /usr/local/bin

docker/build/go-agent-marketing/README.rst

+Usage
+#####
+
+Start the container with this:
+
+``docker run -ti -e GO_SERVER=your.go.server.ip_or_host gocd/gocd-agent``
+
+If you need to start a few GoCD agents together, you can of course use the
+shell to do that. Start a few agents in the background, like this:
+
+``for each in 1 2 3; do docker run -d --link angry_feynman:go-server gocd/gocd-agent; done``
+
+Getting into the container
+##########################
+
+Sometimes, you need a shell inside the container (to create test repositories,
+etc). docker provides an easy way to do that:
+
+``docker exec -i -t CONTAINER-ID /bin/bash``
+
+To check the agent logs, you can do this:
+
+``docker exec -i -t CONTAINER-ID tail -f /var/log/go-agent/go-agent.log``
+
+Agent Configuration
+###################
+
+The go-agent expects it's configuration to be found at
+``/var/lib/go-agent/config/``. Sharing the configuration between containers is
+done by mounting a volume at this location that contains any configuration
+files necessary.
+
+**Example docker run command:**
+``docker run -ti -v /tmp/go-agent/conf:/var/lib/go-agent/config -e GO_SERVER=gocd.sandbox.edx.org 718d75c467c0 bash``
+
+`How to setup auto registration for remote agents`_
+
+Building and Uploading the container to ECS
+###########################################
+
+-  Copy the go-agent GitHub private key to this path:
+
+   -  ``docker/build/go-agent-marketing/files/go_github_key.pem``
+   -  A dummy key is in the repo file.
+   -  The actual private key is kept in LastPass - see DevOps for access.
+   -  WARNING: Do *NOT* commit/push the real private key to the public
+      configuration repo!
+
+-  Insert the correct URL in the FROM statement in the Dockerfile:
+
+   -  ``docker/build/go-agent-marketing/Dockerfile``
+   -  Replace the #'s in:
+   -  ``############.dkr.ecr.us-east-1.amazonaws.com/prod-tools-goagent:latest``
+   -  WARNING: Do *NOT* commit/push the real URL to the public
+      configuration repo!
+
+-  Create image
+
+   -  This must be run from the root of the configuration repository
+   -  ``docker build -f docker/build/go-agent-marketing/Dockerfile .``
+   -  or
+   -  ``make docker.test.go-agent-marketing``
+
+-  Log docker in to AWS
+
+   -  Assume the role of the account you wish to log in to
+
+      -  ``source assume_role.sh <account name>``
+
+   -  ``sh -c `aws ecr get-login --region us-east-1```
+
+      -  You might need to remove the ``-e`` option returned by that command in
+         order to successfully login.
+
+-  Tag image
+
+   -  ``docker tag <image_id> ############.dkr.ecr.us-east-1.amazonaws.com/prod-tools-goagent:latest``
+   -  ``docker tag <image_id> ############.dkr.ecr.us-east-1.amazonaws.com/prod-tools-goagent:<version_number>``
+
+-  upload:
+
+   -  ``docker push ############.dkr.ecr.us-east-1.amazonaws.com/edx/release-pipeline/prod-tools-goagent:latest``
+   -  ``docker push ############.dkr.ecr.us-east-1.amazonaws.com/edx/release-pipeline/prod-tools-goagent:<version_number>``
+
+.. _How to setup auto registration for remote agents: https://docs.go.cd/current/advanced_usage/agent_auto_register.html

docker/build/go-agent-marketing/files/go_github_key.pem

+-----BEGIN RSA PRIVATE KEY-----
+This file is junk, replace with the real key when
+building the container.
+-----END RSA PRIVATE KEY-----

docker/build/go-agent/Dockerfile

@@ -31,19 +31,9 @@ RUN apt-get update && apt-get install -y -q \
     libmysqlclient-dev \
     python3-pip
 
-# Install php
-RUN apt-get update && apt-get install -y \
-    php5-common \
-    php5-cli
-
 # Install dependencies needed for Ansible 2.x
 RUN apt-get update && apt-get install -y libffi-dev libssl-dev
 
-# Install drush (drupal shell) for access to Drupal commands/Acquia
-RUN php -r "readfile('http://files.drush.org/drush.phar');" > drush && \
-    chmod +x drush && \
-    sudo mv drush /usr/local/bin
-
 # Install Docker - for Docker container building by a go-agent.
 COPY docker/build/go-agent/files/docker_install.sh /tmp/docker/
 RUN /bin/bash /tmp/docker/docker_install.sh

util/README.rst

@@ -15,12 +15,20 @@ To modify configuration file:
 2.  Modify docker\_test to include date commands.
 
     Replace
-
-    ``$(docker_test)%: .build/%/Dockerfile.test     docker build -t $*:test -f $< .``
+    
+    ::
+    
+        $(docker_test)%: .build/%/Dockerfile.test
+            docker build -t $*:test -f $< .
 
     with
-
-    ``$(docker_test)%: .build/%/Dockerfile.test     date     docker build -t $*:test -f $< .     date``
+    
+    ::
+    
+        $(docker_test)%: .build/%/Dockerfile.test    
+            date   
+            docker build -t $*:test -f $< .
+            date
 
 3.  Replace the command that runs the dependency analyzer with a line to build
     your Dockerfiles.
@@ -58,7 +66,7 @@ To modify configuration file:
 8.  Round build time to a whole number, and add it to the
     configuration/util/parsefiles\_config.yml file.
 
-9.  Undo steps 1a, 1b, 1c to revert back to the original state of the docker.mk
+9.  Undo steps 2, 3, 4 to revert back to the original state of the docker.mk
     file.
 
 10. Commit and push to your branch. Your Dockerfile should now be built as a

util/parsefiles_config.yml

@@ -8,6 +8,7 @@ docker_plays_paths:
 weights:
   - discovery: 6
   - go-agent: 3
+  - go-agent-marketing: 4
   - xqwatcher: 3
   - analytics_api: 1
   - edxapp: 28