Adding in a new go-agent to use for marketing pipelines. Contains nec… (#4249)

* Adding in a new go-agent to use for marketing pipelines. Contains necessary modules to make requirements in edx-mktg

* Setting up the go-agent with a name and resources so it will autoregister with these.

* Leaving php and drush is goagent so we can fall back. Making docs more clear

* Build go-agent-marketing based on tagged image of go-agent

* Attempting to fix build

* Trying latest

* Adding the acquia github key directly into the goagent (#4264)

* Adding the acquia github key directly into the goagent

* Moving the acquia key to the marketing agent

docker/build/go-agent-marketing/Dockerfile

+FROM edxops/go-agent:latest
+
+# Install necessary modules for running make requirements in edx-mktg
+# Using rvm so we can control the ruby version installed. This also installs gem 2.6.12
+RUN bash -c 'apt-add-repository -y ppa:rael-gc/rvm && apt-get update && \
+    apt-get install -y rvm && source /usr/share/rvm/scripts/rvm && \
+    rvm install ruby-2.4.1 && gem install bundler -v 1.16.0'
+
+# Installs node 8.9.3 and npm 5.5.1 as of 12/13/17. Unlikely to change much since node 9 is a stable version for other OS
+RUN curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash - && \
+    apt-get update && apt-get install -y nodejs
+
+# Install php
+RUN apt-get update && apt-get install -y \
+    php5-common \
+    php5-cli
+
+# Install drush (drupal shell) for access to Drupal commands/Acquia
+RUN php -r "readfile('http://files.drush.org/drush.phar');" > drush && \
+    chmod +x drush && \
+    sudo mv drush /usr/local/bin
+
+# !!!!NOTICE!!!! ---- Runner of this pipeline take heed!! You must replace acquia_github_key.pem with the REAL key
+# material that can checkout private github repositories used as pipeline materials. The key material here is faked and
+# is only used to pass CI!
+# setup the acquia github identity
+ADD docker/build/go-agent-marketing/files/acquia_github_key.pem /var/go/.ssh/acquia_github_key
+RUN chmod 600 /var/go/.ssh/acquia_github_key && \
+    chown go:go /var/go/.ssh/acquia_github_key

docker/build/go-agent-marketing/README.rst

+Usage
+#####
+
+Start the container with this:
+
+``docker run -ti -e GO_SERVER=your.go.server.ip_or_host edx/go-agent-marketing``
+
+If you need to start a few GoCD agents together, you can of course use the
+shell to do that. Start a few agents in the background, like this:
+
+``for each in 1 2 3; do docker run -d --link angry_feynman:go-server edx/go-agent-marketing; done``
+
+Getting into the container
+##########################
+
+Sometimes, you need a shell inside the container (to create test repositories,
+etc). docker provides an easy way to do that:
+
+``docker exec -i -t CONTAINER-ID /bin/bash``
+
+To check the agent logs, you can do this:
+
+``docker exec -i -t CONTAINER-ID tail -f /var/log/go-agent/go-agent.log``
+
+Agent Configuration
+###################
+
+The go-agent expects it's configuration to be found at
+``/var/lib/go-agent/config/``. Sharing the configuration between containers is
+done by mounting a volume at this location that contains any configuration
+files necessary.
+
+**Example docker run command:**
+``docker run -ti -v /tmp/go-agent/conf:/var/lib/go-agent/config -e GO_SERVER=gocd.sandbox.edx.org 718d75c467c0 bash``
+
+`How to setup auto registration for remote agents`_
+
+Building and Uploading the container to ECS
+###########################################
+
+-  Build and tag the go-agent docker image
+
+   -  Follow the README in the go-agent directory to build and tag for go-agent-marketing.
+
+-  Copy the Acquia GitHub private key to this path:
+
+   -  ``docker/build/go-agent/files/acquia_github_key.pem``
+   -  A dummy key is in the repo file.
+   -  The actual private key is kept in LastPass - see DevOps for access.
+   -  WARNING: Do *NOT* commit/push the real private key to the public
+      configuration repo!
+
+-  Create image
+
+   -  This must be run from the root of the configuration repository
+   -  ``docker build -f docker/build/go-agent-marketing/Dockerfile .``
+   -  or
+   -  ``make docker.test.go-agent-marketing``
+
+-  Log docker in to AWS
+
+   -  Assume the role of the account you wish to log in to
+
+      -  ``source assume_role.sh <account name>``
+
+   -  ``sh -c `aws ecr get-login --region us-east-1```
+
+      -  You might need to remove the ``-e`` option returned by that command in
+         order to successfully login.
+
+-  Tag image
+
+   -  ``docker tag <image_id> ############.dkr.ecr.us-east-1.amazonaws.com/prod-tools-goagent-marketing:latest``
+   -  ``docker tag <image_id> ############.dkr.ecr.us-east-1.amazonaws.com/prod-tools-goagent-marketing:<version_number>``
+
+-  upload:
+
+   -  ``docker push ############.dkr.ecr.us-east-1.amazonaws.com/edx/release-pipeline/prod-tools-goagent-marketing:latest``
+   -  ``docker push ############.dkr.ecr.us-east-1.amazonaws.com/edx/release-pipeline/prod-tools-goagent-marketing:<version_number>``
+
+.. _How to setup auto registration for remote agents: https://docs.go.cd/current/advanced_usage/agent_auto_register.html

docker/build/go-agent-marketing/config/autoregister.properties

+agent.auto.register.key=dev-only-override-this-key
+agent.auto.register.resources=edxMarketing
+agent.auto.register.hostname=edxMarketingAgent

docker/build/go-agent-marketing/files/acquia_github_key.pem

+-----BEGIN RSA PRIVATE KEY-----
+This file is junk, replace with the real key when
+building the container.
+-----END RSA PRIVATE KEY-----

docker/build/go-agent/README.rst

@@ -3,12 +3,12 @@ Usage
 
 Start the container with this:
 
-``docker run -ti -e GO_SERVER=your.go.server.ip_or_host gocd/gocd-agent``
+``docker run -ti -e GO_SERVER=your.go.server.ip_or_host edx/go-agent``
 
 If you need to start a few GoCD agents together, you can of course use the
 shell to do that. Start a few agents in the background, like this:
 
-``for each in 1 2 3; do docker run -d --link angry_feynman:go-server gocd/gocd-agent; done``
+``for each in 1 2 3; do docker run -d --link angry_feynman:go-server edx/go-agent; done``
 
 Getting into the container
 ##########################
@@ -53,6 +53,11 @@ Building and Uploading the container to ECS
    -  or
    -  ``make docker.test.go-agent``
 
+-  Tag image for the go-agent-marketing Dockerfile
+
+   - *REQUIRED for go-agent-marketing Dockerfile*
+   - ``docker tag <image_id> edxops/go-agent``
+
 -  Log docker in to AWS
 
    -  Assume the role of the account you wish to log in to

util/README.rst

@@ -16,11 +16,19 @@ To modify configuration file:
 
     Replace
 
-    ``$(docker_test)%: .build/%/Dockerfile.test     docker build -t $*:test -f $< .``
+    ::
+
+        $(docker_test)%: .build/%/Dockerfile.test
+            docker build -t $*:test -f $< .
 
     with
 
-    ``$(docker_test)%: .build/%/Dockerfile.test     date     docker build -t $*:test -f $< .     date``
+    ::
+
+        $(docker_test)%: .build/%/Dockerfile.test
+            date
+            docker build -t $*:test -f $< .
+            date
 
 3.  Replace the command that runs the dependency analyzer with a line to build
     your Dockerfiles.
@@ -58,7 +66,7 @@ To modify configuration file:
 8.  Round build time to a whole number, and add it to the
     configuration/util/parsefiles\_config.yml file.
 
-9.  Undo steps 1a, 1b, 1c to revert back to the original state of the docker.mk
+9.  Undo steps 2, 3, 4 to revert back to the original state of the docker.mk
     file.
 
 10. Commit and push to your branch. Your Dockerfile should now be built as a

util/parsefiles_config.yml

@@ -8,6 +8,7 @@ docker_plays_paths:
 weights:
   - discovery: 6
   - go-agent: 3
+  - go-agent-marketing: 4
   - xqwatcher: 3
   - analytics_api: 1
   - edxapp: 28