Skip to content

C
configuration
Commit a68f2658 by John Jarvis
Options

removing README.md content in favor of the wiki

parent 8adcd235
Showing with 1 additions and 377 deletions
README.md
......@@ -34,381 +34,5 @@ This project is a re-write of the current edX provisioning and
configuration tools, we will be migrating features to this project
over time, so expect frequent changes.
## AWS
### Quick start - Building the stack on a single server
To deploy the entire edX platform on a single ec2 instance
run the following commands:
```
git clone git@github.com:edx/configuration
mkvirtualenv ansible
cd configuration
pip install -r ansible-requirements.txt
cd playbooks
(adjust the settings in edx_sandbox.yml)
`ansible-playbook -vvv --user=ubuntu edx_sandbox.yml -i inventory.ini`
```
This will install the following services on a single instance
* edX lms (django/python) for courseware
* edX studio (django/python) for course authoring
* mysql (running locally)
* mongo (running locally)
* memcache (running locally)
Note: In order for mail to work properly you will need to add AWS credentials for an account that
has SES permissions, see `secure\_example/vars/edxapp\_sandbox.yml`
### Building the stack with CloudFormation
The first step is to provision the CloudFormation stack. There are
several options for doing this.
* The [AWS console](https://console.aws.amazon.com/cloudformation/home)
* The AWS [CloudFormation CLI](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-installing-cli.html)
* Via Ansible
If you don't have experience with CloudFormation, the web console is a
good place to start because it will use a form wizard to gather
configuration parameters, it will give you continuous feedback during
the process of building the stack and useful error messages when
problems occur.
Before you create the stack you will need to create a key-pair that can
be used to connect to the stack once it's instantiated. To do this
go to the 'EC2' section and create a new key-pair under the 'Key Pairs'
section. Note the name of this key and update
`cloudformation_templates/edx-reference-architecture.json` file. Under the
'KeyName' section change the value of 'Default' to the name of the key-pair
you just created.
Details on how to build the stack using Ansible are available below.
#### Building with the AWS Console
From the AWS main page that lists all the services you can use. Click on the
CloudFormation link. This will take you to a list of cloud stacks you currently
have active. Here click the 'Create Stack' button. In the wizard you can give a
name for your stack and pass in a template which defines the edX stack. Use the
`edx-reference-architecture.json` template in the `cloudformation_templates` directory.
#### Building with the CloudFormation CLI
To build from the CloudFormation CLI you will have to first upload the configuration
file to an S3 Bucket. The easiest way to do this is to use `s3cmd`.
```
s3cmd put /path/to/edx-reference-architecture.json s3://<bucket_name>
aws cloudformation create-stack --stack-name <stack_name> --template-url https://s3.amazonaws.com/<bucket_name>/edx-reference-architecture.json --capabilities CAPABILITY_IAM
```
### Post Bringup Manual Commands
Unfortunately there is some infrastructure that we need that is currently not supported
by CloudFormation. So once your stack is created by CloudFormation you need to run
a few manual commands to fill in those gaps.
This requires that you've installed the command line utilities for [ElastiCache][cachecli]
and [EC2][ec2cli]. Note that we requrire at least version 1.8 of the ElastiCache CLI due
to some newer commands that we rely on.
[cachecli]: http://aws.amazon.com/developertools/2310261897259567
[ec2cli]: http://aws.amazon.com/developertools/351
At the end of the CloudFormation run you should check the "Outputs" tab in
Amazon UI and that will have the commands you need to run (see
screenshot).
![CloudFormation Output (Amazon console)](doc/cfn-output-example.png)
Run the commands shown here before moving onto the next step.
### Connecting to Hosts in the Stack
Because the reference architecture makes use of an Amazon VPC, you will not be able
to address the hosts in the private subnets directly. However, you can easily set
up a transparent "jumpbox" so that for all hosts in your vpc, connections are
tunneled.
Add something like the following to your `~/.ssh/config` file.
```
Host *.us-west-1.compute-internal
ProxyCommand ssh -W %h:%p vpc-us-west-1-jumpbox
ForwardAgent yes
Host vpc-us-west-1-jumpbox
HostName 54.236.202.101
ForwardAgent yes
```
This assumes that you only have one VPC in the ```us-west-1``` region
that you're trying to ssh into. Internal DNS names aren't qualified
any further than that, so to support multiple VPC's you'd have to get
creative with subnets, for example ip-10-1 and ip-10-2...
Test this by typing `ssh ip-10-0-10-1.us-west-1.compute.internal`,
(of course using a hostname exists in your environment.) If things
are configured correctly you will ssh to 10.0.10.1, jumping
transparently via your basion host.
Getting this working in important because we'll be using Ansible
with the SSH transport and it will rely on this configuration
being in place in order to configure your servers.
### Finding your hosts via boto
Boto is how fabric looks up metadata about your stack, most importantly
finding the names of your machines. It needs your access information.
This should be the contents of your ```~/.boto``` file. Make sure
to customize the region:
```ini
[Credentials]
aws_access_key_id = AAAAAAAAAAAAAAAAAAAA
aws_secret_access_key = BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
[Boto]
debug = 1
ec2_region_name = us-west-1
ec2_region_endpoint = ec2.us-west-1.amazonaws.com
```
### Tagging
Tagging is the bridge between the provisioning and configuration
phases. The servers provisioned in your VPC will be stock Ubuntu
12.0.4 LTS servers. The only difference between them will be the tags
that CloudFront has applied to them. These tags will be used by Ansible
to map playbooks to the correct servers. The application of the
appropriate playbook, will turn each stock host into an appropriately
configured service.
The *Group* tag is where the magic happens. Every AWS EC2 instance
will have a *Group* tag that corresponds to a group of machines that
need to be deployed/targeted to as a group of servers.
**Example:**
* `Group`: `edxapp_stage`
* `Group`: `edxapp_prod`
* `Group`: `edxapp_some_other_environment`
Additional tags can be added to AWS resources in the stack but they should not
be made necessary deployment or configuration.
## Ansible
Ansible is a configuration management tool that edX is evaluating to replace
the puppet environment that is currently being used for edX servers.
http://ansible.cc/docs
_Note: Because the directory structure changes in v1.2 we are using the dev
version instead of the official v1.1 release._
* __Hosts__ - The ec2.py inventory script generates an inventory file where
hosts are assigned to groups. Individual hosts can be targeted by the "Name"
tag or the instance ID. I don't think there will be a reason to set host
specific variables.
* __Groups__ - A Group name is an identifier that corresponds to a group of
roles plus an identifier for the environment. Example: *edxapp_stage*,
*edxapp_prod*, *xserver_stage*, etc. For the purpose of targeting servers
for deployment groups are created automatically by the `ec2.py` inventory
script since these group names will map to the _Group_ AWS tag.
* __Roles__ - A role will map to a single function/service that runs on
server.
## Organization
### Secure vs. Insecure data
As a general policy we want to protect the following data:
* Usernames
* Public keys (keys are OK to be public, but can be used to figure out usernames)
* Hostnames
* Passwords, API keys
The following yml files and examples serve as templates that should be overridden with your own
environment specific configuration:
* vars in `secure_example/vars`
* files in `secure_example/files`
Directory structure for the secure repository:
```
ansible
├── files
├── keys
└── vars
```
The same directory structure, required yml files and files are
in the secure_example dir:
```
secure_example/
├── files
├── keys
└── vars
```
The default `secure_dir` is set in `group_vars/all` and can be overridden by
adding another file in group_vars that corresponds to a deploy group name.
The directory structure should follow Ansible best practices.
http://ansible.cc/docs/bestpractices.html
* At the top level there are yml files for every group where a group name is an
identifier that corresponds to a set of roles plus an environment.
* The standard environments are _stage_ and _production_.
* Additional environments can be named as well, below an example is given
called _custom_.
### Variables
* The ansible.cfg that is checked into the playbook directory has hash merging
turned on, this allows us to to merge secure and custom data into the default
variable definitions for every role.
* For example, `vars/lms_vars.yml` (variables needed for the lms role) sets the
`env_config` which has keys that can be overridden by
`vars/secure/edxapp_stage_vars.yml` for setting passwords and hostnames.
* If needed, additional configuration can be layered, in the example
`vars/secure/custom_vars.yml` changes some paramters that are set in
`vars/secure/edxapp_stage_vars.yml`.
__TODO__: _The secure/ directories are checked into the public repo for now as an
example, these will need to be moved to a private repo or maintained outside of
github._
### Users and Groups
There are two classes of users, admins and environment users.
* The *admin_users* hash will be added to every server and will be put into a
group that has admin bits.
* The *env_users* hash are the class of users that can be optionally included
in one of the group-environment playbooks.
Example users are in the `vars/secure` directory:
* [*env_users* for staging environment](/vars/secure/edxapp_stage_users.yml)
* [*admin_users* will be realized on every server](/vars/secure/users.yml)
```
cloudformation_templates <-- official edX cloudformation templates
   └── examples <-- example templates
playbooks
└──
edxapp_prod.yml <-- example production environment playbook
edxapp_stage.yml <-- example stage environment playbook
edxapp_custom.yml <-- example custom environment playbook
├── files <-- edX cloudformation templates
│   └── examples <-- example cloudformation templates
├── group_vars <-- var files that correspond to ansible group names (mapped to AWS tags)
├── keys <-- public keys
├── roles <-- edX services
│   ├── common <-- tasks that are run for all roles
│   │   └── tasks
│   ├── lms
│   │   ├── tasks <-- tasks that are run to setup an LMS
│   │   ├── templates
│   │ └── vars <-- main.yml in this directory is auto-loaded when the role is included
│   │
│   └── nginx
│   ├── handlers
│   ├── tasks
│   ├── vars
│   └── templates
│   (etc)
└── vars <-- public variable definitions
└── secure_example <-- secure variables (example)
```
### Installation
```
mkvirtualenv ansible
pip install -r ansible-requirements.txt
util/sync_hooks.sh
```
### Launching example cloudformation stack - Working example
#### Provision the stack
**This assumes that you have workng ssh as described above**
```
cd playbooks
ansible-playbook -vvv cloudformation.yml -i inventory.ini -e 'region=<aws_region> key=<key_name> name=<stack_name> group=<group_name>'
```
* _aws_region_: example: `us-east-1`. Which AWS EC2 region to build stack in.
* _key_name_: example: `deploy`. SSH key name configured in AWS for the region
* _stack_name_: example: `EdxAppCustom`. Name of the stack, must not contain
underscores or cloudformation will complain. Must be an unused name or
otherwise the existing stack will update.
* _group_name_: example: `edxapp_stage`. The group name should correspond to
one of the yml files in the `playbooks/`. Used for grouping hosts.
While this is running you see the cloudformation events in the AWS console as
the stack is brought up. Loads the `playbooks/cloudformation.yml` template
which creates a single small EBS backed EC2 instance.
_Note: You should read the output from ansible and not necessarily trust the
'ok'; failures in cloudformation provisioning (for example, in creating the
security group), may not cause ansible-playbook to fail._
See files/examples for
adding other components to the stack.
##### If ansible-playbook gives you import errors
Ansible really wants to call /usr/bin/python and if you have good virtualenv
hygeine, this may lead to ansible being unable to import critical libraries
like cloudfront. If you run into this problem, try exporting PYTHONPATH inside
your virtualenv and see if it runs better that way. E.g.:
```
export PYTHONPATH=$VIRTUAL_ENV/lib/python2.7/site-packages/
ansible-playbook playbooks/cloudformation.yml -i playbooks/inventory.ini
```
If that works fine, then you can add an export of PYTHONPATH to
`$VIRTUAL_ENV/bin/postactivate` so that you no longer have to think about it.
### Configure the stack
* Creates admin and env users
* Creates base directories
* Creates the lms json configuration files
Assuming that the edxapp_stage.yml playbook targets hosts in your vpc
for which there are entiries in your `.ssh/config`, do the
following to run your playbook.
```
cd playbooks
ansible-playbook -v --user=ubuntu edxapp_stage.yml -i ./ec2.py -c ssh
```
*Note: this assumes the group used for the edx stack was "edxapp_stage"*
For more information including installation instruction please see the [Configuration Wiki](https://github.com/edx/configuration/wiki).
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment