Add an introduction to docs

This is the first in a series of commits that aims to update this documentation in an effort to make it more useful to other developers and the Open edX community at large. The introduction provides some background information on the discovery service that was missing before. It also explains key concepts necessary to understand the service. This change also brings these docs into line with those written for other services by switching to using the edX theme.

LEARNER-1399

.travis/run_tests.sh

@@ -18,9 +18,9 @@ echo '{ "allow_root": true }' > /root/.bowerrc
 
 make requirements
 make requirements.js
+
 # Ensure documentation can be compiled
-cd docs && make html
-cd ..
+make docs
 
 # Check if translation files are up-to-date
 make validate_translations

Makefile

@@ -3,7 +3,7 @@ NODE_BIN=./node_modules/.bin
 
 .PHONY: accept clean clean_static compile_translations detect_changed_source_translations dummy_translations extract_translations \
 	fake_translations help html_coverage migrate open-devstack production-requirements pull_translations quality requirements.js \
-	requirements start-devstack static stop-devstack test validate validate_translations
+	requirements start-devstack static stop-devstack test validate validate_translations docs
 
 include .travis/docker.mk
 
@@ -87,3 +87,6 @@ detect_changed_source_translations: ## Check if translation files are up-to-date
 	cd course_discovery && i18n_tool changed
 
 validate_translations: fake_translations detect_changed_source_translations ## Install fake translations and check if translation files are up-to-date
+
+docs:
+	cd docs && make html

README.rst

@@ -6,13 +6,7 @@ Course Discovery Service  |Travis|_ |Codecov|_
 .. |Codecov| image:: http://codecov.io/github/edx/course-discovery/coverage.svg?branch=master
 .. _Codecov: http://codecov.io/github/edx/course-discovery?branch=master
 
-The Course Discovery service is a data aggregator with several purposes:
-
-1. Allow external parties to access data about Courses in an OpenEdX installation from a single central location (no matter which the system of record for that data is), in a way that can be secured and will not have operational impact on the running OpenEdX installation.
-2. Allow other services inside an OpenEdX installation to consume a consolidated source of course content for presentation to users (for instance, for the purposes of course marketing and discoverability).
-3. Provide a facility for naming dynamic groups of courses (catalogs) for use by other services in the system (such as coupon fulfillment and external course discovery filtering).
-
-To aid in these goals, the Course Discovery Service collects data from several systems internal to the Open edX installation. These include Otto (the E-Commerce Service), Studio, and in the edX.org implementation, the Drupal marketing site. The data loading framework is designed to make adding additional systems easy.
+Service providing access to consolidated course and program metadata.
 
 Documentation
 -------------

docs/conf.py

@@ -10,7 +10,8 @@
 # serve to show the default.
 import datetime
 import os
-import sys
+
+import edx_theme
 
 # on_rtd is whether we are on readthedocs.org
 on_rtd = os.environ.get('READTHEDOCS', None) == 'True'
@@ -44,8 +45,8 @@ source_suffix = '.rst'
 master_doc = 'index'
 
 # General information about the project.
-project = u'edX Discovery Service'
-copyright = u'2015-{year}, edX Inc.'.format(year=datetime.date.today().year)
+project = 'edX Discovery Service'
+copyright = '2015-{year}, edX Inc.'.format(year=datetime.date.today().year)
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
@@ -95,7 +96,7 @@ pygments_style = 'sphinx'
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
-# html_theme = 'sphinx_rtd_theme'
+html_theme = 'edx_theme'
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
@@ -103,7 +104,7 @@ pygments_style = 'sphinx'
 # html_theme_options = {}
 
 # Add any paths that contain custom themes here, relative to this directory.
-# html_theme_path = []
+html_theme_path = [edx_theme.get_html_theme_path()]
 
 # The name for this set of Sphinx documents.  If None, it defaults to
 # "<project> v<release> documentation".

docs/index.rst

-.. course_discovery documentation master file, created by
-   sphinx-quickstart on Sun Feb 17 11:46:20 2013.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
+edX Discovery Service
+=====================
 
-Course Discovery Service
-========================
-A service for serving course discovery and marketing information to partners, mobile, and edX.org
+**Discovery** is a service that provides access to consolidated course and program metadata. It does this primarily through a REST API that supports courses, course runs, programs, catalogs, and search.
+
+This guide begins with some background information on the service, then focuses on what you need to know to run and develop for the service.
 
 .. toctree::
    :maxdepth: 2
 
+   introduction
    getting_started
    api
    catalogs
@@ -19,4 +18,4 @@ A service for serving course discovery and marketing information to partners, mo
    features
    internationalization
    oidc
-   edx_extensions
\ No newline at end of file
+   edx_extensions

docs/introduction.rst

+Introduction
+============
+
+The distribution of edX's data has grown over time. Any given feature on edx.org may need information from Studio, the LMS, the Ecommerce service, and/or the Drupal marketing site. Discovery is a data aggregator whose job is to collect, consolidate, and provide access to information from these services.
+
+Discovery allows services internal to an Open edX installation to consume a consolidated source of metadata for presentation to users. For example, search on edx.org is provided by Discovery. Discovery also allows external parties to access data about content in an Open edX installation from a single, central location in a secure way that doesn't impact performance of said installation.
+
+Courses and Course Runs
+-----------------------
+
+One of Discovery's distinguishing features is the way it formalizes the relationship between courses and course runs. For example, ``course-v1:foo+bar+fall`` and ``course-v1:foo+bar+spring`` identify fall and spring runs of the same course, ``foo+bar``. You can think of courses as collections of course runs. Discovery infers this relationship when collecting data from other services. This hierarchy is the foundation for catalogs and programs, two additional structures provided by Discovery.
+
+Catalogs
+--------
+
+Catalogs are dynamic groups of courses. A catalog is defined with an Elasticsearch query. Catalogs are used to give external parties scoped views of edX content. They are also used to implement coupons on the Ecommerce service. For example, a coupon providing a 25% discount on courses from a specific organization would be tied to a catalog identifying those courses.
+
+Programs
+--------
+
+Programs are fixed collections of courses whose completion results in the awarding of a credential. Discovery only stores program metadata. For example, Discovery is responsible for keeping track of which courses belong to a program. Other program-related features such as calculating completion and awarding credentials are the responsibilities of separate systems.
+
+Data Loading
+------------
+
+Data about courses and course runs is collected from Studio, the LMS, the Ecommerce service, and, for edx.org, the Drupal marketing site. The data loading pipeline used to collect this data can be run with a management command called ``refresh_course_metadata``. edX runs this command several times a day using a Jenkins job. It can be manually run to populate a local environment with data. The data loading framework is designed to make adding additional systems easy.
+
+API
+---
+
+Access to information about courses, course runs, catalogs, programs, and more is provided by a REST API. For more about the API, use your browser to visit the ``/api-docs`` hosted by a running Discovery instance.
+
+Search
+------
+
+Discovery uses Elasticsearch to index data about courses, course runs, and programs. Indexing can be run at any time with a management command called ``update_index``. The Discovery API can be used to run search queries against the Elasticsearch index.