Skip to content

C
course-discovery
Commit b0ed8340 by Renzo Lucioni
Options

Add quickstart and advanced sections to docs 

This change completes a streamlining of the project docs. They're now organized into three sections. The (previously added) introduction provides context to new developers, explaining why the service exists and what it does. The quickstart section covers information new developers need to know to run and develop for the service. The advanced section covers additional, important information that isn't strictly necessary to run the service.

LEARNER-1399
parent 2838f97a
Showing with 177 additions and 565 deletions
README.rst
......@@ -11,7 +11,7 @@ Service providing access to consolidated course and program metadata.
Documentation
-------------
`Documentation <https://edx-discovery.readthedocs.io/en/latest/>`_ is hosted on Read the Docs. The source is hosted in this repo's `docs <https://github.com/edx/course-discovery/tree/master/docs>`_ directory. To contribute, please open a PR against this repo.
`Documentation <https://edx-discovery.readthedocs.io/en/latest/>`_ is hosted on Read the Docs. The source is hosted in this repo's `docs <https://github.com/edx/course-discovery/tree/master/docs>`_ directory. The docs are automatically rebuilt and redeployed when commits are merged to master. To contribute, please open a PR against this repo.
License
-------
......
docs/advanced.rst 0 → 100644
Advanced Usage
==============
This section contains information about advanced usage and operation of the Discovery service.
Elasticsearch
-------------
Discovery uses Elasticsearch 1.5 to provide search functionality.
Index Aliasing
++++++++++++++
Discovery application code uses an `index alias`_ to refer to the search index indirectly. For example, the timestamped ``course_discovery_20160101113005`` index may be assigned and referred to by the alias ``catalog``. Using an alias prevents index maintenance (e.g., the indexing and index swapping performed by ``update_index``) from affecting service uptime.
.. _index alias: https://www.elastic.co/guide/en/elasticsearch/reference/1.5/indices-aliases.html
Boosting
++++++++
Discovery uses Elasticsearch's `function score`_ query to modify ("boost") the relevance score of documents retrieved by search queries. You can find the service's boosting config at ``course_discovery/apps/edx_haystack_extensions/elasticsearch_boost_config.py``, complete with comments explaining what each part does and how it's been tuned.
.. _function score: https://www.elastic.co/guide/en/elasticsearch/reference/1.5/query-dsl-function-score-query.html
Querying Elasticsearch
++++++++++++++++++++++
In addition to running search queries through the Discovery API, you can make HTTP requests directly to Elasticsearch. This is especially useful if you want to tune how relevance scores are computed. These examples show curl being used from a Discovery shell:
.. code-block:: bash
$ curl 'edx.devstack.elasticsearch:9200/_cat/indices?v'
$ curl 'edx.devstack.elasticsearch:9200/catalog/_search?pretty=true' -d '{"explain": true, "query": {YOUR QUERY HERE}}'
The `explain`_ parameter tells Elasticsearch to return a detailed breakdown of how relevance scores were calculated. You can get yourself a query to run by intercepting queries made by the application. Add logging to ``course_discovery/apps/edx_haystack_extensions/backends.py::SimpleQuerySearchBackendMixin::build_search_kwargs`` that prints the final value of ``search_kwargs``, then run a search query through the API.
.. _explain: https://www.elastic.co/guide/en/elasticsearch/reference/1.5/search-request-explain.html
Extensions
----------
edX manages two "extension" apps located at ``course_discovery/apps/edx_catalog_extensions`` and ``course_discovery/apps/edx_haystack_extensions`` as part of Discovery. These apps provide edX-specific customizations. They include data migrations, management commands, and search backends specific to edX. We'd like to move these apps to separate repos at some point in the future to avoid confusion. They live here for now until we can determine what other edX-specific components need to be extracted from the general project.
``edx_catalog_extensions`` is disabled by default. edX developers should add ``course_discovery.apps.edx_catalog_extensions`` to ``INSTALLED_APPS`` in a ``private.py`` settings file.
Catalogs
--------
Catalogs are dynamic groups of courses modeled as access-controlled Elasticsearch queries. You can find the ``Catalog`` model in ``course_discovery/apps/catalogs/models.py``.
Permissions
+++++++++++
A catalog's ``viewers`` property returns the users who are allowed to view the catalog and the courses within it. These per-object permissions are implemented using `django-guardian`_.
.. _django-guardian: https://github.com/django-guardian/django-guardian
Administration
++++++++++++++
You can administer catalogs through the LMS at ``/api-admin/catalogs``. You can also modify catalogs using Discovery's Django admin at ``/admin/catalogs/``. The admin interface provides a preview button you can use to view the list of courses contained in a catalog, as well as the standard ``django-guardian`` admin interface for managing user permissions.
Waffle
------
Discovery uses `django-waffle`_ to control the release of new features. This allows us to gradually increase traffic to new features and divert traffic quickly if problems are discovered. Please refer to Waffle's `documenation`_ for an overview of the models you may encounter throughout the codebase.
.. _django-waffle: https://github.com/jsocol/django-waffle
.. _documentation: https://waffle.readthedocs.io/en/latest/
Internationalization
--------------------
All user-facing strings should be marked for translation. edX runs this application in English, but our open source users may choose to use another language. Marking strings for translation ensures our users have this choice. Refer to edX's `i18n guidelines`_ for more details.
.. _i18n guidelines: http://edx.readthedocs.io/projects/edx-developer-guide/en/latest/conventions/internationalization/index.html
Updating Translated Strings
+++++++++++++++++++++++++++
Like most edX projects, Discovery uses `Transifex`_ to translate content. Most of the translation process is automated: Transifex pulls new strings from this repo on a daily basis, and translated strings are merged back into the repo every week.
If you change or introduce new strings, run ``make fake_translations`` to extract all strings marked for translation, generate fake translations in the Esperanto (eo) language directory, and compile the translations. You can trigger the display of the translations by setting your browser's language to Esperanto (eo), and navigating to a page on the site. Instead of plain English strings, you should see specially-accented English strings that look like this:
Thé Fütüré øf Ønlïné Édüçätïøn Ⱡσяєм ι# Før änýøné, änýwhéré, änýtïmé Ⱡσяєм #
OpenID Connect
--------------
`OpenID Connect`_ (OIDC) is a simple identity layer on top of the OAuth 2.0 protocol which allows clients to verify the identity of and obtain basic profile information about an end-user based on authentication performed by an authorization server. Discovery uses the edX's OIDC provider for login at ``/login``. The LMS currently serves as the authentication provider.
.. _OpenID Connect: https://openid.net/specs/openid-connect-core-1_0.html
If you're using `devstack`_, OIDC should be configured for you. If you need to configure OIDC manually, you need to register a new client with the authentication provider (the LMS) and update Discovery's Django settings with the newly created client credentials.
.. _devstack: https://github.com/edx/devstack
You can create a new OAuth 2.0 client on the LMS at ``/admin/oauth2/client/``:
1. Click the ``Add client`` button.
2. Leave the user field blank.
3. Specify the name of this service, ``discovery``, as the client name.
4. Set the ``URL`` to the root path of this service: ``http://localhost:18381``.
5. Set the ``Redirect URL`` to the OIDC client endpoint: ``http://localhost:18381/complete/edx-oidc/``.
6. Select ``Confidential (Web applications)`` as the client type.
7. Click ``Save``.
Designated the new client as trusted by creating a new entry for it at ``/admin/edx_oauth2_provider/trustedclient/``. Finally, copy the newly created ``Client ID`` and ``Client Secret`` values to Discovery's settings (in ``course_discovery/settings/private.py``, if running locally).
Publisher
---------
"Publisher" is an information management tool meant to support the course authoring, review, and approval workflow. The tool can be used to manage course metadata and is designed for use with the Drupal site that hosts edx.org.
docs/api.rst deleted 100644 → 0
API
===
The meat of the Course Catalog API is located in this app. ``v1`` is the current version of the API.
API Views
---------
.. automodule:: course_discovery.apps.api.v1.views
:members:
Serializers
-----------
.. automodule:: course_discovery.apps.api.serializers
:members:
Renderers
---------
.. automodule:: course_discovery.apps.api.renderers
:members:
Permissions
-----------
.. autoclass:: course_discovery.apps.api.filters.PermissionsFilter
:members:
docs/catalogs.rst deleted 100644 → 0
Catalogs
========
A catalog is modeled as an Elasticsearch query (see :doc:`elasticsearch`) returning a
list of courses. The ``Catalog`` model lives in :file:`course_discovery/apps/catalogs/models.py`.
.. autoclass:: course_discovery.apps.catalogs.models.Catalog
:members:
Permissions
-----------
The ``viewers`` property of a catalog gives the users who are allowed to view the catalog and the courses it
contains. We use `django-guardian`_ for per-object permissions.
.. _django-guardian: https://django-guardian.readthedocs.io/en/stable/
Administration
--------------
Catalog administration is primarily done through the LMS at ``/api-admin/catalogs``. However, if you need to modify
a catalog or its query directly, you can do so using the course catalog Django admin at ``/admin/catalogs/``. The
admin interface provides a preview button to directly view the list of courses contained in a catalog, as well as
``django-guardian``'s standard admin for user permissions.
docs/course_metadata.rst deleted 100644 → 0
Course Metadata
===============
The ``course_metadata`` app at :file:`course_discovery/apps/course_metadata` primarily deals with getting data from
various external systems and pulling it into the course catalog.
Data Loaders
------------
:file:`course_discovery/apps/course_metadata/data_loaders.py` contains code for retrieving data from different
sources. The ``AbstractDataLoader`` class defines the basic interface, and is currently subclassed by the three
concrete implementations shown below. In the future we may add more data loaders as the requirements of the course
catalog change.
.. automodule:: course_discovery.apps.course_metadata.data_loaders
:members:
Retrieving Course Metadata
--------------------------
The ``refresh_course_metadata`` command in :file:`course_discovery/apps/course_metadata/management/commands/refresh_course_metadata.py` is used to retrieve metadata. This is run daily in production through a Jenkins job, and can be manually run to set up your local environment. The data loaders are each run in series by the command. The data
loaders should be idempotent -- that is, running this command once will populate the database, and if nothing has
changed upstream then running it again should not change the database.
The command retrieves a JWT access token from the configured OpenID Connect provider.
.. code-block:: bash
$ ./manage.py refresh_course_metadata
QuerySets
---------
We use a custom ``QuerySet`` for retrieving active courses, based on the definition of "active" that the LMS uses.
.. autoclass:: course_discovery.apps.course_metadata.query.CourseQuerySet
:members:
Views
-----
The ``QueryPreviewView`` provides a simple interface to test a query before saving it to a catalog.
.. autoclass:: course_discovery.apps.course_metadata.views.QueryPreviewView
:members:
Models
------
The ``course_metadata`` contains most of the models used in the course catalog API.
.. automodule:: course_discovery.apps.course_metadata.models
:members:
:undoc-members:
Searching for Courses
---------------------
The fields of ``CourseIndex`` and ``CourseRunIndex`` are the fields that can be used in ES queries.
.. automodule:: course_discovery.apps.course_metadata.search_indexes
:members:
:undoc-members:
docs/edx_extensions.rst deleted 100644 → 0
.. _edx-extensions:
edX Extensions
==============
edX.org has its own app, ``edx_catalog_extensions``, to contain edX-specific customizations. These include data
migrations, management commands, etc. specific to edX. Non-edX users should NOT use this app. This app
is explicitly disabled by default in all non-test environments.
At some point in the future this app will be moved to a separate repository to avoid confusion. It exists here now
until we can determine what other edX-specific components need to be extracted from the general project.
edX developers should add ``'course_discovery.apps.edx_catalog_extensions'`` to the ``INSTALLED_APPS`` setting in a
``private.py`` settings file.
Settings
========
``HAYSTACK_INDEX_RETENTION_LIMIT``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Default: ``3``
This field sets an upper bound for the number of indexes that will be retained after
a purge triggered by the 'remove_unused_indexes' command. This command will never delete the currently used index.
\ No newline at end of file
docs/elasticsearch.rst deleted 100644 → 0
Elasticsearch
=============
This service uses Elasticsearch to power the course catalog functionality. This allows users to search courses by
various criteria related to data collected from the E-Commerce Service (Otto) and other potential data sources.
The service is configured to use the `course_discovery` index by default. If you'd like to change the index, or the
URL of the Elasticsearch service, update the `ELASTICSEARCH` setting.
Elasticsearch has a feature called `aliases <https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html>`_.
This feature allows indices to be referenced by an alias. For example, the timestamped course_discovery_20160101113005
index could be assigned the alias course_discovery. It is common practice to reference aliases in code, rather than
indices, to allow for index swapping, reindex, and other maintenance without affecting service uptime. We recommend
following this practice.
Creating an index and alias
---------------------------
The `install_es_indexes` management command should be used when initializing a new alias and index. This command will
check to see if the alias exists, and is linked to an open index. If that check is true, the command will exit
successfully. If that check fails, a new index with a timestamped name (e.g. course_discovery_20160101113005) will be
created; and, the alias will be assigned to the new index.
.. code-block:: bash
$ ./manage.py install_es_indexes
Query String Syntax
-------------------
We use the query string syntax to search for courses. See `the Elasticsearch documentation`_ for a guide to the
query string syntax, and :doc:`course_metadata` for a list of fields which can be searched.
.. _the Elasticsearch documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-syntax
Result Boosting
---------------
We use the `function_score` to boost relevance based on configureable factors. See `the function_score documentation`_
for a guide to how this function can be used to provide more relevant results.
Example functions:
Closest to today's date
.. code-block:: json
{
"linear":{
"start":{
"origin":"now",
"scale":"1d",
"decay":0.5
}
},
"weight":10.0
}
Specific result type
.. code-block:: json
{
"filter":{
"term":{
"type":"micromaster"
}
},
"weight": 5.0
}
.. _the function_score documentation: https://www.elastic.co/guide/en/elasticsearch/reference/1.7/query-dsl-function-score-query.html
docs/features.rst deleted 100644 → 0
Feature Toggling
================
All new features/functionality should be released behind a feature gate. This allows us to easily disable features
in the event that an issue is discovered in production. This project uses the
`Waffle <http://waffle.readthedocs.org/en/latest/>`_ library for feature gating.
Waffle supports three types of feature gates (listed below). We typically use flags and switches since samples are
random, and not ideal for our needs.
Flag
Enable a feature for specific users, groups, users meeting certain criteria (e.g. authenticated or staff),
or a certain percentage of visitors.
Switch
Simple boolean, toggling a feature for all users.
Sample
Toggle the feature for a specified percentage of the time.
For information on creating or updating features, refer to the
`Waffle documentation <http://waffle.readthedocs.org/en/latest/>`_.
Permanent Feature Rollout
-------------------------
Over time some features may become permanent and no longer need a feature gate around them. In such instances, the
relevant code and tests should be updated to remove the feature gate. Once the code is released, the feature flag/switch
should be deleted.
docs/getting_started.rst deleted 100644 → 0
Getting Started
===============
This guide will walk you through the steps necessary to run the course-discovery service locally, on your host. In the future, we intend to switch to Docker to avoid the need for many of these manual steps.
Install dependencies
--------------------
If you have not already done so, create and activate a virtualenv running Python 3.5. We suggest using `virtualenvwrapper`_.
.. _virtualenvwrapper: https://virtualenvwrapper.readthedocs.org/en/latest/
.. note:: Installing virtualenvwrapper with pip on OS X El Capitan may result
in a strange OSError due to `a compatibility issue with the six package
<https://github.com/pypa/pip/issues/3165>`_. In this case, instruct pip to
ignore six:
.. code-block:: bash
$ pip install virtualenvwrapper --upgrade --ignore-installed six
Install dependencies as follows:
.. code-block:: bash
$ mkvirtualenv --python=$(which python3) discovery
$ workon discovery
$ make requirements
Unless otherwise stated, assume all commands below are executed within the virtualenv.
Install Elasticsearch
---------------------
The course-discovery service uses Elasticsearch (ES) to allow searching for course and program data. Elasticsearch is built using Java, and requires at least Java 8 in order to run. You will need to install the `JDK`_ if you haven't already. To install ES, download and unzip a file of your choice from https://www.elastic.co/downloads/past-releases/elasticsearch-1-5-2. It doesn't matter where you leave it. Locate the ES binary and start the server with:
.. code-block:: bash
$ bin/elasticsearch
.. _JDK: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
Navigating ES can be challenging if you don't have much experience with it. The `elasticsearch-head`_ plugin
offers a web-based front end to help with this. To install elasticsearch-head run the following from the root of the elasticsearch-1.5.2 directory:
.. code-block:: bash
$ bin/plugin -install mobz/elasticsearch-head/1.x
.. _elasticsearch-head: https://mobz.github.io/elasticsearch-head/
You should now be able to access the front end at http://localhost:9200/_plugin/head/.
Run migrations
--------------
By default, local installations use SQLite to create a database named ``default.db``. To apply migrations, run:
.. code-block:: bash
$ make migrate
If you don't have ES running when you run this, you will see an error after migrations are applied. The error comes from a post-migration hook we use for creating an index and a corresponding alias in ES after migrations are run.
Start the server
----------------
You can now start the server with:
.. code-block:: bash
$ ./manage.py runserver 8008
Access http://localhost:8008 in your browser and you should see the query preview page. You can run the service at any port of your choosing; these docs will use 8008.
Having a superuser will make it easy for you to sign into the Django admin. Do so as follows:
.. code-block:: bash
$ ./manage.py createsuperuser
Use the username and password you provided to sign into the Django admin at http://localhost:8008/admin. You should be able to see tables representing all of the application's models.
LMS integration
---------------
To integrate with the LMS, bring up the LMS and navigate to http://localhost:8000/admin/catalog/catalogintegration/. Click "Add catalog integration," and add the URL to the course-discovery service running on your host: ``http://192.168.33.1:8008/api/v1/``.
.. note:: When inside the Vagrant VM, you need to use a special IP to refer to your host. You can find it by running ``ifconfig`` and looking at the IPV4 address for vboxnet0. It's usually 192.168.33.1.
In order for the LMS running in the Vagrant VM to access course-discovery, you will need to run it at 0.0.0.0:8008.
.. code-block:: bash
$ ./manage.py runserver 0.0.0.0:8008
Private settings
----------------
When developing locally, it may be useful to have settings overrides that you do not wish to commit to the repository.
If you need such overrides, create a file :file:`course_discovery/settings/private.py`. This file's values are
read by :file:`course_discovery/settings/local.py`, but ignored by Git.
If you are an edX employee, see :ref:`edx-extensions`.
Configure partners
------------------
The Catalog Service is designed to support multiple collections of API endpoints to construct its search
indexes. These collections are represented in the system's domain model as "Partner" entities. In addition to indexing,
Partners link related top-level system entities -- Courses, Organizations, and Programs -- in order to create logical
index partitions for use during search operations.
To configure a Partner, add a new entry to the system via the Catalog Service administration console found at
``https://catalog.example.com/admin``. Alternatively you may execute the ``create_or_update_partner`` management
command via the terminal. This command, found in
:file:`course_discovery/apps/core/management/commands/create_or_update_partner.py`, allows service operators to specify
any/all Partner attributes as command arguments for both new and existing Partners, including marketing site
and OIDC authentication credentials.
Required arguments include the ``code`` and ``name`` fields, as follows:
.. code-block:: bash
$ ./manage.py create_or_update_partner --code='abc' --name='ABC Partner'
Additional optional attributes can be specified:
+-------------------------------+-----------------------------------------+----------------------------------------------------+
| Attribute/Argument | Description | Notes / Example Values |
+===============================+=========================================+====================================================+
| courses-api-url | LMS Courses API Endpoint | https://lms.example.com/api/courses/v1/ |
+-------------------------------+-----------------------------------------+----------------------------------------------------+
| ecommerce-api-url | Ecommerce API Endpoint | https://ecommerce.example.com/api/v2/ |
+-------------------------------+-----------------------------------------+----------------------------------------------------+
| organizations-api-url | Organizations API Endpoint | https://orgs.example.com/api/v1/organizations/ |
+-------------------------------+-----------------------------------------+----------------------------------------------------+
| programs-api-url | Programs API Endpoint | https://programs.example.com/api/v1/programs/ |
+-------------------------------+-----------------------------------------+----------------------------------------------------+
| marketing-site-url-root | Drupal-based Marketing Site URL | https://www.example.com/ |
+-------------------------------+-----------------------------------------+----------------------------------------------------+
| marketing-site-api-url | Drupal Courses API Endpoint | https://www.example.com/api/v1/courses/ |
+-------------------------------+-----------------------------------------+----------------------------------------------------+
| marketing-site-api-username | Drupal Courses API Account Username | (This value comes from the Drupal user account) |
+-------------------------------+-----------------------------------------+----------------------------------------------------+
| marketing-site-api-password | Drupal Courses API Account Password | (This value comes from the Drupal user account) |
+-------------------------------+-----------------------------------------+----------------------------------------------------+
| oidc-url-root | Open edX OpenID Connect URL | https://lms.example.com/oauth2 |
+-------------------------------+-----------------------------------------+----------------------------------------------------+
| oidc-key | Open edX OpenID Connect Client Key/ID | (This value comes from the LMS Client record) |
+-------------------------------+-----------------------------------------+----------------------------------------------------+
| oidc-secret | Open edX OpenID Connect Client Secret | (This value comes from the LMS Client record) |
+-------------------------------+-----------------------------------------+----------------------------------------------------+
docs/index.rst
......@@ -9,13 +9,5 @@ This guide begins with some background information on the service, then focuses
:maxdepth: 2
introduction
getting_started
api
catalogs
course_metadata
elasticsearch
testing
features
internationalization
oidc
edx_extensions
quickstart
advanced
docs/internationalization.rst deleted 100644 → 0
Internationalization
====================
All user-facing text content should be marked for translation. Even if this application is only run in English, our
open source users may choose to use another language. Marking content for translation ensures our users have
this choice.
Follow the `internationalization coding guidelines`_ in the edX Developer's Guide when developing new features.
.. _internationalization coding guidelines: http://edx.readthedocs.org/projects/edx-developer-guide/en/latest/internationalization/i18n.html
Updating Translations
~~~~~~~~~~~~~~~~~~~~~
This project uses `Transifex`_ to translate content. After new features are developed the translation source files
should be pushed to Transifex. Our translation community will translate the content, after which we can retrieve the
translations.
.. _Transifex: https://www.transifex.com/
Pushing source translation files to Transifex requires access to the edx-platform. Request access from the Open Source
Team if you will be pushing translation files. You should also `configure the Transifex client`_ if you have not done so
already.
.. _configure the Transifex client: http://docs.transifex.com/client/config/
The `make` targets listed below can be used to push or pull translations.
.. list-table::
:widths: 25 75
:header-rows: 1
* - Target
- Description
* - pull_translations
- Pull translations from Transifex
* - push_translations
- Push source translation files to Transifex
Fake Translations
~~~~~~~~~~~~~~~~~
As you develop features it may be helpful to know which strings have been marked for translation, and which are not.
Use the `fake_translations` make target for this purpose. This target will extract all strings marked for translation,
generate fake translations in the Esperanto (eo) language directory, and compile the translations.
You can trigger the display of the translations by setting your browser's language to Esperanto (eo), and navigating to
a page on the site. Instead of plain English strings, you should see specially-accented English strings that look like this:
Thé Fütüré øf Ønlïné Édüçätïøn Ⱡσяєм ι# Før änýøné, änýwhéré, änýtïmé Ⱡσяєм #
docs/introduction.rst
......@@ -25,12 +25,12 @@ Data Loading
Data about courses and course runs is collected from Studio, the LMS, the Ecommerce service, and, for edx.org, the Drupal marketing site. The data loading pipeline used to collect this data can be run with a management command called ``refresh_course_metadata``. edX runs this command several times a day using a Jenkins job. It can be manually run to populate a local environment with data. The data loading framework is designed to make adding additional systems easy.
API
---
Access to information about courses, course runs, catalogs, programs, and more is provided by a REST API. For more about the API, use your browser to visit the ``/api-docs`` hosted by a running Discovery instance.
Search
------
Discovery uses Elasticsearch to index data about courses, course runs, and programs. Indexing can be run at any time with a management command called ``update_index``. The Discovery API can be used to run search queries against the Elasticsearch index.
API
---
Access to information about courses, course runs, catalogs, programs, and more is provided by a REST API. For more about the API, use your browser to visit ``/api-docs`` hosted by a running Discovery instance.
docs/oidc.rst deleted 100644 → 0
Configure edX OpenID Connect (OIDC)
===================================
This service relies on the edX OIDC (`OpenID Connect`_) authentication provider for login. Note that OIDC is built atop
OAuth 2.0, and this document may use the terms interchangeably. Under our current architecture the LMS serves as our
authentication provider.
Configuring Course Discovery Service to work with OIDC requires registering a new client with the authentication
provider and updating the Django settings for this project with the client credentials.
.. _OpenID Connect: http://openid.net/specs/openid-connect-core-1_0.html
A new OAuth 2.0 client can be created on the LMS at ``http://127.0.0.1:8000/admin/oauth2/client/``.
1. Click the :guilabel:`Add client` button.
2. Leave the user field blank.
3. Specify the name of this service, ``Course Discovery Service``, as the client name.
4. Set the :guilabel:`URL` to the root path of this service: ``http://localhost:18381/``.
5. Set the :guilabel:`Redirect URL` to the OIDC client endpoint: ``http://localhost:18381/complete/edx-oidc/``.
6. Copy the :guilabel:`Client ID` and :guilabel:`Client Secret` values. They will be used later.
7. Select :guilabel:`Confidential (Web applications)` as the client type.
8. Click :guilabel:`Save`.
Now that you have the client credentials, you can update your settings (in
:file:`course_discovery/settings/private.py`, if running locally). The table below describes the relevant settings.
+-----------------------------------------------------+----------------------------------------------------------------------------+--------------------------------------------------------------------------+
| Setting | Description | Value |
+=====================================================+============================================================================+==========================================================================+
| SOCIAL_AUTH_EDX_OIDC_KEY | OAuth 2.0 client key | (This should be set to the value generated when the client was created.) |
+-----------------------------------------------------+----------------------------------------------------------------------------+--------------------------------------------------------------------------+
| SOCIAL_AUTH_EDX_OIDC_SECRET | OAuth 2.0 client secret | (This should be set to the value generated when the client was created.) |
+-----------------------------------------------------+----------------------------------------------------------------------------+--------------------------------------------------------------------------+
| SOCIAL_AUTH_EDX_OIDC_URL_ROOT | OAuth 2.0 authentication URL | http://127.0.0.1:8000/oauth2 |
+-----------------------------------------------------+----------------------------------------------------------------------------+--------------------------------------------------------------------------+
| SOCIAL_AUTH_EDX_OIDC_ID_TOKEN_DECRYPTION_KEY | OIDC ID token decryption key. This value is used to validate the ID token. | (This should be the same value as SOCIAL_AUTH_EDX_OIDC_SECRET.) |
+-----------------------------------------------------+----------------------------------------------------------------------------+--------------------------------------------------------------------------+
docs/publisher.rst deleted 100644 → 0
Publisher
=========
The ``publisher`` tool is an information management system that supports the course authoring, review, and approval workflow. The tool manages courses as they transition through the lifecycle of creation (across various subsystems), release/publication to the edX.org marketing site, and eventual retirement/archival.
Note: This app is primarily built for the edX.org use case.
Configure Emails
----------------
The publisher tool supports notifying responsible parties when users comment on courses and course runs. Notifications are sent via Django's email backend. In order to send emails the ``enable_publisher_email_notifications`` switch must be activated, and the following settings must be defined:
+------------------------------------+----------------------------------------------------------------------------+--------------------------------------------------------------------------+
| Setting | Description | Value |
+====================================+============================================================================+==========================================================================+
| PUBLISHER_FROM_EMAIL | Official email address for sending emails. | Email address |
+------------------------------------+----------------------------------------------------------------------------+--------------------------------------------------------------------------+
Since the publisher tool uses the built-in Django email functionality, any Django email backend can be used. For info on configuring email backends see `django_email`_.
.. _django_email: https://docs.djangoproject.com/en/1.10/topics/email/
We at edX.org use Amazon's Simple Email Service (SES) (`amazon_ses`_). If you'd also like to use SES, the django-ses (`django_ses`_) is installed as a base requirement. Simply define values for the settings below to configure the backend.
.. _amazon_ses: https://aws.amazon.com/ses/
.. _django_ses: https://github.com/django-ses/django-ses
+------------------------------------+----------------------------------------------------------------------------+--------------------------------------------------------------------------+
| Setting | Description | Value |
+====================================+============================================================================+==========================================================================+
| EMAIL_BACKEND | django_ses.SESBackend | The backend to use for sending emails. |
+------------------------------------+----------------------------------------------------------------------------+--------------------------------------------------------------------------+
| AWS_ACCESS_KEY_ID | YOUR-ACCESS-KEY-ID | (This should be set to the value from the AWS account.) |
+------------------------------------+----------------------------------------------------------------------------+--------------------------------------------------------------------------+
| AWS_SECRET_ACCESS_KEY | YOUR-SECRET-ACCESS-KEY | (This should be set to the value from the AWS account.) |
+------------------------------------+----------------------------------------------------------------------------+--------------------------------------------------------------------------+
| AWS_SES_REGION_NAME | Region your SES service is using. | (This should be set to the value from the AWS account.) |
+------------------------------------+----------------------------------------------------------------------------+--------------------------------------------------------------------------+
| AWS_SES_REGION_ENDPOINT | Region your SES service is using. | (This should be set to the value from the AWS account.) |
+------------------------------------+----------------------------------------------------------------------------+--------------------------------------------------------------------------+
docs/quickstart.rst 0 → 100644
Quickstart
==========
This section covers information you need to know to run and develop for the Discovery service.
Devstack
--------
Discovery is part of edX's Docker-based "devstack." To run the service locally, follow along with the instructions in the https://github.com/edx/devstack repo's `README`_.
.. _README: https://github.com/edx/devstack/blob/master/README.rst
Devstack will allow you to run all edX services together. If you only need Discovery, you can run just the services it requires:
.. code-block:: bash
$ DEVSTACK_WORKSPACE=.. docker-compose -f docker-compose.yml -f docker-compose-host.yml up -d mysql elasticsearch memcached discovery
Data Loaders
------------
Run the data loaders using the ``refresh_course_metadata`` management command to populate a Discovery instance with data. Open a Discovery shell with ``make discovery-shell``, then run:
.. code-block:: bash
$ ./manage.py refresh_course_metadata
By default, ``refresh_course_metadata`` loads data for every "partner" in the system. Partners are site tenants, like edx.org. You can view and create tenants using the Django admin at ``/admin/core/partner/``. To load data for a specific tenant:
.. code-block:: bash
$ ./manage.py refresh_course_metadata --partner_code <SHORT CODE HERE>
Search Indexing
---------------
Once you've loaded data into your Discovery instance, you may want to run Elasticsearch queries against it. Doing so requires indexing the data you've loaded, which you can do by running the ``update_index`` management command. Open a Discovery shell with ``make discovery-shell``, then run:
.. code-block:: bash
$ ./manage.py update_index --disable-change-limit
Once indexing completes, you can run search queries against the newly created index through the API. For more on this, visit ``/api-docs``.
Tests
-----
Use Docker Compose to run tests just like Travis does. The ``.travis.yml`` file is a good reference if you want to run the entire test suite. You'll notice that a Docker Compose file hosted in this repo is used to run tests instead of the Compose files in the devstack repo. This Compose file defines special test settings and has yet to be consolidated with the Compose files in the devstack repo.
To run specific tests, bring up the services used for testing with ``make travis_up``. To run the tests in ``course_discovery/apps/api/v1/tests/test_views/test_programs.py``:
.. code-block:: bash
$ docker-compose -f .travis/docker-compose-travis.yml exec course-discovery bash -c '. /edx/app/discovery/venvs/discovery/bin/activate && cd /edx/app/discovery/discovery && pytest course_discovery/apps/api/v1/tests/test_views/test_programs.py'
When you're done, take down the services you brought up with ``make travis_down``.
docs/testing.rst deleted 100644 → 0
Testing
=======
The command below runs the Python tests and code quality validation—Pylint and PEP8.
.. code-block:: bash
$ make validate
Code quality validation can be run independently with:
.. code-block:: bash
$ make quality
httpretty
---------
edX uses `httpretty <http://httpretty.readthedocs.org/en/latest/>`_ a lot to mock HTTP endpoints; however,
`a bug in httpretty <https://github.com/gabrielfalcao/HTTPretty/issues/65>`_ (that is closed, but still a problem)
prevents us from using it in this repository. Were you to use `httpretty`, you would find that, although you might
mock an OAuth2 endpoint, `httpretty` blocks requests to Elasticsearch, leading to test failures.
Given our extensive use of Elasticsearch, and need to mock HTTP endpoints, we use the
`responses <https://github.com/getsentry/responses>`_ library. Its API is practically the same as that of `httpretty`.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment