This documentation has moved! Please click `Creating Peer Assessments <http://edx.readthedocs.org/projects/edx-open-response-assessments/en/latest/>`_ to see the latest information about peer assessments.
Introduction to Peer Assessments
********************************
.. note:: **The peer assessment feature is in limited release.** If you're at an edX consortium university and you plan to include peer assessments in a MOOC, you'll work with your edX project manager (PM) to enable the peer assessment feature and create peer assessment assignments.
Peer assessments allow instructors to assign questions that may not have definite answers. Students submit a response to a question, and then compare their peers' responses to a rubric that you create. Peer assessment problems also include self assessments, in which students compare their own responses to the rubric.
For more information, see the following sections:
* :ref:`PA Elements`
* :ref:`PA Scoring`
* :ref:`PA Create a PA Problem`
* :ref:`PA Access Status`
.. _PA Elements:
********************************
Elements of a Peer Assessment
********************************
When you create a peer assessment problem, you'll specify several elements:
* The number of responses and assessments.
* The assessment type or types.
* The due dates for each step (optional).
* The question.
* The rubric.
For step-by-step instructions, see :ref:`PA Create a PA Problem`.
=======================================
Number of Responses and Assessments
=======================================
In the problem code, you'll indicate the **number of responses** each student has to assess and the **number of peer assessments** each response has to receive.
.. note:: Because some students may submit a response but not complete peer assessments, some responses may not receive the required number of assessments. To increase the chance that all responses will receive enough assessments, you must set the number of responses that students have to assess to be higher than the number of assessments that each response must undergo. For example, if you require each response to receive three assessments, you could require each student to assess five responses.
If all responses have received assessments, but some students haven't completed the required number of peer assessments, those students can assess responses that other students have already assessed. The student who submitted the response sees the additional peer assessments when he sees his score. However, the additional peer assessments do not count toward the score that the response receives.
For more information, see :ref:`PA Specify Name and Assessment Types`.
=====================
Assessment Type
=====================
In your problem, you'll also specify the **assessment type or types**. You can see the type and order of the assessments when you look at the problem. In the following example, after students submit a response, they complete peer assessments on other students' responses ("Assess Peers") and then complete self assessments ("Assess Yourself").
.. image:: /Images/PA_AsmtWithResponse.png
:alt: Image of peer assessment with assessment steps and status labeled
:width: 600
You can set the assignment to include a peer assessment only, a self assessment only, or a peer assessment followed by a self assessment.
For more information, see :ref:`PA Specify Name and Assessment Types`.
===================================
Start and Due Dates (optional)
===================================
You can specify **start dates** and **due dates** for students to submit responses, perform peer assessments, and perform self assessments.
You can set different dates for each step, and these dates can overlap. For example, you can allow students to submit responses and complete peer and self assessments starting on March 1. You can require all responses to be submitted by March 7, but allow students to continue peer and self assessments until March 14, a week after all responses are due.
If you don't specify dates, the deadline for all elements--responses, peer assessments, and self assessments--is the due date that you set for the subsection that contains the peer assessment. If you do specify dates, those dates take precedence over the subsection due date.
.. note:: We don't recommend that you use the same due date and time for response submissions and assessments. If a student submits a response immediately before the due date, other students will have very little time to assess the response before peer assessment closes. In this case, a student's response may not receive a score.
For more information, see :ref:`PA Add Due Dates`.
==============
Question
==============
You'll also specify the **question** that you want your students to answer. This appears near the top of the component, followed by a field where the student enters a response.
When you write your question, you can include helpful information for your students, such as what students can expect after they submit responses and the approximate number of words or sentences that a student's response should have. (A response cannot have more than 10,000 words.)
For more information, see :ref:`PA Add Question`.
.. _PA Rubric:
=======
Rubric
=======
Your problem must include a **rubric** that you design. The same rubric is used for peer and self assessments, and the rubric appears when students begin grading. Students compare their peers' responses to the rubric.
Rubrics are made of *criteria* and *options*.
* Each criterion has a *name*, a *prompt*, and two or more *options*.
* The name is a very short summary of the criterion, such as Ideas or Content. Criterion names generally have just one word. **The name for each criterion must be unique.** The system uses the criterion name for identification. Criterion names do not appear in the rubric that students see when they are completing peer assessments, but they do appear on the page that shows the student's final grade.
.. image :: /Images/PA_CriterionName.png
:alt: Image of a final score page with call-outs for the criterion names
* The prompt is a description of the criterion.
* Each option has a *name*, an *explanation*, and a *point value*.
.. image:: /Images/PA_Rubric_LMS.png
:alt: Image of a rubric in the LMS with call-outs for the criterion prompt and option names, explanations, and points
When you create your rubric, decide how many points each option will receive, and make sure that the explanation for each option is as specific as possible. For example, one criterion and set of options may resemble the following.
**Criterion**
Name: Origins
Prompt: Does this response explain the origins of the Hundred Years' War? (5 points possible)
**Options**
.. list-table::
:widths: 8 20 50
:stub-columns: 1
:header-rows: 1
* - Points
- Name
- Explanation
* - 0
- Not at all
- This response does not address the origins of the Hundred Years' War.
* - 1
- Dynastic disagreement
- This response alludes to a dynastic disagreement between England and France, but doesn't reference Edward III of England and Philip VI of France.
* - 3
- Edward and Philip
- This response mentions the dynastic disagreement between Edward III and Philip VI, but doesn't address the role of Salic law.
* - 5
- Salic law
- This response explains the way that Salic law contributed to the dynastic disagreement between Edward III and Philip VI, leading to the Hundred Years' War.
For more information about writing effective rubrics, see Heidi Goodrich Andrade's `Understanding Rubrics <http://learnweb.harvard.edu/alps/thinking/docs/rubricar.htm>`_.
Note that different criteria in the same assignment can have different numbers of options. For example, in the image above, the first criterion has three options and the second criterion has four options.
For more information, see :ref:`PA Add Rubric`.
.. _PA Scoring:
***********************
Peer Assessment Scoring
***********************
Peer assessments are scored by criteria. An individual criterion's score is the median of the scores that each peer assessor gave that criterion. For example, if the Ideas criterion in a peer assessment receives a 10 from one student, a 7 from a second student, and an 8 from a third student, the Ideas criterion's score is 8.
A student's final score for a peer assessment is the sum of the median scores for each individual criterion.
For example, a response may receive the following scores from peer assessors:
.. list-table::
:widths: 25 10 10 10 10
:stub-columns: 1
:header-rows: 1
* - Criterion Name
- Peer 1
- Peer 2
- Peer 3
- Median
* - Ideas (out of 10)
- 10
- 7
- 8
- **8**
* - Content (out of 10)
- 7
- 9
- 8
- **8**
* - Grammar (out of 5)
- 4
- 4
- 5
- **4**
To calculate the final score, add the median scores for each criterion:
**Ideas median (8/10) + Content median (8/10) + Grammar median (4/5) = final score (20/25)**
Note, again, that final scores are calculated by criteria, not by individual assessor. Thus the response's score is not the median of the scores that each individual peer assessor gave the response.
.. _PA Create a PA Problem:
********************************
Create a Peer Assessment Problem
********************************
.. warning:: Peer assessments are in limited release and are only available in a few courses. To enable the peer assessment feature in your course, contact your edX program manager. After the feature has been enabled, you can create peer assessments by following the steps below.
To create a peer assessment problem, you'll edit the XML code in a Problem component, similar to creating other problems. The following image shows what a peer assessment component looks like when you edit it in Studio, as well as the way that students see that peer assessment in the courseware.
.. image:: /Images/PA_XML_LMS_All.png
:alt: Image of a peer assessment in Studio and LMS views
:width: 800
Creating a peer assessment is a multi-step process:
* :ref:`PA Create Component`
* :ref:`PA Specify Name and Assessment Types`
* :ref:`PA Add Due Dates`
* :ref:`PA Add Question`
* :ref:`PA Add Rubric`
* :ref:`PA Test Problem`
Each of these steps is covered in detail below.
.. _PA Create Component:
============================
Step 1. Create the Component
============================
#. In Studio, open the unit where you want to create the assessment.
#. Under **Add New Component**, click **Advanced**, and then click **Peer Assessment**.
#. In the Problem component that appears, click **Edit**.
The component editor opens, and you can see sample code that includes the assignment's title, the assessment type or types, the number of assessments that students must complete, a sample question ("prompt"), and a rubric. You'll replace this sample content with the content for your problem in the next steps.
Note that you won't use the **Settings** tab in the component editor when you create peer assessments.
* **The title of the assignment**. In this example, because there is no text between the ``<title>`` tags, the assignment does not have a specified title.
* **The type and order of the assessments**. This information is in the **name** attribute in the two ``<assessment>`` tags. Assessments run in the order in which they're listed. In this example, the peer assessment runs, and then the student performs a self assessment.
* **The number of responses that each student must assess** (for peer assessments). This information is in the **must_grade** attribute in the ``<assessment>`` tag for the peer assessment. In this example, each student must grade five peer responses before he receives the scores that his peers have given him.
* **The number of peer assessments each response must receive** (for peer assessments). This information is in the **must_be_graded_by** attribute in the ``<assessment>`` tag for the peer assessment. In this example, each response must receive assessments from three students before it can return to the student who submitted it.
To specify the name and assessment types, follow these steps.
#. Between the ``<title>`` tags, add a name for the problem.
#. Specify the type of assessments you want students to complete.
- If you want students to complete a peer assessment only, delete the ``<assessment name="self-assessment"/>`` tag.
- If you want students to complete a self assessment only, delete the ``<assessment name="peer-assessment" must_grade="5" must_be_graded_by="3""/>`` tag.
- If you want students to complete a peer assessment and then a self assessment, leave the default tags.
#. If your students will complete a peer assessment, replace the values for **must_grade** and **must_be_graded_by** in the ``<assessment name="peer-assessment">`` tag with the numbers that you want.
.. note:: The value for **must_grade** must be greater than or equal to the value for **must_be_graded_by**.
.. _PA Add Due Dates:
==========================================
Step 3. Add Start and Due Dates (optional)
==========================================
Setting start and due dates is optional. If you don't specify dates, the deadline for all student responses and assessments is the due date that you set for the subsection that contains the peer assessment. If you do specify dates, those dates take precedence over the subsection due date.
To specify due dates and times, you'll add code that includes the date and time inside the XML tags for the problem and for each specific assessment. The date and time must be formatted as ``YYYY-MM-DDTHH:MM:SS``.
.. note:: You must include the "T" between the date and the time, with no spaces. All times are in universal coordinated time (UTC).
* To specify a due date for response submissions, add the ``submission_due`` attribute with the date and time to the ``<openassessment>`` tag (this is the first tag in your problem).
* To specify start and end times for an assessment, add ``start`` and ``due`` attributes with the date and time to the ``<assessment>`` tags for the assessment.
In this example, the problem is set at the subsection level to open on February 24, 2014 at midnight UTC. (This information does not appear in the code.) Additionally, the code specifies the following:
* Students must can begin submitting responses on February 24, 2014 at midnight UTC, and must submit all responses before March 1, 2014 at midnight UTC:
* Students can begin peer assessments on February 24, 2014 at midnight UTC, and all peer assessments must be complete by March 8, 2014 at midnight UTC:
* Students can begin self assessments on February 24, 2014 at midnight UTC, and all self assessments must be complete by March 8, 2014 at midnight UTC:
.. note:: We don't recommend that you use the same due date and time for response submissions and peer assessments. If a student submits a response immediately before the due date, other students will have very little time to assess the response before peer assessment closes. In this case, a student's response may not receive a score.
.. _PA Add Question:
============================
Step 4. Add the Question
============================
The following image shows a question in the component editor along with the way the question appears to students.
.. image:: /Images/PA_Question_XML-LMS.png
:alt: Image of question in XML and the LMS
:width: 800
To add the question:
#. In the component editor, locate the first set of ``<prompt>`` tags. The opening ``<prompt>`` tag appears directly below the opening ``<rubric>`` tag.
#. Replace the sample text between the ``<prompt>`` tags with the text of your question. Note that the component editor respects paragraph breaks and new lines inside the ``<prompt>`` tags. You don't have to add ``<p>`` tags to create individual paragraphs.
Add Formatting or Images to the Question
****************************************
In this initial release, you cannot add text formatting or images in the Peer Assessment component. If you want to include formatting or images in the text of your prompt, you can add an HTML component that contains your text above the Peer Assessment component, and then remove the prompt from the Peer Assessment component. The instructions for the peer assessment still appear above the **Your Response** field.
.. image:: /Images/PA_HTMLComponent.png
:alt: A peer assessment that has an image in an HTML component
:width: 500
To remove the prompt from the Peer Assessment component, open the component editor, and then delete the first set of ``<prompt>`` tags together with all the text between the tags. The first few lines of XML for the problem will then resemble the following.
<prompt>Determine if there is a unifying theme or main idea.</prompt>
<option points="0">
.. _PA Add Rubric:
============================
Step 5. Add the Rubric
============================
To add the rubric, you'll create your criteria and options in XML. The following image shows a highlighted criterion and its options in the component editor, followed by the way the criterion and options appear to students.
.. image:: /Images/PA_RubricSample_XML-LMS.png
:alt: Image of rubric in XML and the LMS, with call-outs for criteria and options
For more information about criteria and options, see :ref:`PA Elements`.
To add the rubric:
#. In the component editor, locate the following XML. This XML contains a single criterion and its options. You'll replace the placeholder text with your own content.
.. code-block:: xml
<criterion>
<name>Ideas</name>
<prompt>Determine if there is a unifying theme or main idea.</prompt>
<option points="0">
<name>Poor</name>
<explanation>Difficult for the reader to discern the main idea.
Too brief or too repetitive to establish or maintain a focus.</explanation>
</option>
<option points="3">
<name>Fair</name>
<explanation>Presents a unifying theme or main idea, but may
include minor tangents. Stays somewhat focused on topic and
task.</explanation>
</option>
<option points="5">
<name>Good</name>
<explanation>Presents a unifying theme or main idea without going
off on tangents. Stays completely focused on topic and task.</explanation>
</option>
</criterion>
.. note:: The placeholder text contains indentations and line breaks. You don't have to preserve these indentations and line breaks when you replace the placeholder text.
#. Under the opening ``<criterion>`` tag, replace the text between the ``<name>`` tags with the name of your criterion. Then, replace the text between the ``<prompt>`` tags with the description of that criterion.
Note that **every criterion must have a unique name.** The system uses the criterion name for identification. For more information about criteria, see :ref:`PA Rubric`.
#. Inside the first ``<option>`` tag, replace the value for ``points`` with the number of points that you want this option to receive.
#. Under the ``<option>`` tag, replace the text between the ``<name>`` tags with the name of the first option. Then, replace the text between the ``<explanation>`` tags with the description of that option.
#. Use this format to add as many options as you want.
You can use the following code as a template:
.. code-block:: xml
<criterion>
<name>NAME</name>
<prompt>PROMPT TEXT</prompt>
<option points="NUMBER">
<name>NAME</name>
<explanation>EXPLANATION</explanation>
</option>
<option points="NUMBER">
<name>NAME</name>
<explanation>EXPLANATION</explanation>
</option>
<option points="NUMBER">
<name>NAME</name>
<explanation>EXPLANATION</explanation>
</option>
</criterion>
.. _PA Provide Comment Options:
=============================================
Step 6 (optional). Provide Comment Options
=============================================
After students fill out the rubric, they can provide additional comments for the responses they've assessed. By default, students see a field for comments below the rubric.
.. image:: /Images/PA_CommentsField.png
:alt: Contents field
:width: 500
You can change the text that appears above this comment field. Additionally, you can provide a comment field for each individual criterion.
.. _PA Change Comments Prompt:
Change the Default Prompt Text
*******************************
By default, the prompt text for the comment field is the following:
``(Optional) What aspects of this response stood out to you? What did it do well? How could it improve?``
You can replace this default text with your own text.
To change this text:
#. Locate the ``<feedbackprompt>`` tags between the last closing ``</criterion>`` tag for the rubric and the closing ``</rubric>`` tag for the problem:
.. code-block:: xml
<option points="3">
<name>Excellent</name>
<explanation>Includes in-depth information and exceptional supporting details that are fully developed. Explores all facets of the topic.</explanation>
</option>
</criterion>
<feedbackprompt>(Optional) What aspects of this response stood out to you? What did it do well? How could it improve?</feedbackprompt>
</rubric>
</openassessment>
2. Change the text between the ``<feedbackprompt>`` tags to the text that you want.
By default, students see only a single comment field below the entire rubric. However, you can add a comment field to an individual criterion or to several individual criteria. The comment field can contain up to 300 characters.
In the following image, the first criterion has a comment field, but the second does not.
.. image:: /Images/PA_Comments_Criterion.png
:alt: Comment box under an individual criterion
:width: 500
To add a comment field:
#. Locate the opening ``<criterion>`` tag for the criterion that you want to change.
#. Add the ``feedback="optional"`` attribute to this tag:
.. code-block:: xml
<criterion feedback="optional">
<name>NAME</name>
<prompt>PROMPT TEXT</prompt>
<option points="NUMBER">
<name>NAME</name>
<explanation>EXPLANATION</explanation>
</option>
<option points="NUMBER">
<name>NAME</name>
<explanation>EXPLANATION</explanation>
</option>
</criterion>
If you want to provide comment fields below additional criteria, add the ``feedback="optional"`` attribute to the opening tag for each criterion.
.. _PA Test Problem:
============================
Step 7. Test the Problem
============================
To test your assignment, set up the assignment in a test course, and ask a group of beta users to submit responses and grade each other. The beta testers can then let you know if they found the question and the rubric easy to understand or if they found any problems with the assignment.
.. _PA Access Status:
******************************
Access Status of Problem Steps
******************************
After your problem has opened, you can check the current number of students who are in each step--that is, how many students have submitted responses, have completed peer and self assessments, are waiting to assess responses or receive grades, or have finished the problem entirely.
To find this information, open the problem in the LMS, scroll to the bottom of the problem, and then click the black **Course Staff Information** banner.
You may want to let your students know what to expect when they complete peer assessments. This guide walks students through each step of the process.
This documentation has moved! Please click `Creating Peer Assessments <http://edx.readthedocs.org/projects/edx-open-response-assessments/en/latest/PeerAssessment_Students.html>`_ to see the latest peer assessment information for students.
\ No newline at end of file
********************
Student Instructions
********************
Peer assessment problems have several steps--you'll submit a response, then assess (grade) peer responses, and then assess your own response. When you come to a peer assessment problem in the course, you'll see the question and a response field, followed by grayed-out steps that show you what you'll do next.
:alt: Peer assessment example with question, response field, and assessment types and status labeled
:width: 550
Here, we'll walk you through the process of performing a peer assessment:
#. Submit your response to a question.
#. Assess, or grade, responses that other students have submitted.
#. Assess your own response to the question.
#. Receive your score and provide feedback on the peer assessment.
At any time during the assessment, you can see your status at the bottom of the page under **Your Grade**. A message tells you the steps that you still have to perform before you can receive your grade. For example, you may see the following message:
Not Completed
You have not completed the Peer Assessment step and Self Assessment step of this problem.
Submit Your Response
********************
Read the question carefully. Some instructors include important information in the question, such as how long your response must be or specific topics your response must cover.
.. note:: Your response must contain fewer than 10,000 words (approximately the equivalent of 20 pages of 8.5x11 paper, with text single-spaced).
After you compose a response, type it into the response field under **Your Response**, and then click **Submit your response and move to the next step**. If you can't finish your response all at once, you can click **Save Your Progress** to save a draft of your response, and then come back and submit it later.
After you submit your response, if no other students have submitted responses yet, you'll see the following message:
**Waiting for Peer Responses**
**All submitted peer responses have been assessed. Check back later to see if more students have submitted responses. You'll receive your grade after you complete the peer assessment and self assessment steps, and after your peers have assessed your response.**
If other students have already submitted responses, the peer assessment step starts immediately after you submit your response. However, you don't have to start grading right away. If you want to stop working and come back later, just refresh or reopen your browser when you come back. New peer responses will be available for you to grade.
Note that you can view your response at any time after you submit it. To do this, click the **Your Response** heading to expand the response field.
.. image:: /Images/PA_S_ReviewResponse.png
:alt: Image of the Response field collapsed and then expanded
:width: 550
Assess Peer Responses
**********************
When peer assessment starts, you'll see the original question followed by another student's response and a rubric. Above the rubric you can also see how many responses you'll assess, how many you've already assessed, and the amount of time you can expect to spend on each assessment. Below the rubric is a field where you can provide comments about the student's response.
The rubric contains *criteria*, each of which has several *options*. Read the student's response and the rubric carefully, and then select the option that you think best reflects the student's response.
.. image:: /Images/PA_LMS-PARubric.png
:alt: Image of the question, response, and rubric
:width: 550
When you've selected options in the rubric, provide additional comments about the response in the field below the rubric. When you're done, click **Submit your assessment and move to response #<number>**.
.. note:: Some assessments may have an additional **Comments** field for one or more of the assessment's individual criteria. You can enter up to 300 characters in these fields. In the following image, the first of the criteria has a separate **Comments** field, but the second does not.
.. image:: /Images/PA_Students_CommentBoxes.png
:alt: Rubric with call-outs for comment boxes
:width: 500
When you submit your assessment of the first student's response, another response opens for you. Assess this response in the same way that you assessed the first response, and then submit your assessment. You'll repeat these steps until you've assessed the required number of responses. The number in the upper-right corner of the rubric updates as you assess each response.
Assess Your Response
********************
When you've completed enough peer assessments, your self assessment opens. You'll see your response along with the same rubric that you used in the peer assessment step. Assess your response, and then click **Submit Your Assessment**.
Receive Your Score and Provide Feedback
****************************************
After you submit your self assessment, if other students are still assessing your response, you'll see the following message under the **Assess Your Response** step.
**Your Grade: Waiting for Peer Assessment**
**Your response is still undergoing peer assessment. After your peers have assessed your response, you'll see their feedback and receive your final grade.**
If you see this message, keep checking back periodically until peer assessment is complete.
When peer assessment is complete, you can see the scores you received from all of your peers, as well as your self assessment. You can also see any additional comments that your peers have provided.
.. image:: /Images/PA_AllScores.png
:alt: A student's response with peer and self assessment scores
:width: 550
If you want to, you can provide feedback on the scores that you received under **Provide Feedback on Peer Assessments**.
Assess Additional Peer Responses (optional)
*******************************************
If you've assessed the required number of peer responses and completed your self assessment, you can assess additional peer responses. To do this, click the **Assess Peers** heading. If any responses remain to be assessed, a new response opens.
***********************
Peer Assessment Scoring
***********************
Peer assessments are scored by criteria. An individual criterion's score is the median of the scores that each peer assessor gave that criterion. For example, if the Ideas criterion in a peer assessment receives a 10 from one student, a 7 from a second student, and an 8 from a third student, the Ideas criterion's score is 8.
A student's final score for a peer assessment is the sum of the median scores for each individual criterion.
For example, a response may receive the following scores from peer assessors:
.. list-table::
:widths: 25 10 10 10 10
:stub-columns: 1
:header-rows: 1
* - Criterion Name
- Peer 1
- Peer 2
- Peer 3
- Median
* - Ideas (out of 10)
- 10
- 7
- 8
- **8**
* - Content (out of 10)
- 7
- 9
- 8
- **8**
* - Grammar (out of 5)
- 4
- 4
- 5
- **4**
To calculate the final score, the system adds the median scores for each criterion:
**Ideas median (8/10) + Content median (8/10) + Grammar median (4/5) = final score (20/25)**
Note, again, that final scores are calculated by criteria, not by individual assessor. Thus the response's score is not the median of the scores that each individual peer assessor gave the response.
- Added warning that version 1 of the :ref:`Open Response Assessments<Open Response Assessment>` suite has been deprecated and added a link to the `new ORA information <http://edx-open-response-assessments.readthedocs.org/en/latest/>`_.
* - 05/16/14
* - 05/16/14
- Updated :ref:`Working with Video Components` to reflect UI changes.
- Updated :ref:`Working with Video Components` to reflect UI changes.
.. warning:: **Version 1 of of our Open Response Assessments suite has been deprecated.** This documentation is meant for past courses that have used this first version of open response assessments. We no longer support new open response assessments of this type.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. warning:: Open response assessments are still in beta. We recommend that
Version 2 of our Open Response Assessments suite was released to a few courses in April 2014 and more broadly in May 2014. If you want to create a new open response assessment in your course, see `the new ORA documentation <http://edx.readthedocs.org/projects/edx-open-response-assessments/en/latest/>`_.
you test them thoroughly in a practice course and only add them to
courses that are **not** already running.
.. warning:: Problems can occur when you move your course from edge.edx.org to edx.org, or vice versa, if your course has the same URL in both places. To work around this problem, make sure that the URLs are different by specifying a different university or course number when you create each course. For example, if your university is edX and your course number is edX101, you can specify the course number as **edx_101** (note the underscore) on Edge and **edX101** on edX.
*********************************************
Introduction to Open Response Assessments
*********************************************
.. note:: Problems can occur when you move your course from edge.edx.org to edx.org, or vice versa, if your course has the same URL in both places. To work around this problem, make sure that the URLs are different by specifying a different university or course number when you create each course. For example, if your university is edX and your course number is edX101, you can specify the course number as **edx_101** (note the underscore) on Edge and **edX101** on edX.
Open response assessments allow instructors to assess student learning through questions that may not have definite answers. Tens of thousands of students can receive feedback on written responses of varying lengths as well as files, such as computer code or images, that the students upload. Open response assessment technologies include self assessment and peer assessment. With self assessments, students learn by comparing their answers to a rubric that you create. With peer assessments, students compare their peers' answers to the rubric.
Open response assessments allow instructors to assess student learning through questions that may not have definite answers. Tens of thousands of students can receive feedback on written responses of varying lengths as well as files, such as computer code or images, that the students upload. Open response assessment technologies include self assessment and peer assessment. With self assessments, students learn by comparing their answers to a rubric that you create. With peer assessments, students compare their peers' answers to the rubric.
...
@@ -28,8 +29,9 @@ For more information, see the following:
...
@@ -28,8 +29,9 @@ For more information, see the following:
.. _ORA Notes:
.. _ORA Notes:
*********************************************
A Few Notes about Open Response Assessments
A Few Notes about Open Response Assessments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
*********************************************
**Open response assessment technology is still in beta.** For a good
**Open response assessment technology is still in beta.** For a good
experience with open response assessments, you'll need to follow a few
experience with open response assessments, you'll need to follow a few
...
@@ -53,8 +55,9 @@ edX project manager (PM).
...
@@ -53,8 +55,9 @@ edX project manager (PM).
.. _ORA Components:
.. _ORA Components:
*********************************************
Components of an Open Response Assessment
Components of an Open Response Assessment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
*********************************************
An open response assessment has three elements:
An open response assessment has three elements:
...
@@ -86,8 +89,9 @@ An open response assessment has three elements:
...
@@ -86,8 +89,9 @@ An open response assessment has three elements:
.. _ORA Types:
.. _ORA Types:
******************************
Open Response Assessment Types
Open Response Assessment Types
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
******************************
There are three types of assessments for ORAs: self assessment, AI
There are three types of assessments for ORAs: self assessment, AI
assessment, and peer assessment.
assessment, and peer assessment.
...
@@ -109,8 +113,9 @@ assessment.
...
@@ -109,8 +113,9 @@ assessment.
.. _ORA Effective Questions:
.. _ORA Effective Questions:
******************************
Effective Questions
Effective Questions
~~~~~~~~~~~~~~~~~~~
******************************
When you write your question, we recommend that you specify an
When you write your question, we recommend that you specify an
approximate number of words or sentences that a student's response has
approximate number of words or sentences that a student's response has
...
@@ -123,8 +128,9 @@ that a student will be able to submit a response for the problem.
...
@@ -123,8 +128,9 @@ that a student will be able to submit a response for the problem.
.. _ORA Rubrics:
.. _ORA Rubrics:
***************
Rubrics
Rubrics
~~~~~~~
***************
The same rubric is used for all three ORA types, and it can include
The same rubric is used for all three ORA types, and it can include
anything that you want it to include.
anything that you want it to include.
...
@@ -143,8 +149,9 @@ options.
...
@@ -143,8 +149,9 @@ options.
.. _ORA Create an ORA Problem:
.. _ORA Create an ORA Problem:
*********************************************
Create an Open Response Assessment Problem
Create an Open Response Assessment Problem
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
*********************************************
.. warning:: Open response assessments are still in beta. To enable open response assessments in your course, you'll need to work with your edX program manager.
.. warning:: Open response assessments are still in beta. To enable open response assessments in your course, you'll need to work with your edX program manager.
...
@@ -166,8 +173,9 @@ Each of these steps is described in detail below.
...
@@ -166,8 +173,9 @@ Each of these steps is described in detail below.
.. _ORA Step 1:
.. _ORA Step 1:
================================
Step 1. Create the ORA Component
Step 1. Create the ORA Component
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
================================
#. Modify your course's advanced settings to allow open response assessments. For more information about how to do this, contact your edX program manager.
#. Modify your course's advanced settings to allow open response assessments. For more information about how to do this, contact your edX program manager.
#. In Studio, open the unit where you want to create the ORA.
#. In Studio, open the unit where you want to create the ORA.
...
@@ -185,8 +193,9 @@ Step 1. Create the ORA Component
...
@@ -185,8 +193,9 @@ Step 1. Create the ORA Component
.. _ORA Step 2:
.. _ORA Step 2:
========================
Step 2. Add the Question
Step 2. Add the Question
^^^^^^^^^^^^^^^^^^^^^^^^
========================
#. In the component editor, locate the [prompt] tags.
#. In the component editor, locate the [prompt] tags.
...
@@ -202,8 +211,9 @@ Step 2. Add the Question
...
@@ -202,8 +211,9 @@ Step 2. Add the Question
.. _ORA Step 3:
.. _ORA Step 3:
========================
Step 3. Add the Rubric
Step 3. Add the Rubric
^^^^^^^^^^^^^^^^^^^^^^^^
========================
.. note:: After you release your problem to students in your live course, make sure
.. note:: After you release your problem to students in your live course, make sure
you don't make any changes to the rubric that affect scoring, such as adding or removing an option
you don't make any changes to the rubric that affect scoring, such as adding or removing an option
...
@@ -241,8 +251,9 @@ Step 3. Add the Rubric
...
@@ -241,8 +251,9 @@ Step 3. Add the Rubric
.. _ORA Step 4:
.. _ORA Step 4:
============================================
Step 4. Set the Assessment Type and Scoring
Step 4. Set the Assessment Type and Scoring
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
============================================
.. note:: After you release your problem to students in your live course, make sure
.. note:: After you release your problem to students in your live course, make sure
you don't make any changes to the code for the assessment type and scoring. Changing
you don't make any changes to the code for the assessment type and scoring. Changing
...
@@ -291,7 +302,7 @@ The problem that includes this code has the following characteristics.
...
@@ -291,7 +302,7 @@ The problem that includes this code has the following characteristics.
assessment,
assessment,
Set the Assessment Type and Scoring
Set the Assessment Type and Scoring
###################################
************************************
#. In the component editor, locate the [tasks] tags.
#. In the component editor, locate the [tasks] tags.
...
@@ -302,8 +313,9 @@ Set the Assessment Type and Scoring
...
@@ -302,8 +313,9 @@ Set the Assessment Type and Scoring
.. _ORA Step 5:
.. _ORA Step 5:
============================
Step 5. Set the Problem Name
Step 5. Set the Problem Name
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
============================
.. note:: After you release your problem to students in your live course, make sure
.. note:: After you release your problem to students in your live course, make sure
you don't change the name of the problem. Changing the display name when the problem
you don't change the name of the problem. Changing the display name when the problem
...
@@ -331,8 +343,9 @@ To change the name:
...
@@ -331,8 +343,9 @@ To change the name:
.. _ORA Step 6:
.. _ORA Step 6:
=========================
Step 6. Set Other Options
Step 6. Set Other Options
^^^^^^^^^^^^^^^^^^^^^^^^^
=========================
If you want to change the problem settings, which include the number of
If you want to change the problem settings, which include the number of
responses a student has to peer grade and whether students can upload
responses a student has to peer grade and whether students can upload
...
@@ -436,8 +449,9 @@ Open response assessments include the following settings.
...
@@ -436,8 +449,9 @@ Open response assessments include the following settings.
.. _ORA Step 7:
.. _ORA Step 7:
========================
Step 7. Save the Problem
Step 7. Save the Problem
^^^^^^^^^^^^^^^^^^^^^^^^
========================
- After you have created the prompt and the rubric, set the assessment
- After you have created the prompt and the rubric, set the assessment
type and scoring, changed the name of the problem, and specified any
type and scoring, changed the name of the problem, and specified any
.. warning:: In peer assessments, the **due date** that you set for the subsection that contains the ORA problem is the date by which students must not only submit their own responses, but finish grading the required number of peer responses.
.. warning:: In peer assessments, the **due date** that you set for the subsection that contains the ORA problem is the date by which students must not only submit their own responses, but finish grading the required number of peer responses.
...
@@ -708,11 +727,13 @@ to see if any other students have submitted responses.
...
@@ -708,11 +727,13 @@ to see if any other students have submitted responses.
.. _ORA Access Scores:
.. _ORA Access Scores:
******************************
Access Scores and Feedback
Access Scores and Feedback
~~~~~~~~~~~~~~~~~~~~~~~~~~
******************************
=======
Scoring
Scoring
^^^^^^^
=======
Scores for open response assessment problems that have more than one assessment type are not cumulative. That is, if a problem has a value of 10 points and it contains both a self assessment and an AI assessment, the total score is out of 10 points rather than 20.
Scores for open response assessment problems that have more than one assessment type are not cumulative. That is, if a problem has a value of 10 points and it contains both a self assessment and an AI assessment, the total score is out of 10 points rather than 20.
...
@@ -737,9 +758,9 @@ Note that if the same 20-point problem changes the assessment order, the student
...
@@ -737,9 +758,9 @@ Note that if the same 20-point problem changes the assessment order, the student
.. note:: Because the assessment order can affect a student's final score, we recommend that you specify peer assessments as the last assessments for ORA problems. If the last assessment is a self assessment, the student can award herself full points for a response, even if the response received low grades from the AI and peer assessments.
.. note:: Because the assessment order can affect a student's final score, we recommend that you specify peer assessments as the last assessments for ORA problems. If the last assessment is a self assessment, the student can award herself full points for a response, even if the response received low grades from the AI and peer assessments.
================
Accessing Scores
Accessing Scores
^^^^^^^^^^^^^^^^
================
You access your scores for your responses to AI and peer assessment problems through the **Open Ended Console** page.
You access your scores for your responses to AI and peer assessment problems through the **Open Ended Console** page.
