After your open response assessment assignment has been released, you can access information about the number of students in each step of the assignment or the performance of individual students. This information is available in the **Course Staff Information** section at the end of each assignment. To access it, open the assignment in the courseware, scroll to the bottom of the assignment, and then click the black **Course Staff Information** banner.
After your open response assessment assignment has been released, you can access information about the number of learners in each step of the assignment or the performance of individual learners. This information is available in the **Course Staff Information** section at the end of each assignment. To access it, open the assignment in the courseware, scroll to the bottom of the assignment, and then click the black **Course Staff Information** banner.
:alt: The Course Staff Information banner at the bottom of the peer assessment
...
...
@@ -15,7 +15,7 @@ After your open response assessment assignment has been released, you can access
View Metrics for Individual Steps
************************************************
You can check the number of students who have completed, or are currently working through, the following steps:
You can check the number of learners who have completed, or are currently working through, the following steps:
* Submitted responses.
* Completed peer assessments.
...
...
@@ -25,42 +25,42 @@ You can check the number of students who have completed, or are currently workin
To find this information, open the assignment in the courseware, scroll to the bottom of the assignment, and then click **Course Staff Information**.
The **Course Staff Information** section expands, and you can see the number of students who are currently working through (but have not completed) each step of the problem.
The **Course Staff Information** section expands, and you can see the number of learners who are currently working through (but have not completed) each step of the problem.
:alt: The Course Staff Information box expanded, showing problem status
.. _Access Information for a Specific Student:
.. _Access Information for a Specific Learner:
***********************************************
Access Information for a Specific Student
Access Information for a Specific Learner
***********************************************
You can access information about an individual student's performance on a peer assessment assignment, including:
You can access information about an individual learner's performance on a peer assessment assignment, including:
* The student's response.
* The peer assessments that other students performed on the student's response, including feedback on individual criteria and on the overall response.
* The peer assessments that the student performed on other students' responses, including feedback on individual criteria and on the overall responses.
* The student's self assessment.
* The learner's response.
* The peer assessments that other learners performed on the learner's response, including feedback on individual criteria and on the overall response.
* The peer assessments that the learner performed on other learners' responses, including feedback on individual criteria and on the overall responses.
* The learner's self assessment.
In the following example, you can see the student's response. The response received one peer assessment, and the student completed a peer assessment on one other student's response. The student also completed a self assessment.
In the following example, you can see the learner's response. The response received one peer assessment, and the learner completed a peer assessment on one other learner's response. The learner also completed a self assessment.
.. image:: /Images/PA_SpecificStudent.png
:width: 500
:alt: Report showing information about a student's response
:alt: Report showing information about a learner's response
For an example that shows a student's response with more assessments, see :ref:`Access Student Information`.
For an example that shows a learner's response with more assessments, see :ref:`Access Learner Information`.
Accessing information about a specific student has two steps:
Accessing information about a specific learner has two steps:
#. Determine the student's course-specific anonymized ID.
#. Access information for that student.
#. Determine the learner's course-specific anonymized ID.
To determine a student's course-specific anonymized ID, you'll need two .csv spreadsheets from the Instructor Dashboard: the grade report (**<course name>_grade_report_<datetime>.csv**) and the list of course-specific anonymized student IDs (**<course name>-anon-ids.csv**).
To determine a learner's course-specific anonymized ID, you'll need two .csv spreadsheets from the Instructor Dashboard: the grade report (**<course name>_grade_report_<datetime>.csv**) and the list of course-specific anonymized learner IDs (**<course name>-anon-ids.csv**).
#. In the LMS, click the **Instructor** tab.
#. On the Instructor Dashboard, click **Data Download**.
...
...
@@ -72,41 +72,41 @@ To determine a student's course-specific anonymized ID, you'll need two .csv spr
.. note:: Generating a grade report for a large class may take several hours.
5. When the link to the grade report appears in the **Reports Available for Download** list, click the link to open the spreadsheet.
#. When you have both spreadsheets open, view the **<course name>_grade_report_<datetime>.csv** spreadsheet. Locate the student that you want by username or e-mail address. Make a note of the number in the ID column (column A) for that student. In the following example, the student ID for e-mail address ``amydorrit@example.com`` (username ``lildorrit``) is ``18557``.
#. When you have both spreadsheets open, view the **<course name>_grade_report_<datetime>.csv** spreadsheet. Locate the learner that you want by username or e-mail address. Make a note of the number in the ID column (column A) for that learner. In the following example, the learner ID for e-mail address ``amydorrit@example.com`` (username ``lildorrit``) is ``18557``.
.. image:: /Images/PA_grade_report.png
:width: 500
:alt: Spreadsheet listing enrolled students and grades
:alt: Spreadsheet listing enrolled learners and grades
7. Go to the **<course name>-anon-ids.csv** spreadsheet, locate the user ID that you noted in step 6, and then copy the value in the "Course Specific Anonymized user ID" column (**column C**) for the user. The value in column C is the student's anonymized user ID for the course. In the following example, the anonymized user ID for student ID ``18557`` is ``ofouw6265242gedud8w82g16qshsid87``.
7. Go to the **<course name>-anon-ids.csv** spreadsheet, locate the user ID that you noted in step 6, and then copy the value in the "Course Specific Anonymized user ID" column (**column C**) for the user. The value in column C is the learner's anonymized user ID for the course. In the following example, the anonymized user ID for learner ID ``18557`` is ``ofouw6265242gedud8w82g16qshsid87``.
.. image:: /Images/PA_anon_ids.png
:width: 500
:alt: Spreadsheet listing students' anonymous user IDs
:alt: Spreadsheet listing learners' anonymous user IDs
.. note:: Make sure that you don't copy the value in column B. You need the *course-specific* anonymized user ID from **column C**.
.. _Access Student Information:
.. _Access Learner Information:
=======================================
Access the Student's Information
Access the Learner's Information
=======================================
#. In the LMS, go to the peer assessment assignment that you want to see.
#. Scroll to the bottom of the problem, and then click the black **Course Staff Information** banner.
#. Scroll down to the **Get Student Info** box, paste the student's course-specific anonymized user ID in the box, and then click **Submit**.
#. Scroll down to the **Get Learner Info** box, paste the learner's course-specific anonymized user ID in the box, and then click **Submit**.
The student's information appears below the **Get Student Info** box.
The learner's information appears below the **Get Learner Info** box.
The following example shows:
* The student's response.
* The learner's response.
* The two peer assessments for the response.
* The two peer assessments the student completed.
* The student's self assessment.
* The two peer assessments the learner completed.
* The learner's self assessment.
For a larger view, click the image so that it opens by itself in the browser window, and then click anywhere on the image that opens.
.. image:: /Images/PA_SpecificStudent_long.png
:width: 250
:alt: Report showing information about a student's response
\ No newline at end of file
:alt: Report showing information about a learner's response
In this document, we describe the architecture for:
* Training a classifier using a supervised machine learning algorithm.
* Grading student essays using a trained classifier.
* Grading learner essays using a trained classifier.
Both training and grading require more time than is acceptable within the
request-response cycle of a web application. Therefore, both
...
...
@@ -31,11 +31,11 @@ Requirements
------------
* Grading tasks *must* be completed within hours after being scheduled.
Ideally, the delay would be within several minutes, but students could
Ideally, the delay would be within several minutes, but learners could
tolerate longer delays during periods of high usage or failure recovery.
The AI Grading API does not implement deadlines, so if a submission
is submitted for grading (allowed when the problem is open),
the student will receive a grade for the AI assessment step.
the learner will receive a grade for the AI assessment step.
* Grading task queues must tolerate periods of high usage,
as the number of submissions will likely increase when
...
...
@@ -63,7 +63,7 @@ Entities
* **AI Grading API**: An API that encapsulates all interactions with AI-grading database models and the task queue. All inputs and outputs are JSON-serializable, so the API calls can be made in-process (likely the initial implementation) or through the network.
* **Submission**: An essay submitted by a student to a problem in a course.
* **Submission**: An essay submitted by a learner to a problem in a course.
* **Assessment**: Specifies the scores a submission received for each criterion in a rubric.
...
...
@@ -129,9 +129,9 @@ Parameter: AI Grading Workflow ID
Procedure:
1. A student submits an essay, creating a **submission** in the database.
1. A learner submits an essay, creating a **submission** in the database.
2. The student updates the workflow, and the **Workflow API** uses the **AI Grading API** to:
2. The learner updates the workflow, and the **Workflow API** uses the **AI Grading API** to:
a. Retrieve the most recent **ClassifierSet** for the current rubric definition (possibly none if training hasn't yet finished).
b. Create an **AI Grading Workflow** record in the database, associated with a Submission ID and **ClassifierSet**.
...
...
@@ -141,8 +141,8 @@ Procedure:
a. Retrieve the submission and classifiers from persistent storage or a cache.
i. If the **ClassifierSet** is null, then the classifier wasn't available when the student created the submission.
ii. Since we cannot grade the student without a classifier, we create the **AI Grading Workflow** record but do not schedule the **Grading Task**. This means that the workflow will not be marked complete.
i. If the **ClassifierSet** is null, then the classifier wasn't available when the learner created the submission.
ii. Since we cannot grade the learner without a classifier, we create the **AI Grading Workflow** record but do not schedule the **Grading Task**. This means that the workflow will not be marked complete.
iii. When a **Training Task** completes, update incomplete **Grading Tasks** with null **ClassifierSets** with the newly created **ClassifierSet**, then schedule the **GradingTasks**.
b. **Optimization**: Check whether a completed **AI Grading Workflow** exists for this submission using the same **ClassifierSet**.
...
...
@@ -156,7 +156,7 @@ Procedure:
e. Create an **AssessmentPart** for each rubric criterion, containing the score assigned by the classifier for that criterion.
f. Mark the **AI Grading Workflow** as complete by associating the **Assessment** with the workflow.
4. When a student checks the status of the submission, the **AI Grading API**:
4. When a learner checks the status of the submission, the **AI Grading API**:
a. Queries the database for the latest **AI Grading Workflow** matching the submission.
b. Reports whether the workflow is started or complete.
ORA2 supports AI assessment for student responses, but currently does not support authoring of AI problems. In order to migrate an existing AI assessment problem into ORA2, you will need to:
ORA2 supports AI assessment for learner responses, but currently does not support authoring of AI problems. In order to migrate an existing AI assessment problem into ORA2, you will need to:
1. Create a problem with example-based assessment enabled.
...
...
@@ -53,4 +53,4 @@ ORA2 supports AI assessment for student responses, but currently does not suppor
.. image:: course_staff_ai.png
3. At this point, students can submit essays and receive grades.
3. At this point, learners can submit essays and receive grades.
"errors":["If your assignment includes a student training step, the rubric must have at least one criterion, and that criterion must have at least one option."]
"errors":["If your assignment includes a learner training step, the rubric must have at least one criterion, and that criterion must have at least one option."]
{% if 'example_based_assessment' in editor_assessments_order %}
{% trans "In this assignment, you can include steps for student training, peer assessment, self assessment, and example based assessment. Select the steps that you want below, and then drag them into the order that you want. If you include an example based assessment step, it must precede all other steps. If you include a student training training step, it must precede peer and self assessment steps." %}
{% trans "In this assignment, you can include steps for learner training, peer assessment, self assessment, and example based assessment. Select the steps that you want below, and then drag them into the order that you want. If you include an example based assessment step, it must precede all other steps. If you include a learner training training step, it must precede peer and self assessment steps." %}
{% else %}
{% trans "In this assignment, you can include steps for student training, peer assessment, and self assessment. Select the steps that you want below, and then drag them into the order that you want. If you include a student training step, it must precede all other steps." %}
{% trans "In this assignment, you can include steps for learner training, peer assessment, and self assessment. Select the steps that you want below, and then drag them into the order that you want. If you include a learner training step, it must precede all other steps." %}
{% trans "Enter feedback text that students will see before they enter their own feedback. Use this text to show students a good example peer assessment." %}
{% trans "Enter feedback text that learners will see before they enter their own feedback. Use this text to show learners a good example peer assessment." %}
<pid="student_training_description_closed"class="openassessment_description_closed {% if assessments.student_training %} is--hidden {% endif %}">
{% trans "Students learn to assess responses by scoring pre-assessed sample responses that you provide. Students move to the next step when the scores they give match your scores. Note that if you add this step, you must also add a peer assessment step. This step must come before the peer assessment step." %}
{% trans "Learners learn to assess responses by scoring pre-assessed sample responses that you provide. Learners move to the next step when the scores they give match your scores. Note that if you add this step, you must also add a peer assessment step. This step must come before the peer assessment step." %}
</p>
<divid="student_training_settings_editor"class="assessment_settings_wrapper {% if not assessments.student_training %} is--hidden {% endif %}">
<h3class="title">{% trans "Student Response" %}</h3>
<h3class="title">{% trans "Learner Response" %}</h3>
<divclass="student__answer__display__content">
{% if workflow_cancellation %}
{% blocktrans with removed_by_username=workflow_cancellation.cancelled_by removed_datetime=workflow_cancellation.created_at|utc|date:"N j, Y H:i e" %}
Student submission removed by {{ removed_by_username }} on {{ removed_datetime }}
Learner submission removed by {{ removed_by_username }} on {{ removed_datetime }}
{% endblocktrans %}
<br>
<!-- Comments: Reason for Cancellation-->
...
...
@@ -21,7 +21,7 @@
Comments: {{ comments }}
{% endblocktrans %}
{% else %}
{% include "openassessmentblock/oa_submission_answer.html" with answer=submission.answer answer_text_label="The student's response to the question above:" %}
{% include "openassessmentblock/oa_submission_answer.html" with answer=submission.answer answer_text_label="The learner's response to the question above:" %}
{% endif %}
</div>
...
...
@@ -55,7 +55,7 @@
<li>
<divclass="has--warnings">
<divclass="warning">
{% trans "Caution: Removing a student's submission cannot be undone." %}
{% trans "Caution: Removing a learner's submission cannot be undone." %}
gettext("You've added a criterion. You'll need to select an option for the criterion in the Student Training step. To do this, click the Settings tab.")
gettext("You've added a criterion. You'll need to select an option for the criterion in the Learner Training step. To do this, click the Settings tab.")
gettext("You've deleted an option. The system has removed that option from its criterion in the sample responses in the Student Training step. You may have to select a new option for the criterion.")
gettext("You've deleted an option. The system has removed that option from its criterion in the sample responses in the Learner Training step. You may have to select a new option for the criterion.")
gettext("You've deleted all the options for this criterion. The system has removed the criterion from the sample responses in the Student Training step.")
gettext("You've deleted all the options for this criterion. The system has removed the criterion from the sample responses in the Learner Training step.")
