Personal formative feedback

They need to make artifacts, like programs, databases, and lab reports. They might also finish artifacts. This happens with fading worked examples. Authors start with a complete worked example. Then they give students another one, with the last step missing. The students' task is to finish the artifact.

When we're talking about exercises here, we're talking about tasks like making artifacts. We're not talking about multiple-choice questions (MCQs). There's nothing wrong with MCQs. They're useful for activating prior knowledge, diagnosing misconceptions, testing fact recall, and other things. There is an MCQ learning object type in Skilling, and you should use it. However, MCQs are not enough by themselves. To learn skills, students have to make things, and turn them in.

Let's look at the components of the recommendation.

• Hands-on: students create artifacts, like Web pages, navigation bars, and so on. There can be multiple-choice quizzes and fill-in-the-blanks, but they are used to, for example, activate prior knowledge, or automate (in a psychological sense) partial tasks. A classic example of automation is the times table. Memorizing it reduces cognitive load when doing arithmetic in problem solving.
• Low-stakes: each exercise contributes a small amount of the student's grade. This reduces risk, so that students are more free to experiment.
• Many: rather than turning in, say, five artifacts, students turn in dozens. Courses can be designed so that each exercise requires a relatively small amount of new knowledge, keeping cognitive load within reasonable limits. If exercises are cumulative (which they would be in this course), knowledge learned early in the course is reinforced often.
• Formative feedback: instead of just a grade, students get a list of things they have done right and wrong for each exercise. This helps them identify and correct their mistakes.
• Personal: each student receives feedback about his/her own work.
• Resubmission: students get a chance to correct their mistakes. This gives them a reason to read the feedback.

The system streamlines assessment. First, authors of course material create rubrics for each exercise they write. Here is a screenshot of an exercise with rubric items.

This simple exercise has five rubric items. More complex exercises have fifteen or more.

Each rubric item has responses. Graders choose responses when assessing students' work on each rubric item. For example, here are responses for the "Dog page HTML is valid" rubric item:

For a particular student's submission, more than one response might be appropriate for the rubric item. For example, a student might have made two mistakes in their dog page HTML:

To summarize so far:

• Authors write exercises.
• They create rubrics for each exercise. The rubrics are reusable across exercises.
• Each rubric item has several responses. Each response is an evaluation of a student's work on that item. Multiple responses might be appropriate when evaluating a submission with a rubric item.

Here is part of the interface shown when starting to assess a student's work.

A grader (perhaps the instructor, perhaps someone else) clicks on a response for each rubric item. Here is the rubric item list after assessing a student's work:

Sometimes students' mistakes are hard to predict, so new rubric item responses can be added during grading. Clicking "Create message" generates a message summarizing the feedback.

Note that:

• Students get individual formative feedback.
• The "clickable response" system means that each submission can be assessed in a minute or two.
• Graders do not need to be experts, although they need to know enough to understand the rubric items and responses. Graders can be former students, house-bound workers, or anyone with minimum expertise, and access to the Internet.
• Feedback is standardized. The same items and responses are used across all sections, across time.