Tag: rubrics

Posted on

The Benefits of Rubrics

Today we have a guest post from Sarah E. James (sarah [dot] james [at] g [dot] harvard [dot] edu), Colin Brown (colin [dot] brown [at] northeastern [dot] edu), and George Soroka (soroka [at] fas [dot] harvard [dot] edu.)

At the 2020 APSA Teaching and Learning Conference, we led a workshop on writing rubrics for the political science classroom. Rubrics may not be a cure-all for student writing, but we do believe that college faculty underestimate their benefits for teaching advanced writing skills. They can also be a powerful tool for developing a mindful teaching practice.

The use of rubrics is extremely common at the K-12 level (at least here in the U.S.), and there is considerable evidence to support their effectiveness at teaching beginning writing skills (Heidi Andrade’s work is a good starting point, for example, here, here, and here). There has been less evidence for their effectiveness at the university level, but the few existing studies point in the same general direction as the elementary ed studies:

  • Rubrics help students learn specific skills, and make them aware of their own learning processes.
  • Rubrics make grading more efficient, while simultaneously increasing student perceptions of how fair the grading is.
  • Rubrics help expose aspects of the “hidden curriculum” in our writing expectations and may help the most disadvantaged writers the most.

Our key takeaway for participants in our workshop: rubrics let you deal with subjective measures, but in a transparent way, and without being arbitrary. Generating a good rubric requires you to be clear about what you actually value, and on what you expect students to be able to demonstrate. From the students’ side, this is a clear signal of where you want them to spend most of their time. From the instructor’s side, this is a good way to make sure that you’re following the adage of “if you didn’t teach it, don’t test it.” And when we think of the kind of genre-specific writing skills we demand of students, this sort of clarity can be extremely helpful for students who may “write well” in a general sense but who may have no experience in how things like evidence, counterarguments, citations, and literature reviews work in political science specifically.

Rubrics can only capture so much, and when you use one, you are limited to only a certain number of skills or aspects in your assessment. At our TLC workshop, the most common concern our participants had was, “what happens if we end up having to give a good grade to a bad paper?” This is a (small) risk, but we encouraged our participants to step back for a second and think about the rubric as a mindful teaching exercise. If a paper feels like it should get a lower grade than the rubric suggests, are there skills that should be included explicitly in the rubric? (They can be added next time!) If not, then what’s causing you to think the grade should be lower—and is it really something that should be entering into your assessment?

For those interested in designing their own rubrics, we provided examples for an introductory and an upper-level course, as well as a worksheet to help in setting it up. Our examples are designed to focus much more on discipline-specific skills (using evidence, critical thinking, professional norms) than on the quality of prose itself, and our instinct (tested to limited effect in our JPSE article) is that this is the most productive use of rubrics in the college-level classroom. But the structure of rubrics allows them to be adapted to the instructor’s aims, whatever they are—and they force the instructor to make those aims clear to themselves and to their students.

Posted on

Pre-Post Post-Its 3: Rubrics or Goobrics?

This post was inspired by an accidental encounter with an assignment rubric created by someone, or a committee, at my university. I do not know if the rubric is being used. I hope not, because it has 5 rows X 5 columns that contain a total of 860 remarkably ambiguous words, followed by a 362 word explanation — nearly four pages of information. No student is going to wade through such complex and lengthy verbiage for guidance on their work.

In contrast, I use a 3 X 4 rubric with 112 words. Much easier for students to decipher, but just as pointless if students don’t read it given that it is attached to about two dozen course assignments. So I decided to find out whether students do read it, with a third Post-It note survey, comprised of these items:

  • How do I feel right now?
  • I have read the rubric: a) before starting assignments, b) after receiving a grade for an assignment, c) both a and b, or d) I have never looked at the rubric.
  • For your answer to the previous question, why?

Similar to the previous Post-It note surveys, sixteen out of the twenty-one students present, or nearly 80 percent, stated that they felt badly in the one-word check-in. Twelve of these students wrote that they were tired. Three had neutral responses, and only two reported positive feelings. I’ll write more about this recurring problem in a future post.

For the second question, only two students commented that they had never read the rubric, but one of them wrote that they had already taken one of my courses and were familiar with it — so in effect 95 percent of the students who were in the classroom that day had used the rubric at some point. Nine students wrote that they read the rubric before starting to write, while ten students said that they had read the rubric both before and after completing an assignment.

Students’ answers to the last question included statements such as:

  • “So I know what to expect when my work is graded.”
  • “Wanted to know before [starting the assignment] how to do it, and after to see if I did anything wrong.”
  • “To make sure I can get the full amount of points.”
  • “That’s what you’re supposed to do.”

This is all good — the rubric seems to be serving its intended purpose and is not wasted effort or based on an incorrect assumption on my part.

Posted on

A Basic Mistake

Back in July I wrote about a new writing assignment that I would be inserting into the undergraduate course that I’m now teaching. As is my standard practice, the directions for the assignment included a statement that students should refer to the rubric accompanying the assignment for an indication of how their work would be evaluated. I also instructed students to write about some of the concepts discussed by Tomer Perry in his series on best practices in game design (I put links to the relevant blog posts in the assignment directions).

Here is my mistake: the rubric did not have any criteria specific to the game design principles that students were supposed to address. When grading students’ essays, I didn’t think it was fair of me to penalize them for not writing about something that I had failed to include in the rubric. This was a reminder of a lesson I have learned repeatedly over the years but occasionally forget.

Posted on

Abandon: Fall 2017 edition

As promised in my last post about teaching risk-averse students, I am going to again apply Simon’s ABC technique to last semester’s teaching. And since I taught two sections of my first-year seminar, I’ll focus on that.

First item on the “abandon” list: in-class peer review of student-designed games. Although I think the rubric that students use to evaluate classmate-designed games is good, they simply refuse to use it to provide honest feedback. I know that the majority of the students understand at least some of the principles reflected by the rubric because of the way they have analyzed the games in the final exam. In the classroom, though, they rate the games as perfect. A potential replacement for the peer review process — and this is really more of a “begin” item — is a short writing assignment after each round of game design in which they compare the game their team designed with another team’s game that they played in class.

Second thing to abandon: my organization of memo-writing assignments. I have assumed, incorrectly, that first-semester college students can grasp the purpose and format of a memo with minimal instruction on my part. After three separate iterations of the assignment, complete with an opportunity to rewrite each memo, I didn’t see significant improvement in the quality of students’ work, which was the same thing that happened in the course last year. A possible solution is to walk students step by step through the mechanics of writing a memo in class, so that by the end of the process they have in their hands a  document that they can submit for a “perfect” grade. But this would remove pretty much any opportunity for students to independently engage in creative thinking, which is another term for problem solving. More holding of students’ hands to protect them from anything they might find unpleasant. I’ll have to think more about how to better organize an assignment like this.

Third item on the list, which is speculative at this point: abandon the whole course. I’ve been teaching this first-year seminar since its inception four years ago, when it came into being through a revision to my university’s general education requirements. The developmental rationale for the course is not well-defined, and the learning outcomes associated with it are mostly not measurable. Valid data on how the course may be of benefit to students simply isn’t being collected, which means that it is really nothing but an empty three-credit curricular requirement. While I think the topic on which I have built the course is a good one, I am finding it less enjoyable to teach over time. And interaction with university administrators about aspects of teaching it have been less than satisfactory. So, if I have the opportunity in future fall semesters not to teach the course, I might take it.

Posted on

Making Students Make: Classroom Game Design Part 3

As promised, here is information about the final exam I have created for my first-year seminar. As I mentioned in my previous two posts in this series, my goal is to make students more aware of what is now often referred to as “design thinking”: in this course, they are not just learning how to build games, but to solve problems.

Instructions for the exam:

Write an essay that compares one of the games that you helped design and one of the games that you played in this course. Identify which game was better by analyzing how well the games:

  • Incorporated the principles of the “golden rule” and “magic circle” in relation to events in the real world.
  • Utilized elements of conflict and cooperation.
  • Had clear written and unwritten rules.
  • Facilitated meaningful decision making.

Use information from assigned course readings to support your argument. Be sure to include references to your sources in your essay to avoid plagiarism – this applies whether you are using direct quotations or just the ideas of another author. Use parenthetical in-text citations as with reading responses to save space. An essay with few or no relevant references to the course readings will suffer a reduction in grade.

  • The essay should be double-spaced, in 11 or 12 point font, and the equivalent of 3-4 pages long.
  • There is no need for a separate bibliography or title page; please do not include them.
  • Work independently; do not discuss your essay with other students.

A screenshot of the rubric I’ll be using to grade the exams is below. As I mentioned about the game design beta test rubric, I am not really concerned with the exam scores generated by the rubric — my primary goal is getting students to become more aware of how their experiences translate into learning.

Posted on

Making Students Make: Classroom Game Design Part 2

This is the second post on my process for game design by students. The first post, where I mention changing the beta test rubrics with which teams of students evaluate each other’s games, is here. The rubric originally looked like this:

Ok, but not great.

My new rubric more clearly references the principles discussed by Tomer Perry in his series on game design:

What is most important here is not the rubric’s criteria or point values, but the fact that it will be used by students rather than the instructor. The purpose of the rubric is to get students to benchmark the quality of their effort against that of their peers, and possibly make them into more autonomous learners in the process.

Posted on

Specification Grading In An Online Course

Today we have another post by guest contributor William R. Wilkerson, Professor of American Government and Politics at SUNY-Oneonta. He can be reached at bill [dot] wilkerson [at] oneonta [dot] edu.

This summer for the first time I am teaching an online version of my judicial process course, Law Courts and Politics. I adopted a specifications grading system, something that has been discussed by people like Linda Nilson at Inside Higher Ed and Amanda Rosen on this blog. With specifications grading all assignments are graded on a satisfactory or unsatisfactory basis and course grades are based on assignment bundles.

My course is five weeks long with a distinct theme for each week’s lesson. Each lesson includes an online quiz made up of multiple choice and short essay questions on the textbook (Corley, Ward and Martinek’s American Judicial Process ), various discussion topics on the text, other assigned readings, video and audio, as well as a 600-750 word writing assignment. Each of these elements—quizzes, discussion, and the writing assignment, along with a summative assignment for those wishing earn a B or an A—are tied to course learning objectives. The grade bundles are as follows: Continue reading “Specification Grading In An Online Course”

Posted on

Specifications Grading, Attempt 1, Day 0

Hello, ALPS readers! I’m back after a long summer and spring sabbatical, and am eager to get back in the classroom and talk all things pedagogy here on ALPS. I’m starting a new series where I outline in excruciating detail my experiences using Specifications Grading.  I’ll be sharing my materials, talking about the ups and downs, and reflecting on this unique grading system throughout the semester.

We’ve given quite a bit of attention to specifications grading in the past few months. I did a presentation on it at the ALPS workshop at the University of Surrey in May as I started working on adapting one of my own courses to this new system. I also consulted several former students and children-of-friends about what they thought of the system in abstract, and the general consensus ranged from “shrug” to “that might be cool.” Experts in analysis, my young consultants.

In a nutshell, Specifications Grading is a system where all assignments are clearly linked to course learning outcomes, given clear specifications on what students need to do to earn a passing mark, and graded on a pass/fail style system, where a pass is a high bar (typically a B). Assignments are bundled together by learning outcome, and course grades are assigned based on the bundles that students complete. So, higher grades go to students that either complete more bundles (achieving more learning outcomes) or higher-level bundles that demand students complete more complex work. The course also employs flexibility mechanisms such as tokens to let students revise or reattempt a failing assignment, forgive a course absence, or gain some other kind of benefit. This system is supposed to ensure that all students who pass the class are achieving the minimum learning outcomes for the course, but also puts their grade into their hands by removing the mystery behind grades (no longer 170 out of 200 points, but ‘excellent’ ‘satisfactory’ or ‘unsatisfactory) and letting them choose which grade bundle to achieve.

Check out our previous posts for more general information on Specs Grading, or check out this great community of scholars working with the system.. For this new series, I am going to write throughout the semester about my experience in adapting and teaching my research methods course to this system.

Continue reading “Specifications Grading, Attempt 1, Day 0”

Posted on

Teaching Methods: Reliability and Validity at the AP Reading

Last month I participated in the annual Reading of the Advanced Placement US Government exams. In May, over 300,000 students in US high schools took this exam, consisting of a series of sixty multiple choice questions and four free responses, usually in the hope of earning college credit or placement in a higher level course. The multiple-choice items are easily dealt with, but someone has to score the free response questions. Enter the AP Readers, a collection of several hundred college professors and high school teachers who voluntarily spend a week sitting in a convention center, scoring essays for eight hours a day, seven days straight. Sound awful? It gets worse: we actually score responses to the same essay question, over and over again. On a good day, I score about 450 essays, all answering the same question.

So why have I put myself through this tedious exercise for nine of the last ten years?

Continue reading “Teaching Methods: Reliability and Validity at the AP Reading”

Posted on

Grading on Spec III

The first-year seminar I mentioned in my last post will have twenty-two students; each will be (or should be) submitting at least thirty-five reading responses during the semester. While the reading responses make students read and write about what they’ve read, I have to grade them within twenty-four hours of submission if my feedback is going to have any effect. To make grading these writing assignments as easy as possible, I’ve used a form of specifications grading with a 2 X 2 rubric.

I’ve noticed that any long-form comments I make about what a student submits — besides compliments like “Good work” — almost always relate to mechanical errors in the student’s writing. I see spelling mistakes, disagreement between the singular and plural, incorrect verb tenses, and other problems that detract from the ideas students are trying to express. My rubric hasn’t included a criteria for writing mechanics, hence the perceived need for me to type something in the tiny comment box when grading.

I’ve decided to add that criteria to the rubric to further reduce the amount of time spent grading. The rubric now looks like this:

30 point rubric

Since each reading response will be worth 30 points rather than 20, I will probably also boost my grading scale for the course from 1,000 to 2,000 points.

Previous posts on specifications grading here, here, and here.