Assessing Student Preparation

This is follow-up to a previous guest post by Joel Moore on the benefits of assessing students’ preparation for class discussion rather than their participation in it. Students are asked to rate themselves on the amount of assigned reading they have completed prior to class, and this forms the basis of their preparation grade.

He has created a web app that simplifies the process. The app is available at https://joeldmoore.com/apps/preparation/.

A video that discusses how to use the app is at https://www.youtube.com/watch?v=SQdK1FwycKw.

Transparent Teaching in Action: Sometimes you aren’t as good as you think you are

In the last few months I’ve been interested in transparent teaching. Basically, transparency is a commitment to all of your students to be transparent about your expectations for a class, particularly when it comes to course assignments. Transparent teaching requires being clear and precise about 3 things in an assignment:

  1. Purpose–explain to the students WHY you are asking them to do this assignment.  Don’t just assume they know how it connects to the material.  Talk to them about the skills and knowledge they will gain and how that is relevant to the course, major, program, or other aspects of their lives.
  2. Tasks–be explicit about the tasks students must follow to complete the assignment.  Sometimes we tell our students to ‘write a 5 page essay’ but don’t realize that many of our students might not know what we mean by an essay–it means different things in different fields, and not all of your students will have taken a polisci class before. Of the three areas, this is where we tend to be the most detailed in our assignments, but even here we can probably be more clear about exactly what we want than we area.
  3. Criteria for Success–the students need to know how their work will be evaluated to determine whether they have successfully completed the assignment or not.  This might mean a rubric, or just a list of what you are going to be looking at.  But one of the most important elements–and the one I messed up on–is it provide examples of successful work.  This might be student work, or something written by a professional working in the field. Want students to write a strong literature review? Show them what a strong one looks like, and talk to them about what makes it strong.  Contrast it with unsuccessful or weak work.  Better yet, give them the examples and the rubric, and have THEM score the work, so they understand how the criteria is applied.

Yes, all of that takes time.  But we owe it to our students to give them every chance to achieve success in the work we assign them.  If we think it is valuable for them to do this work, then we need to give them the detail and time it takes so that success is entirely in their hands.  Plus, doing this has wider benefits.

Research by the Transparency in Learning and Teaching (TILT) team at the University of Nevada in Las Vegas has shown that increasing transparency in our assignments doesn’t just impact their work on those assignments. It leads to big gains in student confidence and sense of belonging in college, particularly for disadvantaged populations who come to college less prepared.  At UNLV, they saw a 20% increase in retention for these students who participated in classes where just two assignments were transformed to become more transparent. I led a project at my university last spring on transparency, and while I’m still evaluating the data, those who participated indicated they found the training sessions and transformation process very useful.  If you want to attend a training on TILT, review their extensive resources, or read the ongoing research, head to their Resources page.

So here I am, someone who has trained on transparent teaching, conducted multiple trainings myself, and doing research in the area.  I’ve also used specifications grading in the past, which is in the transparent teaching wheelhouse.  I’m a pro, right?

Yeah, right.  Every time I think I’ve ‘got’ something when it comes to teaching, I end up learning how much more I have to learn.

I’m teaching a new course this semester called Sex, Marriage, and Violence, and I’m running it largely as a seminar.  Students have to write weekly papers on the readings and pose discussion questions that form the basis for class.  In the syllabus I included a purpose statement for these papers as well as a rubric, and the instructions (I thought) were clear: I wanted a full APA style bibliographic entry for each article or chapter, a 1-2 sentence summary of their main claims or findings, a 500-1000 word analysis of the themes, debates, gaps or issues raised by the readings for the week, and 1-2 discussion questions.

The first papers showed up on Monday, and all but one of them failed to follow this format.  Only a few did the bibliographic entries or summaries; some put the summaries in the main body of the paper (something I explicitly said not to do); only a handful gave discussion questions at the end; and a few wrote essentially annotated bibliographies, with no effort to compare or synthesize the readings.

A couple of students messing up is probably their fault; almost everyone messing up is clearly mine.

The good news is that this is fixable.  Because this is an assignment that starts early and repeats weekly, I was able to catch the problem in week 2.  This is an advantage of using smaller stakes assignments throughout the class–there is less harm to the student if they are completely off base in their efforts, and you have a chance to teach them how to improve. 

To fix this, I recognized a key mistake on my part: while I had provided a rubric, I had not provided an example of successful work.  So I drew up a 2 page outline of a paper.  I didn’t write an entire paper myself.  But the outline showed them the formatting I wanted to see in the paper, such as showing them that I wanted the bibliographic entires and summaries BEFORE the paper proper, and the discussion questions AFTER, not embedded in the text.  And I wrote an example of a thesis statement based on the readings for Week 2 along with a single paragraph showing them how to analyze one of the readings according to that theme.  Drawing up this document took me about an hour, but the students SO appreciated it.  They now had a visual aid to see what I wanted with examples, not just a list of requirements and criteria.  I also allowed them to resubmit their paper by the next class, and throughout the course, will ask students to keep anonymized versions of their work so I will have student-written examples to show future classes.

All of this goes to show that even when you are trying to be transparent, sometimes you might not be able to anticipate what kind of information or models the students need to really understand what you want them to do. When that happens, it is typically OUR fault as instructors, not theirs as students. Now, some students will get it wrong no matter how clear you are, but there are definitely times when we are wrong when we think we are being clear.  I’ve sometimes thought that the ‘provide examples of successful work, and show students why it is successful’ is optional, but on reflection I think this is perhaps the most important thing we can do*. This was an important wake-up call for me that even when I think I’m doing a good job, I can still get it wrong. 

*There are those who question providing examples of successful work lest students simply copy it.  I completely understand the critique, but i think with creativity on our part, this can be overcome. Often I am demonstrating formatting and technique, not content, so as long as you change the content out, there’s little risk of copying.  My go-to example is to make arguments about why cats are better pets than dogs, and I can illustrate the expected structure of an essay, a bibliographic entry or citation, a thesis statement, and using evidence to defend a claim using this very non-political example.

Another point here is that providing a structural template for students who aren’t strong writers is super useful to them.  I’m fine with them imitating my structure–I’m teaching them how to write a strong paper.  I encourage students who have more advanced writing skills to branch away from the suggested structure in my outline/template to find their own style.  

Changing a Course on Development, Part 8

I want my final exam for this course to be its pièce de résistance — a vehicle for students to demonstrate how well they can apply their knowledge about the relationships between economics and the environment. I also want the application of knowledge to happen in an authentic, real-world context, where writing has a clearly-defined role, audience, purpose, and format. So here is the exam:

A Plan for Louisiana’s Future

Role

You are the Director of the Office of Planning and Budget for the State of Louisiana.

Audience

The governor of Louisiana

Purpose

Recommend to the governor whether Louisiana should either:

  1. Raise taxes to build the southern part of the state to a 10,000 year flood standard, or
  2. Stop all public infrastructure spending in areas unprotected by existing levees.

These are your only policy options. Write a 2-3 page rationale for choosing one of them. Discuss why your choice is economically best for the state. For evidence in support of your rationale, refer to relevant course readings and Continue reading

Additional Snippets from Comparative Politics

A few more quick thoughts about my comparative politics course, which just ended . . .

On the last day of class, I distributed pieces of paper and asked students to write down what they thought were the two best and two worst assigned readings, and to include brief explanations of their choices. Nine students were in the room — out a class of ten. Despite such a small sample, I will go out on a limb and draw some conclusions, all of which relate to transparency:

  • I should preface each writing assignment with a very brief overview of how assigned readings relate to the topic at hand, because students don’t automatically know this. For the question “Does Chinese culture promote authoritarianism?” the class read an interview with Singapore’s former leader Lee Kuan Yew. Two of the students didn’t understand that Yew, being of Chinese ancestry like many other Singaporeans, was discussing his views on Chinese cultural norms.
  • I should also explicitly inform students that readings are written in different styles for different audiences, which means that some of the readings will be more difficult and less enjoyable to read than others. For example, in the section of the course on revolution, I assign Theda Skocpol’s 1976 article “France, Russia, China: A Structural Analysis of Social Revolutions.” Two students said this article was a least favorite reading, because of its style. Oddly, in my opinion, a few other students complained that the long-form journalism found in some of The New York Times articles was hard to understand.
  • Students have difficulty placing unfamiliar historical events in chronological order and parsing out how what happened before might have affected what happened after (the “history just happens” mentality). For an assignment about Iran, students felt confused about the relationship between the 1979 revolution, popular support for the post-revolutionary Islamic Republic, and more recent anti-government protests. While I find it easy to connect the dots, students don’t — for them, context can be confusing. So for some assignments I need to narrow down the reading list.

After students finished identifying readings they liked or disliked, I asked them if they would have preferred formal mid-term and final exams, or regular quizzes, as an alternative to some of the writing assignments. I was surprised that they said no, since I had been under the impression that they used test scores to reassure themselves of their performance. One senior said that he didn’t remember a thing from test-based courses he had taken, while he remembered a lot from writing-based courses, and therefore he regarded the latter as much more beneficial. Two others said that, as studies have shown, they and their friends promptly forgot whatever facts they dumped into short-term memory the night before an exam, so they regarded exams as anxiety-producing events irrelevant to their learning. If next year I use quizzes as a stick to improve attendance in class, they will probably need to be the “write a paragraph that answers the question” type.

 

The Not-So-Pop Quiz and Absent Students

More thoughts on the management of extremely small classes . . . especially those that convene at 8:00 a.m. Mine has only ten students, and recently attendance has sometimes been as low as 40 percent, which empirically demonstrates that a substantial portion of the class thinks regularly attending class is unimportant.

I am not one of those professors who penalizes students’ grades for non-attendance — my students are legal adults with their own lives and get to suffer the consequences of their decisions. I also don’t give many in-class exams. For the subjects that I teach, I believe that writing is a better assessment tool than tests composed of multiple choice questions.

I also believe that reading and writing before class, and discussion during class, are in and of themselves critical for learning. But, to borrow a phrase from Lenin, what is to be done?

Last night I remembered the physics exams I had as an undergraduate. Often students were allowed to bring along a single sheet of paper with notes written on it, to reference during the exam. The real purpose of the “crib sheet” was to get students to review their notes and identify for themselves the concepts and processes that were important for an understanding of physics. Also the act of writing and rewriting to make the most critical information fit on a single piece of paper helped strengthen memories of this information.

I decided to employ a similar method. This morning I announced that class on Tuesday of next week would include a quiz. I explained in detail the format of the quiz, its intent, and the fact that it would be an extremely low-stakes event — worth only 10 points out of an available total of 1,200 from all other assignments. I gave each student a blank 3 X 5 note card to write notes on; they can use the card during the quiz.

The downside is that I now have to create and grade a quiz. But even if the entire class decides to show up, I will still have only ten quizzes to grade. A reasonable trade-off, in my opinion.

I’ll report on how this goes next week.

Changing a Course on Globalization, Part 1

The first of what promises to be a semester-long series on granular-level changes to my globalization course, in which students will be partnering with a local non-profit organization:

Historically my students have been reluctant to evaluate their performance in relation to that of their peers, whether via Monopoly money, rubrics, or anonymous online rank-order surveys. Given that team-based projects account for twenty to twenty-five percent of the final grade in several of my courses, I do think it’s important to ensure some degree of individual accountability in any formally-assessed collaboration between students. No one likes being forced to work with free riders, especially in a course like this one, where students will be conducting research outside of scheduled class time by gathering information from (gasp!) members of the local community. As I have explained to students before — without much success — I can’t be aware of how hard or how well every single one of them has worked with their teammates on a group project, because I’m not always present when the work happens.

So, I racked my brain for a new method of peer evaluation, and came up with the Project Contribution Award:

Please select four people from class, other than yourself, whom you think each deserves 40 points for their outstanding contributions to the project. I will tally the results and the individuals with the most votes will win the award.

Students will submit their choices via an ungraded survey on the Canvas LMS.

The class has only twenty students, so the Project Contribution Award translates into one-fifth of them winning an additional 40 points in a course with a 1,200 point grading scale, a three percent bonus for being perceived by classmates as having performed well on the project. Although I will be dividing students into teams that will take on different responsibilities, the quality of the final product for the community partner will be dependent in some fashion on everyone’s contribution. So I’m hoping that the one in five possibility of earning the award is a sufficient incentive. If not, I can increase the weight of the award in future semesters, or assign one award to each team instead of having multiple awards for the whole class.

A mechanical note: constructing this kind of survey on Canvas requires repeatedly copying and pasting the names of all the students on the class roster. In my case, the survey has four multiple choice questions, each with the same set of twenty names, so the process only took a few minutes. But the inability to generate multiple iterations of a survey or quiz question with a “copy” command probably reduces the utility of Canvas’s quiz feature for courses with large enrollments. No one wants to paste each student’s name over and over again for a 200-student class. In this scenario, I would probably use Google Forms, which does allow the copying of questions. However, I would not be able to just send a link to the survey to students, because responses would be anonymous (allowing students to vote for themselves). Everyone in the class would need Gmail accounts. If your university uses Gmail as its email client, that’s great, but if not, this option requires some extra work on the instructor’s part getting students to create Google accounts with easily-identifiable usernames.

Links to all posts in this series:

Changing a Course on Globalization, Part 1

Changing a Course on Globalization, Part 2

Changing a Course on Globalization, Part 3

Changing a Course on Globalization, Part 4

Changing a Course on Globalization, Part 5

Changing a Course on Globalization, Part 6

Changing Course on Globalization, Part 7

 

 

Follow-Up on Slack and Specification Grading

Today we have another post by guest contributor William R. Wilkerson, Professor of American Government and Politics at SUNY-Oneonta. He can be reached at bill [dot] wilkerson [at] oneonta [dot] edu.

Earlier this summer I wrote about two changes that I made to my five-week online summer course, Law, Courts and Politics: using Slack for class communication and specifications grading. Both experiments were a success.

Slack

Slack was a great addition. I found it easy to set up and to use. Students liked it. Thanks to the resources I noted in my earlier post, I created a simple structure: channel for each week was the home of announcements, files, links, and discussion — the center of the course. The introduction channel gave students the ability to practice and the questions forum got some use, especially early in the term.

Because Slack has excellent apps for all mobile and computer platforms, I hoped that it would encourage regular communication, which it did. Total posts in the weekly channels ranged from 62 in week 2 to 90 in week 4. I posted reminders and introduced topics, but most posts were from the students. Nine or ten students active each week; one student never posted in the weekly discussion forums. I was pleased that a group of students began posting mid-week and continued through the end of the week. Students picked up quickly on hashtags for topics and connecting to their fellow students via the @ symbol, which facilitated interaction. Posts were fairly long too, especially when you consider they were writing on their phones. I had expected phone use to result in short responses to comments, but that didn’t happen. Continue reading

Making Students Make: Classroom Game Design Part 3

As promised, here is information about the final exam I have created for my first-year seminar. As I mentioned in my previous two posts in this series, my goal is to make students more aware of what is now often referred to as “design thinking”: in this course, they are not just learning how to build games, but to solve problems.

Instructions for the exam:

Write an essay that compares one of the games that you helped design and one of the games that you played in this course. Identify which game was better by analyzing how well the games:

  • Incorporated the principles of the “golden rule” and “magic circle” in relation to events in the real world.
  • Utilized elements of conflict and cooperation.
  • Had clear written and unwritten rules.
  • Facilitated meaningful decision making.

Use information from assigned course readings to support your argument. Be sure to include references to your sources in your essay to avoid plagiarism – this applies whether you are using direct quotations or just the ideas of another author. Use parenthetical in-text citations as with reading responses to save space. An essay with few or no relevant references to the course readings will suffer a reduction in grade.

  • The essay should be double-spaced, in 11 or 12 point font, and the equivalent of 3-4 pages long.
  • There is no need for a separate bibliography or title page; please do not include them.
  • Work independently; do not discuss your essay with other students.

A screenshot of the rubric I’ll be using to grade the exams is below. As I mentioned about the game design beta test rubric, I am not really concerned with the exam scores generated by the rubric — my primary goal is getting students to become more aware of how their experiences translate into learning.

Specification Grading In An Online Course

Today we have another post by guest contributor William R. Wilkerson, Professor of American Government and Politics at SUNY-Oneonta. He can be reached at bill [dot] wilkerson [at] oneonta [dot] edu.

This summer for the first time I am teaching an online version of my judicial process course, Law Courts and Politics. I adopted a specifications grading system, something that has been discussed by people like Linda Nilson at Inside Higher Ed and Amanda Rosen on this blog. With specifications grading all assignments are graded on a satisfactory or unsatisfactory basis and course grades are based on assignment bundles.

My course is five weeks long with a distinct theme for each week’s lesson. Each lesson includes an online quiz made up of multiple choice and short essay questions on the textbook (Corley, Ward and Martinek’s American Judicial Process ), various discussion topics on the text, other assigned readings, video and audio, as well as a 600-750 word writing assignment. Each of these elements—quizzes, discussion, and the writing assignment, along with a summative assignment for those wishing earn a B or an A—are tied to course learning objectives. The grade bundles are as follows: Continue reading

Making assessment relevant

Insert metaphor here

Reading Martin’s post yesterday, just as I’m finishing my duties as an external examiner, makes me think about assessment formats.

Too often, we fall into the essay-and-exam approach: it’s simple, and easy and hardly anyone questions it. Of course, as the institution I external at is about to find out, I’m one of the people who does question it.

Assessment has a terrible reputation to deal with: in essence, it’s a hassle to do as a student, a hassle to set and mark as an instructor and the source of more academic complaints than anything else. No-one has a good word to say about it, it seems.

In our hearts, we know that it matters and that there has to be some kind of means of evaluating student performance, for their sakes and ours. But surely there’s a better way of doing it. Continue reading