Signal or Noise: what do I really want to assess?

By Carolyn Hoessler

“The essential requirements of a course/program are the knowledge and skills which must be acquired or demonstrated in order for a student to successfully meet the learning objectives of the course/program.” (University of Saskatchewan policy)

When I was in undergrad, I thought I was a good student. I would take notes during classes, remember the material, tutor my classmates, complete assignments, and so on. Then would come the final exam including the multiple-choice section with its numbered bubble sheet. I could read, circle the right answer, even write a rationale, but I could not fill in the correct bubble – I actually could not fill in any bubble, at least not 50 times. A prior injury to a particular ligament meant the motion of shading in the bubbles was painful and resulted in severe swelling of my hand, as did (at the time) extended periods of neat writing. I could type, I could write in bigger letters but could not complete a bubble sheet. Does this mean I was a bad student or that I did not know the material? Few would claim that my knowledge of the content was missing, yet I would not have been able to fill in the correct answer.

Assessment embeds multiple abilities into a single performance, rated often on the output. Answering a computation exam question, such as “What is the variance of the first three numbers in the following set { 3 6 8 9 4 }?”, involves steps like:

• Seeing the words and translating them from symbols to concepts
• Recognizing all of the words including concepts such as “average” and “set” taught in the course, as well as words like “following” and “first” that may not be known
• Selecting the correct formula to compute a variance including distinguishing the colloquial understanding from the statistical concept.
• Applying the formula and completing the calculations correctly
• Recalling the correct number of units to report
• Completing all of these steps in a restricted period of time
• Lastly reporting the computed value in the medium required (e.g., bubble sheet, online form, written paper test, or verbal), including not shading the wrong bubble.

What do we really want to measure in our students: Their recall of the formula for variance, the ability to complete mathematical calculations, their ability to write with a pencil? All of the above is what would be assessed as any question or task we ask embeds both the important knowledge or skills that we seek to measure as well as less relevant side pieces. With this wide net we capture variation in our students that we may not care about, such as their penmanship or their familiarity with words such as “following” along with the valuable measures of relevant ability.

If the aim is to measure ability by capturing the signal while minimizing the noise, then we need to differentiate between the essential requirements that characterize knowledge of a field and the happenstance of usual process.

Showing academic excellence means demonstrating knowledge or ability for those essential requirements regardless of the particular approach (see Resources below). Luckily for me, most of my instructors during undergrad agreed; it did not matter if I wrote the A, B, C, or D on paper in big letters or shaded in a bubble as they were assessing my knowledge of the material and not my ligament strength. But my classmates were still tested on their ability to find and shade the correct bubbles. Creating relevant assessment extends beyond accommodating a few students to broader questions about what and how we assess every student. So, what are your assessment methods measuring, besides the skills or knowledge taught in class?.

P.S. Pardon the metaphors.

Resources:

• Accessibility and disability offices on campuses, including Disability Services for Students at the University of Saskatchewan
• Determining Essential Requirements for Courses/Programs, Office of Accessible Education, Stanford University – definition for essential requirements including open-ended questions for identifying the essential requirements or components in your own course or program
• Resources for Instructors, Student Disability Office, University of Windsor – definition for essential requirements including open-ended questions for identifying the essential requirements or components in your own course or program
• Instructor’s Handbook: Accommodating Students with Disabilities, The Adaptive Technology Centre, Queen’s University – defines essential requirements under general guidelines and offers examples of accommodations
• Academic Accommodation and Access for Students with Disabilities, University of Saskatchewan Policies  – defines essential requirements