Portraits of Thinking: A Test Taker

I want to continue the discussion of cognition, and to do so through a series of portraits drawn from the writing I’ve done over the years.

As I’ve been arguing during the year of this blog’s existence—and for some time before—we tend to think too narrowly about intelligence, and that narrow thinking has affected the way we judge each other, organize work, and define ability and achievement in school. We miss so much.

I hope that the portraits I offer over the next few months illustrate the majesty and surprise of intelligence, its varied manifestations, its subtlety and nuance. The play of mind around us. And, though not all the portraits will be of young people in school, I hope, as well, that collectively the portraits help us think in a richer way about teaching, learning, achievement, and the purpose of education—a richer way than that found in our current national policy and political discourse about school.

This first portrait comes from an adult literacy and developmental education program that I describe in Lives on the Boundary. The focus is on standardized testing, a close look at one test-taker. And though this woman is in her forties, I think there’s a lot here worth considering for all ages, especially in our current test-intensive culture.

***

When they entered the program, Ruby and Alice and Sally and all the rest were given several tests, one of which was a traditional reading inventory. The test had a section on comprehension—relatively brief passages followed by multiple-choice questions—and a series of sections that tested particular reading skills: vocabulary, syllabication, phonics, prefixes and roots. The level of the instrument was pretty sophisticated, and the skills it tested are the kind you develop in school: answering multiple-choice questions, working out syllable breaks, knowing Greek and Latin roots, all that.

What was interesting about this group of test takers was that—though a few were barely literate—many could read and write well enough to get along, and, in some cases, to help those in their communities who were less skilled. They could read, with fair comprehension, simple news articles, could pay bills, follow up on sales and coupons, deal with school forms for their kids, and help illiterate neighbors in their interactions with the government. Their skills were pretty low-level and limited profoundly the kinds of things they could read or write, but they lived and functioned amid print.

The sad thing is that we have few tests of such naturally occurring competence. The typical test focuses on components of reading ability tested in isolation (phonetic discrimination, for example) or on those skills that are school-oriented, like reading a passage on an unfamiliar topic unrelated to immediate needs: the mating habits of the dolphin, the Mayan pyramids. Students then answer questions on these sorts of passages by choosing one of four or five possible answers, some of which may be purposely misleading.

To nobody’s surprise, Ruby and her classmates performed miserably. The tasks of the classroom were as unfamiliar as could be. There is a good deal of criticism of these sorts of reading tests, but one thing that is clear is that they reveal how well people can perform certain kinds of school activities. The activities themselves may be of questionable value, but they are interwoven with instruction and assessment, and entrance to many jobs is determined by them. Because of their centrality, then, I wanted to get some sense of how the students went about taking the tests. What happened as they tried to meet the test’s demands? How was it that they failed?

My method was simple. I chose four students had each of them take sections of the test again, asking them questions as they did so, encouraging them to talk as they tried to figure out an item.

The first thing that emerged was the complete foreignness of the task. A sample item in the prefixes and roots section (called Word Parts) presented the word “unhappy,” and asked the test-taker to select one of four other words “which gives the meaning of the underlined part of the first word.” The choices were very, glad, sad, not. Though the teacher giving the test had read through the instructions with the class, many still could not understand, and if they chose an answer at all, most likely chose sad, a synonym for the whole word unhappy.

Nowhere in their daily reading are these students required to focus on parts of words in this way. The multiple-choice format is also unfamiliar—it is not part of the day-to-day literacy—so the task as well as the format is new, odd.

I explained the directions again—read them slowly, emphasized the same item—but still, three of the four students continued to fall into the test maker’s trap of choosing synonyms for the target word rather than zeroing in on the part of the word in question. Such behavior is common among those who fail in our schools, and it has led some commentators to posit the students like these are cognitively and linguistically deficient in some fundamental way: they process language differently, or reason differently from those who succeed in school, or the dialect they speak in some basic way interferes with their processing of Standard Written English.

Certainly in such a group—because of malnourishment, trauma, poor health care, environmental toxins—you’ll find people with neurolinguistic problems or with medical difficulties that can affect perception and concentration. And this group—ranging in age from nineteen to the mid-fifties—has a wide array of medical complications: diabetes, head injury, hypertension, asthma, retinal deterioration, and the unusual sleep disorder called narcolepsy. It would be naïve to deny the effect of all this on reading and writing.

But as you sit alongside these students and listen to them work through a task, it is not damage that most strikes you. Even when they’re misunderstanding the test and selecting wrong answers, their reasoning is not distorted and pathological. Here is Millie, whose test scores placed her close to the class average—and average here would be very low just about anywhere else.

Millie is given the word ““kilometer” and the following list of possible answers:

a. thousand
b. hundred
c. distance
d. speed

She responds to the whole word—kilometer—partially because she still does not understand how the test works, but also, I think, because the word is familiar to her. She offers speed as the correct answer because: “I see it on the signs when I be drivin’.” She starts to say something else, but stops abruptly. “Whoa, it don’t have to be ‘speed’—it could be ‘distance.’”

“It could be ‘distance,’ couldn’t it?” I say.

“Yes, it could be one or the other.”

“Okay.”

“And then again,” she says reflectively, “it could be a number.”

Millie tapped her knowledge of the world—she had seen kilometer on road signs—to offer a quick response: speed. But she saw just as quickly that her knowledge could logically support another answer (distance), and, a few moments later, saw that what she knew could also support a third answer, one related to number. What she lacked was specific knowledge of the Greek prefix kilo, but she wasn’t short on reasoning ability. In fact, reading tests like the one Millie took are constructed in such a way as to trick you into relying on commonsense reasoning and world knowledge—and thereby choosing a wrong answer. Take, for example, this item:

Cardiogram
a. heart
b. abnormal
c. distance
d. record

Millie, and many others in the class, chose heart. To sidestep that answer, you need to know something about the use of gram in other words (versus its use as a metric weight), but you need to know, as well, how these tests work.

After Millie completed five or six items, I have her go back over them, talking through her answers with her. One item that had originally given her trouble was “extraordinary”: a) “beyond”; b) “acute”; c) “regular”; d) “imagined.” She had been a little rattled when answering this one. While reading the four possible answers, she stumbled on “imagined”: “I…im…”; then, tentatively, “imaged”; a pause again, then “imagine,” and, quickly, “I don’t know that word.”

I pronounce it.

She looks up at me, a little disgusted, “I said it, didn’t I?”

“You did say it.”

“I was scared of it.”

Her first time through, Millie had chosen regular, the wrong answer—apparently locking onto ordinary rather than the underlined prefix extra—doing just the opposite of what she was supposed to do. It was telling, I thought, that Millie and two or three others talked about words scaring them.

When we come back to “extraordinary” during our review, I decide on a strategy. “Let’s try something,” I say. “These tests are set up to trick you, so let’s try a trick ourselves.” I take a pencil and do something the publishers of the test tell you not to do: I mark up the test booklet. I slowly began to circle the prefix extra, saying, “This is the part of the word we’re concerned with, right?” As soon as I finish she smiles and says “beyond,” the right answer.

“Did you see what happened there?  As soon as I circled the part of the word, you saw what it meant.”

“I see it,” she says. “I don’t be thinking about what I’m doing.”

I tell her to try what I did, to circle the part of the word in question, to remember that trick, for with tests like this, we need a set of tricks of our own.

“You saw it yourself,” I say.

“Sure I did. It was right there in front of me—‘cause the rest of them don’t even go with ‘extra.’”

I am conducting this interview with Millie in between her classes, and our time is running out. I explain that we’ll pick this up again, and I turn away, checking the wall clock, reaching to turn off the tape recorder. Millie is still looking at the test booklet.

“What is this word right here?” she asks. She had gone ahead to the other, more difficult, page of the booklet and was pointing to “egocentric.”

“Let’s circle it,” I say. “What’s the word? Say it.”

“Ego.”

“What’s that mean?”

“Ego. Oh my.” She scans the four options—self, head, mind, kind—and says “self.”

“Excellent!”

“You know, when I said ‘ego,’ I tried to put it in a sentence: ‘My ego,’ I say. That’s me.”

I ask her if she wants to look at one more. She goes back to “cardiogram,” which she gets right this time. Then to “thermometer,” which she also gets right. And “bifocal,” which she gets right without using her pencil to mark the prefix. 

Once Millie saw and understood what the test required of her, she could rely on her world knowledge to help her reason out some answers. Cognitive psychologists talk about task representation, the way a particular problem is depicted or reproduced in the mind. Something shifted in Mille’s concept of her task, and it had a powerful effect on her performance.