The Mismeasure of Teaching and Learning in Contemporary School Reform, Part II

This post continues the post from May 6, 2011.

Just as learning and achievement get narrowly defined in this reform world, so does teaching. Priscilla’s story is emblematic not only of the mechanical and restrictive pedagogy that is frequently laid on teachers in a test-driven environment, but also of the attitude toward teachers and the walling off of their participation and on-the-ground knowledge. Teachers today live in a bipolar world. They are praised as central to student achievement and routinely condemned as the cause of low performance. And, the overriding measure of competence or (the term used these days) effectiveness is the standardized test score in math and reading. The teacher becomes a knowledge-delivery system, and the better the students do, the more effective the teacher is judged.

No doubt, a focus on the K-12 teacher is an important feature of current school reform. Though parental income continues to be the strongest predictor of academic achievement, within the school the teacher is key. And in comparison to most of the countries (Finland, South Korea) whose educational system our policy-makers admire, teachers in the U.S. are paid less, have less substantial professional development, and enjoy less occupational status. So it could be a good thing to have the teacher’s role acknowledged on a national scale. And, as noted, some of the problems the reformers target are important ones. There is wide variation in competence in the teaching profession. How could there not be with a work force of close to four million, the largest profession in the country?

Many of these teachers – but by no means all – work within a poorly executed evaluation system, and they complain about inadequate assessment of their performance, and, therefore, inadequate guidance. One reason for this state of affairs is that principals are so awash in administrative duties that they neither have the time to conduct careful evaluations nor the training to do what their title originally signified: be the principal teacher, able to provide guidance about pedagogy. The current managerial orientation of school reform will do little to remedy that.

And sadly it is true that some teachers hold low expectations for their students, especially those who are less advantaged. I remember sitting in Elena Castro’s warm and stimulating classroom – she’s the teacher who asked her student to apply the word “admire” – when a group of teachers from a neighboring district with similar working-class demographics walked through on a visit. I heard one of the visitors whisper to another, “Our students couldn’t handle this.” Low expectations can come from out-and-out bigotry, or from jaded weariness, or from misguided sympathy – and the reformers are right to assail them.

I want to dwell on this business of low expectations for a moment longer – particularly on NCLB’s response to them, for the NCLB approach reveals a lot about the one-dimensional way teachers are understood in our reform environment.

For NCLB, it is lack of effort and low expectations that lead to low student achievement. That’s quite a claim, given all the other factors that contribute to student achievement, but let’s grant it momentarily. You’ll recall that the mechanism that will correct those low expectations is the threat of high-stakes testing. This is a pretty simplified notion of motivation: raise your expectations or you’ll be punished, what a friend of mine calls the caveman theory of motivation. And there’s an even more simplistic theory of cognitive and behavioral change at work: threat will lead to a change in beliefs, whether these beliefs come from prejudice or pity.

This kind of reductive understanding of teachers and teaching characterizes a good deal of post-NCLB reform as well. I’ll select two phenomena in current reform as illustration: the discounting of experience and the attempt to determine effective teaching practices.

For the standardized test score to be locked in as the reigning measure of teacher effectiveness, other indicators of competence need to be discounted. One is seniority – which reformers believe, not without reason, overly constrains an administrator’s hiring decisions. Another is post-baccalaureate degrees and certifications in education, a field many reformers hold in contempt. Fortunately for the reformers, there are studies that do report low correlations between experience (defined as years in the profession) and student test scores. There are also studies that report similarly low correlations between student scores and post-baccalaureate degrees and certifications. These studies lead to an absolute claim heard frequently these days that neither experience nor schooling beyond the bachelor’s degree make any difference in teacher effectiveness – and that the test score remains our only legitimate measure of competence.

On the face of it, this is a remarkable assertion. Can you think of any other profession – from hair styling to fire fighting to neurosurgery – where we wouldn’t value experience and training? If reformers had a more comprehensive understanding of teaching, they would at least consider the possibility that something is amiss with the studies. The problem is that the studies for the most part deal in simple aggregates and define experience or training in crude ways. Experience is defined as years on the job, and it’s no surprise that years alone don’t mean much. But if you define experience in one of the ways Webster suggests—“activity that includes training, observation of practice, and personal participation and knowledge gained from this”— then you would most likely find a connection between experience and competence. What people do with their time on the job is crucial, becomes the foundation of expertise. As for the question about post-baccalaureate work, the same principle applies: What kind of training? Where? What was the curriculum? The quality of supervision? I’ll be the first to admit that a number of education programs leave a lot to be desired, but to discount experience and training in blanket fashion is not only wrong-headed but also undercuts attempts to create better working conditions for teachers, more robust professional development, and opportunities for career advancement – all things the reformers say they want.

The qualities of good work—study and experimentation, the accumulation of knowledge and refinement of skill – are thinly represented in descriptions of teacher quality, overshadowed by a (often inadequately understood) language of testing. In a similar vein, it is telling that the long history of Western thought on education – Plato to Septima Clark – is rarely if ever mentioned in the reform literature. History as well as experience and inquiry are replaced with a technological metric.

The cases of experience and training provide an example of the way many of the reformers’ foundational assumptions about teaching and learning are constructed. There is much made about science in current reform, talk of “data-driven decision making” and “evidence-based practice.” But some policy decisions (like the expansion of charter schools) are made without much research support, and bedrock claims such as “experience and training don’t matter” are based on narrowly conceived and executed analyses.

***

Last year there was a long article in the New York Times Magazine about attempts to pinpoint the techniques that comprise good teaching—from standing still when giving directions to ways to pose a math problem and call on students—and teaching those techniques to teachers. The article was titled “Building a Better Teacher,” and that title captured for me both the promise and the potential limitations of a powerful strand in current school reform: A technique-and-testing orientation to improving teacher quality.

Teaching is a complex activity that, when done well, requires, among other qualities, subject matter knowledge as well as skill in teaching it. Teaching requires a knowledge of young people’s minds and hearts and, more particularly, a knowledge of one’s students and the ability to read them, read the dynamics of the room, and react appropriately. The good teacher holds a belief in human ability and a commitment to fostering it—and one manifestation of that belief is the creation of a safe and respectful as well as demanding classroom. Effective techniques are an important part of this mix, and good mentorship includes a close analysis of what a teacher is doing—like a coach and athlete watching a film—and providing corrective feedback and better ways to do things. Contrary to the story reformers tell, teachers of teachers have been doing this for a long time. What is new is the strong focus on techniques, the increased role of electronic and testing technology to study them, and the attempt to define “effective” by seeking positive correlations between specific techniques or clusters of techniques and, you guessed it, students’ standardized test scores. What is also new is the magnitude of the effort, punched up considerably by a 45 million dollar project funded by the Gates Foundation. (I am involved in another Gates-funded project.) There is, indeed, a considerable push among some reformers to build a better teacher.

Because teaching does involve a good deal of craft, I’m sympathetic to this attempt to analyze useful techniques and make them available to new and developing teachers. But given the technocratic orientation of contemporary school reform, I worry that other aspects of teaching less easily observed and circumscribed—from bearing and pacing to beliefs about learning—will get short shrift. The building of effective teachers will occur through the accumulation of techniques. And given the need in reform-initiated research to find correlations between techniques and test scores, researchers will veer toward those techniques that are most readily definable, leading to a possible narrowing of the repertoire of techniques themselves.

There is a further issue: The use of any technique gains meaning in a time and place. Consider the Chicago math teacher Michelle Smith’s decision to move her student from the back to the front of the room. She’s rearranging seating to insure order in the room, an activity that could be considered an effective technique for classroom management. Imagine, however, the other unpleasant ways that decision could have played out: The student refusing to move, or insulting or threatening her, or stirring up his comrades sitting nearby. But Michelle’s action occurs in the context of a relationship with the class and with that boy, a legacy of care and learning. (“Miss Smith,” he later told me, “she’s teaching us how to do things we couldn’t do before.”) Michelle knows local culture, understands rituals of masculinity and the huge importance of allowing that boy a little space to save face. She has developed a classroom persona that blends sass and seriousness and uses it strategically. Technique is contextual and part of a performative flow of events.

When you focus on technique without regard to context, you can get analyses like the following, taken from a New York Times column on the Gates project. Two researchers are rating the videotape of a teacher they don’t know. They zero in on a segment where she doesn’t see or ignores a boy who is raising his hand repeatedly. The teacher gets a low mark on “respect and rapport.” That’s a legitimate possible rating. But what if that boy frequently takes up conversational space, and the teacher has spoken with him about it, explaining that she can’t always call on him. I and other teachers I know have done this. Then that teacher’s actions would be seen in a different light—demonstrating a potential error in rating.

A further level of error can occur as we move to the center of this machine, finding what practices correlate with test scores, for the scores themselves are typically not stable. (In value-added measures, for example, a significant number of teachers who are in the top quartile of scores one year will not be in the top the next; the same holds for the bottom quartile.) And finally there is the attempt to correlate practice to test scores which, precedent for this kind of study suggests, could yield a slew of inconsequential correlations. The research design is mechanistic and reductive—it segments human activity—and is complicated with layers of potential error.

But even if you grant my concerns, isn’t it worthwhile to at least call attention to the tricks of the trade that good teachers use and make those widely available? Absolutely. And here is where we run into perhaps the most considerable problem for reformers as they turn their attention to teacher development.

The entire history of this reform movement has not been built on teacher development but rather on a punitive accountability system of high-stakes testing. (Recall the NCLB feet-to-the-fire approach to motivating teachers.) Lip service was paid to helping teachers use their students’ scores to improve instruction, but little broad-scale infrastructure was put in place to enable even that kind of development to happen. For the current program to be successful, there will need to be a sea-change in attitudes toward and understanding of teachers and the teaching profession. Now would be the time to start.

Though I am disheartened by how few reformers express a robust vision of public education, there is no doubt that they are committed to education itself, and particularly for those who have not been well-served by our schools. But to get the kind of teaching we saw in the opening vignettes, to make more of our schools the potent democratic institutions they can and should be, the reform movement will have to generate within itself a much richer sense of teaching and learning. We have a strong belief in our country that to find a measure for something is to understand it; we confuse counting with analysis. Education reform needs a conceptual framework that would certainly include testing and technique but embedded in the cognitive and emotional world of the classroom.

I would like to thank Megan Franke, Judy Johnson, Felipe Martinez, John Rogers, Shirin Vossoughi, Noreen Webb, and Kyo Yamashiro for their help with this article.

The Mismeasure of Teaching and Learning in Contemporary School Reform Part I

Because of its length, this entry will come in two parts. I'll post part two in a week or so. This essay originally appeared in Dissent,, Spring, 2011. If you share or repost it, please acknowledge the magazine.

The good classroom is rich in small moments of intelligence and care. There is the big stuff of course – the week-long science experiment, the dramalogue, the reporting of one’s research – but important as well are the spontaneous question, the inviting gesture, the tone in a voice. They reveal the cognitive and philosophical intimacy of a room.

In the border town of Calexico, California, third-grade teacher Elena Castro is working with a group of students when a boy who is still learning English comes over from a book he’s reading to ask what the word “admire” means. She turns and gives him a definition, and, as he is walking away, she calls after him asking if he admires the farmer in a story the class had read that morning.

In a combined middle school classroom in Chicago, Kim Day and Dianna Shulla begin class by listening to their students’ distress about a new bus schedule that is getting some kids home late and missing other stops entirely. The teachers suggest that the class collectively discuss the problem and try to develop some strategies to solve it. Several students raise their hands. Dianna walks close to them, leans in slightly, and says, “Talk to me.”

In a one-room schoolhouse in Polaris, Montana, teacher Andy Bayliss is having his students keep a journal on the willows in the creek behind the school. He is working with one of the older students; both are bent over the boy’s sketches and measurements. Andy points to one nicely detailed drawing and asks his student why he thinks the willows grow in these dense clusters, rather than long and snaky up a tree. The boy has fished in these creeks for years, Andy later explains, and “I just wanted him to take a little different look at what he already knows.”

Back in Chicago, high school mathematics teacher Michelle Smith is calling her class to order and sees that one of the young men who plays the class clown is sitting way in the back. She calls him by name, then, with flourish says, “My young gentleman, I’d like you to sit up here where I can see you.” The student groans, uncurls himself from his desk, and walks to the front, sauntering for the benefit of his peers. “C’mon darlin’,” Michelle adds, head tilted, hand on hip, “humor me.” She watches; he sits. “Thank you, Sir. I feel better.” Then “O.K. people. We have work to do today. Let’s go!”

Stephanie Terry’s first-grade classroom in Baltimore, Maryland is packed with books, botanical experiments, children’s drawings and writing. There are areas in the room for students to read, to do science, to write and read their writing aloud. On this day, she introduces a visiting writer to the class. “We’re going to have an author stay with us.” “Ooooo, Miss Terry,” one of the students exclaims, waving his hand, “We’re authors, too!”

These vignettes are drawn from my book Possible Lives, an account of a cross-country journey observing good teachers, and, through their work, trying to capture the meaning of public education in our time. Such episodes are commonplace, available to anyone who would take the time to see them. In the first vignette, a teacher creates a moment of instruction on the fly, leading a student to apply a word he just learned. In that Chicago middle school, an erratic bus schedule is turned into a problem to collaboratively solve – and what students have to say is invited and taken seriously. In rural Montana a science experiment leads a boy to see the familiar in a new light. Next, a math teacher, with a mix of humor and direction, deftly allows a boy to save face while bringing order to her classroom. And finally the embrace of literacy is revealed as a first-grader claims the role of an author.

It might come as a surprise to some, given the thirty-year drumbeat of public education failure, that these vignettes are drawn from public schools, and, except for the one-room schoolhouse, they serve low-income communities of color. It is also worth noting that, with the possible exception of the boy learning the word “admire,” none of this would be captured by current measures of accountability. Yet most parents would want their children taught and treated this way.

***

This is an essay about what is missed or distorted in current school reform and the consequences for educational practice. Reform is guided by a technocratic-managerial orientation to education that is not informed by deep knowledge of the classroom, in fact, can be dismissive of it. This orientation has already led to some sizable policy blunders and results in a restricted definition of teaching and learning – and therefore a restricted sense of the person. It is telling, I think, that you rarely find portrayals of classroom life in the thousands of pages of school reform documents. Students and teachers are discussed, to be sure, but as abstractions, stick figures on a policy grid.

The purpose of education in this environment is thoroughly economic, and, to be fair, has been for decades before the current reforms, coinciding with our nation’s precarious position in the emerging global economy. From the President to the Secretary of Education to governors and mayors, the purpose of education is “to prepare students for 21^st Century jobs.” True, a major goal of American education is to prepare the young to make a living. But in a democracy we send children to school for many other reasons as well: intellectual, social, civic, ethical, aesthetic. Historically, these justifications for schooling have held more importance. Not today.

The reformers are a varied lot, representing a wide range of ideology and motive – including free-marketeers who would like to see public education shrunk or dismantled. But overall, reformers are addressing issues of real importance. Though they tend to downplay disparities in resources between affluent and poor districts or the effects of poverty and discrimination on young people’s lives in school, they rightly target the education of low performing students and low expectations for what these students can achieve. As well, they criticize the recruitment, education, and evaluation of teachers; the structure and the anonymity of schooling, particularly in the large high school; and the state of school governance, especially in big districts, the bureaucratic inertia and seemingly intractable school politics. No wonder that a number of prominent liberals and Civil Rights groups support the reforms.

But it is with the remedies, the methods of reform, that problems arise, for it is the methods, and the assumptions behind them, that directly affect what happens in the classroom. The federal No Child Left Behind Act (NCLB) provides some unfortunate examples, the features of which are present in more current reforms.

The fundamental mechanism of NCLB has been an accountability system of high-stakes standardized tests of the core subjects, mathematics and reading. How schools and districts perform on the tests have big consequences, can ultimately lead to sanctions, withholding of federal funding, and a change in leadership. The assumptions underlying NCLB are that teachers and administrators hold low expectations for their students, particularly those who are less-advantaged, and aren’t putting in enough effort to educate them. NCLB, its framers claim, holds administrators’ and teachers’ feet to the fire.

NCLB did jolt some low-performing schools to evaluate their inadequate curricula and engage in staff development aimed at improving their students’ mastery of the basic math and reading skills measured by the tests. But the use of such tests and the high stakes attached to them led to other outcomes as well, and any student of organizational behavior could have predicted them. A number of education officials at various levels reacted to the high stakes by manipulating the system. They lowered the cut-off test scores for proficiency, or withheld from testing students who would perform poorly, or, occasionally, flat out-fudged the results. So some cases of remarkable improvement in test scores (remember the “Texas Miracle”?) turned out to be unstable or simply fraudulent, not miraculous at all.

Studies of what went on in classrooms were equally troubling – and again predictable. The high-stakes tests led many administrators and teachers to increase math and reading test preparation and reduce time on other subjects: science, history, and geography received less attention, and the arts were, in some cases, drastically reduced or eliminated. (Think here of losing that time-intensive study of the willows in the creek behind the one-room school or not having time to engage those Chicago middle-schoolers in solving the bus scheduling problem in their community.) Also trimmed were activities involving the core subjects of math and reading that didn’t directly map onto the tests but could lead to broader understanding and appreciation. Group reading, the writing of stories, children’s public reading of their stories – all that led to the Baltimore first-grader proclaiming an author’s identity – were jeopardized in the pedagogical calculus of high-stakes testing.

There is certainly an argument to be made for concentrating on the basics of math and reading, for they are so central to success in school, and an unacceptable number of students don’t master them. And a score on a standardized test seems like a straightforward measure of mastery. But in addition to the kinds of manipulation I discussed, there are a host of procedural and technical problems in developing, scoring, and interpreting such tests. “In most cases,” writes measurement specialist Robert Linn, “the instruments and technology have not been up to the demands placed on them by high-stakes accountability.” Thus there is debate among testing experts about what, finally, can be deduced from the scores about a student’s or a school’s achievement. Similar debates surround the currently popular use of “value-added” methods to determine teacher effectiveness.

There is a second, related, issue. Tests embody definitions of knowledge and learning. A test that would include, say, the writing of an essay, or a music recital, or the performance of an experiment embodies different notions of instruction and achievement than do the typical tasks on standardized tests: multiple choice items, matching, fill-ins. I have given both kinds of tests, both have value, but they get at different things, represent knowledge in different ways, can require different kinds of teaching. When one kind of test is emphasized and when the stakes are high, the tests, as we just saw, can drive and compress a curriculum.

This concern about the nature of a school’s response to high-stakes pressure is especially pertinent for those students at the center of reform: poor children, immigrants, racial and ethnic minority students. You can prep kids for a standardized test, get a bump in scores, yet not be providing a very good education. The end result is the replication of a troubling pattern in American schooling: poor kids get an education of skills and routine, a lower-tier education, while students in more affluent districts get a robust course of study.

It’s important to remember how far removed standardized tests are from the cognitive give and take of the classroom. That’s one reason why there is a debate as to whether a test score—which is, finally, a statistical abstraction—is really an accurate measure of learning. Yet the scores on standardized tests have become the gold standard of excellence – and this is the case for post-NCLB initiatives, most notably the Obama administration’s Race to the Top. Though the Department of Education is calling for multiple measures of achievement, to date the standardized tests of math and reading dominate, and because of their ease of use and aura of objectivity, there is good reason to believe they will continue to dominate.

Let me bring all this down to the level of a particular teacher in a particular school. Over the past few years, I’ve been privy to a lot of classrooms caught in the high-stakes machinery. Here’s Priscilla’s story. She’s a thirty-year veteran teaching in an elementary school in a working-class community. The school has 30 students in each of its first, second, and third grade classrooms; 36 to 41 children in grades four through six.

The school’s test scores were not adequate last year, so the principal, under immense pressure, mandated a “scripted” curriculum, that is, a regimented curriculum focused on basic math and literacy skills followed by all teachers. The principal also directed the teachers not to change or augment this curriculum. So Priscilla cannot draw upon her cabinets full of materials collected over the years to enliven, extend, or individualize instruction. (Though like any experienced teacher, she figures out ways to use what she can when she can.) The teachers have also been directed by the principal to increase the time spent on the literacy and math curriculum and trim back science and social studies. Art and music have been cut entirely. “There is no joy here,” she told me, “only admonition.” All this led her to do a remarkable thing, right out of a Jimmy Stewart movie. With her own money she flew back to Washington D.C. intent on telling the story of her school to somebody, anybody in authority. At the National Education Association, the nation’s second largest teachers union, she met with one of the officers.