When Reading Tests Attack (Content)

by Guest Blogger
August 3rd, 2011

By Rachel Levy

During  my first year of full-time teaching ESOL and Social Studies at an inner-city Washington, DC, high school, my principal approached me and told me that my students had come to her saying how much they were enjoying history class. I explained to her my intent to teach content but with a reading and writing intensive emphasis, to build those skills which were quite low among our school’s students. She was enthusiastic and I was thrilled.

A few weeks later, she attended our Social Studies department meeting where she explained to us that since there were no standardized tests in Social Studies, from that point forward, we were required to spend one-fifth of our class time teaching the Stanford-9 Reading Test.  For each of my students, I had to make charts based on testing data showing the skill (for example “context clues”) and how they did on that skill.  Then I was supposed to target my lesson plans to teach and remedy each student’s individual weaknesses. This didn’t seem right, but there was no protesting this: I wanted to help my students, she was my boss, and she was telling me what to do. Furthermore, such instruction and data collection had to be documented in our lesson plan books and during classroom observations.

This is where and how NCLB-applied pressure and high-stakes testing cause poor practices. Some counter, “The testing itself doesn’t cause teaching to the test in an ineffective way. Why don’t teachers simply adopt effective practices, like Core Knowledge?” While testing shouldn’t (in principle) encourage poor practice, unfortunately, my experiences in the classroom and now as a parent shows that national policy incentives mandating high stakes testing change classroom teaching for the worse. Ground-level feedback can help us to see how to fix accountability better than philosophical debates about the nature of testing.

I agree that it’s completely logical, obvious even, as Andrei Radulescu-banu put it on Robert’s recent post, that A=>B (Please read  his comment in its entirety). Certainly, test scores will gradually rise if a well-rounded and knowledge-rich curriculum is implemented. However, many educators are hindered in following this logic by performance pressure and by belief.

There were vague and all-encompassing standards (think horoscopes), however there was no social studies curriculum in DCPS at the time (there still isn’t).  By collaborating with my colleagues and relying on my own education and knowledge of social studies topics I came up with unit and lesson plans pretty easily. However, I really struggled to come up with lesson plans for teaching the Stanford-9 Reading Test.  I thought at the time that it was because I didn’t have much background in reading instruction, that I was missing something. Eventually I figured out that it wasn’t that I was missing something, it was because “teaching” the Stanford-9 Reading Test made absolutely no sense (and Tim Shanahan explains here that such an approach doesn’t work). So I taught history and geography as much as I could and I taught what I imagined “teaching the Stanford-9 Reading Test” was only when I had to, and made sure I had passable documentation in my lesson plan book.

In my second year, I got braver. In faculty meetings when we talked in small groups about how to get our test scores up, I voiced my opinion that the way to get test scores up was via an implicit route—to teach content and have students read and write as much as possible. I stated my skepticism that one could teach the Stanford-9 Reading Test or that students could learn the Stanford-9 test, but except in private asides, I had no supporters. And this was before Michelle Rhee came to town, mind you. Before NCLB, teaching content to struggling students was unappreciated; now, it’s practically an act of subversion.

Now, don’t get me wrong, I have little tolerance for the secondary teachers who say their job isn’t to help struggling readers. Yes, kids should be reading well by the time they get to middle and high school. But for whatever reason (and it’s probably a good idea to try and figure out that reason), many aren’t. Accept it and adjust your practice accordingly. If you’re not prepared to teach and help struggling readers or even non-readers, you have no business teaching in most American public school classrooms. But this does not mean giving a student reading on a first grade level a tenth grade textbook and telling her to read it. Rather it means finding content-relevant materials appropriate to students’ age and maturity and books on their level, scaffolding, and building up.

I also don’t blame my principal. Downtown was breathing down her neck, judging her on reading and math scores.  Rather than fight a losing battle that could cost her her position (while she was ambitious for herself, she also cared deeply about the students in her charge and about the school she had built from scratch), she embraced it all, including putting on Stanford-9 pep rallies. I am not making that up; these pep rallies happen.

In addition to what I describe above and the bankrupty of the content of the tests themselves, described by Diana Senechal here, another fundamental problem is that many of the advocates of reforms centered on testing-based accountability actually believe that kids who can’t read (decode) at all well should not be learning content, that they have to learn reading first and then they can learn content, that teaching a content-based curriculum is useless if kids can’t read. “Let’s focus on teaching reading and get the reading scores up and then we can worry about content.” And let’s be honest, even many teachers and educators who are opposed to testing-based accountability believe this. I encounter this all the time, as a teacher in both inner city and high-performing suburban districts, as a parent in my children’s high-performing district, and in my interactions with readers as an education blogger and writer.

I encountered this attitude, that language proficiency is a prerequisite to learning content frequently as an ESOL teacher. People insisted that English Language Learners should master English first before learning content. However, a very effective way to teach the English language is through the “sheltered content” model, where the content is a vehicle to teach the language. Of course, both English Language Learners and struggling readers need intensive and explicit language and reading instruction, but not beyond its utility and not without pairing it with content.

In the vast majority of cases, the “belief” that students have to learn to read before they can learn content is not a result of dysfunction, laziness, or poor intentions. Quite the contrary–this belief is based on the proven correlation between strong literacy skills and academic success and on the understandable urgency to get kids to master such skills. Unfortunately, it is also based on a fundamental misunderstanding of how people learn and of how such literacy skills develop. Some are celebrating NCLB’s increasing poor and minority students’ ability to read the tests they’re taking, but just because they can read the tests doesn’t mean they actually have a grasp of the content or that they know more. Being able to read the road signs is well and good, but it won’t get you very far if you don’t know how to drive or where to go.

Rachel Levy is a writer and a former (and likely future) ESOL and Social Studies teacher who lives in Ashland, Virginia, with her husband and three children. She blogs about education at All Things Education.

Teach Now, Test Later

by Robert Pondiscio
July 20th, 2011

Over at Joanne Jacobs, they’re talking about Sol Stern’s recent article on the New York City Core Knowledge Language Arts program. Regular commenter Stuart Buck, as he is wont to do, looks to turn the discussion into a referendum on what he perceives to be the anti-reform stance of Diane Ravitch and others.  Stern’s piece, he writes,

“supports the idea that we need a broad curriculum, etc. On the other hand, it completely undermines their insistence that testing inevitably leads poor beleaguered educators to teach to the test, to narrow the curriculum, and even to cheat and lie out of the sheer pressure. After all, if kids can actually do BETTER on the tests with none of the latter misbehavior, then testing isn’t the horror it’s made out to be.”

Later Buck offers that it is not possible to hold these two ideas in one’s head at the same time:

1. “It’s the STAKES attached to the testing that inevitably lead educators to teach to the test, narrow the curriculum, and cheat.”

2. Broad and rich curricula like Core Knowledge would actually allow educators to IMPROVE test scores above and beyond a narrow test-prep curriculum.

True, a patient investment in knowledge and language growth will raise scores over time, but the key phrase is over time.  There is no reason to expect an instant dividend from a knowledge-rich curriculum.  Indeed, because reading tests are de facto tests of background knowledge, there is every reason NOT to expect the results to show for several years when the cumulative effect of broad knowledge acquisition asserts itself. 

The high stakes associated with reading tests may not preclude teaching a knowledge-rich curriculum, but it arguably disincentivizes it.  If you are expected to show at least one year’s growth in one year’s time (a concept I’ve never been able to wrap my mind around) you are far more likely to resort defensively to test-prep and “reading strategies” instruction rather than teach material that might not show up on a state exam this year, or ever. 

The entire proposition is that knowledge and vocabulary are a “slow growing plant,” as E.D. Hirsch has said. The results show up in the long term. That’s hard to reconcile with high stakes reading tests that demand results now.

There’s No Such Thing as a Reading Test

by Robert Pondiscio
June 16th, 2010

“Children who do not learn to read proficiently by the end of third grade are unlikely ever to read at grade level,” writes Sara Mead in the July/August issue of The American Prospect.  The issue features a special section titled “Reading By Grade Three” that examines the crisis in early childhood literacy.  In addition to Sara’s piece, which lays out the case for national action on early childhood literacy, Cornelia Grumman,  executive director of the First Five Years Fund, looks at the need to get kids off to a good start even before formal schooling even begins; the New America Foundation’s Lisa Guernsey cautions against letting the clear need for improved early literacy translate into classrooms that are all skills and no play; Gordon MacInnes of The Century Foundation describes how providing low-income kids with “stable, high-quality preschool and kindergarten” has made a difference in New Jersey.  Lots more good reads; the whole package can be found here.   E.D. Hirsch and I contributed a piece as well, a version of which is below.  It looks at how a fundamental misconception about the nature of reading leads to mischief in how we teach and test it.   

There’s No Such Thing as a Reading Test
E.D. Hirsch Jr. and Robert Pondiscio

It is among the most common nightmares.  You dream of taking a test for which you are completely unprepared having never studied or even attended the course.  For millions of American schoolchildren, however, it is a nightmare from which they cannot wake, a Kafkaesque trial visited upon them each year when they are required by law to take a reading tests with little preparation.  Eyebrows are already being raised.  Not prepared!?  Why, preparing for reading tests has become more than just an annual ritual for schools.  It is practically their raison d’être!

Schools and teachers may indeed be making a Herculean effort to raise reading scores, but paradoxically these efforts do little to improve reading achievement and to prepare children for college, career and a lifetime of productive, engaged citizenship.  This wasted effort is not because, as many would have it, our teachers are lazy or of low quality.   Rather, too many of our schools labor under fundamental misconceptions about reading comprehension, how it works, how to improve it, and how to test it.

Reading is not a Skill

Reading, like riding a bike, is an ability we acquire as children and generally never lose.  Some of us are more confident on two wheels than others and some of us, we believe, are better readers than others.  We view reading ability as a broad, generalized skill that is easily measured and assessed.  We judge our schools and increasingly individual teachers based on their ability to improve the reading ability of our children.  When you think about your ability to read—if you think about it at all—the chances are good that you perceive it as not just a skill, but a readily transferable skill.  Once you learn how to read you can competently read a novel, a newspaper article, or the latest memo from corporate headquarters.  Reading is reading is reading.  Either you can do it, or you cannot. 

This view of reading is only partially correct. The ability to translate written symbols into sounds, commonly called “decoding,” is indeed a skill that can be taught and mastered.  This explains why you are able to “read” nonsense words such as “rigfap” or “churbit.”  Once a child masters “letter-sound correspondence,” or phonics, we might say she can “read” since she can reproduce the sounds represented by written language.  But clearly there’s more to reading making sounds.  To be fully literate is to have the communicative power of language at your command—to read, write, listen and speak with understanding.  As nearly any elementary school teacher can attest, it is possible to decode skillfully yet struggle with comprehension.  And reading comprehension, the ability to extract meaning from text, is not transferable. 

Cognitive scientists describe comprehension as “domain specific.”  If a baseball fan reads “A-Rod hit into a 6-4-3 double play to end the game” he needs not another word to understand that the New York Yankees lost when Alex Rodriguez came up with a man on first base and one out; he hit a groundball to the shortstop, who threw to the second baseman, who relayed to first in time to catch Rodriguez for the final out.  If you’ve never heard of A-Rod or a 6-4-3 double play and cannot reconstruct the game situation, you are not a poor reader.  You merely lack the domain specific knowledge of baseball to fill in the gaps.  

Even simple texts, like the ones our children read on their all-important reading tests, are filled with gaps—presumed domain knowledge—that the writer assumes the reader knows.  Research also tells us that familiarity with domain knowledge increases fluency, broadens vocabulary (you can pick up words in context), and enables deeper reading and listening comprehension.

A simple model, then, would be to think of reading as a two-lock box, requiring two keys to open. The first key is decoding skills. The second key is oral language, vocabulary and domain-specific or background knowledge sufficient to understand what is being decoded.   Even this simple understanding of reading enables us to see that the very idea of an abstract skill called “reading comprehension” is ill-informed.  Yet most U.S. schools teach reading as if both decoding and comprehension are transferable skills (more on that in a moment). Worse, we test our children’s reading ability without regard to whether or not we have given them the requisite background knowledge they need to be successful.    

           
Who is a “good reader?”

Researchers have consistently demonstrated that in order to understand what you’re reading, you need to know something about the subject matter.   Students who are identified as “poor readers” often comprehend with relative ease when asked to read passages on familiar subjects, outperforming even “good readers” who lack relevant background knowledge.  One well-known study looked at junior high school students judged to be either good or poor readers in terms of their ability to decode or read aloud fluently. Some knew a lot of about baseball, while others knew little. The children read a passage written at an early 5th-grade reading level, describing the action in a game. As they read, they were asked to move models of ballplayers around a replica baseball diamond to illustrate the action in the passage. If reading comprehension was a transferable skill that could be taught, practiced and mastered then the students who were “good” readers should have had no trouble outperforming the “poor” readers.  In fact (and perhaps intuitively) just the opposite happened.  Poor readers with high content knowledge outperformed good readers with low content knowledge.  Such findings should challenge our very idea of who is or is not good reader: if reading is the means by which we receive ideas and information, then the good reader is one who best understands the author’s words.

You have probably felt the uncomfortable sensation of feeling like a poor reader when struggling to understand a new product warranty, directions for installing a computer operating system, or some other piece of writing where your lack of background knowledge left you feeling out of your depth.  Your rate of reading slows.  You find yourself repeating sentences over and over to make sure you understand.  If this happens only rarely to you, it is because you possess a broad range of background knowledge—the more you know, the more you are able to communicate and comprehend. The implications of this insight for teaching children to read should be obvious: The more domain knowledge our children receive the more capable readers they will become.

The message has not reached American classrooms, however.  A stubborn belief in reading comprehension as a transferable skill combined with the immense pressures of testing and accountability has created something like a perfect storm—ever more time is being wasted (and wasted is not too strong a word) on scattered, trivial and incoherent reading.   A study sponsored by the National Institute of Child Health and Human Development found that only four percent of 1st grade class time in American elementary schools is spent on science, and two percent on social studies.  In third grade, about five percent of class time goes to each of these subjects.  Meanwhile a whopping 62% in 1st grade and 47% in 3rd is spent on language arts. 

Most young American children spend anywhere from 90 minutes to two-and-a-half hours a day in something educators call the “literacy block,” an extended period which might include  reading aloud, small group “guided reading,” independent writing, and other activities aimed at increasing children’s verbal skills.  Reading instruction largely focuses on teaching and practicing all-purpose “reading comprehension strategies”—helping students to find the main idea of passage, make inferences or identify the author’s purpose.  The general idea is to arm young readers with a suite of all-purpose tricks and tips for thinking about reading that can be applied to any text the child may encounter.   Careful readers may be thinking, “if the ability to understand what you read is a function of your domain-specific background knowledge, then how is it possible to teach all-purpose reading strategies?”  It’s a question well worth asking.  And one that seldom is.  

Reading strategies figured prominently in the 2000 report of the National Reading Panel, and reading strategies work, to a point. Reading comprehension scores tend to go up after instruction in strategies, but it’s a one-time boost.  The major contribution of such instruction is to help beginning readers know that text, like speech, is supposed to make sense.  If someone says something you don’t understand, you can always ask that person to repeat, explain, or give an example.  Reading strategies offer similar workarounds for print.  They’re not useless, but repeated practice seems to have little or no effect.

“The mistaken idea that reading is a skill—learn to crack the code, practice comprehension strategies and you can read anything—may be the single biggest factor holding back reading achievement in the country,” wrote Daniel T. Willingham, Professor of Psychology at the University of Virginia, recently in the Washington Post.  “Students will not meet standards that way. The knowledge base problem must be solved.”

Tests Worth Teaching To

Once the connection between content and comprehension becomes clear, two conclusions come almost unbidden.  Our present system of testing reading ability is inherently unfair.  Since reading comprehension is not a transferable skill, unless we ensure that all children have access to the same body of knowledge, the student with the greater store of background knowledge will always have a strong advantage.  The second conclusion is even worse: the content-neutral way that we teach reading, as a discrete set of skills and strategies, is counterproductive and even irresponsible.

If our schools understood and acted upon the clear evidence that domain-specific content knowledge is foundational to literacy, reading instruction might look very different in our children’s classrooms.  Rather than idle away precious hours on trivial stories or randomly chosen nonfiction, reading, writing and listening instruction would be built into study of ancient civilizations in first grade, for example, Greek mythology in second , or the human body in third.  Recently, the Core Knowledge Foundation has been piloting precisely such a language arts program in a small number of schools in New York City and elsewhere.  Initial results are promising, however it is crucial to remember that building domain knowledge is a long-term proposition.  All reading tests are cumulative.  The measurable benefit of broad background knowledge can take years to reveal itself.

At present, teachers are tacitly discouraged from taking the long view.  Indeed, what incentive would a 2nd grade teacher have to emphasize content that might not show up on a test until 6th grade, if even then?  There is more upside for teachers in doing exactly what they chiefly do now – test prep, skills and strategies – unless we actively incentivize a domain-specific approach to language arts.

Let us propose a reasonable, simple, even elegant alternative to replace the vicious circle of narrowed curriculum and comprehension skills of limited efficacy, which over time depress reading achievement.  By tying the content of reading tests to specific curricular content, the circle becomes virtuous.   Here’s how it would work:  let’s say a state’s 4th grade science standards includes the circulatory system, atoms and molecules, electricity, Earth’s geologic layers and weather; Social Studies standards include world geography, Europe in the Middle Ages, the American Revolution and the U.S. Constitution, among other domains.  The state’s reading tests should include not just fiction and poetry, but nonfiction readings on those topics and others culled from those specific curriculum standards.  Teachers would still teach to the test, emphasizing domain specific knowledge (because it might be on the test), but no one would object, since it would help students not only pass the current year’s test, but to build the broad background knowledge that enables them to become stronger readers in general. 

The benefits of such “curriculum-based reading tests” would be many:  tests would be fairer, and a better reflection (teacher quality advocates take note) of how well a student had learned the particular year’s curriculum.  The tests would also exhibit “consequential validity” meaning they would actually improve education.  Instead of wasted hours of mind-numbing test prep and reading strategy lessons of limited value, the best test-taking strategy would be to spend time learning the material in the curriculum standards—a true virtuous circle.

By contrast let’s imagine what it is like to be a 4th grade boy in a struggling South Bronx elementary school, sitting for a high-stakes reading test.  If you do not pass, you are facing summer school or repeating the grade.  Because the school has large numbers of students below grade level, they have drastically cut back on science, social studies, art, music—even gym and recess—to focus on reading and math.  You have spent the year learning and practicing reading strategies.  Your teacher, worried about her performance, has relentlessly hammered test-taking strategies for months.

The test begins and the very first passage concerns the customs of the Dutch colony of New Amsterdam.  You do not know what a custom is; neither do you know who the Dutch were, or even what a colony is.  You have never heard of Amsterdam, old or new.  Certainly it’s never come up in class.  Without background knowledge you struggle with most of the passages on the test.  You never had a chance.   Meanwhile, across town, more affluent students take and pass the test with ease. They are no more bright or capable than you are, but because they have wider general knowledge—as students who come from advantaged backgrounds so often do—the test is not much of a challenge.  Those who think reading is a transferable skill and take their background knowledge for granted may well wonder what all the fuss is about.  Those kids and teachers in the Bronx struggle all year and fail to get ready for this? Why, all the answers are right there on the page!

It ends, as it inevitably must, in the finger pointing that plagues American education.  Do not blame the tests.  Taxpayers are entitled to know if the schools they support are any good, and reading tests, all things considered, are quite reliable.  Do not blame the test writers.  They have no idea what topics are being taught in school and their job is done when tests show certain technical characteristics. It is unfair to blame teachers, since they are mainly operating to the best of their ability using the methods in which they were trained.  And let’s certainly not blame the parents of our struggling young man in the South Bronx.  Is it unreasonable to assume that a child who dutifully goes to school every day will gain access to the same rich, enabling domains of knowledge that more affluent children take for granted?

It’s not unreasonable at all.  That’s what schools are supposed to be for.  The only thing unreasonable is our refusal to see reading for what it really is, and to teach and test reading accordingly.

What Works is Boring

by Robert Pondiscio
December 6th, 2009

Important, but frustrating piece in the Washington Post this morning about the difficulty of sustaining test-score growth in underperforming schools after dramatic one-time boosts.  “Studies across the country show that many low-performing schools falter after big one-year gains in test scores.  Of the seven D.C. public schools that increased proficiency rates by 20 percentage points or more in both reading and math in 2008,” Bill Turque reports, only showed growth in 2009.  “Most of the schools that surged 20 points or more in a single category last year also had difficulty building on the increase this year.”

The piece looks at any number of reasons–from turnover to cheating–why scores might spike in a given year and then plateau or decline.  But if the piece is any indication, DC schools are overlooking the obvious: a key to long-term growth in reading scores is the steady buildup of background knowledge.  Without knowing anything about the particular schools discussed in the Post piece, I’d bet real money that we’re talking about mediocre schools that got religion (or were forced to get religion) about testing and focused on it.  Hard.  But no one should be surprised to see one-time gains.

“Given the relationship between academic background knowledge and academic achievement, one can make the case that it should be at the top of any list of interventions intended to enhance student achievement,” wrote Robert Marzano in his 2006 book Building Background Knowledge for Academic Achievement. ”If not addressed by schools, academic background knowledge can create great advantages for some students and great disadvantages for others.”

E.D. Hirsch has obviously spent much of his life banging on this same drum, pointing over and over that reading tests are essentially tests of background knowledge.  If DC school leaders understand this, the Post piece doesn’t say. 

Test prep and simplistic reading strategy instruction that focus on trivial stories–students learn to predict, to summarize, to infer — does nearly nothing to add to a child’s store of knowledge, making an such a one-time boost nearly inevitable.   An absence of background knowledge is the difference-maker and left unattended it eventually shows up in the test scores. 

At a recent Aspen Institute panel discussion with Hirsch, Randi Weingarten observed that the reason we’re not seeing more of this is because “what works is boring.”  Building background knowledge is a slow, steady process.  Boring as hell.  And absolutely effective. 

And While You’re At It, Drop Off a Copy of “The Knowledge Deficit”…

by Robert Pondiscio
June 24th, 2009

Dr. Yvonne Fournier, an education columnist for Scripps Howard, takes up E.D. Hirsch’s common sense call for state reading tests to reflect the content taught in each grade.  Responding to a mother who complains that her A-student son’s poor performance on a reading comprehension test is keeping him out of his school of choice, Fournier notes the advantage conferred by the content knowledge accumulated by higher SES kids — “the type of kids who get to learn not just through school but through the vocabulary their educated parents use, the trips they can take, the camps they attend, the extracurricular activities their parents can pay for, weekends at the lake house, Internet access and more.”

Fournier’s advice for the parent?  Clip Hirsch’s recent New York Times op-ed on testing and march to the district office:

Take a copy of Hirsch’s article to the person in charge of setting policy as to who gets into the better schools. Notify your school board and, if need be, the NAACP and your newspaper. Insist on your child’s report card to be taken as proof of his intellectual ability. He needs no further testing unless the school board wants to infer that your son’s teachers gave him his grades.