Reading Test Developers Call Knowledge a Source of Bias

by Lisa Hansel
October 1st, 2014

You might expect to see a headline like this in the Onion, but you won’t. The Onion can’t run it because it isn’t just ironic—it’s 100% true.

A few years ago, a researcher at one of the big testing companies told me that when developing a reading comprehension test, knowledge is a source of bias. He did not mean the obvious stuff like knowledge of a yacht’s anemometer. He meant typical K–12 subject matter.

Since reading comprehension depends chiefly on knowledge of the topic (including the vocabulary) in the passage, the student with that knowledge has a large advantage over the student without it. And since there have always been great educational inequities in the United States, students’ knowledge—acquired both at home and at school—is very strongly correlated with socioeconomic status.

A logical solution would be to test reading comprehension using only those topics that students have been taught. Teachers can do this, but testing companies can’t—how would they have any idea what topics have been taught in each grade? It’s rare for districts, much less states, to indicate what or when specific books, people, ideas, and events should be taught.

Without a curriculum on which to base their assessments, testing companies have devised their own logic—which is sound given the bind they’re in. They distinguish between common and specialized knowledge, and then they select or write test passages that only have common knowledge. In essence, they’ve defined “reading comprehension skill” as including broad common knowledge. This is perfectly reasonable. When educators, parents, etc. think about reading comprehension ability, they do not think of the ability to read about trains or dolphins or lightning. They expect the ability to read about pretty much anything one encounters in daily life (including the news).

I already had this basic understanding, but still I found the “ETS Guidelines for Fairness Review of Assessments” eye opening. Guideline 1 is to “avoid cognitive sources of construct-irrelevant variance…. If construct-irrelevant knowledge or skill is required to answer an item and the knowledge or skill is not equally distributed across groups, then the fairness of the item is diminished” (p. 8). It continues, growing murkier:

Avoid unnecessarily difficult language. Use the most accessible level of language that is consistent with valid measurement…. Difficult words and language structures may be used if they are important for validity. For example, difficult words may be appropriate if the purpose of the test is to measure depth of general vocabulary or specialized terminology within a subject-matter area. It may be appropriate to use a difficult word if the word is defined in the test or its meaning is made clear by context. Complicated language structures may be appropriate if the purpose of the test is to measure the ability to read challenging material.

Avoid unnecessarily specialized vocabulary unless such vocabulary is important to the construct being assessed. What is considered unnecessarily specialized requires judgment. Take into account the maturity and educational level of the test takers in deciding which words are too specialized.

On page 10, it offers this handy table that “provides examples of common words that are generally acceptable and examples of specialized words that should be avoided…. The words are within several content areas known to be likely sources of construct-irrelevant knowledge”:

ETS table 1

Since having good reading comprehension means being able to read about a wide variety of common topics, table 1 seems just fine. But testing companies’ silence about what their reading comprehension tests actually measure is not. They say they are measuring “reading comprehension skill,” but their guidelines show that they are measuring a vaguely defined body of “common knowledge.”

Common words are not common to all. Even “common” knowledge is knowledge that must be taught, and right now—at home and at school—far too many children from low-income homes don’t have an opportunity to learn that knowledge (which is common to youth from middle-class and wealthy homes). That’s why reading comprehension scores are so strongly and stubbornly correlated with socioeconomic status.

These tests of “common” knowledge are accurate assessments and predictors of reading comprehension ability; but they are not fair or productive tests for holding children (and their teachers) accountable before an opportunity to learn has been provided.

If all testing companies would clearly explain that their reading comprehension tests are tests of knowledge, and if they would explain—as the ACT’s Chrys Dougherty does—that the only way to prepare for them is to build broad knowledge, then we could begin to create a fair and productive assessment and accountability system. Before the end of high school, all students should have broad enough knowledge to perform well on a reading comprehension test. But what about in third, fourth, or even seventh grade? In the early and middle grades, is a test drawn only from topics that have been taught in school the only fair way to test reading comprehension? How many years of systematically teaching “common” knowledge are needed before a reading comprehension test that is not tied to the curriculum is fair, especially for a student whose opportunities to learn outside of school are minimal?

The answer depends not so much on the test as on what is done with the scores. If we accepted the fact that reading comprehension depends on broad knowledge, we would radically alter our accountability policies. Scores on “common knowledge” reading comprehension tests would be recognized as useful indicators of where students are in their journey toward broad knowledge—they would not be mistaken for indicators of teaching quality or children’s capacity. Instead of holding schools accountable for scores on tests with content that is not tied to the curriculum, we would hold them accountable for creating a content-rich, comprehensive, well-sequenced curriculum and delivering it in a manner that ensures equal opportunity to learn. To narrow the inevitable gaps caused by differences in out-of-school experiences, we would dramatically increase free weekend and summer enrichment opportunities (for toddlers to teenagers) in lower-income neighborhoods. (We would also address a range of health-related disparities, but that’s a topic for another day.)

In sum, reading comprehension really does rely on having a great deal of common knowledge, so our current reading comprehension tests really are valid and reliable. To make them fair and productive, children from lower-income families must be given an equal opportunity to learn the knowledge that is “common” to children from higher-income homes.

shutterstock_200841857

Reading is always a test of knowledge (image courtesy of Shutterstock).

“Houston, we have a problem”

by Lisa Hansel
July 23rd, 2014

We do indeed have a crisis on our hands, but year after year we fail to diagnose and address it. With 21st century skills, learning styles, comprehension strategies, blame-the-teacher “reforms,” and dozens of other fads clouding our thinking, research-driven common sense improvements get little attention.

It’s frustrating, but our Core Knowledge community is dedicated to spreading the word on rigorous academics. For anyone out there who needs yet more evidence of the desperate need for building broad knowledge and skills, two new reports are worth examining.

“Just the facts, ma’am”

Cold, hard facts are what we get from ACT and Mathematica Policy Research. We learn (yet again) that there are massive disparities in preparation for college and kindergarten.

ACT’s The Condition of College & Career Readiness tackles the high school problem with stark graphics. The one below, showing the massive gaps among youth by race and ethnicity, is especially striking:

ACT 7-22-14 A

Then, a ray of hope. Taking a “core curriculum” in high school appears to greatly increase the odds that a young adult is well prepared. In the chart below, “Core” stands for core curriculum, which ACT defines as “4 years of English and 3 years each of mathematics, science, and social studies” in high school.

ACT 7-22-14 B

That gives us one clear step to take in closing college- and career-readiness gaps. But things are never so simple. You see, most students are already taking a core curriculum:

ACT 7-22-14 C

Clearly, all core curricula are not created equal. But we know better that to lay all blame at the high school doorstep. And in case we forget, Mathematica’s Kindergartners’ Skills at School Entry: An Analysis of the ECLS-K reminds us. This study is interesting because it does not look just at the usual race/ethnicity and income factors. Instead, it focuses on four specific “risk factors”: “the child lives in a single-parent household, the child’s mother has less than a high school education, the child’s household income is below the federal poverty line, and the primary language spoken in the home is not English.”

You may be surprised to see that nearly half—44%—of entering kindergartners face at least one of these risk factors:

Mathematica 7-22-14 A

Sadly, you may be even more surprised to see how devastating even just one risk factor is in terms of reading, math, and working memory:

Mathematica 7-22-14 B

Mathematica 7-22-14 C

(Note: IRT stands for “item response theory.” The children were given two-stage assessments in which their performance in the first stage determined the difficulty of the test items they were given in the second stage.)

If these two new reports tell us anything, it’s that we must intervene early. Gaps that exist at kindergarten entry still exist at the end of high school—ripe for replication when our underprepared young adults have children of their own.

“May the Force be with you”

Schools with coherent, cumulative curricula that build academic knowledge, vocabulary, and skills are intervening. Curriculum is not the solution, of course, but it is a necessary part of the foundation for student (and teacher) learning. Unfortunately, far too many school, district, and policy leaders are unaware of how to make their curricula stronger, much less how to harness a rigorous curriculum for benefits such as early identification of students’ needs and increased teacher collaboration. For those looking to take the first step, I strongly recommend Harvard’s Lead for Literacy series. In 16 one-page memos, Lead for Literacy clearly identifies best practices for literacy programs, assessments, professional development, and program selection. The series may not be as powerful as the Force, but they’ll give leaders a good shot at dramatically increasing students’ knowledge and skills, and enabling them to learn more both in and out of school.

Why My Brother’s Keeper Should Look to ACT and Common Core

by Lisa Hansel
June 3rd, 2014

My Brother’s Keeper, a new Obama-administration initiative focused on boys and young men of color, appears to be off to a strong start. The Task Force’s 90-day report is impressive in terms of breadth and focus. At its heart are “six universal milestones” that “serve as the basis for the Task Force’s work and recommendations:”

  1. Entering school ready to learn
  2. Reading at grade level by third grade
  3. Graduating from high school ready for college and career
  4. Completing postsecondary education or training
  5. Successfully entering the workforce
  6. Reducing violence and providing a second chance

One of the report’s best features is an explicit rejection of any silver-bullet solutions. As we all know, far too many of America’s boys of color face multi-faceted, severe challenges. Thankfully, the Task Force recognizes that viable solutions must be comprehensive, coordinated, and long term. Its recommendations reflect as much, and also a desire to “continue to listen, gather input, engage experts and stakeholders, [and] develop additional recommendations.”

Great! I have a recommendation: Learn from ACT and the Common Core standards. Specifically, realize that meeting the six milestones will require a much greater emphasis on building knowledge and vocabulary in early childhood.

shutterstock_41669881

Image courtesy of Shutterstock.

Let’s start with ACT, which offers both grim data and doable recommendations, and then move to Common Core, which—if properly understood—offers sound guidance.

Many of us think of ACT as just a testing company, but it has a research arm that mines ACT data and the broader literature to figure out how to improve educational outcomes. Chrys Dougherty, ACT senior research scientist, has produced three must-read briefs showing just how difficult it is for youth who are behind academically to catch up—and therefore how crucial it is to intervene early.

In his most recent brief, Dougherty shows that at least half of fourth- and eighth-grade Hispanic and African American students in the states whose data he analyzed are not doing well in reading—and almost none who are doing poorly catch up by the end of high school. Using longitudinal student outcome data, ACT has established benchmark scores that indicate college readiness (or, for younger students, being on a trajectory to end high school college ready). Students who score at or above those benchmarks are “on track,” while students who score more than one standard deviation below them are “far off track.”

Drawing data from Dougherty’s new brief, let’s look at fourth-to-eighth-grade results in reading on ACT Explore.

Fourth graders
who are
“far off track”:

“Far off track” fourth graders who caught up by eight grade:

Non-low income:

29%

10%

Low income:

53%

6%

Hispanic:

56%

5%

African American:

64%

3%

Note: These data are from Arkansas and Kentucky; see the brief for details.

As Dougherty shows, the data tracking students from eighth grade to the end of high school are just as depressing. Worse, keep in mind that these results are for all students, boys and girls. Girls tend to do better in reading than boys. In draft working papers, Dougherty and his colleagues have broken out results by gender, finding an even great challenge for My Brother’s Keeper (and all of us).

Knowing that being ready for college means having acquired an enormous store of academic knowledge, vocabulary, and skills, Dougherty’s first recommendation for school and district leaders is to:

Teach a content-rich curriculum in the early grades. Ensure that all students receive a content- and vocabulary-rich curriculum beginning in the early years, spanning a range of subject areas including not only English language arts and mathematics, but also science, history, geography, civics, foreign language, and the arts…. Such a curriculum—the basis for preparing students long term for college, careers, and informed citizenship—is valuable for all students but is likely to be especially beneficial for students from at-risk demographic groups, who are more likely to arrive from home with limited knowledge and vocabulary.

Let’s assume the Task Force heeds Dougherty’s advice and adds “Teach a content-rich curriculum in the early grades” to its list of recommendations. Where could it find out what that looks like? The Common Core State Standards for English language arts and literacy. Not in the individual standards, but in the narrative that accompanies the standards. There, the Task Force will find something absolutely essential, but so far missing from its report: an understanding that reading comprehension comes not just from mastering reading skills, but also from learning a great deal of academic subject matter and vocabulary.

The Task Force emphasizes having parents talk to their children more (and in more encouraging ways), improving reading skills instruction, and having children read more. These are necessary but insufficient recommendations. To accelerate knowledge and vocabulary acquisition, which will greatly increase the odds of meeting the Task Force’s milestones, parents and educators need to be as efficient as possible and start as early as possible.

The Common Core explains how. Start with the standards’ research appendix:

Word acquisition occurs up to four times faster … when students have become familiar with the domain of the discourse and encounter the word in different contexts…. Hence, vocabulary development for these words occurs most effectively through a coherent course of study in which subject matters are integrated and coordinated across the curriculum and domains become familiar to the student over several days or weeks.

Then, take a look at Common Core’s blueprint for a coherent course of study in K–5, where we learn that “texts—within and across grade levels—need to be selected around topics or themes that systematically develop the knowledge base of students. Within a grade level, there should be an adequate number of titles on a single topic that would allow children to study that topic for a sustained period.” Even better, we learn how to build knowledge before children can read: “Children in the early grades (particularly K–2) should participate in rich, structured conversations with an adult in response to the written texts that are read aloud, orally comparing and contrasting as well as analyzing and synthesizing.”

Everyone on the Task Force is busy, so I’ll boil it down. Parents shouldn’t just talk more; they should also read aloud more. And parents and teachers shouldn’t read aloud just one book on a topic; they should pick a topic and spend a couple of weeks reading aloud and discussing several books on that topic. If they do, many more boys of color will enter school ready to learn and will read at grade level.

 

Dear 8th Grader: You Have a 3% Chance of Getting Ready for College

by Lisa Hansel
April 2nd, 2013

What are the odds that an eighth grader in a high-poverty school who is far behind academically will catch up? You know the odds are low, but single-digit low? According to research from ACT, catching up in high school is rare—if by “catching up” we mean getting poorly prepared eighth graders ready for college by twelfth grade. An eighth grader in a high-poverty school who is far from meeting ACT’s college readiness benchmarks has just a 6% chance of catching up in reading—in science and mathematics, that student has a mere 3% chance. What about catching up before high school? Not likely. A fourth grader in a high-poverty school who is far behind has just a 7% chance of catching up in reading by eighth grade and an 8% chance in mathematics.

For some readers, the obvious conclusion is that the schools need to get better at closing the gap. But the ACT’s report also has findings for all schools, the top 10% of all schools, the top 10% of low-poverty schools, and the top 10% of high-poverty schools. All of the results on catching up are depressing.

Once gaps exist, we certainly have to do everything we can to close them. At the same time, we must start earlier to prevent these enormous gaps from opening up. The path to college begins in preschool.

And, by the way, after preschool, children should go to kindergarten. Why do I state the obvious? While many of us have been chattering about Obama’s universal preschool plan, ECS has just reminded us that some kids do not have access to kindergarten. Across states, access to high-quality kindergarten is so unequal that it “perpetuates, if not exacerbates, the achievement gap.” While 15 states require children to attend kindergarten, five states do not even require school districts to offer kindergarten. (Those five are Alaska, Idaho, New Jersey, New York, and Pennsylvania. Four of them have adopted the Common Core State Standards, which begin in kindergarten. Are you scratching your head yet?)

Ultimately, the ECS report reminds us that we have a long, long way to go in developing a strong system of early childhood education in this country.

Silver lining time: The one benefit of having waited so long to get serious about early learning is that we have an enormous body of research to draw from. We have so much research that sorting through it is a challenge. For that, I’m turning to an unsung hero of the school improvement world: Chrys Dougherty. He is a senior research scientist with ACT and a former teacher. He knows the education and cognitive science research—and he gets kids and classrooms.

ACT recently published Dougherty’s College and Career Readiness: The Importance of Early Learning. Everyone involved in early childhood and early grades education should read it. For that matter, everyone interested in school improvement and closing the achievement gap should read it.

I’m assuming that you are going to read it—it’s only 8 pages! So, instead of offering a CliffsNotes version, I’m providing my favorite parts (you’ll have to go to the report for the endnotes):

Students who do not have a good start usually do not thrive later on. That is due not only to the fact that students in stressful environments with limited learning opportunities often remain in those environments, but also because early learning itself facilitates later learning—students who already know more about a topic often have an easier time learning additional information on the same topic, and early exposure to knowledge can stimulate students to want to learn more….

Educators have long emphasized the importance of learning to read well in the early grades, a belief supported by longitudinal research. Reading consists of two abilities: the ability to identify the words on the page (decoding), and the ability to understand the words once they are identified (comprehension)…. Ensuring that students learn to decode well depends, among other things, on using activities and methods in preschool, kindergarten, and first grade that develop children’s phonological (sound) awareness and their knowledge of the relationship between letters and sounds. Meanwhile, children’s comprehension can be developed in the early grades by reading aloud to them from books that develop their knowledge and vocabulary….

One study found that kindergarteners’ general knowledge of the world was a better predictor of those students’ eighth-grade reading ability than were early reading skills. This is consistent with research showing that reading comprehension, particularly in the upper grades, depends heavily on students’ vocabulary and background knowledge….

Accountability systems have been designed to create a sense of urgency about improving test scores. However, this has often had the undesirable effect of shortening educators’ time horizons so that they emphasize changes aimed at improving accountability ratings over the short run. These changes can include narrowing the curriculum to deemphasize subjects not tested in the current grade, and spending inordinate amounts of time coaching students on how to answer sample test questions.

By contrast, many steps to improve academic learning and behaviors take time to bear fruit and may not immediately result in higher test scores. For example, implementing an excellent kindergarten and first-grade reading, mathematics, science, social studies, or fine arts program will not immediately affect test results in the older grades. Neither will field trips to science and art museums, nature areas, and historical sites—all of which develop knowledge of the world. Accountability incentives should be modified to recognize efforts that increase student learning over the longer run and promote learning in grades and subject areas not covered on state tests.

If we actually followed Dougherty’s advice, our students would have a great chance of getting ready for college.

 

Playing Catch-Up

by Robert Pondiscio
December 5th, 2012

An important new report from ACT’s National Center for Educational Achievement (NCEA) turns the lights up on a point that cannot be made often or strongly enough: when it comes to academic readiness, it’s easier to keep up than to catch up.  Over at Education Week, Sara Mead summarizes the findings, which she correctly describes as “sobering.”

“Among students who were ‘far off track’ in reading in 8th grade, only 10 percent achieved college and career ready standards 4 years later. In math and science, the percentage was even lower. And over 40 percent of African American students taking ACT’s EXPLORE exam in 8th grade scored ‘far off track’ in reading–as did 50% in math and 74% in Science. Put that together and you can’t like those odds.”

Policymakers take note of this from the report itself:

“Efforts to improve students’ academic preparation have often been directed at the high-school level, although for many students, gaps in academic preparation begin much earlier. Large numbers of disadvantaged students enter kindergarten behind in early reading and mathematics skills, oral language development, vocabulary, and general knowledge. These gaps are likely to widen over time because of the ‘Matthew effects,’ whereby those who start out behind are at a relative disadvantage in acquiring new knowledge.”

Bingo.  Old hat to followers of this blog, perhaps, but if “better schools” or even “better teachers” is your go-to response, here’s some cold water for you: “’Far off track’ 8th graders who attended schools in the top 10 percent of performance were roughly 3 times as likely to get back on track by 12th grade as the total sample,” Mead observes. ”But even looking at the top 10 percent of schools, the percentage of ‘far off track’ students getting back on track never exceeded 30%.”

Sobering, indeed.

The report’s takeaway emphasizes the need for “a realistic view of the difficulty of closing these gaps,” hence the need to start earlier.

“Underestimating the time and effort required could lead educators and policymakers to underfund prevention efforts and choose intervention strategies that are too little and too late. Underestimating the difficulty could also lead policymakers to hold schools to unrealistic accountability targets, creating strong incentives at various levels in the system to lower standards and artificially inflate test scores.”

In Mead’s view, this means “high-quality pre-k and early childhood education, particularly for African American, Hispanic, low-income, and other children from groups with higher percentages of students falling behind in school.”  I agree.  But critically, it must also mean a clear and focused understanding of what we mean when we say “high quality pre-k.”  Gaps in language proficiency are fundamentally gaps in knowledge and vocabulary–and the deficits are readily apparent on Day One.  To my mind, “high quality preschool” means aggressive interventions aimed at building language skill and knowledge acquisition before the dreaded Matthew Effect becomes a runaway train.

Oh Say Can You C?

by Robert Pondiscio
August 19th, 2009

More than three out of four college-bound high school graduates are unprepared to earn a “C” or higher in first-year college courses in English, math, reading and science.

That’s the news from 1.5 million ACT tests taken by of the class of 2009, but curiously it’s not the lede.  A press release from the Iowa City-based ACT frames the results in the opposite manner, noting “the percentage of graduates ready to earn at least a “C” or higher in first-year college courses in all four subject areas tested on the ACT increased from 22 percent in 2008 to 23 percent in 2009.”  USA Today, the  New York Times and lots of others repeat the 23% figure or otherwise lead with the “slight improvement” in scores over 2008 results. The Wall Street Journal alone among major papers seems to catch the obvious story.  “Only about a quarter of the 2009 high school graduates taking the ACT admissions test have the skills to succeed in college,” the paper notes.

In other news, 254 million Americans have health insurance.

Update:  EdWeek weighs in and gets the headline right: “ACT Scores Show Most Students Aren’t Ready for College.”  Catherine Gewertz’s piece also features a great quote from FairTest’s Robert Schaeffer on the failure of NCLB to improve college readiness: “Politicians can make all the claims they want that it is raising achievement, but even when there are improvements in state test scores, they don’t show up in college-admissions test data, or on [the National Assessment of Educational Progress].  So where is the beef?”

A Band-Aid on the Unkindest Cut?

by Robert Pondiscio
July 18th, 2009

Someone must have been unhappy with the reaction to news that Shakespeare would have no place in the emerging national standards currently being drafted by Achieve, ACT and the College Board.  The over-the-top quote from Chris Minnich, director of standards at the Council of Chief State School Officers, has suddenly disappeared from Linda Kulman’s Politics Daily piece, which was cited by Common Core, Joanne Jacobs and the Core Knowledge Blog (HT:  MagisterGreen). 

Original Quote:

“They’re really looking for what students should be able to do to truly be ready for college.  It means taking out some of the things that aren’t really important,” including, he says, “whether or not kids should read Shakespeare. Most of the studies say Shakespeare is not critical.”

New Quote:

“They’re really looking for what students should be able to do to truly be ready for college. It means taking out some of the things that aren’t really important. At this point, we don’t know yet what that will be.”  (Emphasis mine).

A misquote? A “clarification?”  Something uttered “off the record” not intended for the light of day?  The bigger problem with what Minnich said remains not his dismissal of Shakespeare, but the clear and unambiguous focus on skills at the expense of content.  That went unchanged in both versions.

Shakespeare in or out, if Mr. Minnich would like to explain the place of specific curricular content in the draft standards, the floor is his.  An invitation has been issued for him to write a guest post on this blog.