Smarter Balanced Confuses Fairness and Validity

by Lisa Hansel
October 15th, 2014

Over the past two weeks, we’ve looked the ETS guidelines for fair assessments that PARCC adopted, as well as a sample item from PARCC. Now let’s turn to the “Bias and Sensitivity Guidelines” ETS developed for Smarter Balanced. While I can’t say that ETS’s guidelines for Smarter Balanced contradict those adopted by PARCC, they are different.

In the introduction, validity and fairness are equated: “if an item were intended to measure the ability to comprehend a reading passage in English, score differences between groups based on real differences in comprehension of English would be valid and, therefore, fair…. Fairness does not require that all groups have the same average scores. Fairness requires any existing differences in scores to be valid” (p. 6).

By this logic, since youth from higher-income homes, on average, have more academic and common knowledge than youth from lower-income homes, the test that conflates reading comprehension ability with opportunity to learn is perfectly fair. Valid I can agree with. Fair I cannot.

A couple pages later, further explanation is offered (p. 8):

Exposure to information

Stimuli for English language arts items have to be about some topic…. Which topics and contexts are fair to include in the Smarter Balanced assessments? One fairness concern is that students differ in exposure to information through their life experiences outside of school. For example, some students experience snow every winter, and some have never experienced snow. Some students swim in the ocean every summer, and some have never seen an ocean. Some students live in houses, some live in apartments, some live in mobile homes, and some are homeless.

Even though curricula differ, the concepts to which students are exposed in school tend to be much more similar than are their life experiences outside of school. If students have become familiar with concepts through exposure to them in the classroom, the use of those concepts as topics and contexts in test materials is fair, even if some students have not been exposed to the concepts through their life experiences. For example, a student in grade 4 should know what an ocean is through classroom exposure to the concept, even if he or she has never actually seen an ocean. A student does not have to live in a house to know what a house is, if there has been classroom exposure to the term. Similarly, a student does not have to be able to run in a race to know what a race is. Mention of snow does not make an item unacceptable for students living in warmer parts of the country if they have been exposed to the concept of snow in school.

Let’s pause here: “Even though curricula differ, the concepts to which students are exposed in school tend to be much more similar than are their life experiences outside of school.” Maybe. Maybe not.

It might be the case that all elementary schools teach snow, oceans, houses, races, and deserts. But does Smarter Balanced really test such banal topics? No. As far as I can tell from its sample items, practice tests, and activities for grades three to five, Smarter Balanced (like PARCC) tests a mix of common and not-so-common knowledge. Passages include Babe Ruth, recycling water in space, how gravity strengthens muscles, papermaking, the Tuskegee Airmen, tree frogs, murals, and much more.

The sample items strike me as comprehensible for third to fifth graders with broad knowledge, but I am highly skeptical that we can safely assume that children are acquiring such broad knowledge in their elementary schools.

As Ruth Wattenberg explained in “Complex Texts Require Complex Knowledge” (which was published in Fordham’s Knowledge at the Core: Don Hirsch, Core Knowledge, and the Future of the Common Core), students in the elementary grades have minimal opportunities to acquire knowledge in history and science. Reviews of basal readers in 1983 and 2003 revealed that they contained very little content. This would be a lost opportunity, not a serious problem, but for the fact that elementary schools tend to devote a substantial amounts of time to ELA instruction, and very little to social studies and science instruction. Wattenberg’s table (p. 35) should be shocking:

Grade and subject 1977 2000 2012
K–3 social studies 21 21 16
4–6 social studies 34 33 21
K–3 science 17 23 19
4–6 science 28 31 24

Even worse, Wattenberg found that “When elementary teachers were asked during what time period struggling students received extra instruction in ELA or math, 60 percent said that they were pulled from social studies class; 55 percent said from science class.”

In their home environments, the schools they attend, and the curriculum to which they are exposed, lower-income children do not have an equal opportunity to learn. As Smarter Balanced guidelines state, the assessment is fair “if students have become familiar with concepts through exposure to them in the classroom.” That’s a big if.

Making matters worse, Smarter Balanced (like PARCC) asserts that it’s just fine for some kids to have to learn during the test. Returning to the “Bias and Sensitivity Guidelines” (p. 8):

Information in the stimulus

A major purpose of reading is to learn about new things. Therefore, it is fair to include material that may be unfamiliar to students if the information necessary to answer the items is included in the tested material. For example, it is fair to test the ability of a student who has never been in a desert to comprehend an appropriate reading passage about a desert, as long as the information about deserts needed to respond to the items is found in the passage.

Last week, we explored how difficult it is to learn from one passage and how greatly such test items advantage students who already know the content that the passage is purportedly teaching. Smarter Balanced clearly disagrees with me. Here’s the introduction it its fourth grade Animal World activity:

The Classroom Activity introduces students to the context of a performance task, so they are not disadvantaged in demonstrating the skills the task intends to assess. Contextual elements include: an understanding of the setting or situation in which the task is placed, potentially unfamiliar concepts that are associated with the scenario; and key terms or vocabulary students will need to understand in order to meaningfully engage with and complete the performance task.

Please take a look at the activity—it assumes an enormous amount of knowledge. Even if it did not, the notion of learning and immediately demonstrating ability flies in the face of well-established research on human’s limited working memory capacity. There’s no getting around it: the students with relevant prior knowledge have a huge advantage.

One (sort of) positive note: I am cautiously optimistic that Smarter Balanced’s computer adaptive testing will help—a little. Here’s how it’s described:

Based on student responses, the computer program adjusts the difficulty of questions throughout the assessment. For example, a student who answers a question correctly will receive a more challenging item, while an incorrect answer generates an easier question. By adapting to the student as the assessment is taking place, these assessments present an individually tailored set of questions to each student and can quickly identify which skills students have mastered…. providing more accurate scores for all students across the full range of the achievement continuum.

In a hierarchical subject like math, the benefits of this adaptation are obvious. In reading, adaptation might help, but it might be misleading. Once a student has mastered decoding, what makes one passage “easier” to comprehend than another is driven primarily by the topic. If the student knows a lot about the topic, then factors like rare vocabulary (which isn’t rare to the reader with the relevant knowledge) and complex sentence structure are of little import. If a student does not know about the topic, then making the vocabulary and sentence structure easier will only help a little. The main way in which adaptive testing might be helpful is in varying the topics; “easier” passages would consist of more common topics, while more “challenging” passages would consist of less common, more academic topics. Then, if we examined the results carefully, we might see that a child lacks essential—teachable—academic knowledge.

Yet, I am only cautiously optimistic because the knowledge that drives reading comprehension is accumulated more haphazardly than hierarchically. One can have some academic knowledge while missing some common knowledge. A student whose grandparents lived most of their lives in Greece may know a great deal about ancient and modern Greece and be ready for a highly sophisticated passage comparing and contrasting ancient and modern Greece. That same student may have no knowledge of China, gravity, Harlem’s Jazz age, or other topics that might appear on the test. Without assessing topics that have been taught, I see no way to truly gauge a students’ comprehension ability (or what the teacher or school has added).

To reinforce the most important message—that comprehension depends on knowledge, and thus schools must systematically build knowledge—the tests need to be tied to the content taught or the high stakes need to be removed so schools will no longer take time out of regular instruction for test preparation.

PARCC Demonstrates the Benefits of Broad Knowledge

by Lisa Hansel
October 8th, 2014

Last week I explored the “ETS Guidelines for Fairness Review of Assessments.” These guidelines were adopted by PARCC, so I decided to take a look at PARCC’s sample items for English language arts. (PARCC is one of the two consortia of states with massive federal grants to create assessments aligned with the Common Core State Standards. Smarter Balanced is the other consortium; ETS developed somewhat different guidelines for it—I’ll take a look at those next week).

The knowledge demands in PARCC’s sample items are very broad, from cougars to Amelia Earhart to DNA testing. While I am happy to see some substantive questions—and hopeful that such test items will reinforce the standards’ call for systematically building knowledge with content-rich curriculum—I worry about the fairness of these assessments given how they are being used.

As I mentioned last week, it would be perfectly fair to have test passages on topics that had been taught. But, since the test developers do not know which topics are taught in each grade, they have to assess “common” knowledge. Due to well-documented differences in opportunities to learn at home and at school, some children know a good bit more common knowledge than others.

Let’s take a look at one of PARCC’s sample items for third grade. Three questions are asked based on the 631-word passage “How Animals Live.” There’s a typical main-idea question paired with a supporting-evidence question, and then a narrower question that assesses the “skills of rereading carefully to find specific information and of applying the understanding of a text.” Here’s the first section of the passage:

What All Animals Need

Almost all animals need water, food, oxygen, and shelter to live.

Animals get water from drinking or eating food. They get food by eating plants or other animals.

Animals get oxygen from air or water. Many land animals breathe with lungs. Many water animals breathe with gills.

Animals need shelter. Some animals find or build shelter. Other animals grow hard shells to protect themselves.

Many words here are undefined: oxygen, shelter, lungs, and gills. Are these words common to all third graders? Probably not, but much of the content is likely familiar to the vast majority of third graders—and perhaps enough content is familiar for most third graders to grasp the section (if not every word). Nonetheless, children who have learned about oxygen, shelter, lungs, and gills start out with a big advantage. They are reading and comprehending more quickly (which is extremely important in a timed test), and they are comfortable as they move into the more difficult content in the rest of the passage.

Here is the second section, and the beginning of the third:

Ways Of Grouping Animals

Animals can be grouped by their traits. A trait is the way an animal looks or acts. Animals get traits from their parents. Traits can be used to group animals.

Animals with Backbones

Animals with backbones belong to one group. A vertebrate is an animal with a backbone. Vertebrates’ backbones grow as they get older. Fish, snakes, and cats are all vertebrates. Vertebrates can look very different.

Let’s ignore the stiff, unengaging style. What really concerns me is the delusion that it is fair for content to be learned and applied during a high-stakes assessment. (As I noted last week, I do not dispute that the assessment is valid and reliable, so my concerns are with accountability policies, not really with this type of assessment.)

Since a definition of trait is given, it’s clear that some significant portion of third graders is not expected to know that word. Now imagine this is the first time you’ve encountered trait and examine the text:

A trait is the way an animal looks or acts…. Traits can be used to group animals…. Vertebrates can look very different.

What is a third grader to make of this? Clearly, vertebrates are not grouped by how they look.

A trait is the way an animal looks or acts…. Fish, snakes, and cats are all vertebrates.

Clearly, vertebrates are not grouped by how they act. How is the backbone (which is not defined) a trait, since it does not seem at all related to how all these animals look or act?

Making matters worse, understanding trait is essential to correctly answering the main-idea and supporting-evidence questions.

shutterstock_57834808

Do these vertebrates look or act alike?
(Image courtesy of Shutterstock.)

The fact is, vocabulary is not learned by being given a definition. Definitions can be helpful, but they are always incomplete. Words are learned through multiple exposures in multiple contexts. Even with simple words, multiple contexts are necessary: What, exactly, makes hoagies and gyros and PB&Js all sandwiches? I can’t even attempt a concise answer—I just know a sandwich when I see one.

Third graders who have had a unit on vertebrates and invertebrates will breeze through this passage; its inadequate definition of trait won’t matter. But students relying on this definition will surely be at least a little confused, possibly totally lost. The assessment will accurately tell us that children without knowledge of traits have limited comprehension of this passage—but it will not accurately tell us anything about their teachers or schools, for no one alerted the educators that the test would measure knowledge of traits.

Reading Test Developers Call Knowledge a Source of Bias

by Lisa Hansel
October 1st, 2014

You might expect to see a headline like this in the Onion, but you won’t. The Onion can’t run it because it isn’t just ironic—it’s 100% true.

A few years ago, a researcher at one of the big testing companies told me that when developing a reading comprehension test, knowledge is a source of bias. He did not mean the obvious stuff like knowledge of a yacht’s anemometer. He meant typical K–12 subject matter.

Since reading comprehension depends chiefly on knowledge of the topic (including the vocabulary) in the passage, the student with that knowledge has a large advantage over the student without it. And since there have always been great educational inequities in the United States, students’ knowledge—acquired both at home and at school—is very strongly correlated with socioeconomic status.

A logical solution would be to test reading comprehension using only those topics that students have been taught. Teachers can do this, but testing companies can’t—how would they have any idea what topics have been taught in each grade? It’s rare for districts, much less states, to indicate what or when specific books, people, ideas, and events should be taught.

Without a curriculum on which to base their assessments, testing companies have devised their own logic—which is sound given the bind they’re in. They distinguish between common and specialized knowledge, and then they select or write test passages that only have common knowledge. In essence, they’ve defined “reading comprehension skill” as including broad common knowledge. This is perfectly reasonable. When educators, parents, etc. think about reading comprehension ability, they do not think of the ability to read about trains or dolphins or lightning. They expect the ability to read about pretty much anything one encounters in daily life (including the news).

I already had this basic understanding, but still I found the “ETS Guidelines for Fairness Review of Assessments” eye opening. Guideline 1 is to “avoid cognitive sources of construct-irrelevant variance…. If construct-irrelevant knowledge or skill is required to answer an item and the knowledge or skill is not equally distributed across groups, then the fairness of the item is diminished” (p. 8). It continues, growing murkier:

Avoid unnecessarily difficult language. Use the most accessible level of language that is consistent with valid measurement…. Difficult words and language structures may be used if they are important for validity. For example, difficult words may be appropriate if the purpose of the test is to measure depth of general vocabulary or specialized terminology within a subject-matter area. It may be appropriate to use a difficult word if the word is defined in the test or its meaning is made clear by context. Complicated language structures may be appropriate if the purpose of the test is to measure the ability to read challenging material.

Avoid unnecessarily specialized vocabulary unless such vocabulary is important to the construct being assessed. What is considered unnecessarily specialized requires judgment. Take into account the maturity and educational level of the test takers in deciding which words are too specialized.

On page 10, it offers this handy table that “provides examples of common words that are generally acceptable and examples of specialized words that should be avoided…. The words are within several content areas known to be likely sources of construct-irrelevant knowledge”:

ETS table 1

Since having good reading comprehension means being able to read about a wide variety of common topics, table 1 seems just fine. But testing companies’ silence about what their reading comprehension tests actually measure is not. They say they are measuring “reading comprehension skill,” but their guidelines show that they are measuring a vaguely defined body of “common knowledge.”

Common words are not common to all. Even “common” knowledge is knowledge that must be taught, and right now—at home and at school—far too many children from low-income homes don’t have an opportunity to learn that knowledge (which is common to youth from middle-class and wealthy homes). That’s why reading comprehension scores are so strongly and stubbornly correlated with socioeconomic status.

These tests of “common” knowledge are accurate assessments and predictors of reading comprehension ability; but they are not fair or productive tests for holding children (and their teachers) accountable before an opportunity to learn has been provided.

If all testing companies would clearly explain that their reading comprehension tests are tests of knowledge, and if they would explain—as the ACT’s Chrys Dougherty does—that the only way to prepare for them is to build broad knowledge, then we could begin to create a fair and productive assessment and accountability system. Before the end of high school, all students should have broad enough knowledge to perform well on a reading comprehension test. But what about in third, fourth, or even seventh grade? In the early and middle grades, is a test drawn only from topics that have been taught in school the only fair way to test reading comprehension? How many years of systematically teaching “common” knowledge are needed before a reading comprehension test that is not tied to the curriculum is fair, especially for a student whose opportunities to learn outside of school are minimal?

The answer depends not so much on the test as on what is done with the scores. If we accepted the fact that reading comprehension depends on broad knowledge, we would radically alter our accountability policies. Scores on “common knowledge” reading comprehension tests would be recognized as useful indicators of where students are in their journey toward broad knowledge—they would not be mistaken for indicators of teaching quality or children’s capacity. Instead of holding schools accountable for scores on tests with content that is not tied to the curriculum, we would hold them accountable for creating a content-rich, comprehensive, well-sequenced curriculum and delivering it in a manner that ensures equal opportunity to learn. To narrow the inevitable gaps caused by differences in out-of-school experiences, we would dramatically increase free weekend and summer enrichment opportunities (for toddlers to teenagers) in lower-income neighborhoods. (We would also address a range of health-related disparities, but that’s a topic for another day.)

In sum, reading comprehension really does rely on having a great deal of common knowledge, so our current reading comprehension tests really are valid and reliable. To make them fair and productive, children from lower-income families must be given an equal opportunity to learn the knowledge that is “common” to children from higher-income homes.

shutterstock_200841857

Reading is always a test of knowledge (image courtesy of Shutterstock).

Ending the Cultural Divide

by Lisa Hansel
September 24th, 2014

It is possible to enjoy NASCAR and polo, FX and PBS, Homer Simpson and Homer’s epics—but it is not always easy. As much as we Americans claim to value diversity and multiculturalism, we tend to shun the idea that one person can embrace and embody more than one culture.

This struggle, writes Vicki Madden (an instructional coach in New York City) in the New York Times, is a reason why many students from low-income homes do not complete college:

Once those from lower socioeconomic backgrounds arrive on campus, it’s often the subtler things, the signifiers of who they are and where they come from, that cause the most trouble, challenging their very identity, comfort and right to be on that campus. The more elite the school, the wider that gap. I remember struggling with references to things I’d never heard of, from Homer to the Social Register. I couldn’t read The New York Times—not because the words were too hard, but because I didn’t have enough knowledge of the world to follow the articles. Hardest was the awareness that my own experiences were not only undervalued but often mocked, used to indicate when someone was stupid or low-class: No one at Barnard ate Velveeta or had ever butchered a deer.

We know that Core Knowledge narrows the knowledge gap. Students in schools using the Core Knowledge Sequence have Homer covered, leaning about Winslow Homer in kindergarten and sixth grade, and reading The Iliad and The Odyssey (or adaptations of them) in sixth grade.

That alone should help students feel that they belong on their college campuses. Explaining how to butcher a deer may be a point of pride for those who can also weave Franz Liszt and Duke Ellington (both studied in the seventh grade via the Sequence) into their conversations. But let’s be honest; it’s the unique student who is resilient when faced with insidious, frequent reminders that grandpa’s fishing boat can’t get you to St. Martin.

shutterstock_164385578

Grandpa’s boat courtesy of Shutterstock.

Just as important as the academic challenge is the personal one students face when they find the strength to stay and earn their degrees. Madden relays her own experience, summing up that of her students:

In college, I read Richard Rodriguez’s memoir, “Hunger of Memory,” in which he depicts his alienation from his family because of his education, painting a picture of the scholarship boy returned home to face his parents and finding only silence. Being young, I didn’t understand, believing myself immune to the idea that any gain might entail a corresponding loss. I was keen to exchange my Western hardscrabble life for the chance to be a New York City middle-class museumgoer. I’ve paid a price in estrangement from my own people, but I was willing. Not every 18-year-old will make that same choice, especially when race is factored in as well as class.

As the income gap widens and hardens, changing class means a bigger difference between where you came from and where you are going. Teachers like me can help prepare students academically for college work. College counselors can help with the choices, the federal financial aid application and all the bureaucratic details. But how can we help our students prepare for the tug of war in their souls?

To this I’d like to add two more questions: Is there a way to prevent students from feeling such a tug of war? Educators in Core Knowledge schools—and any schools that focus on closing the knowledge gap—have your students experienced similar challenges?

Maybe it’s just my own very fortunate experience—I grew up in West Virginia, yet attended rigorous schools and had close friends at “regular” schools—but my hope is that these challenges are minimized by an educational environment that values home culture and academic culture from the very first day of school. In passing, I’ve heard of Core Knowledge schools that have book clubs and study groups for parents, as well as cultural festivals in which families share the rich, varied heritages that make up their school community. I’d love to learn more about such opportunities for whole families to grow together. Please submit a comment below, or email me at lhansel@coreknowledge.org. Thank you!

Buckling Down (Not Under): Thoughtfully Embracing the Common Core

by Lisa Hansel
September 18th, 2014

Over the past few years, the Common Core standards have been hailed and reviled. But do these standards merit such passion? For great schools—with rigorous curricula, collaborative educators, and supportive administrators—state standards make little difference. For schools that need to improve, the Common Core standards offer some useful guidance—but it takes far more than standards to provide a sound education.

What really matters is how the standards are interpreted and implemented. Amidst the cacophony of voices weighing in, I found three level-headed pieces this week.

shutterstock_204463663

One courtesy of Shutterstock.

Let’s start with what just might be the Common Core’s biggest bugaboo: close reading. It’s the perfect example of a pedagogical strategy that can be useful if used occasionally to focus students’ attention, and harmful if taken to extremes. Daniel Willingham of the University of Virginia (and Core Knowledge’s board) weighed in to remind us that reading without drawing on prior knowledge is not possible—nor is comprehension without sufficient knowledge:

Writers count on their audience to bring knowledge to bear on the text…. Things get [tricky] when it’s not plain to the reader that he lacks information that is important to understanding the text. Researchers Eli Gottlieb and Sam Wineburg offered a wonderful example…. They asked Clergy, scientists, and historians to read George Washington’s Thanksgiving Proclamation of 1789. Clergy and scientists focused on Washington’s invoking the “providence of Almighty God,” and other religious phrasing, with clergy applauding the Christian tone, and scientists upset by it. Historians, in contrast, focused on what the document did not say; it did not mention Jesus, nor salvation, nor Christianity. They saw the document as Washington’s self-conscious attempt to craft a statement that would be acceptable to the diversity of religions practiced in the United States.

Willingham concludes that attending to language and word choices and rereading make sense. But if students lack the knowledge needed to fill the inevitable gaps in language, then the text alone will not foster learning. (For yet more sage advice on close reading, see this post by Harvard’s Catherine E. Snow.)

shutterstock_116435485

Two courtesy of Shutterstock.

Next up is a brief report on exactly the kind of Common Core implementation I’d love to see all over the country: teachers working together, using the standards as a guide to create a more rigorous, knowledge-building curriculum. Last week, Robert Pondiscio (who is now at the Fordham Institute) had the pleasure of visiting Washoe County, Nevada, where Aaron Grossman is leading the homegrown Core Task Project. Obviously impressed, Pondiscio writes:

Those who see Common Core as a curricular monoculture, a boondoggle for publishers, or a violation of local control would do well to come to Reno. They’re doing it all without enriching an army of publishers or consultants—and with regional flair. Like children everywhere under Common Core, Reno kids will be expected to cite evidence in their reading and writing. But unlike anywhere else, they’ll be doing so by studying Nevada mining, conflict, and compromise among native tribes in Northern Nevada, and Nevada statehood and the state’s role in the Civil War—Common Core units developed by the Core Task Project. “We’re able to explain to the electorate that everything we’re doing is in-house, matched to our community values and to things we think are important,” Grossman observes.

Among the many resources the Washoe teachers are drawing on is Core Knowledge Language Arts. By thoughtfully crafting their curriculum, they have found that they have plenty of time for both the broad knowledge all students need and the knowledge of Nevada that they cherish.

shutterstock_199648589

Three courtesy of Shutterstock.

Lastly, in a wide-ranging podcast, Linda Darling-Hammond of Stanford offers sound advice on how the Common Core (or any college- and career-focused standards) can improve teacher and student learning. She also highlights research on assessments and suggestions for more productive accountability policies.

Curriculum Doesn’t Matter, Unless You Care about Achievement and Mobility

by Lisa Hansel
September 11th, 2014

Five years ago, Russ Whitehurst published an important paper comparing the effects of various education reforms. Better teachers and curriculum rose to the top, with what is taught being just as important as who is doing the teaching. But that finding didn’t fit with reformers’ obsession with teachers, so the paper was largely ignored. Two years ago, Whitehurst and Matthew Chingos did a more extensive look, confirming the previous findings and challenging states to begin gathering data on which materials are being used in schools.

Now, with pressure to interpret and meet the Common Core standards, curriculum, textbooks, and other instructional materials are finally getting wider attention. Much of that attention has been negative, as those who are against the standards seem to enjoy finding misinterpretations of the standards’ intent. I find that gotcha game silly; no one really expects initial stabs at Common Core–aligned materials to be terrific. Over time they will improve—and with support they will improve more quickly.

I’m thrilled to see growing interest in providing that support. The Helmsley Charitable Trust, the Hewlett Foundation, and the Bill & Melinda Gates Foundation are leading the way by funding several efforts, the most promising of which just might be EdReports.org. Preparing to launch in the winter, EdReports.org is involving teachers in intensive reviews of K–12 math and ELA materials, and the reviews will be free online. At the same time, Rachel Leifer and Denis Udall, program officers with Helmsley and Hewlett (respectively), have penned a plea for “Disrupting the Textbook Status Quo.” Since Core Knowledge is a small non-profit trying to offer better materials for free online, I am heartened by their call for more philanthropies to support development and dissemination efforts so that “a marketplace for instructional materials that rewards quality and innovation” can be created.

For philanthropies that aren’t quite convinced that curriculum matters, here’s one more study to add to the great work noted by Leifer and Udall. In “The Role of the School Curriculum in Social Mobility,” Cristina Iannelli shows that the content of the curriculum has lasting effects.* This study is important not only because of its findings, but because relatively few studies look at the actual courses students take. Iannelli is a professor in the UK; she used the UK’s National Child Development Study (NCDS), which tracks all babies born in the UK in 1958, gathering data at ages 7, 11, 16, 23, 33, 42, 46, and 50. As Iannelli writes (p. 910):

The people in the study were in secondary schools between 1969 and 1976 during the period of reorganisation of British secondary education from a selective to a comprehensive system. The coexistence of different secondary school systems at the end of the 1960s provides an excellent opportunity to study the effect of studying different curricula and attending different types of school on individuals’ chances of reaching the highest social classes of destination.

Focusing on social and occupational status at ages 23, 33, and 42, Iannelli’s findings at age 23 were what you’d expect: parental education and school selectivity had a big impact. Curriculum did too, but what’s really interesting is that the relative importance of curriculum went up—and the importance of parental education and school selectivity went down—as people aged (p. 923–924):

Selective schools, languages, English, mathematics and science subjects had a positive and significant effect on the chances of being in the top social classes and reduced the chances of entering the bottom classes…. [The] results suggest that the indirect effects of parental education via school types and curricula are stronger at the beginning of respondents’ occupational career than at later stages. The opposite is true for social class of origin: it is in the long run that school and curricular choices emerge more powerfully as transmitters of social advantages….

We tested whether the effect of curriculum and school type at age 33 and 42 was simply a result of their effect at age 23 and 33…. The results, before and after controlling for prior occupational destinations, barely change in the analysis of class of destination at 33, indicating that the effect of studying different subjects and attending various types of schools continues beyond the point of career entry. However, when analysing destinations at age 42 after controlling for destination at age 33, while the effect of subjects remains the same the effect of school types reduces and is no longer significant. These results suggest that the subjects studied at school are very good predictors of individuals’ destinations at all three stages of occupational career. On the other hand, the school type attended has a significant short-term and medium-term effect on individuals’ occupational destinations but they become less important for explaining later destinations. This may indicate that cognitive effects may be more persistent than institutional status effects…. The long-lasting effects of some school subjects may indicate that they provide skills, such as critical thinking and complex reasoning, which are useful for individuals’ future occupational careers. [Emphasis added.]

Studying languages, English, mathematics, and science does indeed enhance critical thinking and complex reasoning abilities. In fact, these crucial abilities can only be increased by developing rich knowledge. Cognitive science on how knowledge builds on knowledge, and skills depend on knowledge, would predict these occupational findings. The students who took many courses in these subjects began their careers with a solid foundation of knowledge and skills that enabled them to learn and grow at work.

We won’t ever be able to predict where each young person’s career will go, but we have plenty of evidence as to the type of education that offers the best preparation: a broad, rich, academic curriculum that builds content knowledge and skills together.

shutterstock_196046843

Mobility by knowledge courtesy of Shutterstock.

 

* Many thanks to Webs of Substance for finding this study!

Pariah to Politico 50

by Lisa Hansel
September 4th, 2014

After 30 years of being misunderstood, E. D. Hirsch’s dedication to equalizing opportunity is being widely recognized. Today, he has the honor of being included in the Politico 50.

E D Hirsch

As noted in Politico’s encapsulation of his work, “Hirsch’s argument was revolutionary: All children, regardless of background, should be taught the shared intellectual foundation—from Euclid to Shakespeare to Seneca Falls—needed ‘to thrive in the modern world.’”

Thriving, regardless of the accident of birth, has always been his driving force. In her brilliant article, “‘I’ve Been a Pariah for So Long,’” Peg Tyre explores how Hirsch has been revered and reviled:

In 1978, between stints as head of UVA’s English Department, Hirsch was conducting research at a nearby community college. There, he observed that the largely African-American low-income students could read short works of narrative fiction but could barely wring meaning from a piece about Gen. Robert E. Lee’s surrender to Gen. Ulysses S. Grant at Appomattox because they lacked basic knowledge about the Civil War. “What I saw is that background knowledge really mattered,” Hirsch says….

To level the playing field between rich and poor, schools should intentionally build background knowledge in all children in a wide range of subjects, or, says Hirsch, “It will be impossible to break the cycle of illiteracy that persists from parent to child.”

Which is where his List came in. To make it, Hirsch, along with two professors, scanned newspapers and popular journals for high-frequency concepts and then polled 500 professionals—lawyers and writers—to determine which of those concepts were the most crucial for cultural literacy….

OK, I say, that might have made sense in the 1970s, but what about now when the complexion of American public schools is changing? … How can any dictionary of cultural literacy keep pace with such a rapidly changing world? Hirsch grows crisp. “Why do you think one’s color or ethnicity would affect one’s vocabulary? Without a doubt, Latino culture is having a big influence on America, and the language of culture will change around the margins. But educated conversation is still going on. And you want to make sure those kids—particularly those kids—have the tools they need to be included.”

Tyre concludes, “In the autumn of life, with three grown children (two of whom are teachers), he’ll take vindication where he can. ‘The point wasn’t to perpetuate the culture of power,’ he tells me. ‘It was to open the door to kids who don’t have the keys to power.’”

What Can Preschools Learn about American History?

by Lisa Hansel
August 21st, 2014

To those who answer “nothing” or “they don’t need to learn it,” I have two responses—you’re wrong and you’re not trying hard enough to see the world through the preschoolers’ eyes.  Do they have some grand concept of human history and arrive eager to see how the American experiment fits in? No. But they are routinely confronted with hints that some unknown past exists, and that it must be pretty important—important enough to take a day or so off school.

Although in later grades the calendar is unlikely to be the most logical, efficient, or coherent way to teach content, for preschoolers, all those special days off with funny names offer a way to connect their present to our shared past. (In a nutshell, that’s why CKLA Preschool’s intro to American history follows the calendar, but the U.S. history domains in CKLA K–2 are in chronological order.)

Most of the content in CKLA Preschool is the standard stuff of early childhood: families, animals, Goldilocks, etc. It’s important, but too typical to capture my interest. The “Important People in American History” domain is different, raising questions about what we ought to expect of 4 and 5 year olds and what the purpose of preschool is. What I find particularly interesting in this domain is both how complex concepts are introduced and that a simplified (simplistic, even) introduction is enough.

Take the lessons on Barack Obama (which are done in conjunction with lessons on Martin Luther King, Jr., before and after his birthday). Preschoolers learn that Obama is our president—which is likened to the school principal—and that he became president because “many people” voted for him. (You weren’t expecting the Electoral College, were you?) In a read-aloud and discussion, the role of president is boiled down to things like talking on the phone to important people, reading and signing important papers, and thinking about what laws (or rules) the country should have. Some images and text are devoted to conveying that Obama is a real man with daughters who have to do homework and walk their dog. Fair enough. To me, the most interesting aspects of the lesson come after the read-aloud, when time is spent on two essential concepts: laws and voting. The teacher begins with a discussion of laws; while the teacher is free to do the discussion as s/he sees fit, the domain guide has suggestions for how to explain what laws are, such as:

Laws are special rules that everyone in the country must follow. Laws keep everyone safe and help everyone get along with each other. There is a law that we must wear a seatbelt in a car. There is also a law that all children must go to school. Laws are rules that everyone in our country obeys.

And

Laws are like the rules in our classroom. Rules in our classroom keep us safe and help us all get along with each other. One rule in our classroom is _______. What are some other rules in our classroom?

A teacher could take this as far as s/he likes, expanding into family and school rules and/or helping the children create laws.

To understand voting, teachers are encouraged to pick something the class may choose, such as whether to have goldfish or graham crackers for their next snack, and have the students cast ballots. The choice itself is not monumental (though some 4 year olds may beg to differ), yet critical aspects of voting become concrete: marking one’s choice, dropping the ballot in the ballot (shoe)box, counting the votes, and—perhaps most importantly—having to peacefully accept the majority’s choice. It’s a social skills lesson as much as a citizenship, history, and academic vocabulary lesson.

Those lessons alone are worthwhile, and they become even more so in later lessons. (You were expecting that—it’s Core Knowledge after all. CKLA Preschool’s content is coherent and cumulative even when it’s following the calendar.) The presidency is reviewed while learning about Abraham Lincoln (for Presidents’ Day) and the concept of laws is revisited in the context of learning about Justice Sonia Sotomayor (during March, Women’s History Month).

I thought the concept of a judge would be difficult to convey, but the domain guide offers suggestions to make it comprehensible—especially for children who have already thought about what laws are and why we need them:

A judge’s job is to listen to different people and help make decisions about rules and laws.

And

We have a rule in our class that everyone plays nicely, but sometimes not everyone knows that this means sharing your toys. Sometimes, I act like a judge and help everyone understand what playing nicely means.

After a discussion, the children are ready for the read-aloud, which focuses as much on Sotomayor’s childhood as her career. Which brings me back to the purpose of preschool—not just preparing children to do well in school, but to embrace school as an opportunity to find out what life has to offer. When we introduce preschoolers to important people in American history, we begin showing them just how diverse and significant the choices before them are. Those are the lessons that will last a lifetime; the sooner they grasp them, the better.

CKLA Preschool Sotomayor

This photo of Sotomayor being sworn into office (with her mother holding the Bible) by Chief Justice Roberts is image 7A-7 in CKLA Preschool’s “Important People in American History” domain.

The Skills Myth Might Kill You

by Lisa Hansel
August 13th, 2014

All of us in the Core Knowledge community are well aware of the risks of the skills myth: inadequate reading comprehension, limited critical thinking ability, inability to responsibly fulfill basic citizenship duties like researching issues before voting, etc.

Here’s a new risk: death.

Shockingly, I’m only kind of kidding.

As Annie Murphy Paul explained last week, the vast majority of new doctors think they don’t have to memorize all those pesky medical facts—they can just look them up:

A young doctor-in-training examines a new patient. Should she draw information for the diagnosis from her “E-memory”—electronic memory, the kind that’s available on a computer? Or should she dip into her “O-memory”—organic memory, the old-fashioned sort that resides in the brain?

Research shows that apprentice doctors are increasingly relying on E-memory, often in the form of a digital resource called UpToDate. This is an electronic reference tool, accessible on physicians’ laptops or mobile phones; tap in the patient’s symptoms, and up comes a potential diagnosis and a recommended course of treatment. A recent study found that 89 percent of medical residents regard UpToDate as their first choice for answering clinical questions.

I’m all for reference tools. The more important the decision, the more care we should take in checking our thinking. But doctors who believe such tools make memorization unnecessary are putting us all at risk. They are not developing their own web of knowledge, and so their ability to make connections will be limited. They will be less able to perform the human, artful, problem solving that is the heart of good medicine. This has consequences, as Paul reveals through Jerome P. Kassirer, a professor of medicine at Tufts University:

In medicine, writes Kassirer in an essay in the British Medical Journal, “we don’t always know what we need to know, and searches that are constrained to information we need at a given moment may not generate information that may be critically useful later.”… Kassirer offers an example from his own experience: “From the beginning of my third year at medical school I subscribed to two general medical journals, and I scoured each issue. Then, during my first week of internship, I was asked to examine a patient with hypotension, flushing, diarrhea, and hepatomegaly. About a year earlier a report on the carcinoid syndrome had caught my eye in one of the journals because of its unique metabolic characteristics. I correctly made the diagnosis because the article I had found in browsing had evoked the diagnosis.”

The more important the job, the more care we should take in building our expertise. Depending on E-memory results in shallow knowledge. As Paul writes:

Medical residents are, or should be, in the process of becoming experts, and that process involves building a rich and interconnected database of knowledge in one’s own mind. Research in cognitive science and psychology demonstrates that the ability to make quick and accurate judgments depends on the possession of extensive factual knowledge stored in memory — in internal, organic memory, that is, and not in a device.

shutterstock_90266317

Terrifying photo courtesy of Shutterstock.

 

“Houston, we have a problem”

by Lisa Hansel
July 23rd, 2014

We do indeed have a crisis on our hands, but year after year we fail to diagnose and address it. With 21st century skills, learning styles, comprehension strategies, blame-the-teacher “reforms,” and dozens of other fads clouding our thinking, research-driven common sense improvements get little attention.

It’s frustrating, but our Core Knowledge community is dedicated to spreading the word on rigorous academics. For anyone out there who needs yet more evidence of the desperate need for building broad knowledge and skills, two new reports are worth examining.

“Just the facts, ma’am”

Cold, hard facts are what we get from ACT and Mathematica Policy Research. We learn (yet again) that there are massive disparities in preparation for college and kindergarten.

ACT’s The Condition of College & Career Readiness tackles the high school problem with stark graphics. The one below, showing the massive gaps among youth by race and ethnicity, is especially striking:

ACT 7-22-14 A

Then, a ray of hope. Taking a “core curriculum” in high school appears to greatly increase the odds that a young adult is well prepared. In the chart below, “Core” stands for core curriculum, which ACT defines as “4 years of English and 3 years each of mathematics, science, and social studies” in high school.

ACT 7-22-14 B

That gives us one clear step to take in closing college- and career-readiness gaps. But things are never so simple. You see, most students are already taking a core curriculum:

ACT 7-22-14 C

Clearly, all core curricula are not created equal. But we know better that to lay all blame at the high school doorstep. And in case we forget, Mathematica’s Kindergartners’ Skills at School Entry: An Analysis of the ECLS-K reminds us. This study is interesting because it does not look just at the usual race/ethnicity and income factors. Instead, it focuses on four specific “risk factors”: “the child lives in a single-parent household, the child’s mother has less than a high school education, the child’s household income is below the federal poverty line, and the primary language spoken in the home is not English.”

You may be surprised to see that nearly half—44%—of entering kindergartners face at least one of these risk factors:

Mathematica 7-22-14 A

Sadly, you may be even more surprised to see how devastating even just one risk factor is in terms of reading, math, and working memory:

Mathematica 7-22-14 B

Mathematica 7-22-14 C

(Note: IRT stands for “item response theory.” The children were given two-stage assessments in which their performance in the first stage determined the difficulty of the test items they were given in the second stage.)

If these two new reports tell us anything, it’s that we must intervene early. Gaps that exist at kindergarten entry still exist at the end of high school—ripe for replication when our underprepared young adults have children of their own.

“May the Force be with you”

Schools with coherent, cumulative curricula that build academic knowledge, vocabulary, and skills are intervening. Curriculum is not the solution, of course, but it is a necessary part of the foundation for student (and teacher) learning. Unfortunately, far too many school, district, and policy leaders are unaware of how to make their curricula stronger, much less how to harness a rigorous curriculum for benefits such as early identification of students’ needs and increased teacher collaboration. For those looking to take the first step, I strongly recommend Harvard’s Lead for Literacy series. In 16 one-page memos, Lead for Literacy clearly identifies best practices for literacy programs, assessments, professional development, and program selection. The series may not be as powerful as the Force, but they’ll give leaders a good shot at dramatically increasing students’ knowledge and skills, and enabling them to learn more both in and out of school.