Choosing Curriculum Without Evidence

by Robert Pondiscio
April 13th, 2012

If you wanted to improve medical care, would you focus on hospital administration and patient insurance?  Or would you look at the treatment doctors were giving patients?  Would you try to improve a sports team’s won-loss record by focusing on stadium layout and the team’s travel schedule?  Then why, ask Brookings’ Matthew Chingos and Russ Whitehurst, do education policy makers focus most of their attention on academic standards, teacher evaluation, and school accountability policies?  Shouldn’t we be looking instead at instructional materials?

“There is strong evidence that the choice of instructional materials has large effects on student learning—effects that rival in size those that are associated with differences in teacher effectiveness,” the two write in a new paper from Brookings, Choosing Blindly: Instructional Materials, Teacher Effectiveness, and the Common Core.

“Whereas improving teacher quality through changes in the preparation and professional development of teachers and the human resources policies surrounding their employment is challenging, expensive, and time-consuming, making better choices among available instructional materials should be relatively easy, inexpensive, and quick.”

There’s one big hurdle to clear in correcting this rather obvious problem: Little effort has been made by the field to differentiate effective curricular materials from ineffective ones.  In fact, in most states, districts and schools, it’s nearly impossible to know what materials are being used at all.

“In every state except one, it is impossible to find out what materials districts are currently using without contacting the districts one at a time to ask them. And the districts may not even know what materials they use if adoption decisions are made by individual schools. The National Center for Education Statistics (NCES), which has the mission of collecting and disseminating information related to education in the U.S., collects no information on the usage of particular instructional materials.”

Chingos and Whitehurst predict the blindness on curriculum will become a critical problem for Common Core Standards implementation.  “Publishers of instructional materials are lining up to declare the alignment of their materials with the Common Core standards using the most superficial of definitions,” they note.  “The Common Core standards will only have a chance of raising student achievement if they are implemented with high-quality materials, but there is currently no basis to measure the quality of materials. Efforts to improve teacher effectiveness will also fall short if they focus solely on the selection and retention of teachers and ignore the instructional tools that teachers are given to practice their craft.”

The paper offers up a number of suggestions:  State education agencies should collect data from districts on the instructional materials in use in their schools.   also wants to see the National Governors Association (NGA) and Council of Chief State School Officers (CCSSO) “put their considerable weight behind the effort to improve the collection of information on instructional materials in order to create an environment in which states, districts, and schools will be able to choose the materials most likely to help students master the content laid out in the Common Core standards.”

Chingos and Whitehurst are dead-on in their critique of ed reform’s indifference to curriculum and materials.  When we focus on the mechanism by which schools are  created, managed, financed or evaluated, we are assuming that what kids learn, and with which materials, is pretty much settled, or doesn’t really matter.  All that’s left to do is figure out what works in terms of delivery of instruction and grow it, or figure out what doesn’t work and shut it down.  Any teacher who has worked with different literacy or math programs can easily attest this is not the case.

Whitehurst: Fund Curriculum Research

by Robert Pondiscio
June 11th, 2010

A new report by Brookings’ Russ Whitehurst offers four ideas which “offer substantial promise for improving American education, are achievable and have low costs.”  The first one is to ”choose K–12 curriculum based on evidence of effectiveness.” 

Little attention has been paid to choice of curriculum as a driver of student achievement. Yet the evidence for large curriculum effects is persuasive. Consider a recent study of first-grade math curricula, reported by the National Center for Education Evaluation and Regional Assistance in February 2009. The researchers randomly matched schools with one of four widely used curricula. Two curricula were clear winners, generating three months’ more learning over a nine-month school year than the other two. This is a big effect on achievement, and it is essentially free because the more effective curricula cost no more than the others.

There are myriad moving parts in a good educational outcome, especially for low-SES and minority children.  Curriculum is the least appreciated moving part and isn’t nearly as fun to fight about as structural reforms like charter schools, merit pay, tenure and accountabilty. Among the ed reformers who drive the policy agenda (and have never written a lesson plan), there is a strong tendency to see curriculum, as the National Review’s Jim Manzi put it, as “motherhood and apple pie” or simply see it subsumed within the teacher quality debate, i.e., good teachers make good curricular decisions.  Less appreciated is how a sequenced, structured curriculum can improve teacher quality by allowing teachers to hone their craft, focusing on delivering instruction and differentiation, rather than the enormously difficult and time consuming work of deciding what to teach, and aligning curricular decisions with state and local standards. Curriculum, as Whitehurst has consistently argued, is critical.

“The federal government should fund many more comparative effectiveness trials of curricula, and schools using federal funds to support the education of disadvantaged students should be required to use evidence of effectiveness in the choice of curriculum materials. The Obama administration supports comparative effectiveness research in health care,” Whitehurst notes.  “It is no less important in education.”

Hear, hear.   Whitehurst’s paper also calls for evaluating teachers “in ways that meaningfully differentiate levels of performance”;  accrediting online education providers so they can compete with traditional schools across district and state lines; and providing the public with information that will allow comparison of the labor-market outcomes and price of individual post-secondary degree and certificate programs.

Not Either/Or…It’s AND

by Robert Pondiscio
October 28th, 2009

At Eduwonk, Andy Rotherham catches up to Russ Whitehurst’s paper, Don’t Forget Curriculum.  But he misses the boat when he writes, “I’m not sure when curriculum and reforms like choice, teacher quality, etc…became either/or.”   I’m not sure where Andy’s getting that message, but it’s not from Russ Whitehurst, who went out of his way NOT to say that.  Here’s the relevant quote from his paper:

This is not to say that curriculum reforms should be pursued instead of efforts to create more choice and competition through charters, or to reconstitute the teacher workforce towards higher levels of effectiveness, or to establish high quality, intensive, and targeted preschool programs, all of which have evidence of effectiveness. It is to say that leaving curriculum reform off the table or giving it a very small place makes no sense.

Over at the American Enterprise Institute’s blog, Charles Murray adds his voice to the curriculum choir.

Curriculum: More Reform for Less Money

by Robert Pondiscio
October 15th, 2009

From Day One, among this blog’s raisons d’être has been to say to ed reformers of  every stripe “don’t forget curriculum.”  So it’s great to hear Brookings’ Russ Whitehurst say the same thing–and with cold, hard data to back it up.   In his latest Letter on Education, Whitehurst lays out an argument that should catch the eye of everyone who is focused on charter schools, teacher quality, early childhood ed and standards as the means of boosting student achievement.  He looks at the effect sizes of those reforms and reports curriculum effects have a much greater impact than all of them:

Further, in many cases they are a free good. That is, there are minimal differences between the costs of purchase and implementation of more vs. less effective curricula. In contrast, the other policy levers reviewed here range from very to extremely expensive and often carry with them significant political challenges, e.g., union opposition to merit pay for teachers. This is not to say that curriculum reforms should be pursued instead of efforts to create more choice and competition through charters, or to reconstitute the teacher workforce towards higher levels of effectiveness, or to establish high quality, intensive, and targeted preschool programs, all of which have evidence of effectiveness. It is to say that leaving curriculum reform off the table or giving it a very small place makes no sense. Let’s do what works for the kids, and let’s give particular attention to efficient and practical ways of doing so.

“We conclude that the effect sizes for curriculum are larger, more certain, and less expensive than for the Obama-favored policy levers,” writes Whitehurst, the former director of the Institute of Education Sciences.  He recommends the Administration “integrate curriculum innovation and reform into its policy framework.”

The Department of Education, through the Institute of Education Sciences, should fund many more comparative effectiveness trials of curricula and other interventions, both through its National Center for Education Evaluation and through competitive grants to university-based researchers. The Obama administration has clearly recognized the importance of comparative effectiveness research in health care reform. It is no less important in education reform.”

Can I get an amen?