Can’t Anybody Here Play This Game?

by Robert Pondiscio
May 6th, 2009

 A federal study on the effectiveness of different reading comprehension programs found three of the programs had no impact, while the fourth had a negative impact.  None of the four—Project CRISS, ReadAbout, Read for Real, and Reading for Knowledge—was found to be effective. 

It’s a dispiriting report, but Robert E. Slavin of the Success for All Foundation, makes an important point about it. He tells EdWeek’s Mary Ann Zehr that “IES-sponsored evaluations repeatedly evaluate programs by imposing them on teachers and school leaders who are not interested in them and are likely to implement them haphazardly, if at all, and then find, over and over again, that nothing works.”

There are is no shortage of bogus reading programs out there that overpromise and underdeliver.  That said, fidelity of implementation is huge and nearly impossible to evaluate.  There’s simply no effective way to tell if teachers believe in what they are teaching, simply going through the motions, or not using it at all.   You can’t impose a curriculum on unwilling schools and teachers and expect it to work.  High expectations matter for more than just students.

18 Comments »

  1. Robert
    I agree with Slavin only up to a point.

    Curriculum developer: This curriculum is great!
    Dan: Sweet. How can we be sure that teachers are implementing it correctly?
    CD: We can’t.
    Dan: Can’t you write down on a piece of paper “If teachers do X, Y, and Z, then we predict kids will learn to read?”
    CD: No. What about teacher enthusiasm?
    Dan: Can’t an observer get even a rough idea of enthusiasm?
    CD: No. We can never really measure whether or not teachers are implementing a curriculum as we intend it.
    Dan: See ya.

    Comment by Dan Willingham — May 6, 2009 @ 4:41 pm

  2. What you’re making then is the argument “it-may-not-be-perfect-but-it’s-best-we-can-do” case. It seems impossible to me to draw conclusions about a curriculum unless it’s tried in optimal conditions. A drive-by 30 minute evaluation isn’t even close to acceptable (non teachers, I think, fail to appreciate the degree to which teachers put on a show for visitors). When I taught 5th grade I had Everyday Math foisted on me. I’m not a fan. Let’s just say that as a curriculum, my implementation was more honored in the breach than the observance. It would not have been fair or accurate for my students scores — good OR bad — to be attributed to EDM.

    Comment by Robert Pondiscio — May 6, 2009 @ 5:02 pm

  3. Robert,

    OTOH, would it be fair to attribute them to you? This will become an increasingly non-hypothetical problem when our super-awesome new data systems are implemented and individual teachers are tracked by performance. When a math teacher who has gotten $5,000 performance bonuses three years in a row is told they have to use a curriculum they don’t think will be as effective, well, it is going to be ugly.

    Also, “You can’t impose a curriculum on unwilling schools and teachers and expect it to work.” is pretty much how I feel about CK — I have no doubt that it is very effective in the right schools. Other approaches work better in other contexts and communities. I just don’t get why you don’t take the “different strokes” approach and try to defend curricular diversity and build your market share instead of rolling the dice on a national curriculum.

    Comment by Tom Hoffman — May 6, 2009 @ 9:01 pm

  4. Even if only 1/3 of the teachers had enthusiasm, and the rest stuck with the old methods, shouldn’t there be a measurable improvement if the program is genuinely superior to the old methods?

    I’m not familiar with these programs, but I imagine they are to solid liberal arts curriculum what microwave meals are to whole foods.

    Comment by Ben F — May 6, 2009 @ 9:03 pm

  5. Robert
    I think there are two points to be made:
    1) “It seems impossible to me to draw conclusions about a curriculum unless it’s tried in optimal conditions” I think you could set two different criteria: how the curriculum fares under optimal conditions, or how it fares under real-world conditions. Saying that the test of the curriculum was unfair because it was foisted on teachers who didn’t like it is fine. . . but it restricts the possible applicability of the curriculum. We’re no longer saying “here’s a great way to teach kids to read,” we’re saying “here’s a great way to teach kids to read, under optimal conditions. . . . .” I think anyone would agree that teacher buy-in is crucial, but the sensitivity of the curriculum to this factor surely varies, AND the degree to which curriculum planners realize this and make PD part of their plan varies. Slavin makes it sound as though the problem is the stupid IES studies. I’m arguing that getting teacher buy-in is part of the curriculum. If you’re not getting it, the curriculum fails.
    2) Your point that fidelity of implementation is nearly impossible to evaluate. I disagree. In addition to classroom observation, social psychologists have lots of methods of indirectly evaluating behaviors that people may not want to admit, and of making self-report more valid for these behaviors.

    Comment by Dan Willingham — May 7, 2009 @ 4:30 am

  6. The IES findings are consistent with my personal experience. My district adopted a reading program that seemed to be having a negative effect after the first year. The developers assured the authorities that the positive results would be plentiful after three years..just be patient, etc. Didn’t happen. Lots of money wasted, larger numbers of deficient readers coming through the system.

    Comment by rstanton — May 7, 2009 @ 6:04 am

  7. Slavin has a history of complaining when the scientifically based research demonstrates the inadequacy of the reading programs he develops. Large scale randomly assigned treatment groups are the gold standard in research. If the IES had “cherry-picked” a group of teachers who were fervent supporters of Slavin’s reading program, their results would have been useless. Random assignment is essential in providing a basis for determining the probability that a given treatment is effective.

    Everyday Mathematics is a case in point. Based on the results of studies involving the random assignment of students and teachers to this program, the IES has been able to demonstrate that Everyday Mathematics is a moderately effective intervention. It doesn’t really matter what individual teachers feel about the program. The effectiveness of the program is determined by its overall effect on student outcomes with a wide range of teachers.

    Comment by Judy — May 7, 2009 @ 11:52 pm

  8. [Dan] I’m arguing that getting teacher buy-in is part of the curriculum. If you’re not getting it, the curriculum fails.

    [Nancy] Bingo. (And if the only way to get teacher buy-in is hiring inexperienced or compliant teachers the problem is compounded, but that’s another discussion. “Fidelity” is a highly overrated trait in classroom practice.)

    The flip side of this discussion is the reality that there are curricula that work well in some contexts and not in others. If we can only choose programs based on the “overall effect on student outcomes with a wide range of teachers,” we’re stuck — and I use that word deliberately — with some pretty awful, highly standardized and undistinguished curriculum-instruction models, whose sole purpose is moving the data needle.

    It does matter how individual teachers or schools feel about curricular programs, just as it matters how physicians feel as they select a treatment. Research using “large-scale randomly assigned treatment groups” may suggest an optimum standard of medical care, but in the end, the decision is up to the patient, family and physician. And sometimes, the gold standard is absolutely the wrong choice for the outcomes desired. Heartbreaking stories support this principle, in any number professional fields.

    This process is called human judgment–and the more we remove judgment from the hands of teachers, and put our faith in large-scale, randomly-assigned etc etc etc, the less investment the people actually doing this work, professional teachers, have in the learning outcomes of their students.

    Even Chesley Sullenberger, in his incredibly standardized work, had to make a judgment. Teaching human beings is a messy business.

    Comment by Nancy Flanagan — May 8, 2009 @ 1:47 pm

  9. Please don’t give up on the insight into teaching practices provided by high quality scientific research. Check out what’s actually on the What Works Clearinghouse. (There’s a link on this page.) Better yet, visit their companion site, Doing What Works(dww.ed.gov/index.cfm), which presents the research findings in more accessible, teacher friendly language. What you find might surprise and please you.

    It turns out that rigid, heavily scripted programs that call for clipboard toting monitors to ensure that teachers are sticking to the script instead of using their professional judgment produce mediocre results. What really works? Teachers who “ask higher order questions and provide rich opportunities for students to develop explanations.” The scientific support for this practice is strong based on the rigorous standards of the IES. What is so exciting about the findings of genuinely scientific research studies is the extent to which they support teacher empowerment. Whenever you come across a “research study” that claims to support a mindless, robot-teacher approach to education, try checking it out on the What Works Clearinghouse. You’ll find that such studies rarely, if ever, meet true scientific standards.

    Comment by Ray — May 9, 2009 @ 11:02 am

  10. I’m confused about a few things here. Is there a meaningful distinction between a “method”, a “program”, and a “curriculum”? I am not familiar with any of the four “reading comprehension programs” mentioned. I have been similarly confused in the past when Reading First was under discussion. Was it a success or a failure? Has it disappeared? Have we learned anything from it?

    The discussion on Reading First did prompt me to define my thinking a bit. I realized my perspective is apparently different from the perspective of many others (but I wonder if it is substantially different from practicing teachers of reading.) To me it doesn’t make much sense to expect a reading “program” to improve results. All a program can be is materials, which primarily means books. What a teacher does with those materials, it seems to me, is what really counts. Can a “program” control that? From what I read in the blogs, apparently they try.

    I can understand why a publisher, when promoting a reading “program” would want to convince potential customers that it is a lot more than just materials. But that doesn’t make it true. I don’t know of any reasoning or evidence that it is true. I would consider it all seller’s puff. So this brings to mind an unpleasant picture. High level administrators believe the sales pitch, and teachers “on the ground” are stuck with it. And thus arise the concepts of “resistant teacher” and “fidelity of implementation”. That those concepts arise at all is a red flag to me.

    I have complained before that we lack a solid basis of simple description in the study of education. What happens in the classroom of one of these programs mentioned, and how is it different than what happens in the classroom of another of those programs? And how is it different, if indeed it really is, from what teachers have always been doing? What practices actually make up the everyday routine of teaching young children to read? I would argue that until we do a much better job of answering questions like that we should not expect progress.

    Comment by Brian Rude — May 9, 2009 @ 2:04 pm

  11. In medicine (simplified)
    Pre-Phase-1: Test tubes or mice.
    Phase 1: 30 patients or so.
    Phase 2: 100 or so.
    Phase 3: 1000, randomized.

    What I’d suggest for education:
    Divide Phase 3 into two parts
    Phase 3A: randomized, but with willing/compliant participants
    Phase 3B: randomized, but no check on willingness/compliance

    Any good school leader or teacher would love to know Phase 3A results, no?

    Why do they care if, in the hands of people who resist change, the program doesn’t work? That’s the worry of, say, a superintendent.

    A program with great Phase 3A results could help a lot of kids.

    Comment by Mike G — May 10, 2009 @ 4:47 pm

  12. There are no shortage of bogus reading programs out there that overpromise and underdeliver.

    Who wrote the above? It should be “There is no shortage…” Shortage is singular.

    Comment by Jeremiah Reedy — May 15, 2009 @ 9:01 am

  13. Judy…I couldn”t agree with you more about Slavin. In the real world, when employees are non-compliant…they get fired. Maybe that’s the problem with the protectionist environment we have created in education. Would you put up with this in ANTY other field of work? One of the biggest problems in education is getting faculty buy-in. Who is really our client in this industry called education? The child or the teacher? What it should be and what it is…is two differnt worlds. Mmmm…wonder what we should do?

    Comment by Mike — May 15, 2009 @ 9:33 am

  14. Speaking from the perspective of a producer of decoding and spelling materials, I agree entirely with Dan’s implication that our industry is very bad at looking at things from a teacher’s perspective. All too often, gifted teachers and educators devise complex teaching programs which they can deliver with a very high degree of success. Or you have educational publishers who develop programs which answer to the lastest educational fads (eg, learning styles, 21st century skills) because they know they will sell.

    We’ve approached things from a different angle. Long ago we recognised that most children with ‘learning difficulties’ make poor progress if they don’t have short daily sessions. We also determined that one-to-one instruction for a very short time each day (10 minutes or less) is more effective than longer sessions with small groups. And furthermore, we found that the great majority of primary school teachers fear and loath computer-assisted learning; and that all of them already have far too much to do.

    So we decided to write books which had a simple, but carefully-programmed approach. They use straightforward behaviorist principles. We introduce new material with massed practice, and reinforce it with carefully-sequenced distributed practice. About 95% of each lesson is the latter, so pupils are always ‘getting it right’.

    Our books were designed to be used by untrained classroom assistants, parents, or even older students; thus allowing class teachers to delegate their worst headaches. They require no lesson-planning–and this is also a major factor in our booming sales. You just pick up the book and carry on from where you–or some other classroom assistant– left off the day before. Although there are several different formats which ensure that overlearning isn’t deadening, the formats are constantly repeated so the teacher isn’t always struggling to come to grips with new procedures.

    But most importantly, we have made constant revisions in response to teachers’ suggestions, as well as our own experience working with our own private pupils. Our main series, Dancing Bears, is already in its fifth edition, even though it is only 6 years old.

    However, I would never suggest that any teacher or school should be forced to use our materials (or anyone esle’s). Why in heaven’s name should teachers and schools be forced to use anything? Free markets produce vastly superior results. The Trabant was the ultimate product of socialist engineering–a car which, in most respects, was much worse than a pre-war Volkswagen. Compare a DC school with a KIPP academy–this tells you everything you need to know about relying upon committees of experts.

    Incidentally, I regard the ‘What works clearinghouse’ as useless. They accept studies conducted by the sponsors of the material being evaluated. Only an educator could fail to see the glaring conflict of interest involved.

    Comment by Tom Burkard — May 15, 2009 @ 9:39 am

  15. “Other approaches work better in other contexts and communities.”

    Among the many fundamental misunderstandings about CK (as I see it) is that it is an “approach,” that it “works” for some “learning styles,” but not for others.

    I always thought that CK was about content and knowledge, not about teaching techniques or learning styles. Teach outside, teach standing on your head, teach in a tree…. Doesn’t matter; just get the tanks filled with the right gas.

    When I was a reading “mentor” for 4th graders, I had the kids read outloud(from “What Every 4th-grader Should Know”)while marching up and down the hallway. They got it… I also read to my son from the series every morning over breakfast… He recalled those breakfast sessions (fondly) in one of his college application essays (he’s in!)…

    –pm

    Comment by Peter Meyer — May 15, 2009 @ 12:50 pm

  16. A curriculm should be designed so that it engages the teachers and the students. A “great” curriculum that is hard to deliver for teachers is not a great curriculum.

    At the end of the day, teachers resist change. But one attribute of the best programs are that teachers find them easy to implement and of obvious value to teachers.

    While it’s a tool, not a curriculum, SpellingCity.com is an interesting example. It’s a spelling website for teachers. There is no sales process for it since it’s free. There’s no pressure on teachers to use it from sales people. Yet, over 50,000 teachers found out about it and start using it with their classes this school year. SpellingCity’s site design engaged both the teachers and the students from their initial view. That should be one of the design goal of a curriculum.

    Comment by john edelson — May 15, 2009 @ 3:02 pm

  17. I agree that the programs may have been found ineffective in the area of building reading comprehension. The majority of curriculum doesn’t really “teach” reading comprehension. The majority of curriculum “tests” comprehension. Along with the results of the study, I would like to know if the students were first assessed for reading level. If you ask a 4th grader to read a 4th grade reading level book and answer the questions at the end of each chapter, you could use the results of the end of chapter test to assess whether or not the student comprehended text. However, if that 4th grade student was reading on a 1st grade level, I could guarantee you that he would fail the reading comprehension test. I think Read for Real is a wonderful text for content and reading practice. But I would never advocate its use to teach or to test reading comprehension. I am sure there are many quality programs for sale teach reading comprehension. My favorite is the Lindamood Bell Visualize and Verbalize. I have been using that program for about 15 years, and in that time, only one student failed to significantly improve his reading comprehension level. That program TEACHES reading comprehension rather than test it.

    Comment by Eileen Card — May 16, 2009 @ 6:30 pm

  18. I for one hope we NEVER have one national curriculum, or centralized oversight of the schools, either. Education Law should be in the hands of the state. If I really dislike the law in one state, I can a) fight for better law, or b) move, if I’m really adamant about it. However, if it is a national law, and uses national curriculum, if it doesn’t work for my kids or my family, I am out of luck. I don’t think any one curriculum can meet the needs of all children. I love Core Knowledge, but there are other curricula I love, too.

    Comment by Lee Willis — July 7, 2009 @ 7:46 pm

RSS feed for comments on this post. TrackBack URL

Leave a comment

While the Core Knowledge Foundation wants to hear from readers of this blog, it reserves the right to not post comments online and to edit them for content and appropriateness.