The MET Research Paper: Achievement of What?

by Guest Blogger
December 19th, 2010

by Diana Senechal

A new study by the Measures of Effective Teaching (MET) Project, funded by the Bill and Melinda Gates Foundation, finds that students’ perceptions of their teachers correlate with the teachers’ value-added scores; in other words, “students seem to know effective teaching when they experience it.” The correlation is stronger for mathematics than for ELA; this is one of many discrepancies between math and ELA in the study. According to the authors, “outside the early elementary grades when students are first learning to read, teachers may have limited impacts on general reading comprehension.” This peculiar observation should raise questions about curriculum, but curriculum does not come up in the report.

When the researchers combined student feedback and math value-added (from state tests) into a single score, they found that “the difference between bottom and top quartile was .21 student standard deviations, roughly equivalent to 7.49 months of schooling in a 9-month school year.” For ELA, the difference between top and bottom quartle teachers was much smaller, at .078 student-level standard deviations.

What are students learning in ELA? Beginning in fourth grade, students appear to gain just as much in reading comprehension from April to October as from October to April—that is, the summer months away from school do not seem to affect their gains. According to the researchers, “the above pattern implies that schooling itself may have little impact on standard read­ing comprehension assessments after 3rd grade.” They posit, somewhat innocently, that “literacy includes more than reading comprehension … It involves writing as well.” The lack of teacher effects applied mainly to the state tests;  when the researchers administered the written Stanford 9 Open-Ended Assessment for ELA, the teacher effects were larger than for math.

What explains the relatively low teacher effects on the ELA state tests? The researchers offer two possibilities: (a) teacher effects on reading comprehension are small after the early elementary years and (b) the tests themselves may fail to capture the teachers’ impact on literacy. Both of these hypotheses seem plausible but tangential to the central problem: this amorphous concept of “literacy.” Why should schools focus on “literacy” in the first place? Why not literature and other subjects?

A curious detail may offer a clue to the problem: the correlation between value-added on state tests and the Stanford 9 in ELA is low (0.37). That is, teachers whose students see gains on the ELA state tests are not very likely to see gains on the Stanford 9 as well.  That is, teachers whose students see gains on the ELA state tests are unlikely to see gains on the Stanford 9 as well. (The researchers do not state whether the reverse is true.) The researchers thought some of this might be due to the “change in tests in NYC this year.” When they removed NYC from the equation, the correlation was significantly higher. (But the New York math tests changed this year as well, and this apparently did not affect things—the correlation for math between the state and BAM value-added is “moderately large” at 0.54.)

Is it not possible that NYC suffers from a weak or nonexistent ELA curriculum, more so than the other districts in the study? Certainly curriculum should be considered, if an entire district shows markedly different results from the others.

In math, there usually is a curriculum. It may be strong or weak, focused or scattered, but there is actual material that students are expected to learn. In ELA, this may or may not be the case. In schools and districts with a rigorous English curriculum (as opposed to a literacy program), students read and discuss challenging literary works, study grammar and etymology, write expository essays, and  more. In the majority of New York City public schools, by contrast, this kind of concrete learning is eschewed; lessons tend to focus on a reading strategy, and students practice the strategy on their separate books. New York City has taken the strategy approach since 2003 (and in some cases much earlier); Balanced Literacy, or a version of it, is the mandated ELA program in most NYC elementary and middle schools. The MET researchers do not consider curriculum at all; they seem to assume that a curriculum exists in each of the schools and that it is consistent within a district.

In short, when analyzing teacher effects on achievement gains, the researchers forgot to ask: achievement of what? This is not a trivial question; the answers could shed light on the value-added results and their implications. It may turn out that the curricular differences are too slight or vague to make a difference, or that they do not significantly affect performance on these particular tests. Or the investigation of such differences may turn the whole study upside down. In any case, it is a mistake to ignore the question.

Diana Senechal taught for four years in the New York City public schools and holds a Ph.D. in Slavic languages and literatures from Yale. Her book, Republic of Noise: The Loss of Solitude in Schools and Culture, will be published by Rowman & Littlefield Education in late 2011.

 
 

 

 

23 Comments »

  1. It always surprises me that “core” targets the delivery of hard data information via curriculum as the only deliverable schools may achieve: how much of certain kinds of information do students identify on a multiple choice test as accurately as possible. Ironically, schools teach much more than teachers teach: students learn from each other much more than from a teacher, particularly beyond grade 3 or so. They learn interpersonal skills, manipulative and organizational skills, entrepreneurship. They learn to sell or to buy, drugs, sex, or power. They learn to work within or across cultures, languages, and class. Those are substantive learnings, and measured by no metric known to academics.

    Such learning suggests we may need to know what teachers AVOID TEACHING, or TEACH AGAINST, no less than what books kids read or how elaborate an equation they may solve. And it is particularly ironic that “core knowledge” is as blind to these learnings as it is to the other, more legitimate organizing of information that is critical to school and maturity: variables like the “soft skills” like responsibility, teamwork, inquiry, work across cultures, etc., or like negotiating class, manipulating college or career, or even “basic” information like financial literacy.

    In framing rewards for good teachers, incidentally, it also strikes me that the only rewards that seem credible are financial, while, in spite of such measures, most teachers teach for non-financial incentives like fulfillment, expression, and exploring knowledge. Rewarding a good teacher with a good class, for one example, seems both kind and – ultimately – a disaster for the rest of the school. Yet it is precisely that kind of a reward which tenure, and long standing in a school, tends to deliver.

    Comment by Joe Beckmann — December 19, 2010 @ 1:37 pm

  2. “…it is particularly ironic that “core knowledge” is as blind to these learnings as it is to the other, more legitimate organizing of information that is critical to school and maturity: variables like the “soft skills” like responsibility, teamwork, inquiry, work across cultures, etc., or like negotiating class, manipulating college or career, or even “basic” information like financial literacy.”

    Joey, Joey, Joey. How about these skills from the affective domain you’re so concerned about be taught at home or less emphasized in schools? Ever think schools were there to emphasize cognitive skills FIRST and the soft stuff be relegated to parents or a trickle down philosophy in schools?

    If you’re really looking for teachers to emphasize some of these twenty-first century skills you could try Antioch College in Silver Springs, Ohio. Oh wait. They shut down recently because of insufficient enrollment, didn’t they? I wonder why they couldn’t attract enough students to attend?

    Sorry to sound so sarcastic Joe, but you’re preaching to the wrong choir when you post something like this on the Core Knowledge blog. You’re barking up the wrong tree.

    Comment by Paul Hoss — December 20, 2010 @ 9:38 am

  3. Paul is right of course, but here is my take.

    There is a certain appeal to 21st Century Skills, to de-emphasizing content and focusing on collaboration, inquiry, emotional IQ, reading as a skill, using technology etc. That appeal is that teaching and learning can be made WAY easier. All we need to do is teach a kid to read and use a computer and play nice with others and then that kid can google anything they ever will need to know and then share it with his friends collaboratively. A cure for cancer is sure to follow.

    In reality knowledge and skills are hard won and they come from the deliberate building of knowledge in a logical and purposeful way. Kirschner, Sweller and Clark (see the current American Educator) essentially argue that there is no such thing as reading, researching or collaborating independent of a content domain. Collaborating to build a boat is different than collaborating to write a book. Applying scientific inquiry to cure diseases is different than applying scientific inquiry to find new planets. And yes reading The Scarlett Letter is different from reading the latest issue of Nature.

    There are no shortcuts. Learning and developing skills is hard. Let’s stop pretending it isn’t.

    Comment by Matt — December 20, 2010 @ 10:37 am

  4. Well said, Matt. A case could be made that as educators, we spend our lives in search of the instructional equivalent of the Northwest Passage. And with the same rate of success.

    Comment by Robert Pondiscio — December 20, 2010 @ 10:47 am

  5. Admittedly, I am one teacher in one school in NYC, but I can share my experience as an ELA teacher. We have NO curriculum. Attempts have been made over the years to create one, but have fallen short due to issues of organization, leadership, and money (for materials and additional planning time). Before mayoral control and balanced literacy, my school used the Harcourt Elements of Literature program. While it wasn’t perfect, or a true curriculum, it was pretty comprehensive, addressing multiple genres, reading strategies, vocabulary, grammar skills. We had a very hard-nosed administrator for a few years who was very “my way or the highway” but didn’t really have any insights on managing instruction with 30 kids reading 30 different books. I do the best I can on my own, identifying skills and standards and trying to do the best I can. At least now we have class sets of novels, and they are high-quality literature, but now the emphasis is on non-fiction and we have little of that. My school was also split into 3 small schools, which means I am the only ELA teacher for my grade. There is no one to collaborate with, at least in terms of grade-level planning. Trying to develop a good curriculum by myself, despite my experience, is very, very challenging. More than once I’ve wondered if our kids’ scores would be different if only we could get a solid curriculum together.

    Comment by TJ — December 20, 2010 @ 11:00 am

  6. “…I am one teacher in one school in NYC, but I can share my experience as an ELA teacher. We have NO curriculum.”

    Robert and Diana, is this true? In a school district of a million kids, how could this be?

    Comment by Paul Hoss — December 20, 2010 @ 5:08 pm

  7. Completely, 100% true. My school, like many, was saddled with the content-free Teacher’s College Readers and Writer’s Workshop, which as one staff developer reminded me often “is not a curriculum. It’s a philosophy.” So we had a philosophy.

    Comment by Robert Pondiscio — December 20, 2010 @ 5:23 pm

  8. Mayor Bloomberg and Joel Klein have had no interest in developing curricula to correspond with the state standards? No one clued these two in as to the importance of a quality curriculum?

    These two leaders are in charge of the largest school district in the country and they missed/avoided such an important variable as curricula?

    Say it ain’t so, Joe DiMaggio! On an unrelated note, Robert, are the Yankees in trouble for 2011, or what? They’d be better off if they hired George Costanza as GM.

    Comment by Paul Hoss — December 20, 2010 @ 6:35 pm

  9. Keep in mind, Paul, the tendency to conflate standards and curriculum. If the standards say children should be able to make inferences, then that means you teach the children to make inferences. The idea that making inferences is not a skill, per se, but tied up in the content about which those inferences are made has not yet reached a tipping point.

    Are the Yankees in trouble? As much trouble as a team with the biggest payroll in sports can be. It’s interesting, however, that Cliff Lee decided there are some things more important than the biggest possible paycheck. And that Zack Greinke wants no part of playing under the New York microscope. There’s a metaphor lurking in there somewhere.

    Comment by Robert Pondiscio — December 20, 2010 @ 6:43 pm

  10. How about we start out with a carefully thought-out curriculum framework that reflects the standards and goals for which the district is willing to be held accountable?

    Cliff Lee could be the tip of the iceberg in the twenty-first century free agent world.

    We here in Beantown can only hope the history, mystique, and intimacy of Fenway and the green monster will lure enough of the able bodied to keep us competitive with the evil empire.

    Comment by Paul Hoss — December 20, 2010 @ 8:06 pm

  11. One of the problems, Paul, is that ELA standards tend to reflect a great deal of political compromise. At the state and national level, policymakers are cagey about specifying literature (even broadly) that students should read. Hence the emphasis on skills.

    Thus, if a curriculum “reflects” the standards, it won’t contain very much. It has to go far beyond the standards.

    The Common Core State Standards make gestures toward specific knowledge and study of literature. Even so, they rely on a first-rate curriculum in order to make sense–and somehow policymakers are skipping over curriculum.

    The curriculum need not be confining, nor need it involve a great deal of reinvention. There are excellent curricula out there. Give schools a degree of flexibility and a fine curriculum, and they’ll be in much better shape than they are now.

    It isn’t that hard. But it seems that many assume that standards are already curriculum, and this results in confusion.

    Comment by Diana Senechal — December 20, 2010 @ 8:55 pm

  12. “…somehow policymakers are skipping over curriculum.” Therein lies the mystery; or is it really a mystery as to why some state/district policymakers avoid quality curricula?

    Comment by Paul Hoss — December 21, 2010 @ 10:09 am

  13. Two questions: 1) Who outside of this blog really understand the difference between curricula and standards? and 2)Does anyone have any evidence to support the use of one particular curriculum (for ELA) over another?

    For math it is rather easy (Singapore rules!) but for ELA, all we have are theories and most of those are bad theories. So given that CK is considered a sequence and not curricula, what specific curricula would you suggest?

    Comment by Erin Johnson — December 22, 2010 @ 2:11 am

  14. Erin,

    Any consideration of “evidence” must be accompanied by the question, “evidence of what?”

    When one speaks of “evidence” in favor of a particular curriculum, one assumes that it is already established what the curriculum is supposed to accomplish. That is not always the case.

    I believe that in English class students should become familiar with the best works of literature, literary nonfiction, and some philosophy. (They should read historical works in history class.) So, a curriculum that does this already has “evidence” in its favor, in my view. But others believe ELA should tackle a wide variety of practical tasks, such as writing a business memo, reading instruction manuals, giving a speech at a sports event, etc. In that case, a very different curriculum would have “evidence” in its favor.

    I regard the Core Knowledge sequence as a curriculum. A curriculum need not spell out in great detail how the subjects will be taught (but CK has various levels of support). If teacher education included rigorous preparation in a curriculum, then the curriculum could outline the topics and works that would be taught, and the teacher would know how to take it from there.

    For high school, I particularly like the English offerings at Stuyvesant. One might say, “well, that’s Stuyvesant.” But there’s no reason why many more schools couldn’t have courses like these. Here’s a partial list of works read in the ninth-grade humanities course:
    “Golding’s Lord of the Flies, The Epic of Gilgamesh, excerpts from The Book of Genesis, The Book of Job, excerpts from the Qu’ran, Sophocles’ Oedipus the King, Aristophanes’ Lysistrata, Shakespeare’s Julius Caesar, Lao Tsu’s Tao Te Ching, Shakespeare’s King Lear, Machiavelli’s The Prince, excerpts from More’s Utopia, Marlowe’s Dr. Faustus, various forms of poetry including sonnet and haiku, and excerpts from famous historical doctrines that have in themselves influenced literary development.”

    Comment by Diana Senechal — December 22, 2010 @ 8:31 am

  15. Diana/Erin,

    ELA and no grammar?

    It never ceased to amaze me how most teachers would jump right into having their students write (compositions, essays, “creatively”) without ever first dealing with the mechanics.

    Yes, even high school students should have to demonstrate mastery of these prerequisites before they get into formal writing assignments. When Robert states above there is NO curriculum for NYC schools, kids could be arriving in a freshman English class with exemplary foundations or little to no knowledge of where to begin. Could that be construed as yet another justification for individualizing (pardon my digression)?

    I told my kids every year (grades 2-6) we would deal with grammar, sentence structure, mechanics, etc., in an effort to get them to be able to compose a reasonable sentence first. Then, after demonstrating they had mastered these components, we tackled different forms of writing.

    Some kids would get to a very sophisticated level by June while others muddled their way through the prerequisites. To me, that was fine, as long as they learned as much as they could by June.

    Comment by Paul Hoss — December 22, 2010 @ 10:29 am

  16. Paul,

    Of course. I wasn’t describing the whole curriculum. My ideal ELA curriculum would include literature, grammar, vocabulary, etymology, composition, recitation, and rhetoric.

    Comment by Diana Senechal — December 22, 2010 @ 10:34 am

  17. Paul, I don’t mean this to be provocative or a challenge, but I’m a bit surprised that you’re surprised. The Reading and Writing Workshop, as practiced in NYC and elsewhere, is ostensibly the last word in individualization and differentiation (“Teach the child, not the lesson”). Thus grammar is only germane insofar as a particular student needs help with it (or so we’re led to believe). Beyond this, some believe that grammar is subordinate to engaging children in writing (write now, grammar later). Some see grammar as not important at all if it interferes with a student’s “authentic” voice. Then too, there’s the issue of elementary teachers whose own command of grammar is less than stellar.

    The bottom line is that “write first, mechanics later” is not generally perceived as a problem but the point.

    Comment by Robert Pondiscio — December 22, 2010 @ 10:41 am

  18. Paul is 100% right. ELA is more than just literature (even though that lit is essential).

    And K-8 is very different than high school. Presumably, the kids going into Stuyvesant have already mastered the foundations of academic English (decoding, handwriting, grammar, spelling, sentence writing, composition, vocabulary, logical contructs and reasoning, etc.). And so they are already prepared to delve into the complex texts used in the best high school classes. But we don’t (or shouldn’t) expect students in K-8 to have mastered these elements.

    The fundamental principles underlying Writer’s Workshop et al. is that all of these foundational elements of English should be picked up incidentally and that developing specific curricula is counter to developing good writers/thinkers. Unfortunately this approach has disproportionately penalized kids from family situations where academic English is not spoken in the home and thus have no context for understanding and writing complex academic English.

    So I take it from the comments that no one has a favorite ELA curriculum that they would champion?

    Comment by Erin Johnson — December 22, 2010 @ 12:15 pm

  19. Erin, regarding the importance of grammar, I never wrote or implied otherwise. In the original post, I wrote: “In schools and districts with a rigorous English curriculum (as opposed to a literacy program), students read and discuss challenging literary works, study grammar and etymology, write expository essays, and more.”

    I would favor something along the lines of CK for K-8, and something along the lines of Stuyvesant’s curriculum for 9-12. It doesn’t have to be exactly like them–but something along similar lines.

    A curriculum need not be a package or a program; if the teachers are prepared, it can be a sequence like the CK Sequence, or a sequence of required courses and electives, as at Stuyvesant. It needs good grammar books, though, so that the teachers do not have to come up with the exercises on their own or dig around on the Internet. (The Internet has so many bad explanations and examples of grammar that I wouldn’t rely on it at all, unless there were a superb website.)

    Of course the emphases change from elementary to middle to high school, and even within those levels.

    I doubt that the majority of Stuyvesant students have much explicit background in logical constructs and reasoning (in English, that is) when they enter ninth grade. Nor do they necessarily have top-notch grammar and writing skills. But they do have a stronger background than most, not to mention intellectual curiosity and intensity, which go a long way.

    Comment by Diana Senechal — December 22, 2010 @ 12:37 pm

  20. Robert,

    Where to begin without sounding defensive?

    Write first, mechanics later was an enormous problem as I saw it. If I were going to teach someone how to play golf, I would not hand them a set of clubs and tell them to go out on the course and have it. I’d teach them the fundamentals first; grip posture, set up, first part of the swing, second part of the swing, followed by course management strategies, etiquette, etc. Then, after the requisite work on these fundamentals and assessing their proficiency in each and on the whole, I might be inclined to take them out on the course.

    Yes, yes, I know, teaching ELA is not like teaching golf. However, teaching is teaching; whether it’s algebra or Tiddlywinks. As a teacher I believe the prerequisites are both important and extremely necessary. Many problems are avoided via this approach.

    (SIDEBAR: Why do they call it golf? Because %$#^& and *&#@^ were already taken).

    The readers/writers workshop practiced in my district was reactionary instruction, as I saw it; that is to say, teachers who used this system (many) found out on the run the kids who had learned the prerequisites and the ones had not. They then had to go back and figure out who was where (and I’m not at all convinced this was ever done) and teach what was absent from each kid’s background. To me, that was a bit like putting the cart before the horse or a bit back assward. Why not run everyone through this stuff soup to nuts (elem level) and the kids who knew it would breeze through while those who didn’t had to spend the time learning it. In the end the teacher (me) was sure everyone was getting exposed to the appropriate knowledge and not left guessing who knew what and who didn’t.

    And PLEASE, don’t get me started on “authentic” anything to do with education. Talk about the demise of the progressive movement in education. “Authentic” learning was their poster child.

    Comment by Paul Hoss — December 22, 2010 @ 1:49 pm

  21. Diana, Those textbooks that you would recommend for grammar are the curriculum. If we just take grammar as a sub-set of ELA fundamentals, shouldn’t we all know what textbooks/approaches are effective for grammar study and which ones are less so? You mention that the web is full of bad resources. Which ones are bad and why do you think so? Which ones should teachers look for?

    Homeschoolers are all aware of the available resources and are constantly debating the pros and cons of a textbook or approach. Why don’t public school teachers do the same?

    Also, what do you mean by teachers being prepared and not needing a package or program? What prep do they need? What would you have teachers do on a daily basis to ensure that by the end of 8th grade students have learned to decode, read and analyze non-fiction and fictional texts, write complex and interesting sentences, organize thoughts into coherent theses, etc…? And shouldn’t teachers coordinate their instruction over those 9 years? Standards do not coordinate coherent instruction. Only textbooks/materials/specific lessons do.

    Also re: Stuy. The entrance exam does contain reading comprehension, logical reasoning questions and scrambled paragraphs. Presumably, the students entering Stuy would have had some prep in logical contructs and paragraph construction or they would not have done well on the exam.

    Comment by Erin Johnson — December 22, 2010 @ 5:23 pm

  22. Erin,

    I can’t answer all your questions in detail–I don’t have all the answers, and I have deadlines to meet. But here are a few thoughts–not a policy outline.

    English teachers should be well versed in literature, grammar, writing, etymology, and rhetoric (and phonics, if they are teaching elementary school). They should be capable of taking something like the CK sequence and running with it (though CK offers plenty of materials).

    Now, that doesn’t mean it’s desirable for teachers to be running in different directions. A school should be able to choose its textbooks, and teachers should coordinate the lessons to the degree that is appropriate (probably more at the elementary than at the high school level). But the teacher should have enough command of the subject to teach beyond the textbook and to plan lessons.

    I suspect that if teachers could choose their textbooks (or if a school could), you’d see many teachers debating their pros and cons. They would look for textbooks that best supported the curriculum–the two not being identical.

    As for online resources, teachers should know enough to discriminate among them. Do a search on “coordinating conjunction,” and you’ll find a number of definitions. The teacher should be able to identify the best one and adapt it for the lesson as needed. One resource states: “A coordinating conjunction is a conjunction that links constituents without syntactically subordinating one to the other.” Another states: “The short, simple conjunctions are called coordinating conjunctions.” The latter definition may sound simpler, but it is misleading. Granted, the first one would take some explanation, but if students learned it, they would be much closer to a true understanding of coordinating conjunctions.

    Comment by Diana Senechal — December 22, 2010 @ 8:33 pm

  23. Diana/Erin,

    How do these teacher plans to implement the curriculum in their classroom get balanced against today’s trend of districts mandating their teachers follow scripted lessons? The more I read, the more this appears to be the common trend, especially in urban districts like NYC. Is there any academic freedom remaining?

    Comment by Paul Hoss — December 23, 2010 @ 8:39 am

RSS feed for comments on this post. TrackBack URL

Leave a comment

While the Core Knowledge Foundation wants to hear from readers of this blog, it reserves the right to not post comments online and to edit them for content and appropriateness.