Same SAT, Different Day

by Robert Pondiscio
September 15th, 2011

Still blaming poor SAT scores on test-takers?

SAT reading scores for 2011 high schools grads fell to their lowest point in history. The College Board attributes the decline “to the increasing diversity of the students taking the test,” notes the New York Times. But that argument was effectively dismissed by E. D. Hirsch when scores were announced last year.

“The standard explanation is that our test scores have declined chiefly because of a demographic broadening of the test-taking base. This claim ignores compelling contrary evidence. During the period of the big drop, from 1965 to 1980, verbal scores in the state of Iowa – 98 percent white and middle class – dropped with similar sharpness.

What changed, Hirsch noted, had less to do with the demographics of the test-takers “than the anti-intellectual ideas that fully took over first teacher-training schools and then the teachers and administrators they trained. The result was a retreat from a knowledge-based elementary curriculum — as researchers have shown by analyzing the severe watering down of American school books in the period 1950-to the present. The decline of the elementary curriculum coincided with our sharp decline in verbal ability and test scores.”

A sloppy workman blames his tools.   Or in this case, test-takers.

29 Comments »

  1. I’m surprised at the lack of depth in this post (unusual for this blog). Before you can make causal claims, it’s essential to understand whether the scores reflect changes in performance or changes in the tested population.

    http://www.quickanded.com/2011/09/sat-score-hysteria-and-the-missing-chart.html

    Comment by Bill Tucker — September 15, 2011 @ 4:46 pm

  2. @Bill I agree with your post that that what is happening is more complicated than meets the eye. What I’m confused by is your overlooking the counterfactual. If the change is due to increased numbers of students taking the test–and specifically increased numbers of diverse students, then how does it not follow that in cases where that is NOT the case there is also a drop, e.g. Hirsch’s (quite nondiverse) Iowa example?

    Comment by Robert Pondiscio — September 15, 2011 @ 5:12 pm

  3. I first encountered the observation about the drop in Iowa test scores in a 1978 essay by Christopher Jencks: “What’s Behind the Drop in Test Scores?” The point was taken up in more quantitative detail by John Bishop a decade or so later. They deserve the credit and have the priority on the observation.

    No one would accuse NAEP of not taking demographics into account, and the NAEP long-term verbal trend for 17-year-olds has been downward since 1998. The SAT results may thus be a leading indicator for the next NAEP study of that group, due in 2012. Finally, the demographic “explanation” is at best a correlation, not really an explanation. And it is surely not an exculpation. The demographics of the schools are the demographics of the nation and our future.

    Comment by E D Hirsch — September 15, 2011 @ 6:09 pm

  4. My problem with the diversity issue is that it feels like a smokescreen for saying not-white not smart enough. there is no reason being a minority should dispose you to score lower. I don’t know if the data is parisable this way but one of the interesting questions would be if there is a similar drop at the socio-economic level. If well off kids are not dropping, but poorer kids are then maybe the question of the correlations between SES and the test are all we need to know and the diversity question is really an economic class question.

    Comment by Charlotte Osborn — September 15, 2011 @ 6:33 pm

  5. How do we explain the results for Massachusetts schools/students???

    http://www.boston.com/news/local/breaking_news/2008/08/sat_scores_rise.html.

    Comment by Paul Hoss — September 16, 2011 @ 6:43 am

  6. This explains the results in Massachusetts:

    “Hirsch’s theories, long merely persuasive, now have solid empirical backing in Massachusetts’s miraculous educational
    reforms,” Stern writes. One element of the state’s 1993 Education Reform Act was a “Hirschean knowledge-based curricula for each grade.”

    Comment by alamo — September 16, 2011 @ 9:08 am

  7. Forgot the link:
    http://blog.coreknowledge.org/tag/massachusetts/

    Comment by alamo — September 16, 2011 @ 9:08 am

  8. Thanks for the discussion. A few additional points:

    1) My initial point is that the media, and the public at-large, need to be informed consumers of data. In particular, we need to make sure to understand the information prior to making both conclusions (i.e., performance dropped) and assigning causes to those conclusions (SAT test scores are down because of NCLB). Writers and commenters on this blog argue powerfully that content and facts matter. So, in this case, I was responding to the breezy tone in the original post in which basic statistical issues were dismissed. We must all be careful about making inferences about CHANGES in performance if the group you are comparing with is dissimilar to the initial group tested. In this case, we have evidence that the comparison group is dissimilar, so it is important to be cautious in making conclusions.

    2)The term “demographics” clouds the basic statistical issue. Since the SAT is not taken by all students, we need to think about the population of students that is represented among test-takers. Since the SAT is a college admissions test, the test-taking population is generally drawn from an unrepresentative sample — those students who intend to attend college. So, for example, a district policy that made the SAT universal would very likely depress test scores. We’d need to be very careful to understand the data before claiming that the score declines were at all reflective of curriculum or instructional changes.

    3) As Robert notes, we could see the reverse happen. We could imagine a program that encourages high-achieving lower-income students to pursue college and take the SAT. The inclusion of these new students might increase test scores at the same time the tested population expands and demographics change. It’s about the tested population, not race or income. The confounding part is that unfortunately, race and income are correlated with scores.

    4) I’m not arguing for/against any of the past literature around test score changes in prior decades, particularly those prior to the publication of A Nation at Risk. Nor am I saying that the score declines are meaningless. In the case of 2011 scores, I’m just cautioning against premature conclusions and unsupported causal statements.

    5) The above has nothing whatsoever to do with having high expectations and believing that all students can learn. And, talking about CHANGES in performance over time is very different from discussing absolute performance. The SAT scores are mediocre at best.

    Comment by Bill Tucker — September 16, 2011 @ 9:49 am

  9. DISCLAIMER: I’m not politically correct!

    Lots of BS to support a claim. By diverse, let’s not be limited to race, nationality, and affluence.

    The increase in numbers also includes the dumb farm boys and girls in Iowa that are led to the test with little ambition and/or aptitude for college.

    In Ohio, you’d think the counselors were paid by a head count. “Take the test. You never know.” Well, as a teacher, I have a pretty good idea that the kids who struggle to keep a 2.0 average in HS are not on a par with the bulk who were “college bound” pre-1965.

    Comment by ewaldoh — September 16, 2011 @ 11:23 am

  10. I hope E.D. Hirsch doesn’t get too tired of repeating himself. I sometimes get tired of saying nonfiction books and term papers have a place in our high schools, but as a political consultant once told an exhausted candidate who was extremely tired of repeating the same message again and again: “Soon now, they will begin to listen.” Hope may not be too bright, but it does spring eternal, as the saying goes…

    Comment by Will Fitzhugh — September 16, 2011 @ 11:28 am

  11. Maybe if Asian kids ever wake up to the fact that they belong to a minority group, their scores will go down, too. You think?

    Comment by Will Fitzhugh — September 16, 2011 @ 11:30 am

  12. Paul- While I don’t discount the value of the Mass Standards, you link above is for 2008 data. In 2011 SAT scores dipped in Massahusettes also.

    http://www.boston.com/news/education/k_12/articles/2011/09/15/sat_scores_dip_as_participation_rises/

    Not sure if it is an indication of the economy or a shift in Mass to what is widely seen as less rigours common core standards.

    Comment by Charlotte Osborn — September 16, 2011 @ 12:40 pm

  13. [...] blaming poor SAT scores on test-takers?” wrote Robert Pondiscio on the Core Knowledge Blog. He said that argument “was effectively dismissed by E. D. Hirsch [Core Knowledge’s founder] [...]

    Pingback by The week in blogs « School Board News — September 16, 2011 @ 1:57 pm

  14. Charlotte,

    My apologies. You’re absolutely correct about our scores here for 2011. Perhaps this Associated Press brief sheds a bit more positive spin on our performance.

    http://www.boston.com/news/education/k_12/articles/2011/09/15/sat_scores_dip_as_partic...

    Comment by Paul Hoss — September 16, 2011 @ 5:29 pm

  15. I think there’s more to the expanded pool than diversity. When I graduated from high school the expectation was that something like the top 20% of students would take the SAT and go to 4-year colleges. Now “college for all” has become a rallying cry, and there’s much more sense that anyone who wants a good middle class life needs a college degree.

    So even in un-diverse Iowa, I’d guess a much larger fraction of high school students was taking SATs now, compared to 40 years ago.

    Has anyone looked done a comparison of how the scores of the top 10% or 20% of high school graduates (as measured by high school GPA) have changed over time?

    – Rachel

    Comment by Rachel — September 17, 2011 @ 1:29 am

  16. Good question, Rachel. That’s my point exactly. Too much of an apples-to-oranges concern. There’s nothing to compare when you ask different questions to different kids over a twenty year period.

    Comment by ewaldoh — September 17, 2011 @ 9:33 am

  17. Rachel, to clarify the point about Iowa verbal scores. They did not concern the SAT. (Your hypothesis would make a lot of sense if they did.) The Iowa data is important because these are scores from ALL high school students in the state of Iowa. For many decades all of them took a test called “The Iowa Test of Educational Development” (ITED), and it is from those scores that Jencks, and later Bishop, inferred a nationwide, demographic-wide fall in verbal scores in the 70s, largely owing to less cognitively rich schooling. This hypothesis was greatly strengthened in the 90s by Hayes and his colleagues at Cornell, whose work not only exploded the diversity theory of decline, but also illustrated in detail how schoolbooks had been dumbed down, just as Jeanne Chall had vigorously complained at the time. Last year I made a common metric for SAT and ITED verbal scores, and the two curves follow each other downward as if they were going over the same waterfall.

    Comment by E D Hirsch — September 17, 2011 @ 9:45 am

  18. Back when my parents went through high school, college was seen as something for the brightest kids in the class. The average students did not aspire to higher education. The boys enlisted in the military or took a job in a factory or a trade or maybe a low-level civil service position. The girls maybe worked as a secretary, retail clerk, or waitress for a few years and then get married & become a housewife.

    When I went through high school, most kids aspired to further education, but many still didn’t take the SAT because they were just planning to attend the open-enrollment local junior college or a trade school that didn’t require it.

    I do think that the increase in participation has resulted in a test pool that is much more diverse in terms of cognitive ability even within a particular demographic group.

    Comment by Crimson Wife — September 17, 2011 @ 3:53 pm

  19. @CW The issue is not whether a more diverse pool leads to lower scores. No one would dispute that larger pools of lead to a dilution of talent. When the major leagues expand their rosters, batting averages go down and ERAs go up because players who couldn’t make a team in a 16-team league can when there are 24 or 30 teams.

    But that’s not the point. The question is whether dilution is the only cause of our apparently broad verbal decline, or even the most important one. There is much evidence to suggest it’s not. A 1996 paper titled “Schoolbook Simplification and Its Relation to the Decline in SAT-Verbal Scores” by Hayes, Wolfer, and Wolfe poked a lot of holes in the dilution theory “SAT verbal scores should have declined during the 1952—1963 period because that was when the composition of the test-takers was changing most (from about 5 to over 50% of the senior cohort). To the contrary, ETS reported that scores fluctuated within a narrow range—between 472 and 478. While the average test-taker became less elite (academically) and more diverse in class, race and ethnic background, mean verbal scores did not fall.”

    Likewise when these watershed changes in the number and demographics of those taking the SAT slowed, the declines should have also stabilized if the scores moved with the number of test-takers. Once again, it didn’t happen. “In the 1963-1979 period, mean verbal scores should have leveled off and remained relatively stable. Throughout this period, the fraction of the senior cohort taking this test was stable at just over 50%. it was in this period that verbal scores fell from a high of 478 to 424 (just 2 points above its lowest level ever, 422, recorded in 1991). That is, virtually the entire change in post-World War II American verbal achievement levels — from an initial high plateau of about 475 in the 1950-early 1960s to a much lower plateau of scores from 1979 through 1994 —occurred within this single 16-year period.”

    Finally, they note (and this may be the smoking gun) the entire distribution of verbal scores from top to bottom, shifted to lower levels. “There was not only a proportional decline in top scorers but an absolute decline in the number scoring over 600. There are now 35% fewer scoring over 600. The number scoring over 700 fell from 17,500 in 1972 to just over 10,000 in 1993 — even as the number taking this test grew (Shea, 1993). Highly selective colleges and universities report mean verbal declines on the order of 40 points. The composition hypothesis does not predict this outcome.”

    “An acceptable explanation for the SAT-verbal decline must account for this huge decline in the performance of the academic elite,” the trio concluded.

    Does it?

    Comment by Robert Pondiscio — September 17, 2011 @ 4:42 pm

  20. From the Atlantic: “Fair Test reports that while white students’ overall scores decreased by a mere three points since 2006, black students’ scores decreased by 19 points, Puerto Ricans’ decreased by 17 points, Mexican-Americans’ decreased by nine points, and “other Hispanics’ ” decreased by 14 points. For Asians, the trend is reversed: from 2006 to 2011, Asians’ scores increased by 40 points.”

    Comment by Seth — September 18, 2011 @ 12:27 am

  21. The Viet Nam draft also significantly increased the number of students taking the SATs in the mid-late 60′s and early 70′s. Before the draft, credible high school students were almost exclusively the only college applicants. Once the draft was instituted, everyone and their brother was looking for a college deferment, and for obvious reasons.

    Comment by Paul Hoss — September 18, 2011 @ 6:38 am

  22. More about Hayes, Wolfer and Wolfe’s paper from 1996 that E.D. Hirsch and Robert talked about:

    http://www.indiana.edu/~educy520/sec6342/week_07/hayes96.pdf

    These three gentlemen ran lexical complexity formulas over old collections of school English texts – and were able to correlate the decline in SAT verbal score with the the decline in the difficulty level of English readers had indeed declined since WW2. Some interesting caveats in the textbook decline:

    The grades 1-2 texts declined in complexity between 1946-1962 (this is the period when the Look-Say whole word reading method became popular). In the period 1963-1991, the grades 1-2 texts went back to the level before WW2 – but the level of the later grades declined. For example, in 1996 the mean 6th, 7th and 8th grade readers were simpler than the 5th grade readers from before WW2.

    It’s no coincidence that the recent Common Core standards put such an emphasis on text complexity formulas – that cue perhaps can be traced to the article referenced above. (Although the problem with this approach is that over time we risk ending up with textbooks written ‘to the complexity formula’…)

    In regards to math textbooks – the math SAT decline is harder to explain, because New Math textbooks in the 1960s started by reforming the high school level math and only later tried to penetrate the elementary school. The drop in math SAT scores does coincide with the advent of New Math high school text books, but the catch is that New Math was expanding the curriculum e.g. by the introduction of calculus which was outside of the basic set of skills tested by the SAT.

    Incidentally, if you look back at the comments under E.D. Hirsch’s old piece – there is a reference to a nice series Barry Garelick wrote for Education News about the quality of math textbooks after WW2, which I highly recommend:

    http://www.educationnews.org/articles/it-works-for-me–an-exploration-of-traditional-math-part-i.html

    http://www.educationnews.org/articles/it-works-for-me–an-exploration-of-traditional-math-part-ii.html

    http://www.ednews.org/articles/it-works-for-me–an-exploration-of-traditional-math-part-iii-.html

    Comment by andrei radulescu-banu — September 18, 2011 @ 11:45 pm

  23. Don,

    “How to Stop the Drop in Verbal Scores” in this morning’s Times has not even a sniff of expanded demographics of the test taking population? VERY DISAPPOINTING. VERY.

    While you’re correct to declare a retreat from a knowledge based elementary curriculum has been a major contributor to this dilemma, your ignoring the at least equally causal increase in the demographics of the pool significantly diminishes your posture and with it your credibility in the ed reform debate.

    College for everyone is as big a misnomer as has ever surfaced in this country as education of the ‘whole bloody child’ has been since its introduction into the reform dialogue by the NEA a century ago. You simply cannot ignore either.

    If you’re not going to disclose ‘…the rest of the story’ your Foundation and its (as Robert likes to refer to it) orthodoxy will become as trivialized as any of the blather from people such as Kozol, Meier, Dewey, et. al.

    Your philosophy (Everything Your First Grader Needs to Know, etc.,) has great merit, merit I believed in and practiced for three and a half decades as a public school educator. Please don’t destroy that merit by attempting to discount ALL the variables.

    Comment by Paul Hoss — September 19, 2011 @ 6:24 am

  24. Correlation and Causation. Mr. Hoss, Nobody doubts the current correlations of verbal scores with socio-economic, low-income, ethnic, and racial groups. That is a given, and I don’t doubt that increased participation of previously excluded students (a good thing) correlates with the slight dip this year in SAT verbal scores. What I challenged in my 550-word piece in the Times is the implicit idea that (1) nothing can be done about this correlation, and (2) that this correlation is an even remotely sufficient explanation of the low scores.
    My argument, supported now with a lot of data and analysis, is that the chief cause (among several causes) of the low verbal scores of 17-year-olds after 13 years of schooling is the introduction in the 1930s of a set of ideas about elementary schooling that became completely dominant in our elementary schools by the 1950s and showed their effects in the 60s and 70s. We are so used to those educational ideas, and to the resulting economic, ethnic, racial correlations they perpetuate, that, after 6 decades, we regard the correlations as innate and unavoidable.
    My focusing on the Matthew Effect is meant to help bring the discussion round to the actual causes of low verbal scores, which are inherently cognitive, not social. Indeed it’s true that the social context starts low-income children off with under-par vocabularies and knowledge. The aim of democratic schooling is to do something about that unequal opportunity, which our public schools have failed to do in recent decades.
    Do we know that we can greatly reduce the correlations and raise the scores of 17-year-olds across the board? Yes, but only if we conduct elementary schooling from earliest ages by systematically imparting knowledge in a way that helps overcome the Matthew Effect. It’s been done, and in some places in the world, it’s been done on a big scale.

    Comment by E D Hirsch — September 19, 2011 @ 8:56 am

  25. Improving the curriculum is a worthwhile goal regardless of whether or not it stems the tide of the SAT decline. We’re not going back to the days when only the brightest students aspired to colleges requiring the SAT. But we can ensure that all students have access to a content-rich curriculum.

    Comment by Crimson Wife — September 19, 2011 @ 11:43 am

  26. The SAT scores within my district have decreased as well as the number of students taking the SAT versus the ACT. Some of the research done on the reading scores suggests that students just do not read as much as they used to. I have students at the 12th grade level who do not read at a middle school level. Their vocabulary skills are weak at best. In my opinion this trend is directly linked with the emergence of technology that students have their hands on. Students would rather facebook message or text versus read a book. We have bred a generation of students who do not read, and these students are now parents who do not encourage their children to read, because they don’t. How do we bring back the coolness of reading?

    Comment by Vanessa Johnson — September 19, 2011 @ 11:48 am

  27. I think Ms. Johnson’s comment dovetails with Dr. Hirsch’s overall theme: literature, like that general body of knowledge that CK advocates for K-6, needs to be taught thoroughly and systematically across the entire K-12 curriculum. That means specific titles, authors, genres, and literary periods taught and re-taught across the grade levels in the style of the old trivium. That means a literature curriculum, period.

    Granted, this means students will dislike some, even much, of what they’re required to read because it can’t all be age-appropriate and student-centered. But one hopes that by high school, because of a broad and consistent exposure to a general literary canon, students will have gained that big-picture understanding of literature to where they will have formed attachments to certain styles, authors, and genres, and can articulate their preferences in more mature terms than “this sucks,” “this is boring,” or “I couldn’t get into it.”

    My school district, like many others across Texas, recently adopted an expensive new lesson-plan software which is co-billed as “curriculum.” It is, of course, anything but–just the usual vapid skills-set with no explicit authors or titles. In 11th grade, we are expected to familiarize students with ALL the major periods and genres in American literature in six weeks, before moving on to poetry for another three weeks and analytical/research writing for the rest of the year. I believe such incoherent nonsense passed off as curriculum is the real reason most kids don’t find reading “cool.” You can’t fairly judge what you’ve never been taught.

    Comment by James O'Keeffe — September 21, 2011 @ 1:46 pm

  28. Foreign languages (including Latin) also help reading ability. It used to be routine for college bound students to study two languages, now it is almost unheard of.

    Comment by Harold — September 21, 2011 @ 8:18 pm

  29. I think the admin of this site is actually working hard
    in support of his site, because here every data is quality based information.

    Comment by how to increase sat score how to increase sat score how to increase sat score how to increase sat score how to increase sat score — January 26, 2013 @ 10:00 pm

RSS feed for comments on this post. TrackBack URL

Leave a comment

While the Core Knowledge Foundation wants to hear from readers of this blog, it reserves the right to not post comments online and to edit them for content and appropriateness.