In September, this blog passed on the results of a survey of Oklahoma high school students whose lack of knowledge of basic civics strained credulity. But not far enough, apparently. Via Public School Insights comes word that the results of the survey were “likely fabricated.” The survey by a firm called Strategic Vision LLC for the Goldwater Institute showed, for example, that only 43% of Oklahoma students could correctly identify the nation’s two major political parties, while only 26% could identify the first ten amendments to the Constitution as the Bill of Rights. However the raw numbers, revealed by the website FiveThirtyEight, show not one of the 1,000 students surveyed got more than 7 out of ten questions correct–a figure that definitely doesn’t pass the smell test.
An Oklahoma state representative arranged to survey students in his district and found results sharply at odds — and much better — than the survey seemed to show. Rep. Ed Cannady’s survey showed 95% of Oklahoma students could name the two parties, while 91% knew the Bill of Rights. Cannady’s results strain my credulity as much as the Goldwater survey should have, but didn’t. The day over 90% of high school students can answer basic civics questions is the day I go back to my fields a happy man. At The Quick and The Ed, Chad Alderman says “it’s rather remarkable, in retrospect, that so many people were willing to take these amazingly poor findings as solid evidence of the failings of American public schools.”
Guilty as charged. Excuse me while I wipe the egg off my face.
That said, the survey results didn’t ring any alarms not because I’m prepared to think the worst of U.S. schoolkids, but rather because I spent several years teaching in a school where my 5th graders had no knowledge of government or civics and no attempt had been made to give it to them.
Robert, you’re not going to take all that egg for yourself, are you? I get some of it on my face too!
But the true egg distribution is still unknown. We have yet to find out whether or not the survey results were valid.
That said, even if the survey results were not fabricated, it’s entirely likely that (a) many students were “caught at a bad time” and (b) many students didn’t take the survey seriously.
That raises additional questions. How much do we remember at “bad times” compared to “good times”? Which is more indicative of our knowledge? And under what conditions do we take tests (or questions) seriously?
Diana Senechal
Thanks for the plug, Robert! I also have trouble with Cannaday’s numbers. Did all the tested students really have no prior knowledge of the test and its purpose?
By the way, I believe the firm that did the Oklahoma study carried out a similar study in Arizona, with similarly preposterous findings. Nate Silver made a strong case that the Oklahoma distributions were, statistically speaking, nearly impossible. I wonder whether he’ll review the Arizona results as well.
Both myself and OCPA are investigating the validity of the survey. Thus far we have been provided with the call logs, and are awaiting receipts for the marketing list purchases of students. If the allegations against the polling firm are valid, then both myself and the OCPA will have been victims of a fraud. If not, the polling firm will have been the victim. Once we have gathered the requested evidence, we will consider whether to conduct a replication poll with another firm.
Regarding they “why did I buy this” point consider the following evidence from a multiple choice civics exam given to college students:
http://www.americanciviclitera…..dings.html
Simply put, even college results are abysmal. Like the U.S. citizenship test itself, our survey was given in an open answer format rather than multiple choice, and to high school students rather than college students. One would expect students to perform significantly worse without the correct answer staring them in the face in a multiple choice format.
In short, I’m not buying this alleged replication for a second, but we are doing our best to investigate the situation, and I can only ask for your patience at this point.
Matthew Ladner
I had my 7 year old take an online quiz based on the citizenship exam the other day with similar questions and she got 15/30 correct. Okay, she’s bright for her age but I don’t think she’s any smarter than the typical high school student. She got all the easy ones, missed nearly all the hard ones, and was about 50/50 on the medium difficulty ones.