Comments on: If It Sounds Too Bad To Be True… http://blog.coreknowledge.org/2009/11/10/if-it-sounds-too-bad-to-be-true/ Closing the Achievement Gap: Teaching Content Sat, 20 Mar 2010 17:44:17 -0500 hourly 1 By: Crimson Wife http://blog.coreknowledge.org/2009/11/10/if-it-sounds-too-bad-to-be-true/comment-page-1/#comment-8257 Crimson Wife Thu, 12 Nov 2009 21:54:17 +0000 http://blog.coreknowledge.org/?p=3741#comment-8257 I had my 7 year old take an online quiz based on the citizenship exam the other day with similar questions and she got 15/30 correct. Okay, she's bright for her age but I don't think she's any smarter than the typical high school student. She got all the easy ones, missed nearly all the hard ones, and was about 50/50 on the medium difficulty ones. I had my 7 year old take an online quiz based on the citizenship exam the other day with similar questions and she got 15/30 correct. Okay, she’s bright for her age but I don’t think she’s any smarter than the typical high school student. She got all the easy ones, missed nearly all the hard ones, and was about 50/50 on the medium difficulty ones.

]]> By: Matthew Ladner http://blog.coreknowledge.org/2009/11/10/if-it-sounds-too-bad-to-be-true/comment-page-1/#comment-8238 Matthew Ladner Wed, 11 Nov 2009 06:02:37 +0000 http://blog.coreknowledge.org/?p=3741#comment-8238 Both myself and OCPA are investigating the validity of the survey. Thus far we have been provided with the call logs, and are awaiting receipts for the marketing list purchases of students. If the allegations against the polling firm are valid, then both myself and the OCPA will have been victims of a fraud. If not, the polling firm will have been the victim. Once we have gathered the requested evidence, we will consider whether to conduct a replication poll with another firm. Regarding they “why did I buy this” point consider the following evidence from a multiple choice civics exam given to college students: http://www.americanciviclitera.....dings.html Simply put, even college results are abysmal. Like the U.S. citizenship test itself, our survey was given in an open answer format rather than multiple choice, and to high school students rather than college students. One would expect students to perform significantly worse without the correct answer staring them in the face in a multiple choice format. In short, I'm not buying this alleged replication for a second, but we are doing our best to investigate the situation, and I can only ask for your patience at this point. Matthew Ladner Both myself and OCPA are investigating the validity of the survey. Thus far we have been provided with the call logs, and are awaiting receipts for the marketing list purchases of students. If the allegations against the polling firm are valid, then both myself and the OCPA will have been victims of a fraud. If not, the polling firm will have been the victim. Once we have gathered the requested evidence, we will consider whether to conduct a replication poll with another firm.

Regarding they “why did I buy this” point consider the following evidence from a multiple choice civics exam given to college students:

http://www.americanciviclitera…..dings.html

Simply put, even college results are abysmal. Like the U.S. citizenship test itself, our survey was given in an open answer format rather than multiple choice, and to high school students rather than college students. One would expect students to perform significantly worse without the correct answer staring them in the face in a multiple choice format.

In short, I’m not buying this alleged replication for a second, but we are doing our best to investigate the situation, and I can only ask for your patience at this point.

Matthew Ladner

]]> By: Claus http://blog.coreknowledge.org/2009/11/10/if-it-sounds-too-bad-to-be-true/comment-page-1/#comment-8235 Claus Wed, 11 Nov 2009 03:01:43 +0000 http://blog.coreknowledge.org/?p=3741#comment-8235 Thanks for the plug, Robert! I also have trouble with Cannaday's numbers. Did all the tested students really have no prior knowledge of the test and its purpose? By the way, I believe the firm that did the Oklahoma study carried out a similar study in Arizona, with similarly preposterous findings. Nate Silver made a strong case that the Oklahoma distributions were, statistically speaking, nearly impossible. I wonder whether he'll review the Arizona results as well. Thanks for the plug, Robert! I also have trouble with Cannaday’s numbers. Did all the tested students really have no prior knowledge of the test and its purpose?

By the way, I believe the firm that did the Oklahoma study carried out a similar study in Arizona, with similarly preposterous findings. Nate Silver made a strong case that the Oklahoma distributions were, statistically speaking, nearly impossible. I wonder whether he’ll review the Arizona results as well.

]]> By: Diana Senechal http://blog.coreknowledge.org/2009/11/10/if-it-sounds-too-bad-to-be-true/comment-page-1/#comment-8233 Diana Senechal Wed, 11 Nov 2009 01:09:35 +0000 http://blog.coreknowledge.org/?p=3741#comment-8233 Robert, you're not going to take all that egg for yourself, are you? I get some of it on my face too! But the true egg distribution is still unknown. We have yet to find out whether or not the survey results were valid. That said, even if the survey results were not fabricated, it's entirely likely that (a) many students were "caught at a bad time" and (b) many students didn't take the survey seriously. That raises additional questions. How much do we remember at "bad times" compared to "good times"? Which is more indicative of our knowledge? And under what conditions do we take tests (or questions) seriously? Diana Senechal Robert, you’re not going to take all that egg for yourself, are you? I get some of it on my face too!

But the true egg distribution is still unknown. We have yet to find out whether or not the survey results were valid.

That said, even if the survey results were not fabricated, it’s entirely likely that (a) many students were “caught at a bad time” and (b) many students didn’t take the survey seriously.

That raises additional questions. How much do we remember at “bad times” compared to “good times”? Which is more indicative of our knowledge? And under what conditions do we take tests (or questions) seriously?

Diana Senechal

]]>