Inferencing Test

by Robert Pondiscio
October 1st, 2009

“After failing to move a runner past first base for the entire game, the Giants sent Davis to the plate with the potential tying and winning runs in scoring position.  Unfortunately, he hit into a 6-4-3 double play to end the game.”

  1.  How many outs were there when Davis came to bat?
  2. To whom did he hit the ball?
  3. Describe the kind ball he hit (pop up? Line drive? etc.)
  4. What was the final score of the game
  5. How many runners were on base?

If you are able to answer all five of these questions (#5 is tricky) is it because you have mastered the ”reading skill” of making inferences. Or because your knowledge of baseball fills in the gaps for you?

22 Comments »

  1. Great post. I can answer it and it’s because of my baseball knowledge rather than my ability to make inferences. “Two runners in scoring position…” is the key to 5 but if you don’t know what “scoring position” means, it’s tough to infer it based on the other information.

    Makes a good point about “content knowledge” versus “reading skills”. I’ll use it myself in the future.

    Comment by CaliTeacher — October 1, 2009 @ 11:16 am

  2. 1.) 1
    2.) the shortstop
    3.) groundball
    4.) 2-0
    5.) 3

    And I’ll take the bait: I know those mostly because I’ve spent far too much time following baseball — though spending time reading and doing math didn’t hurt either. Clearly, content is the most important thing in this example that taps into a rather specialized set of knowledge — but, frankly, I think advocates of core knowledge need to be careful that they don’t imply that content is the ONLY important thing. I agree that it’s more important than, or at least a precursor to, these “reading skills” and such, but even I can’t be convinced that teaching those things has absolutely zero value.

    Comment by Corey — October 1, 2009 @ 11:57 am

  3. Oops, 4 is 1-0. I clearly should have been taught how to proofread better.

    Comment by Corey — October 1, 2009 @ 11:58 am

  4. 1) 1
    2) The shortstop
    3) Ground ball
    4) 1-0
    5) Three — there must also have been a runner on first to get the 6-4-3 double play.

    Comment by Rachel — October 1, 2009 @ 12:01 pm

  5. And I would never argue that readig strategies have no value. I’m sure you had the same experience teaching that I did Corey, wherein it’s a bit of an epiphany to some kids that reading should always make sense and that if they don’t understand it, there are ways to address it. But there are limits to fix-up strategies. Consider the opposite issue: When we get standardized test results back and a kid gets an inferencing question wrong, we tend to assume we should teach….inferencing. But my example shows that may not be sensible. It’s not a transferable skill.

    And I’m impressed you and Rachel figured out the bases were loaded. I shouldn’t have signaled it was tricky. No man on first, no 6-4-3 double play!

    Comment by Robert Pondiscio — October 1, 2009 @ 12:09 pm

  6. Robert, I agree with you. My first year of teaching 10 minutes per day had been added on to the day — which we used for “instructional home room”. For the first couple months, there was a skill of the week (e.g. “making inferences” or “determining author’s purpose”. I remember in early October we did an interim assessment. The results came back and the kids hadn’t done well on the section on author’s purpose. The literacy coach couldn’t believe they’d done so poorly on it when it had just been the skill of the week a week or two before the test.

    “What happened?” she said.

    I was befuddled (to say the least). Did she really think that in 50 minutes per week the kids could learn these abstract things we were calling skills? If so, shouldn’t the school year be about a month long and wouldn’t all the kids be passing the test? And don’t get me started on the curriculum I used my second year.

    So, anyway, I absolutely share your frustration with “skills”-based curricula. But I get the distinct feeling in reading many of the posts around the blogosphere that disparage them that many others who read them must believe the authors are arguing that teaching these skills has no utility whatsoever. A curriculum made up of 100% “skills” would be virtually useless, but a curriculum made up of 100% content wouldn’t really be all that much better. I guess what I’m saying is that I’d just like to see it acknowledged more often that while we too often eschew content, content alone is also not the answer.

    Comment by Corey — October 1, 2009 @ 2:34 pm

  7. This reminds me of “tests” that circulated in the 60′s asking questions about when does the eagle fly and what color is a chocolate cake. They were illustrating the effect of cultural bias–and how things that are “common knowledge” in one culture are completely missing, or wrong, in another.

    I am not a baseball fan, but I was able to infer a few things. My recollection/guess/ineference was that a double play is one in which there are two outs. Knowing that the tying and winning runs were on base told me that the current score was x-(x-1). I suppose that 6-4-3 must contain information that might indicate the value of x. So, if this were a real test, I might play around with them. The relative value of a pop-up or line drive is something I can play with–guessing that a pop-up might be caught and render the batter out–and perhaps if it were the first baseman (or someone close to the first baseman) a runner who left base wouldn’t be able to return before the base was tagged.

    Lot of inferring going on here–however, I am inferring from previous knowledge–which I suppose is your point.

    So– is this a good test of a student’s ability to draw inferences from text, or merely a culturally biased question that can be answered easily by someone whose culture provides additional knowledge and hardly at all by the average Britisher whose culture does not? And what then is the solution? Ought we more carefully write tests, screening for bias (as standardized tests routinely are), or ought we command that all grow up in a standardized culture so far as is possible?

    Comment by Margo/Mom — October 1, 2009 @ 3:50 pm

  8. We’re on the same page, Corey. Rather that repeat myself, I’ll point to a post on this subject I wrote a couple of weeks back. I’ve come to see the issue as a question of putting the horse before the cart:

    http://blog.coreknowledge.org/2009/09/16/horses-and-carts-2/

    Comment by Robert Pondiscio — October 1, 2009 @ 3:51 pm

  9. It’s an example of how we tend to misdiagnose reading difficulties as a lack of a skill, when it is a lack of content knowledge. The issue of cultural bias (is background knowledge a cultural bias?) is interesting. I’m tempted to argue that it would be virtually impossible to create a meaningful reading test that did not assume some content knowledge or, if you prefer, cultural bias. Am I wrong, Margo?

    Comment by Robert Pondiscio — October 1, 2009 @ 3:54 pm

  10. Since I know very little about sports in general, I am totally reliant on using reason to answer your questions. Perhaps describing my thought processes in attempting to do so might give some clues to what’s involved. Fortunately I have always been introspective, and accustomed to analyzing my thought processes. And that’s makes me pretty good at observing and remembering my thought processes.

    I did pick up immediately that it was baseball you were talking about. I think that came from the words, “runner”, “base” and “game”. Then the going got a little tougher.

    Thinking back of when I first read this about five minutes ago, I realize that at just about this point a conclusion came to mind. This is baseball. I won’t understand it. On the basis of that conclusion I switched to a scan mode of reading. I didn’t expect to understand, so I didn’t try. I didn’t read for details. I just scanned to confirm the rest of the paragraph was about baseball. Do students do that at times? Should we know about it? What can be done about it?

    Also at about that time I also recognized where you were heading. Content is important. That was an easy conclusion based on what I know. It was confirmed by reading more, but it was a very easy confirmation. Not much brain power was required.

    Then I looked at your questions. I was still mostly scanning, but was reading closely enough to realize I had no idea of the answers. My next thought was to consider the possibility that all of the answers could be arrived at by logic applied to information that was there. So considered the possibility of returning to read very carefully. After all, I’m a logical type of guy. I can dig things out of reading.

    That should change my reading mode, from a scanning mode to critical mode. But that didn’t seem to happen, at least not immediately or easily. It was hard to change modes. I would go back to the beginning, and about two seconds later realize I just still just scanning. Is that a mental habit? I have long believed that mental habits are very important, but have never been able to develop that idea very much. Perhaps this is an example.

    I attained a critical mode of reading at least enough to zero in on the phrase “to the plate” I concluded that meant to go to bat. That was not an easy or automatic conclusion though. When a player is running the bases he does go “to the plate”, does he not? I entertained the notion that that was involved here. I rejected that notion, on what grounds I can’t say, and am confident that “to the plate” means to go to bat.

    So I moved on to “tying and winning”. I first interpreted it as tying and winning the game, as opposed to losing the game, but quickly realized that doesn’t make sense. You don’t tie the game and win the game at the same time. A more careful reading led me to conclude that tying and winning are adjectives to runs, not verbs. That changes things. In my scanning mode I had read, “potential of tying”, but that was a misreading. I do know what a run is. Well, I had to think about that. A player hits the ball, and runs for the bases. That’s what a run is. How does that fit in here?

    So then I went on to “in scoring position”. Is “scoring” a noun or a verb (gerund or participle). That was not immediately obvious. I concluded that the “runs” are in scoring position. How can a run be in a position? So a run is . . . . . Actually I can make some sense out of that, but I’m not sure how to verbalize it at the moment.

    And then I came to the “6-4-3 double play”. I drew a blank on that. Maybe with enough time and imagination I could come up with some hypothesis on that. Or maybe not. And why did it end the game? And why is it unfortunate? I presume it is unfortunate because they thereby lost the game, but I have no idea how.

    And now to the questions. “How many outs . . . .? There must be a clue in the text. Three strikes make an out. Three outs make something, an inning, isn’t it? But what makes a game? I thought nine innings make a game. So this must be the “bottom of the ninth”? So probably there’s two outs already. If Davis strikes out the game must be over. Yes, I can use logic. But my logic feels very shaky. Nothing in my experience confirms my logic.

    Question 2, To whom did he hit the ball? I don’t get it. In baseball you don’t hit the ball “to” anyone, do you?

    I could go on, but at this point I’m pretty fed up with the whole affair. I’m not making any progress in understanding this. Of course my motivation at the moment is to analyze my thinking, and to analyze the contention that knowledge is important. That’s a forgone conclusion. I’ve always known knowledge is important. What’s new in recent years (or maybe not so recent) is that anyone would not realize the importance of knowledge

    In his linked article Dan Willingham says that reasoning like this is difficult mental work, and that it’s slow. Based on my experience described above, I totally agree.

    Comment by Brian Rude — October 1, 2009 @ 4:43 pm

  11. Amazing, Brian! That was an awful lot of work, and no doubt cognitively painful, so thanks very much. I found it fascinating because as teachers, we’re often encouraged to do “think alouds” — model successful reasoning for our students. I’ve never heard or read an *unsuccessful* think aloud, so it was illuminating. Especially revealing was the frustration level that inevitably kicked in. This is clearly what it must be like to read without background knowledge, yet with the idea that reading strategies should be enough.

    How many of my students did I torture like this, I wonder?

    Comment by Robert Pondiscio — October 1, 2009 @ 4:56 pm

  12. Cory,

    Would the runner on third not count as scoring if he crossed the plate before the 2nd baseman threw to first for the final out?

    In which case the final score was 2-1, no?

    Either way, content counts. Point taken.

    And to Margo’s point, a cricket or rugby questions might be appropriate for a test where UK culture was more dominant.

    Either way, sports literature is a genre to which one would hope children are exposed, and can cover a range of topics. Baseball alone could cover morality and ethics (tales of the 1919 White Sox, Jackie Robinson’s arrival in Brooklyn) politics (The Bronx is Burning) and just great writing (anything by Roger Angell).

    These topics are relevant to a student’s intellectual development as a citizen, regardless of gender or ethnic origin.

    Comment by Matthew — October 1, 2009 @ 5:35 pm

  13. Brian, your “think-aloud” was wonderful. It reads like a funny and deep parable, but it was an attempt to answer actual questions. I don’t know a thing about baseball either–so I can relate to this:

    “To whom did he hit the ball? I don’t get it. In baseball you don’t hit the ball ‘to’ anyone, do you?”

    I enjoyed this, too: “If Davis strikes out the game must be over. Yes, I can use logic. But my logic feels very shaky. Nothing in my experience confirms my logic.”

    And the places where language escapes you: “I concluded that the ‘runs’ are in scoring position. How can a run be in a position? So a run is . . . . . Actually I can make some sense out of that, but I’m not sure how to verbalize it at the moment.”

    Comment by Diana Senechal — October 1, 2009 @ 9:45 pm

  14. Me, as a Kiwi, who played softball a bit at school, couldn’t answer any of it. I read the bit about “failing to move a runner past first base for the entire game” and the bit about “potential tying and winning runs in scoring position”, and thought, how can you potentially tie or win if you haven’t got a single runner past first base? Unless the other guys got zero as well. And how can you work out how many outs there were when you don’t know how many people had played before Davis?

    Margo/MomSo– is this a good test of a student’s ability to draw inferences from text, or merely a culturally biased question that can be answered easily by someone whose culture provides additional knowledge and hardly at all by the average Britisher whose culture does not?

    Why can’t both answers be right? And why did you put the word “merely” in there next to “culturally-biased”? Humans are naturally social creatures who virtually always live in a culture. The actions of our ancestors still affect our lives today, for example the reason that so many were so excited by the election of a black president of the USA was in part because of the legacy of slavery and racism in US history. People are still being killed in Northern Ireland because of the political-religious influences of the Tudor Kings and Queens attempt to secure their borders by encouraging Scottish and English Protestants to settle in Ireland. In a slightly less deadly way, class issues affect UK political history as they don’t in NZ, for example the fuss about whether UK politicians went to comprehensive schools, grammar schools or private (public in their terminology) schools.
    Or to shift to science, we now conceptualise disease in a way quite different to our ancestors, for example it used to be that infectious diseases were killed by germs, now many non-medical people can distinguish between bactierial infections (antibiotics treat) and viral infections (for which antibiotics are bad and doctors shouldn’t prescribe them in response to patient pressure). And before the germ-theory of disease, there were other theories, God’s Will, bad air, your humours were out of balance, etc. Other cultures presumably have their own understandings.
    I don’t think that cultural bias should be described as “mere”.

    Comment by Tracy W — October 2, 2009 @ 6:35 am

  15. Here’s another question: If the pitcher was a reliever who only pitched this inning, does he get a the win? If not, what does he get if anything?

    Comment by kderosa — October 2, 2009 @ 8:20 am

  16. Good one, Ken. We also know what his Earned Run Average for this outing is. However, since we don’t know how the three runners reached base, we do not know enough to determine his WHIP.

    Comment by Robert Pondiscio — October 2, 2009 @ 10:58 am

  17. By mentioning WHIP has Robert outed himself as a follower of SABR, Nate Silver or Rob Neyer?

    Personally I like ERC and DIPS since much like educational data, stats that reflects performance are not always the most commonly used or easily understood.

    Comment by Matt — October 2, 2009 @ 12:49 pm

  18. I think that there are some key questions that must be answered here. One is whether a reading test (and specifically a test that attempts to measure the ability to draw inferences from text) can be constructed that does not require (and therefore test for) prior content knowledge, as Robert asked. The other is Tracy’s question about whether a culturally biased question can also be a valid measure of a student’s ability to draw inferences from text. I would suggest that in theory, the answer to Tracy’s question might be “yes” assuming a condition. That condition would be that all students tested were subject to the same cultural upbringing. This is an assumption that modern standardized testing does not generally make.

    It is possible to construct reading passages that are reasonably free of cultural bias. Possibly not the most intriguing of written work–but we are talking about constructing a testing environment, not works of great literature. If we are attempting to determine the ability of a student to draw meaning from an unfamiliar passage–using inference–then we need to have a test item that does in fact test that ability, rather than assessing prior knowledge. In most states there are several committees that review test items. One committee typically reviews the question itself to determine if the question actually assesses what it purports to. That committee would likely have problems with the passage presented. The passage requires information not included on the page in order to arrive at the answer. So this question would very likely not ever make it to a cultural review committee. If it did, however, their slant would be to think about whether any additional information required was culturally loaded. Knowledge of baseball in the United States may or may not qualify as culturally biased–however there may be some gender bias. Again–not too likely to make it through either of these screens, but if so, it would probably trip up on psychometrics. This is where I start swimming out of my league, but my understanding of this is that it has to do with comparing scores of students who scored correctly on other typical items testing the same thing. Do they fall out the same way demographically?

    But, and I know that some disagree with me, it is important to distinguish a test that seeks to measure reading skill, from a test that seeks to measure reading experience. I would agree that reading skill is best supported by a considerable amount of reading experience, and that reading of “good” works is better than reading just any old thing. Matthew suggests that sports literature is a meaningful genre. Well, you couldn’t have proved it by me growing up. In fact I was well into adult years before I even ran into anyone else suggesting that there was such a thing. Likewise, I have never felt particularly drawn to science fiction, although I have read a few works that I consider to be superior.

    And yet, I can see someone constructing a whole and very meaningful curriculum or segment of a curriculum on either of them. And I would grieve if reading standards were to stand in the way of that. So–we can either go in the direction of picking one of everyone’s favorite genre–guaranteeing that no one is happy. Or we assess those works which we find that the preponderance of well-educated people are familiar with and use these as the cornerstone of required reading–forever looking backward and eliminating the valuable life-lessons from something (that I find) as quirky as sports literature, and overlooking historically excluded groups. Or we can take a step back and ask that critical question with regard to what we intend to set as a standard, and to test. Are we intent on ensuring, and testing, the ability to read–or are we set on determining that everyone has read from the books that we deem appropriate to developing and enhancing the ability to read?

    And this, as I see it, is the crux of the skills vs content issue. Those who fall more heavily on the content side are not so concerned with ensuring that there IS content, as in defending a particular set of content.

    Comment by Margo/Mom — October 2, 2009 @ 4:22 pm

  19. Matthew: No, the run would not have counted — it was a force out.

    Ken: He would get a save.

    Robert: It would most likely be 3.00, but there could’ve been a hit batsman or error.

    Matt: Or he just plays fantasy baseball.

    Comment by Corey — October 2, 2009 @ 5:09 pm

  20. Would the runner on third not count as scoring if he crossed the plate before the 2nd baseman threw to first for the final out?

    No, one of the simplifying assumptions of baseball rules is that when a runner is forced out, no other runners have time to score before the out, if the out ends the inning.

    I’m enough of a baseball geek that the combination of “runners in scoring position” and “6-4-3 double play” struck me as odd before I noticed Robert’s hint that it was tricky. It seemed very odd that the passage wouldn’t comment that the bases were loaded.

    Comment by Rachel — October 2, 2009 @ 7:44 pm

  21. I’m sorry Margo/Mom, I wrote poorly. I wasn’t asking whether a “culturally-biased question can also be a valid measure of a student’s ability to draw inferences from text”, I was arguing that culturally-biased questions are a valid and important measure of a student’s ability to draw inferences from a text. Not “can also be” but “are also”. I do not think that teaching reading should confine itself merely to teaching how to understand texts that aren’t culturally-biased.

    I would suggest that in theory, the answer to Tracy’s question might be “yes” assuming a condition. That condition would be that all students tested were subject to the same cultural upbringing.

    I wouldn’t be so restrictive as to say that all students tested were subject to the same cultural upbringing. I’d say that all students tested should have been taught about the same bit of culture that they’re being tested on. For example, my parents were and are atheists, as is most of my extended family but I do know quite a bit about Christian traditions, such as who Moses is, interpretations of what Jesus “dying for our sins” means, etc, even though it’s not something I believe in.

    My brother, who wound up going to a openly Protestant school, for reasons I won’t go into here, also carries a fair bit around in his head about Buddist and Islamic belief systems, which he suddenly displayed in Thailand.

    Do you have any evidence that the same cultural upbringing is a necessary condition for being able to make inferences from a culturally-biased text, as opposed to merely knowing a lot about that culture, even though you weren’t brought up in it?

    It is possible to construct reading passages that are reasonably free of cultural bias.

    Can you give an example of one of these passages?

    More generally, even if it is possible to construct a reading passage that is reasonably free of cultural bias, I don’t think that the only goal of teaching children to read should be to teach them to read such passages. We should also be teaching children to make sense of important passages that do depend on cultural bias. After all, a lot of what they are going to be reading as adults will be written by people who do have their own cultural biases. Politicans and activists don’t necessarily lay out where they are coming from and what assumptions they are making explicilty.

    In other words, I think teaching how to make inferences from work that is culturally-biased should be a goal of teaching reading. Obviously we can’t teach any kid enough to be able to read works from every single source of cultural bias, so I think the questions of what to teach should be done on a country-by-country level, for example NZ schools should place a lot less importance on US history than US schools should, and I think US schools should ignore NZ history entirely.

    So–we can either go in the direction of picking one of everyone’s favorite genre–guaranteeing that no one is happy. Or we assess those works which we find that the preponderance of well-educated people are familiar with and use these as the cornerstone of required reading–forever looking backward and eliminating the valuable life-lessons from something (that I find) as quirky as sports literature, and overlooking historically excluded groups.

    Why do you think we need to overlook historically-excluded groups under your second option? And why you think that we’re obliged to forever look backward? New books are being written all the time, and after time some of them are added to the canon, why do you forsee this process stopping?

    I am also not sure about your point about “eliminating the valuable life-lessons from something…as quirky as sports lessons”. Given that we can’t teach everything at school, aren’t we inevitably going to be eliminating the valuable life-lessons that could be learnt from something, no matter what option we teach?

    Are we intent on ensuring, and testing, the ability to read–or are we set on determining that everyone has read from the books that we deem appropriate to developing and enhancing the ability to read?

    I think, from what the cognitive scientists are saying, that this is a false dichotomy. To ensure the ability to read, you do need students to have a broad set of content knowledge.

    As an aside, I don’t think we need to be as specific as books, for example many biblical stories are available not just in the bible, but in their own books for children (which are probably easier to carry around and manipulate for little hands), there are many retellings of the myths of Ancient Greek and Rome, etc. As long as every book used is reasonably accurate in the knowledge it presents, I don’t think we need to specifiy books.

    And this, as I see it, is the crux of the skills vs content issue. Those who fall more heavily on the content side are not so concerned with ensuring that there IS content, as in defending a particular set of content.

    What puzzles me as is the skills side. Why does the skills side just ignore the question of what children would be able to understand if no skills are taught. For example, in this response here, you ask a lot of questions, but you present no arguments that children who have been taught to make inferences without content-rich knowledge will be equipped to read and understand the general public discourse amongst adults of the country they are living in.

    Comment by Tracy W — October 5, 2009 @ 8:33 am

  22. Um… both.

    Comment by Shelly — October 8, 2009 @ 12:34 am

RSS feed for comments on this post. TrackBack URL

Leave a comment

While the Core Knowledge Foundation wants to hear from readers of this blog, it reserves the right to not post comments online and to edit them for content and appropriateness.