The American Educational Researchers Association (AERA) may find itself in the crosshairs for political correctness and staking out controversial positions, but a true bill of particulars against the organization would include its inability or unwillingness to act as an honest broker in determining the validity of education research.
Writing at the Washington Post’s Answer Sheet blog, Dan Willingham responds to the recent screed by Sharon Begley of Newsweek, who described education research as a “national scandal.” There is a lot of excellent research, but there’s also “a lot of dreck,” he observes. But good luck telling the difference. A professor of psychology at UVA, Willingham points out that psychology “is more vigilant in its self-regulation, particularly through its professional societies,” which he notes are “deeply committed to scientific rigor, they are run by scientists, and they publish very high quality research.” But where do you turn if you’re a teacher looking for research on teaching children to read?
“The American Educational Researchers Association (AERA) ought to be logical place, but it has not shown a lot of interest in taking on the job. I think a large part of the reason for this is that it is an enormous organization that includes scholars from very different disciplines: psychology, economics, political science, critical theory, history, feminist studies, etc. These different fields not only have different criteria by which evidence is evaluated, they have different definitions of what it means to “know” something. Small wonder, then, that AERA is seldom ready to make a flat statement on a research issue.”
This lack of clarity opens the door for “commercial interests and frank snake-oil salesmen” to hijack the conversation on research issues, observes Willingham, which “damages the field, and ultimately harms students.” He suggests AERA start by getting its members to decide which issues within education are amenable to a scientific analysis. “Education researchers frequently lament policy makers cherry-picking research findings to support positions that they advocate for non-research-based reasons. Until researchers get their act together, we continue to invite them to do so,” he concludes.
What are the two psychology professional organizations Dan refers to? How do public policy people and other non-experts avoid the trap of relying on the advice vetted by the wrong organization? How did these two organizations get set up? Is there any reference text describing their history? Why are there two and not one? What were the debates at the time they were set up?
In his piece, Dan seems to take the position that the quality problem in education research is institutional, and that it is up to the AERA needs to get its act straight and take a position as to what constitutes quality research.
Certainly there is a lot of hot pompous air labeled as research at the AERA.
But it’s not entirely clear if you can hope for consensus at the level of the AERA when education departments of reputable universities like Columbia U Teachers College, Chicago U, etc don’t have the stamina to root out bad practices from within their own internal backyard.
Looking comparatively at psychology, it would be interested to see where quality emerged first – in college departments, or in the professional organizations?
Comment by andrei radulescu-banu — May 18, 2010 @ 11:45 am
[...] This post was mentioned on Twitter by Robert Pondiscio. Robert Pondiscio said: Fools rush in where AERA fears to tread. Dan Willingham's take on what's wrong with ed research. http://bit.ly/cYTffK [...]
Pingback by Tweets that mention Fools Rush In Where AERA Fears to Tread « The Core Knowledge Blog, The Core Knowledge Blog -- Topsy.com — May 18, 2010 @ 12:42 pm
The two professional psychology organizations are the American Psychology Association (APA), established in 1910 by several pioneering psychologists of the early 20th century, and the Association for Psycholgical Science (APS), established in 1988, by a group of scientifically-oriented (read: research) psychologist who believed that APA was too much oriented to clinical / psychopathy issues. Their websites are easily accessible via Google or other search engines.
As for AERA, it’s a big-tent organization which welcomes very diverse views regarding the nature of educational research and what it means to “know” something about educational phenomena / learning etc. One can find very compelling gold standard research at AERA annual research meeting and other research efforts that can best be evaluated using other, no less valid (only different) standards of evidence. Buth, hey, you can find some dreck, too, just as in any slice of life.
Regarding Andrei’s last comment, these quality standards for psychologically-oriented educational research (e.g., role of motivation in student achievement) emerged, somewhat reciprocally, from both college/university psychology departments and the professional organizations (which are, of course, made up of faculty members from colleges/universities).
Comment by M C Smith — May 18, 2010 @ 4:16 pm
The logical place for education research is to look at professional organizations or conferences by discipline. AERA is too large for that. I am not familiar with a reading example used in the article. Judging by the broadness of the question, neither is the author. Compare:
“Suppose that I’m a legislator and I want to know what the latest research says about repression of childhood memories and whether recovered memories should be admissable as court testimony.”
“If I’m a legislator–or teacher–interested in a question such as, “What’s the latest research on how kids learn to read and how they respond to different ways of teaching reading?” where do I turn?”
The comparable psychology question would be, “What’s the latest research on memory and ways to improve it?”
In my area, mathematics education, you would currently go to PME (Psychology of Mathematics Education) and JRME (Journal for Research in Mathematics Education) as well as several other topic-specific journals such as JMB (Journal of Mathematical Behavior) for peer-screened research. You can ask any math ed researcher in a particular field to find it out. Or you can search Google Scholar for highly cited articles and track their sources.
“Naturally, a legislator is not going to dig through the research literature himself or herself.”
OUCH!!!
As the saying goes, “It’s everything that is wrong with everything.” Assisted by the staff, legislators have to dig through literature. And they have to be literate enough to understand what is dug up, as well.
Calling for more centralized bodies policing research seems counter-intuitive when everything is getting disintermediated. I think legislators and their staff will just have to learn to search the research webs instead.
Comment by Maria Droujkova — May 19, 2010 @ 6:09 am
I am a critic of educational research, but I’ll have to admit I know nothing about it. But why do I know nothing about it? At times throughout my life I have decided to learn a little about educational research, but nothing ever came of it. I would go to the nearest university library and browse the education section. It’s easy enough to find journals that purport to be about educational research. But when I open such a journal I could never find anything that seemed to be of any interest, or that seemed to go anywhere. How come? There are a few possible answers that come to mind.
Answer number one is that nothing will make sense until you have a starting point, and maybe I never found a starting point. But I’ve been to ed school. Admittedly that was many years ago, but shouldn’t that give me some starting point to make sense out of educational research?
Answer number two is that educational research is like scientific research. You can’t expect to understand anything until you have spent years studying the foundation of the subject. By this perspective a simple starting point is not nearly enough. We must go far beyond a simple starting point. We must amass a substantial body of knowledge. For example you can not expect to make much sense out of research in botany if all you have for background is high school biology. But by this argument we would have to believe that all those education professors in the ed schools have amassed that substantial body of knowledge about teaching and learning. Perhaps they claim to, but all the evidence available to me over a lifetime points to the conclusion that they have not.
A third possibility is that you have to be in a position, an occupational position probably, in which the research is relevant for it to make any sense.
Or maybe, a fourth possibility, the research is just not very good. That would be my expectation. I expect educational research to be not very good simply because I have never found educational research that seemed very good.
But for whatever reason I’ve never found educational research of interest in the past, maybe it’s still there. Maybe someone can help me get started.
So here’s my challenge, to anyone who will accept it. Name a bit of interesting and relevant educational research. Name a study that is:
- available on the internet, for free.
- readable and understandable with a few hours of effort. To be a bit more specific I would say ten to twenty hours of effort. When I find a good book it’s not hard at all to put in that kind of time over a period of a week or so. Project Follow Through, I presume, does not qualify on this count.
- relevant in at least some way. This could be relevant to what we do as practicing teachers. But it could also be relevant in a foundational way, something that explains something fundamental about teaching and learning.
- not a catalog of the obvious. I’m not much interested in plowing through a mountain of statistics in order to establish that at a five percent level of significance girls play with dolls more than boys do.
Summer is just about here. I have time, and I have the internet. Somebody show me the way.
Comment by Brian Rude — May 19, 2010 @ 4:43 pm
Brian, I would suggest asking edubloggers in the field you want to target for research satisfying your parameters. Yours are excellent questions that people will probably like to answer. Meanwhile, here are a couple of my favorites:
http://www.lkl.ac.uk/rnoss/papers/ProportionalReasoningNursingJRME.pdf
http://edorigami.wikispaces.com/Bloom%27s+Digital+Taxonomy
Comment by Maria Droujkova — May 19, 2010 @ 9:03 pm
[...] via Fools Rush In Where AERA Fears to Tread « The Core Knowledge Blog. [...]
Pingback by The scandal of education research « Future Schoolz: Research | Reform — May 23, 2010 @ 12:58 am
Thanks, Maria for those links. I looked up the one on proportional reasoning in nursing practice. I printed it out in its entirety so I could read it carefully. I find much food for thought there. Whether it’s good research or not I couldn’t say, but I do feel it’s the kind of research we need a lot more off. In a comment in “More Bang For The Book”, May 21 on the Core Knowledge Blog, I used this study in a favorable comparison to what I think is a more common, and less useful, type of study.
Comment by Brian Rude — May 24, 2010 @ 5:12 pm
Thank you for your comment at “Core Knowledge” – it made interesting points about “looking closely.” I find the book study you linked curious, and it definitely rings true in my own experience in several countries where I spent any amount of time at people’s homes: US, Russia, Ukraine, and Germany. Actually, I remember my mom teaching me good manners for visiting people’s homes when I was three or four, saying it’s rude to stare at any stuff, EXCEPT for books, which you can examine as closely as you want.
I am glad you found the nurse study valuable. This study and other work by the authors is “good research” in that it opened up many new research questions and provided a new framework for looking at learning mathematics. I found it immediately helpful for my work with students. As you say, the value is in the combination of rich qualitative data from long-term immersive observation, and statistics.
Comment by Maria Droujkova — May 31, 2010 @ 7:32 am