The Broader, Bolder Approach to Education is out today with its recommendations on school accountability. As I write this, the report is not yet on BBA’s website, but many of the recommendations will be familiar to readers of Richard Rothstein’s Grading Education: Getting Accountability Right.
I suspect the recommendation that will be subject to the most discussion is the call for states to “provide for the inspection of districts and schools to ensure their contributions to satisfactory student performance in academic subject areas, as well as in the arts, citizenship, physical fitness and mental and physical health, work and other behavioral skills that will enable them to achieve success in a pluralistic society and complex global economy.” The report points that “school inspections as the core of state accountability systems” have precedent in places like England.
There is a lot to agree with in BBA’s insistence that a narrow focus on testing has had a deleterious impact on schools. But personally, I wonder if a schools inspectorate will make matters better or worse. Spend time in a struggling school in the weeks before a “quality review” and you’ll see an extraordinary amount of teaching and learning time going to cleaning classrooms, updating portfolios, making sure bulletin boards have up-to-date student work, etc. Having lived through a few such inspections, its tempting to suggest judging a school from a formal walk-around is like judging a household from a Thanksgiving dinner. Remember the grief your mom used to give you to clean up and mind your manners before company came? Now imagine mom’s livelihood depends on it. That’s a school in the weeks before quality review. It’s hard not to be skeptical that a visit from the inspectorate would be any less subject to gaming and distractions that a relentless focus on test prep.
Update: The report is here. Patrick “Eduflack” Riccards has a detailed summary here.
Not sure if it’s gauche to pitch an interview here in your comments section, but we just published an interview with former assistant secretary Christopher Cross on the new BBA recommendations. He addresses the inspectorate idea in some detail, though I’m not sure he touches on your concerns specifically: http://www.publicschoolinsights.org/christopher-cross-describes-broader-bolder-approach-accountability.
Also, I would think a good inspectorate system should be able to look beyond the spit and polish of a few days. Practices that are well embedded in school culture would be difficult to fake–if the inspection rubrics are well conceived.
We’ll publish an interview with Diane Ravitch tomorrow.
You might be right, Claus, and I muddied my point in the post. Just like it’s not the testing, but the response to the testing (endless test prep, curriculum narrowing) that’s the problem, I worry less about the inspectorate than the response within the schools. I’m a little skeptical about the inspections too. Lots of manpower and expertise needed. And it’ll be hard not to reduce it to a checklist exercise.
All that said, we need a dialogue on accountability. So BBA’s report is a welcome addition. I need to read through it again, but I thought the authors were very careful in anticipating the idea that they’re soft on accountability.
I think it comes down to something you’ve addressed many times before. Big ideas can often suffer from poor implementation or mindless mechanization. That problem, however, can doom just about any reform, however brilliant and well conceived at the outset.
The policy report is short, but it seems rather oblivious to the scholarly literature on principal evaluations (and if they’re proposing to come up with professional inspectors who are way better than principals, I would have to wonder who is going to “train” those inspectors in the first place).
In a famous 1987 study that summarized previous research, Medley and Coker pointed out that ??“almost all educational personnel decisions are based on judgments which, according to the ?research, are only slightly more accurate than they would be if they were based on pure chance.”? ? ?And in a noted 1995 book, Kenneth Peterson claimed that “seventy years of research on principal ?ratings of teachers shows that they do not work well. Well-designed empirical studies depict ?principals as inaccurate raters both of individual teacher performance behaviors and of overall ?teacher merit.”?
What makes them think that these so-called “professional inspectors” would do any better?
* Medley, Donald M., & Coker, Homer. 1987. “The Accuracy of Principals’ Judgments of ?Teacher Performance.” Journal of Educational Research 80 no. 4 (March/April): 242-247. ?
* Peterson, Kenneth D. 1995. Teacher Evaluation: A Comprehensive Guide to New Directions and ?Practices. Thousand Oaks, CA: Corwin Press. ?
That’s one way to put it, Claus. The issue is really that high-stakes accountability drives practice, and not always sensibly so. Thus you need to think about the most likely response to the accountability measure, not merely the ideal or even preferred response (it’s unlikely that anyone really wanted to drive test prep and take history, art, science and music out of the elementary school curriculum when they passed NCLB, for example). So I think the response to a directorate is just as likely to be as deleterious. I’m also less sanguine than you that it can be well-executed. Frankly, I’m even seen documentation falsified and student work massaged to make intervention plans and portfolios more robust than they actually were.
The good news is it would be easier to do random, unannounced inspections than random, unannounced testing. But somehow I don’t think that’s what BBA has in mind.
While there may be a danger of fraud, it strikes me that anyone who can fake good assignments and truly challenging lesson plans after the fact can, and probably would, create them for more honest purposes. The ability to create first-rate lesson plans and assignments that are aligned with strong curricula and broader school goals is, after all, critically important. The question of how to guard against falsification of the resulting student work is another matter, but the inspection system is important for its ability to look behind test scores.
BBA doesn’t, as far as I know, advocate the abolition of state tests, so a good inspection system could give us a much stronger sense of what’s happening behind the curtain. Test-prep factories would likely have a very difficult time suddenly creating sham lesson plans, assignments and portfolios that would belie their actual practice. The inspectorate reminds me of research by the Chicago Consortium earlier this decade: Consortium researchers reviewed the effects of school reforms on the level of assignments and student work, because they felt test scores alone weren’t complete measures of quality. The results of such research–or inspections–can inform robust conversations about improvement. That’s not necessarily the case with test scores alone.
As for the ability to train effective inspectors…. Research more recent that the studies you cite, Stuart, suggests that principals are actually effective and distinguishing the most effective teachers from the least effective teachers, though they’re less effective with teachers in the middle range. (See, for example, Jacob and Lefgren, 2007–http://econ.byu.edu/faculty/Lefgren/Assets/papers/principals.pdf). According to Jacob and Lefgren, principals are about as effective as value-added measures in identifying teacher performance–though that’s not necessarily a high bar to clear. Also, principals all too often haven’t been given the time or training to perform effective teacher evaluations.
There are models of effective evaluation programs that an inspectorate system could possibly draw on. As with any reform, poor or hasty implementation can do great damage. As always, the devil is in the details.
Yes, I’m familiar with Jacob/Lefgren. They actually found that value-added measures were 50% better at predicting future student achievement than principal evaluations (see page 28). And the correlation between achievement and principal evaluations was about 0.3 (and that’s for the question where principals were specifically asked about how effective the teacher was in raising achievement).
I’m intrigued by the Wilkerson et al. (2000) paper suggesting that student ratings can be the best predictor of whether a teacher is successful at raising student achievement. See http://www.rise.hs.iastate.edu/uploads/reports/360%20wilkerson,manatt,rogers&maughan.pdf
What in the world is a “lesson plan”? And if schools were inspected, would inspectors actually ask to see teachers’ lesson plans, and make judgments on that basis? And if so, isn’t that a giant step backwards?
Yes, I know what a lesson plan is back in ed school. Is that what we’re talking about here? But the lesson plans we learn to make in ed school are just a matter of jumping through hoops to get through the course. Everyone knows that. Don’t they? Haven’t we known that for about a hundred years now? Has anything changed?
Teachers plan. Good teachers obviously make good plans. But would anyone actually claim that a good plan by a teacher would be judged a good plan by ed school standards?
Yes, I know I’m a cynic. Don’t I have good reason? One might ask why I’m talking about ed school. Well, who is going to train and certify those inspectors?
Good teachers plan. They always have and they always will. But that doesn’t mean they write down those plans, or that if they do that writing will be little more than scribbled notes to themselves, or that those scribbled notes will have any meaning to anyone else.
I teach in college, lower level math. That is certainly different in some ways than teaching K-12, but in other ways it is very much the same. I struggle everyday to explain math in ways that students can understand, so that they can do problems and pass tests. I struggle everyday to make assignments that will be just what students need to understand. I struggle to get those homework assignments graded and back to the students the next class period. I then I struggle to figure out how to do it better next semester. I have some success, and I have some failure. But I plan. I plan every semester and I plan everyday. I spend considerably more time in planning than I do in front of a class.
But if an inspector came by and wanted to see my “lesson plan” for tomorrow’s algebra class, what I would show him or her is my assignment sheet, a list of problems. That’s it! That’s all I write down. That’s all I plan to write down. That’s all I have time to write down. That’s all I should write down. I would probably also show the inspector a bit of attitude, an attitude that would not be in my best interests. I have ideas about teaching and learning. I try everyday to refine those ideas, to profit from my past experience, to do things a little more effectively next semester.
Yes, I am capable of writing down those ideas. I have a website full of that writing. But to make a “lesson plan” as we learned in ed school, with goals, objectives, methods, etc, is not a good use of my time. Jumping through hoops seldom is.
A well designed homework assignment, which generally means a well chosen list of problems, is crucial to good teaching. It’s just as important as a good explanation in class. But how is an inspector to look at a list of problems, from a book he has never seen before, in a subject he doesn’t know, and say, “Yup, you’ve got a pretty good plan there.”?
This is not to say that no valuable information could be obtained by spending time in a school and trying to find out what’s going on. But how it could be done to be more constructive than destructive needs a lot of thought.
Brian–While I don’t share your disdain for lesson plans (I used them religiously when I taught), I do understand the need for healthy skepticism. It’s enormously important to determine who does the inspections, with what kind of training, under what conditions, using what kinds of rubrics, motivated by what vision of quality, etc. Unfortunately, any reform idea brings with it all sorts of caveats, and even good ideas can (and do) become destructive if they devolve into bureaucratic nonsense. It’s often frustrating just how quickly people in the reform community jump onto bandwagons without acknowledging complexities or tradeoffs.
Still, my sense of the BBA group is that they’ve acknowledged the complexities of the approach they promote. Chris Cross said quite openly that the development of an inspectorate will take time. At least other countries offer some models.
Stuart–Fair enough about Jacob and Lefgren, who find that value added measures are more effective than principals in predicting future student achievement. Yet they also suggest that principals might be able to identify “strategic behavior on the part of teachers to improve test scores without increasing actual knowledge” (i.e. mindless test prep) because they “observe inputs as well as outputs.” A well-designed inspectorate system could have similar effects. And don’t forget that inspections would be coupled with other forms of data, including state assessment data.
This is an example of a positive school review (27 pages).
http://www.doe.mass.edu/charter/reports/inspect/0484.pdf
This is an example of a negative school review.
http://www.doe.mass.edu/charter/reports/2007/ncces/sitevisit.pdf
* * *
If you feel like reading it, simply search for the word “observe” and you’ll find what the inspectors reported first hand.
For example: “Site visitors observed that the quality of instruction delivered during the essential skills blocks was uneven, with teachers ignoring some off task behavior and sometimes focusing their attention on a small group of engaged students. The tone in some classes was observed to be friendly, casual, and lacking in focus or urgency.”
Thanks to GGW for the links to those two school reports. I looked at both of them. I don’t claim to have read them thoroughly or carefully, but they do bring to mind a few thoughts.
In both cases the immediate purpose of the report is to justify the renewal of the school’s charter. In the second report, the less favorable one, I was struck by the four requirements that they school had to meet for its charter renewal. The first requirement for renewal was that the school had to come up with a report about something. Check. Done. Accomplished. The second requirement was that the school had to come up with a report about something else. Check. Done. Accomplished. The third requirement was that the school had to come up with yet another report on yet something else. Again, check. Done. Accomplished. They’re batting a thousand so far. The fourth requirement was that their students must make adequate yearly progress in NCLB terms. On this they came up a little short.
Some people would say this is all reasonable and sensible. Others, like me would be much more cynical. It appears to me that only one of those requirements is substantial. The other three, in my humble opinion, can be fairly described as jumping through hoops. They assemble some rhetoric saying that they have very good intentions. Is the world a better place now?
All this reinforces my perception that subjecting schools to regular inspections is nothing to get excited about. I stated in my last post that I believe valuable information could be obtained by spending time in a school and trying to find out what’s going on. But that seems much more like a possibility than a realistic expectation.
It seems to me that we don’t really have much agreement about what constitutes good or bad teaching, or what constitutes a good or bad school. We have some agreement, but that seems mainly intuitive. We think we can recognize what’s good or bad when we see it. Nor do we have much agreement about the means to accomplish what we intuitively agree on. So does this give us an adequate basis for making a system of school inspections beneficial, worth the cost?
It is suggested that such inspectors need training. Where will this training come from? Ed school? Where else? Is this good?
The last year I taught in a public school we had a school inspection. I was not much involved, and didn’t give it much thought. I got a copy of the report a month or so later. I read it, but without much interest. Life in the school went on as before. Nothing changed. I suppose no one expected anything to change.
More recently I was working in a factory when the factory had an inspection that was a part of some process for obtaining some desired certification. Again it was all very peripheral to me. It came and went. Nothing changed, of course.
When I was in the Army we had occasional inspections of our unit. They were always unpleasant, but of course there was never any change in anything.
Where will educational improvement come from? School inspections? Well, maybe. Apparently they have them in Massachusetts, and from what I read Massachusetts does pretty well in its education. But correlation, we must constantly remind ourselves, is not causation.
So what is the cause of Massachusetts’ success? I’d really like to know.
Note that only charter schools in Massachusetts are inspected.
There used to be a different inspection office, called EQA. They inspected districts.
They just disbanded EQA.
Districts didn’t like critical reports and complained to the Governor.