The next round in the statistical war between Bush advisor Matt Ladner and Shanker Institute fellow Matt DiCarlo is here. DiCarlo’s last look at Florida’s numbers prompted Ladner find DiCarlo “Bounded in a Nutshell but Counts Himself a King of Infinite Space.” I for one can’t wait for the hissy-fit Ladner will have over DiCarlo’s latest look at Bush’s school grade myth:
Former Florida Governor Jeb Bush was in Virginia last week, helping push for a new law that would install an “A-F” grading system for all public schools in the commonwealth, similar to a system that has existed in Florida for well over a decade.
In making his case, Governor Bush put forth an argument about the Florida system that he and his supporters use frequently. He said that, right after the grades went into place in his state, there was a drop in the proportion of D and F schools, along with a huge concurrent increase in the proportion of A schools. For example, as Governor Bush notes, in 1999, only 12 percent of schools got A’s. In 2005, when he left office, the figure was 53 percent. The clear implication: It was the grading of schools (and the incentives attached to the grades) that caused the improvements.
There is some pretty good evidence (also here) that the accountability pressure of Florida’s grading system generated modest increases in testing performance among students in schools receiving F’s (i.e., an outcome to which consequences were attached), and perhaps higher-rated schools as well. However, putting aside the serious confusion about what Florida’s grades actually measure, as well as the incorrect premise that we can evaluate a grading policy’s effect by looking at the simple distribution of those grades over time, there’s a much deeper problem here: The grades changed in part because the criteria changed.
Uh, oh. There’s not an educator or education reporter in Florida who doesn’t know this. Yet Bush never has to face such realities – especially when in the presence of anoter fawning republican governor like Virginia’s Bob McDonnell. More from DiCarlo:
the number of A-rated schools in six years” argument, but they are far preferable to claiming credit for what’s on the scoreboard after having changed the rules of the game.
In summary, then, it is incredibly misleading to compare the distributions of grades between 1999 and 2005 (to say nothing of attributing the increases, even if they’re “real,” to the system itself). Using a consistent set of criteria, there would almost certainly have been significant improvement in the grades over this time, but ignoring the huge rule changes in 2000 and 2002 severely overstates this positive change.
Again, Governor Bush and supporters of his reforms have some solid evidence to draw upon when advocating for the Florida reforms, particularly the grade-based accountability system. The modest estimated effects in these high-quality analyses are not as good a talking point as the “we quadrupled the number of A-rated schools in six years” argument, but they are far preferable to claiming credit for what’s on the scoreboard after having changed the rules of the game.
“Scoreboard” has emerged as the latest rhetorical trick of ed reformers. Who could ever be against a scoreboard? But what if the scoreboard gets filtered every year? Four outs this inning instead of three. Or maybe let one team start each inning with a player on third. I doubt that the Bushies don’t like having their model metaphorically dismissed so effortlessly.