On Tuesday morning a study was released by the National Education Policy Center at the University of Colorado, Boulder, that is highly critical of the controversial LA Times’ system of teacher evaluations that was published last summer.
The new report, released by U of Colorado researchers, Derek Briggs and Ben Domingue, found that “the research on which the Los Angeles Times relied for its teacher effectiveness reporting was demonstrably inadequate to support the published rankings.”
Harsh words.
After the LA Times folks got an early look at Briggs and Domingue’s study, they rushed out a story of their own regarding the study’s findings.
The headline on Monday’s LAT story read as follows:
Separate study confirms many Los Angeles Times findings on teacher effectiveness
Huh? “Confirms?”
In the sub hed, the Times admitted that the Colorado study raised “…some questions about the precision of ratings as reported in The Times,” but most of the story suggested that the new research was more validating than it was critical.
Okay, well,….now compare those characterizations with the title of the press release for the U of Colorado study:
Research Study Shows L. A. Times Teacher Ratings Are Neither Reliable Nor Valid
Then the press release really revs up:
Based on the results of the Briggs and Domingue research, NEPC director Kevin Welner said, “This study makes it clear that the L.A. Times and its research team have done a disservice to the teachers, students, and parents of Los Angeles. The Times owes its community a better accounting for its decision to publish the names and rankings of individual teachers when it knew or should have known that those rankings were based on a questionable analysis. In any case, the Times now owes its community an acknowledgment of the tremendous weakness of the results reported and an apology for the damage its reporting has done.”
In short, the Colorado study does more than “raises some questions.” It’s an outright frontal attack.
BUT BEFORE WE GO FURTHER, LET’S REVIEW THE BACK STORY TO ALL THIS.
As you will remember this past August the LA Times ranked 6,000 LA Unified School District elementary school teachers into categories ranging from “most effective” to “least effective.” The rankings were based, in the simplest terms, on whether or not the teachers’ students improved, stayed the same, or got worse in their performance on standardized math and English tests. (It’s a little more complicated, but that’s the basic principle.) This criteria for evaluation has come to be known as “value added.”
The Times published the rankings in a searchable database that made public the ranking of the 6000 teachers. This caused the LA teachers’ union, UTLA, to go utterly ballistic. Union prez A.J. Duffy urged his members to cancel their subscriptions to the LA Times. Rallies were held and so on.
Yet, the series of articles, written and reported primarily by Times reporters Jason Song and Jason Felch, jump-started a long-overdue local and national conversation on the subject of merit-based teacher evaluations in a way that nothing else had.
Not surprisingly, the series began winning awards.
ENTER U of COLORADO RESEARCHERS DEREK BRIGGS AND BEN DOMINGUE
In their study, Briggs AND Domingue say that when they attempted to reproduce the Times’ findings, (while also controlling for additional variables that the Times’ researcher did not employ), they got very different results:
For example, when they looked at how the teachers did with reading test outcomes, their findings included the following:
• More than half (53.6%) of the teachers had a different effectiveness rating under the alternative model.
• Among those who changed effectiveness ratings, some moved only moderately, but 8.1% of those teachers identified as “more” or “most” effective under the alternative model are identified as “less” or “least” effective in the L.A. Times model, and 12.6% of those identified as relatively ineffective under the alternative model are identified as effective by the L.A. Times model….
It goes on from there.
The dueling studies first caught my attention when happened to hear U of Colo. researcher Derek Briggs on a segment of the same Monday Patt Morrison show that I had just been on. [See post below.]
Briggs was joined on the segment by the Times’ editor on the teacher evaluations project, David Lauter. For the first half of the segment, Briggs roundly criticized the Times’ findings and methodology. Then, in the following half, Lauter cheerily ignored and/or spun everything that Briggs had to say.
I found the exchange to be very disconnected and perplexing.
You can (listen for yourself here.)
I turns out I was not alone. Education writer Emily Alpert of the Voice of San Diego was similarly flummoxed by the discrepancy between the new study—and the Times take on the new study.
Then late Monday night the National Education Policy Center plus researchers Briggs and Domingue, issued their own unhappy rebuttal to the Times’ article. It began:
Yesterday, on Monday February 7, 2011, the Times published a story about this new study. That story included false statements and was generally misleading….
A point-by-point fact sheet followed.
Look: I have no idea whether or not Briggs and Domingue have a more accurate model for teacher evaluation than the Times does. Or if the truth is somewhere in between.
I do know, however, that if we are continue the important conversation that the Times’ series started, we need to make sure that conversation is fact based.
Monday, it wasn’t.
PS: According to the U of Colorado, their study was embargoed until Tuesday morning, a stricture that the LA Times merrily ignored.
PPS: Both Jason Felch and Jason Song are very good journalists whose work I respect and admire. Thus I can’t help but wonder if this urge to spin the contents of the Colorado study came from above their pay grade.
[…] Over at Witness LA, Celeste Fremon notes that despite the harsh conclusions of university researchers, the Times’ writeup of the study seems to suggest the research “confirms” their findings. The new report, released by U of Colorado researchers, Derek Briggs and Ben Domingue, found that “the research on which the Los Angeles Times relied for its teacher effectiveness reporting was demonstrably inadequate to support the published rankings.” […]
In other words, the LA Times covers education the way the LA Times covers child welfare. And David Lauter responds to criticism of LA Times coverage of education the same way he responds to criticism of LA Times coverage of child welfare, as you described in these very good posts:
http://witnessla.com/los-angeles-times/2010/admin/the-la-times-replies-to-criticism-over-dcfs-coverage-part-1/
http://witnessla.com/los-angeles-times/2010/admin/disagreeing-with-the-la-times-over-foster-coverage-the-sequel/
Better be careful. Sometime soon, Mr. Lauter is bound to accuse you of not caring if LA schoolchildren get a good education.
Richard Wexler
Executive Director
National Coalition for Child Protection Reform
http://www.nccpr.org
Those involved in the education world have consistently said Value Added Methodology is not ready for prime time–even those who support the use of such methodology eventually (see Rick Hess). Many folks expressed serious concern about labeling teachers as “less effective” on such questionable methodologies, in a major newspaper no less. Don’t you agree that if a media outlet is to name and shame individuals it has a responsibility to make sure its information is accurate? The LA Times knew about the VAM controversies and went ahead anyway. And a teacher committed suicide. And the readers are no more informed on who is a good teacher.
But hey, it sold papers, garnered page hits.
Criticize the Times all you want (and there’s plenty to pick from) but if a teacher committed suicide for being upset about the poor ranking, is that person really teacher material? If this person’s reaction to the ranking was suicide, how could they deal with so many other issues in the classroom? How could the Teachers Union, or fellow teachers at that school, not notice how upset the teacher was? If teachers can’t notice that about their coworkers, what does that say about the students they are supposed to monitor at the school?
If LAUSD is going to use value added analysis on the teachers then…
http://www.examiner.com/public-education-in-los-angeles/if-lausd-is-going-to-use-value-added-analysis-on-the-teachers-then
anon at 4:14 – That’s harsh! Many, many factors contributed to the teacher killing himself, but it appears the LAT phony rankings finally pushed him over the edge. I thankfully have not been close to anyone who has committed suicide, but from what I read, those who are really serious about it keep it to themselves.
I’m glad that people besides teachers are starting to speak out against the Times’ rankings, because no one will listen to the teachers over the media and the politicians, who want to sell newspapers and buy votes.
The LAT used to be a really great newspaper, but it hasn’t been for years. They used to do multi part, multi day in depth investigations that were worth spending the hours reading, when the paper was a real broadsheet, not an over sized tabloid. Now I just give the paper a skim on my computer before I dash out in the morning.
First, the issue of the embargo is a red herring. The LAT was not bound to the embargo because it never agreed to the embargo in the first place. That’s because the LAT did not receive the study from UC Boulder and thus was never asked to respect the embargo. One can’t impose an embargo on a third party without their consent, and one can’t impose an embargo retroactively.
Second, the comment from Carol about the paper falling from grace is odd, since she laments the paper no longer doing multi part, multi day in depth investigations on a blog that is in fact discussing a multi part, multi day in depth investigation in the LAT.
You can’t have it both ways, Carol!
I have a choice between believing distinguished university professors and researchers, confirming what nearly all the experts were saying about VAM, versus yellow journalists and amateur statisticians Jason Felch and Jason Song?
It’s bad enough that the Gates Foundation paid for all the VAM pseudo-science hype, but I recently learned that Gates also supports the Discovery Institute to the tune of millions. They believe humans rode dinosaurs, which is about as scientifically correct as VAM.
http://dissidentvoice.org/2011/02/on-anschutz-villaraigosa-lausd-privatization-candidates-and-riding-dinosaurs/
If all Carol said was the LAT had fallen from grace then Anon should feel lucky Carol held back. I won’t. No significant institution has failed Los Angeles more in the last 25 years than the Journalism of the Los Angeles Times.
The Times has become a real joke that people make.
“Yellow” journalism is accurate. No real journalist would throw away his or her lifetime of past and future integrity so easily.
From the LA Times article, this point really resonated with me (a charter school teacher whose entire student population consists of recovered drop-outs): “The authors also found that the way school administrators assign students to teachers — giving especially challenging students to a certain teacher and not to others, for example — could have skewed the value-added results.”
Value-added rankings such as those that appeared in the LA Times provide a huge financial & psychological disincentive to work with students who are habitually truant, who have dropped out, and/or who are in and out of juvenile halls and probation camps. This is a very unstable student population. It may take several months or even years of contact with these students to finally engage them in school in a consistent manner. They are the most difficult students to track down and have incredible obstacles in their lives that make attending school challenging on the best of days.
Those of us who choose to work with these students do so because we believe passionately that these students can and should have access to a public education. We know our students have very limited options in terms of schooling. Publicly comparing our test scores with that of other teachers would be misleading, disheartening, and ultimately unproductive in mobilizing more teachers in more sectors of public education to help us tackle this city’s enormous drop-out problem.