Wed September 19, 2012
Do Scores Go Up When Teachers Return Bonuses?
Originally published on Wed September 19, 2012 8:47 pm
In Chicago, parents were fuming over a weeklong strike by teachers. Around the rest of the country, in the face of growing evidence that many U.S. students are falling behind, administrators have tried to devise different ways to motivate teachers.
Among the contentious issues is whether teachers should be held accountable for their students' performance on standardized tests. Such efforts have produced enormous conflicts between school districts and teachers. In many parts of the country, administrators and teachers have fought one another to a standstill.
That's where a novel social science study may have the potential to shift the conversation.
Economist John List at the University of Chicago recently conducted an unusual field experiment in Chicago Heights, a school district near Chicago. List and his colleagues found a struggling school district: Only 64 percent of students met minimum state requirements on achievement tests. Nearly all the kids qualified for free or reduced-price school lunches, a measure of straitened socio-economic conditions.
List and his colleagues, Roland G. Fryer, Steven Levitt and Sally Sadoff, divided 150 teachers into three groups. One group got no incentive; they just went about their school year as usual. A second group was promised a bonus if their students did well at math.
The third group is where the psychology came in: The teachers were given a bonus of $4,000 upfront — but it had a catch. If student math performance didn't improve, teachers had to sign a contract promising to return some or all of the money.
Other U.S. studies have found limited evidence that traditional bonuses do much to shift student scores.
List said the idea of giving some teachers money upfront — with the threat of taking it away later — builds on a well-known psychological principle: "What we tried to capitalize on in this particular study was a concept called loss aversion," he said. "Once we have something in our possession, we feel it would be really, really painful to have to give it up."
Loss aversion has been shown to be a powerful motivator in many business settings, but List said this was the first rigorous test of the principle in an educational setting. The idea, he said, was that by giving teachers the money upfront, they might work harder to keep the money at the end of the year than they would if the money had been promised as a traditional bonus.
In line with earlier work, List and his colleagues found that students of teachers who received the traditional bonus performed no better than students of teachers who received no incentive at all. But List found that students of teachers who were given the bonus upfront showed significant improvement in math test scores.
"What we found is strong evidence in favor of loss aversion," he said. "Teachers who were paid in advance and [were] asked to give the money back if their students did not perform — their [students'] test scores were actually out of the roof: two to three times higher than the gains of the teachers in the traditional bonus group."
The difference in test scores produced by the incentive system was about the same as that detected in earlier studies that measured differences in student performance when kids were taught by great teachers rather than average teachers. Effectively, List said, the way the incentive was constructed turned average teachers into great teachers.
List said he thinks that the incentive system motivated teachers to be extra vigilant with underperforming students. If Johnny didn't get a concept, a teacher stuck with Johnny — and the concept — until the kid got it.
There are several caveats to keep in mind before anyone can talk about implementing the bonus structure widely, List said. Among them: The study needs replication. It remains to be seen whether the gains in student performance are long-lasting, and whether the same increases in student performance can be found in subjects other than math.
List said future studies might also tweak the incentive system: Teachers may be asked to return the money for underperformance periodically, rather than all at once at the end of the year. Prior studies involving loss aversion have found that the technique is more effective when people feel the threat of periodic and regular losses.
But there's another catch: List warned that the bonus system needed buy-in from teachers. Teaching isn't like making widgets; it requires motivation and passion. If teachers feel they are being manipulated rather than encouraged to improve their performance, they could end up looking for other lines of work.
RENEE MONTAGNE, HOST:
Part of the clash between the Chicago teachers and the city was about an issue that's become increasingly visible in school districts across the country. Should the performance of students on standardized tests, be used to evaluate the performance of teachers?
STEVE INSKEEP, HOST:
NPR science correspondent Shankar Vedantam joins us regularly to discuss social science research and has been looking into a related question: How do you compensate teachers so that students perform better?
SHANKAR VEDANTAM, BYLINE: Hi, Steve.
INSKEEP: OK. So what do you do?
VEDANTAM: Well, I spoke with this economist; his name is John List. He teaches at the University of Chicago. And he's just completed an experiment that looked at teachers in Chicago Heights. Now, this is a struggling school district, not very far from Chicago. And List decided to try something a little different from what's been tried before. He tried to say, is there an incentive system that I can give teachers, that can improve the performance of students? And he used a technique that actually has been used fairly widely in business settings, but hasn't been used before in educational settings. It's not quite a carrot, and it's not quite a stick. It's based, actually, on a psychological principle. Here he is.
JOHN LIST: What we try to capitalize on, in this particular study, is a concept called loss aversion. Once we have something in our possession, we feel that it will be really, really painful to have to give it up.
INSKEEP: Loss aversion.
VEDANTAM: Exactly. So I mean, if I were to make this practical for you, Steve, it would be like saying, I'm going to give you a bonus at the end of the year if you do really well. So that would be the traditional bonus. But the loss aversion would be, I give you your bonus up front. But then I tell you if your performance doesn't stack up, you're going to have to give the money back at the end of the year.
INSKEEP: OK. Subtle difference. He wants to know if this makes a big difference in people's minds. So what was the experiment?
VEDANTAM: So what he did was, he found 150 teachers in Chicago Heights. He divided them into three groups. One group got no incentive at all. One group got a traditional bonus. They were promised $4,000 if their students did really well at the end of the year. A third group was given $4,000 up front and told, if your students don't improve, you're going to have to give the money back at the end of the year.
And what List and his colleagues have found is that student test scores didn't improve when teachers were given a traditional bonus. It made no difference when you promise teachers a bonus at the end of the year. But when you gave the money up front, and told teachers the money could be taken away at the end of the year, it seemed to make a huge difference to student performance. Here he is again.
LIST: What we found is strong evidence in favor of loss aversion. Teachers who were paid in advance and asked to give the money back if their students did not perform, their test scores were actually out of the roof - two to three times higher than the gains of the teachers in the traditional bonus group.
INSKEEP: Wow, that's a huge difference. Why?
VEDANTAM: Well, first of all, List made sure that the teachers weren't cheating. So he made sure that the teachers weren't using their bonus to buy students extra supplies and extra textbooks....
INSKEEP: Oh, OK.
VEDANTAM: ...you know, the teachers weren't teaching longer hours. And the test was standardized, so there was no cheating on the test scores. What he thinks is happening is that the fear of losing the bonus seems to make teachers highly vigilant to underperformance. So if Johnny doesn't get something in class, the teacher doesn't go on to the next subject, or the next concept. The teacher sticks to Johnny, and makes sure Johnny gets it before moving on.
INSKEEP: Because I've already spent that money on the vacation at the Indiana Dunes, and I don't want to have to pay it back because I don't have it anymore.
VEDANTAM: Exactly. But I mean, the effect of this is pretty enormous. So, at least in this one study, the effect of this loss-aversion incentive system seems to turn average teachers into great teachers.
INSKEEP: So what are some of the drawbacks of this system, if any?
VEDANTAM: Well, I think this study would be need to be replicated before you can think about it in terms of policy. This is one school district; they looked specifically at math scores. They don't know if the same thing applies in other ways. It's also possible that the loss-aversion technique could be modified. In other settings, it's been found to be more effective when you take the money back in smaller increments over time, rather than in one, big chunk at the end. But overall, if the results do hold up, I think it could be quite powerful, especially when you consider this could be offered on a purely voluntary basis. You could tell people: If you want the bonus, you're welcome to have it right at the start of the year. There's just a risk that it could get taken back at the end of the year.
INSKEEP: NPR's Shankar Vedantam, solving the problems of the world. Shankar, thanks very much.
VEDANTAM: Thanks, Steve.
INSKEEP: And you can follow him on Twitter @HiddenBrain. You can also follow this program @MorningEdition and @NPRInskeep. Transcript provided by NPR, Copyright NPR.