Last week I posted a link to the long-term study which seemed to show the “value-added” of better teachers, as expressed in lifetime income, and various other factors such as girls who got through teenage without becoming pregnant. Monica quickly expressed her distaste for the whole “value-added” approach to teacher evaluation. Today, Michael Winerip has a wonderful piece in the Times in which he tracks down and reveals a key problem with the study: http://tinyurl.com/6nwskxn He found the flaw through close reading and follow up interviews, but any of us could have picked it up by simple math: the study looks at the impact of teachers on students over decades, that means the students must have been in school decades ago, which is to say, before the era of high stakes testing. So while it indeed may be that better teacher have a clear and identifiable “value added,” (and thus, of course, less effective teachers have an equally clear negative effect on students), nothing in the study links identifying those teachers to our modern high stakes tests.
I admired Mr. Winerip for slowing down, reading the study carefully, asking himself the right questions, and then following up by speaking with the authors of the study — who confirmed his insight. That is a perfect example of the “Slow Thinking” Daniel Kahneman examines and describes in Thinking Fast and Slow and a reminder (to me) of the danger of reading over a study and taking away talking points, without examining it carefully. On the subject of education, Kahneman gives another perfect, and related example — which starts in a very different place, talking about kidney cancer.
It turns out that the lowest incidence of kidney cancer in the US are in rural, sparsely populated, Republican districts in the South and Midwest. Why is that? Surely none of us think that voting preferences influence our kidneys, or that the relative health of our kidneys influences us in the voting booth. Many might assume that these rural areas have less pollution, or that hard-working farmers living close to the land tend to be free of the blights of stressed-out urban and suburban folks. Is that what you thought? Well it also so happens that the highest incidence of kidney cancer in the US are in rural, sparsely populated, Republic districts in the South and Midwest? Why? It can’t be that the soil is both significantly healthy and unhealthy at the same time. The real key is the hyphenated word –“sparsely populated” — because the sample of people studies is smaller, it yields more extremes. Small sample = extreme results. Kahneman urges us (and scientists, who commonly make this error) to linger long on the question of whether the sample size in a study is large enough to overcome this tendency and yield significant results. In the case of the “value-added” study, size is not the issue. But this does tie to education, because Kahneman claims that the entire $1.7 billion Gates Foundation initiative to create smaller schools is based on false results — because smaller schools will always yield more extreme results — smaller schools, smaller samples, more extreme results (students who flourish, students who fail). Therefore making schools smaller does not necessarily make then any better, “the truth is small schools are not better on average; they are simply more variable.” (118)
We all need to slow down, and hats off to those who do.