Teacher Value Add Data Unreliable

2/3/14 1:41 PM EST

Calculations of teachers’ “value-added” scores — which aim to measure teachers’ effectiveness at boosting student learning — may fluctuate significantly due to factors beyond their control, including such random events as a dog barking loudly outside a classroom window, according to a new study funded by the Education Department.

The study looked at more than 1,140 fourth- and fifth-grade teachers in 10 school districts in Indiana. Their students took two different assessments in math and reading: The Indiana state test, known as ISTEP, and the norm-referenced Measures of Academic Progress test, known as MAP. The ISTEP and MAP cover similar content and scores correlate fairly well; students who do well on one tend to do well on the other.
But many teachers’ VAM scores varied considerably depending on which test was used.

The researchers calculated two VAMs for each teacher, one based on their students’ ISTEP scores and one based on the same students’ MAP scores from the same year.
Just one-third of the teachers fell into the same performance quintile with both the ISTEP- and MAP-based value-added measure. Another 38.5 percent got similar results — their two VAM scores were just one quintile apart.

But 19 percent of teachers did significantly better on one VAM than the other, with their scores two quintiles apart, meaning they might have been rated in the bottom 20 percent of all teachers according to their students’ scores on one exam, and squarely in the middle of the pack of teacher effectiveness based on the same students’ scores on the other exam. Seven percent of teachers had scores three quintiles apart. And 2 percent of teachers swung from the top quintile to the bottom depending on which test was used for the calculations.

The researchers, from the Regional Educational Laboratory Midwest, conclude that the variations don’t reflect on the quality of teaching. Rather, they are likely due to “measurement error,” which could derive from conditions such as the time of day the tests were administered or distractions such as a student misbehaving or a dog barking during one exam but not the other. “Such idiosyncratic factors as these will cause student  test scores, and consequently estimates of teacher value- added, to fluctuate from year to year and test to test for reasons unrelated to the quality of teaching and learning in the classroom,” they conclude.
The researchers recommend using several years of data when calculating VAM scores to smooth out some of the fluctuation.

Many districts nationally base their teacher retention and merit pay decisions in part on VAM scores, which typically account for 30 percent to 50 percent of an educators’ job evaluation.
— Stephanie Simon

 POLITICO