Principal Lesson: School Leaders’ Incalculable Effectiveness

by William J. Mathis and Margaret Terry Orr

BOULDER, CO (March 5, 2013) – A recent paper for the National Bureau of Economic Research that attempts to estimate the effects of school principals on student achievement is ambitious in its scope but is undermined by serious methodological limitations, a new review released today concludes.
 
Indeed, the reviewer, Margaret Terry Orr of the Bank Street College of Education in New York City, finds that the most salient lesson to be learned from the paper is that it’s simply impossible to accurately estimate principal effectiveness using student achievement data for performance evaluation decisions given limitations in data and analytic methods – and that, therefore, such estimates should not be used to evaluate principals.
 
Orr reviewed Estimating the effect of leaders on public sector productivity: The case of school principals, by Gregory F. Branch, Eric A. Hanushek and Steven G. Rivkin, and published by the National Bureau of Economic Research. She also included in her review a less technical companion article by the same authors drawing on the same research: “School Leaders Matter,” in Education Next, a journal published by the Hoover Institution.
 
The review was published today by the National Education Policy Center, housed at the University of Colorado Boulder School of Education. Orr has extensively researched the subjects of leadership preparation and graduate evaluation, having served for several years on New York State’s Educator Evaluation task force and as an advisor to several states on their principal evaluation models.
 
Both the NBER paper and the Education Next article describe a study that seeks to establish the value added provided by school principals in improving student achievement. The researchers find large to small effect sizes, depending on the particular assessment model examined.
 
Orr notes that the studies use a large data set, but that methodological issues severely limit its usefulness. Those problems include the age of the data, which ranges from 12 to 18 years old, along with the failure to account for variables such as experience as a principal, effectiveness of comparison principals, nonlinear effects of principals on schools, other influences on high-poverty schools, and teacher mobility.
 
The authors also did not account for district policies and career mobility patterns.  Orr further notes that the authors could not estimate the effects of principals in their first year as a school leader.

“These methodological flaws and data limitations raise serious questions about the actual effect sizes of principals on student test scores and thus the validity of the analyses,” Orr points out.
 
The report is most useful for methodological discussions about technical issues such as value-added estimates for principals, the validity of different models, and principal effects on teacher and principal turnover.
 
“The most important policy-relevant conclusion that can be derived from this report is that estimating principal effectiveness is simply not possible given current methodology and sample size restrictions. Thus, such estimates should not be used to evaluate principals,” Orr concludes.
 
Find Margaret Terry Orr’s review on the NEPC website at:
 http://nepc.colorado.edu/thinktank/review-estimating-effect-principals.
 
Find Estimating the effect of leaders on public sector productivity: The case of school principals, by Gregory F. Branch, Eric A. Hanushek and Steven G.  Rivkin, on the web at
http://www.nber.org/papers/w17803, and the article “School Leaders Matter,” by the same authors, at
http://educationnext.org/school-leaders-matter/.