Is Anyone Listening? Policy Versus Research on Test-Based Accountability and Charter Schools

by Iris C. Rotberg — June 19, 2012

It is particularly important to examine the research on testing and school choice because both are central to federal policies under No Child Left Behind and Race to the Top and both have contributed to fundamental changes in U.S. education. This article analyzes the extent to which current federal policies in these areas are supported by research findings. It argues that as research findings on the consequences of education policies have become clearer, federal policies pertaining to teacher and principal accountability and school choice have moved in a direction directly contradictory to the preponderance of research evidence.

The slogan “evidence-based practice” appears frequently in government documents as a reminder to base policies and programs on research rather than on conjecture. It is ironic, therefore, that as research findings on the consequences of recent education policies have become clearer, federal policies pertaining to teacher and principal accountability and school choice have moved in a direction directly contrary to the preponderance of research evidence. Some have argued, however, that the research findings are ambiguous and leave policymakers with little information beyond “on the one hand, on the other hand.” In fact, the research findings on a number of education policy issues are quite clear.

It is particularly important to examine the research on testing and school choice because both issues are central to federal policies under No Child Left Behind and Race to the Top, and both have contributed to fundamental changes in U.S. education. This article analyzes the extent to which current federal policies in these areas are supported by research findings. It draws largely on research conducted by the National Research Council and by the RAND Corporation. These organizations both have strong reputations for their expertise and objectivity, and both have conducted extensive reviews that take into consideration a wide range of studies and many of the factors that have contributed to variations in research findings.

The main focus of this article is Race to the Top, a competitive grants program that builds on No Child Left Behind. The federal funding provided by Race to the Top is highly dependent on whether states adopt specific policies. Among the most controversial policies are those designed to hold teachers and principals accountable for their students’ scores on standardized tests and to increase the number of charter schools.

LINKING STUDENT TEST SCORES TO SPECIFIC TEACHERS AND PRINCIPALS

States were only eligible to apply for funding under Race to the Top if they had no barriers to linking data on student achievement or growth to specific teachers and principals for purposes of evaluation. Although the selection criteria for funding stated that school districts should evaluate the effectiveness of teachers and principals using multiple measures, student growth (as measured by value-added techniques) was expected to be a significant factor (U.S. Department of Education, 2009). In practice, this meant that test scores typically trumped other measures. These evaluations were to be used in making decisions on compensating, promoting, and retaining teachers and principals. Highly effective teachers and principals were defined as those who achieve high rates of student growth—for example, one and one-half grade levels in an academic year. Therefore, to be rated highly effective, teachers and principals had to exceed the proverbial “above average” standard set by Lake Wobegon.

In its response, the National Research Council cited the wide array of problems inherent in using value-added measures to evaluate teachers. The council emphasized that without random assignment of teachers to schools and students to teachers, value-added methods could not control for unobserved differences among students (for example, differences in motivation and parental involvement) and, therefore, could not provide objective evaluations of teachers. The council also cited a number of other problems, including unreliability of value-added measures (teachers might be rated high one year and low the next); variation depending on the specific value-added method used; difficulty in linking students with teachers by subject; difficulty in isolating the effects of individual teachers when students are taught by multiple teachers; low transparency of value-added measures (that is, the statistics remain a mystery to most people); and uncertainty about whether the measures are valid indicators of the effectiveness of teachers.

The council strongly advised, therefore, that further research be conducted before implementing large-scale evaluations of teachers using value-added measures. It expressed “significant concerns” about using measures of student achievement that “have not yet been adequately studied for the purposes of evaluating teachers and principals” and “that face substantial practical barriers to being successfully deployed in an operational personnel system that is fair, reliable, and valid” (National Research Council, 2009, p. 8).

The policies currently implemented under Race to the Top suggest that the advice was not taken.

A subsequent report by the National Research Council on test-based incentives came to a similar conclusion: “Despite using them for several decades, policy makers and educators do not yet know how to use test-based incentives to consistently generate positive effects on achievement and to improve education.” As in the previous report, the council advised that the “programs be used with caution and that substantial further research is required to understand how they can be used successfully” (National Research Council, 2011, p. 5).

The council also expressed concern about the incentive to narrow the curriculum inherent in test-based accountability plans. Indeed, that risk is magnified by the fact that high-poverty schools, which are under the greatest pressure to raise test scores, also have the greatest incentive to narrow the curriculum. Test-based accountability, therefore, might widen the achievement gap if low-income children are deprived of the educational opportunities available to their more affluent peers.

INCREASING THE NUMBER OF CHARTER SCHOOLS

The Race to the Top competition also encouraged states to ease or eliminate any caps limiting the number of charter schools—even though the preponderance of research evidence did not show that charter schools, on average, had higher student achievement than traditional public schools. A comprehensive study by RAND of choice programs in the United States and other countries concluded: “None of the studies suggest that charter-school achievement outcomes are dramatically better or worse on average than those of conventional public schools” (Gill, Timpane, Ross, Brewer, & Booker, 2007, p. xiv).

This finding is not surprising. Charter schools vary widely, as do traditional public schools; the terms “charter school” and “traditional school” provide little information about school quality. In some settings, charter schools are superior academically; in others, they have serious academic problems. These differences are actually quite predictable given the varied settings in which the studies are conducted. For example, some studies are conducted in districts where charter schools have high per-pupil expenditures and can offer a wide range of services, while others are conducted in districts where charter schools face serious fiscal problems.

Moreover, the risk that charter schools will lead to increased student stratification also varies depending on the characteristics of the choice program and its setting. RAND’s analysis of research found a potential for stratification in large, unregulated school choice programs, both in the United States and other countries. Although most charter schools in the United States appear to have racial/ethnic distributions comparable to those of local public schools, in some states many charter schools are racially homogeneous (Gill et al., 2007).

To date, a large number of charter schools are located in districts where public schools are already highly segregated by race and ethnicity. Therefore, RAND’s finding that most charter schools have racial/ethnic distributions similar to those in local public schools is not unexpected; it would be virtually impossible, for example, for charter schools to segregate students any more than they are already segregated in most inner cities. However, a national expansion of charter schools in response to federal incentives could lead to greater student stratification if states increased the number of charter schools in districts that are currently integrated.

The potential for increased stratification has clear implications for federal policy. An expansion of charter schools in heterogeneous school districts is likely to have very different consequences from an expansion in inner cities. The risk is that widespread expansion of charter schools in integrated communities will result in school enclaves based on socioeconomic status, race, and ethnicity. The potential for adverse effects is too great.

There is a further risk in federal policies promoting the expansion of charter schools: the potential for charter schools to stratify by characteristics in addition to socioeconomic status, race, and ethnicity. RAND, for example, found that students with disabilities are less likely to attend charter schools. Anecdotal evidence has shown that charter schools also have the potential to increase stratification by religion and even political values—developments that have not yet been systematically studied in the United States. However, we know from international experience that a number of countries, including Sweden, the Netherlands, and Israel, have struggled with the stratification effects of their school choice policies.

Federal charter school policy applies, without distinction, to all states receiving Race to the Top funds and, therefore, to all settings and programs across the country. It does not distinguish between settings in which charter schools might be a positive influence and those in which they are likely to fail academically or stratify a previously integrated system. The fact is the federal government cannot control the quality or stratification of schools, despite its admonition to the states in its Race to the Top guidelines that charter schools should be “high-performing.”

An additional observation: the federal government’s advocacy for both charter schools and test-based accountability has increased substantially over the past 10 years. These two policies are an improbable pair. The original purpose of charter schools was to encourage innovation, while test-based accountability has led to an excessive focus on teaching to the test. It is difficult to measure, or even define, innovation and, therefore, difficult to assess the extent to which charter schools have achieved their original goal. We do know, however, that charter school students are required to take the same tests as students in traditional public schools and that the reputation of charter schools depends on the test results. It would be realistic, therefore, to expect innovation in charter schools to be as constrained by test-based accountability as it is in traditional public schools.

IMPLICATIONS

Any federal policy has to consider the potential for adverse unintended consequences along with the potential for overall advantages. In the case of test-based accountability and charter schools, the adverse consequences are predictable and the advantages, on balance, highly questionable.

As we continue to institute policies that are unsupported by research evidence, we ignore a long history of research showing the overwhelming contribution of family and school poverty to the achievement gap. The United States is at risk of winning the race to be first in the industrialized world in percentage of children living in poverty. And, inevitably, as income inequality continues to rise in the United States, so too does the achievement gap based on family income (Reardon, 2011). Even apart from the research described earlier, it should have been obvious from the beginning that test-based accountability and school choice would not address the underlying problems of poverty contributing to low achievement.

The 2009 Program for International Student Assessment found that the achievement differences based on socioeconomic status within member countries in the Organization for Economic Cooperation and Development (OECD) far exceeded any achievement differences between countries. Only 11% of the variation in reading scores in the OECD comparisons can be attributed to differences between countries; the remainder of the variation is accounted for by differences within countries. A substantial portion of this within-country variation is related to socioeconomic differences among students and among schools. On average, almost 60% of the variation in reading performance within OECD countries can be explained by socioeconomic differences. Not surprisingly, given the child poverty rate, the percentage accounted for in the United States is almost 80% (OECD, 2010).

These problems are difficult, but not intractable. We have many years of experience and research, both in the United States and internationally, on which to build. We know that civil rights policies, combined with increased access to educational and professional opportunities, were major factors in increasing the socioeconomic status of generations of students from minority backgrounds, thereby decreasing the achievement gap based on race and ethnicity. We know that our policies on access to preschool and higher education, student financial aid, affirmative action, and school integration can facilitate gains in educational achievement. So too can broader social, employment, and fiscal policies. The differences these policies make in poverty rates and the achievement gap are apparent across various time periods. They are also apparent in the significant variation among countries in the achievement gap based on socioeconomic status. I do not claim these policies will “solve” the problem. And I know they are not politically popular. Far from it. But I believe these are policies that can make a difference in providing opportunities for the large numbers of children who are at the greatest risk.

Current federal policy has touched on some of these issues—for example, preschool education, student financial aid, and employment. The efforts, however, have been small, both in comparison to the need and in the context of broader social and fiscal policies that over the past decade have contributed to an increasing achievement gap based on family income. At the same time, our education rhetoric perpetuates the myth that we can fix the educational consequences of concentrations of poverty by holding teachers and principals accountable for their students’ test scores and by opening charter schools—policies that we know from research evidence will, at best, be ineffective and, more likely, lead to further increases in the gap.

References

Gill, B., Timpane, P. M., Ross, K. E., Brewer, D. J., & Booker, K. (2007). Rhetoric versus reality: What we know and what we need to know about vouchers and charter schools. Santa Monica, CA: RAND.

National Research Council. (2009). Letter report to the U.S. Department of Education on the Race to the Top fund. Washington, DC: National Academies Press.

National Research Council. (2011). Incentives and test-based accountability in education. Washington, DC: National Academies Press.

Organization for Economic Cooperation and Development. (2010). PISA 2009 results: Overcoming social background, equity in learning opportunities and outcomes (Vol. II). Paris, France: Author.

Reardon, S. F. (2011). The widening academic achievement gap between the rich and poor: New evidence and possible explanations. In R. Murnane & G. Duncan (Eds.), Whither opportunity? Rising inequality and the uncertain life chances of low-income children. New York, NY: Russell Sage Foundation Press.

U.S. Department of Education. (2009, November). Race to the Top program: Executive summary. Washington, DC: Author.

Cite This Article as: Teachers College Record, Date Published: June 19, 2012 http://www.tcrecord.org ID Number: 16798, Date Accessed: 11/9/2012 2:09:12 PM