PISA: It's Poverty Not Stupid

by Mel Riddle,  National Association of Secondary School Principals

There are three kinds of lies; lies, damn lies, and statistics."--Mark Twain

"There are three kinds of lies; lies, damn lies, and statistics."--Mark Twain

The release of the 2009 PISA results this past week has created quite a stir and has provided ample fodder for public school bashers and doomsayers who further their own philosophical and profit-motivated agendas by painting all public schools as failing. For whatever reason, these so-called experts, many of whom have had little or no actual exposure to public schools, refuse to paint an accurate picture of the state of education.

Secretary of Education, Arne Duncan, should be providing the nation with a proper vision and focus for public education. He knows our challenges all too well. He confirmed that he gets it when he recently wrote me saying, "We must build a culture nationally where great educators ... choose to work with children and communities who need the most help." I believe his message is sincere and heartfelt and it is spot on. However, overstating a problem in order to increase the sense of urgency around school improvement is just as bad as understating the problem.

This week, Duncan had a golden opportunity to use the PISA results to provide focus for our education efforts and to point us in the right direction. Instead, he dug himself deeper into the pseudo-reformers' hole--more charter schools, more reliance on competition and free-market strategies, more testing, more use of test scores to evaluate teachers, more firing of principals and teachers, more closing of low-scoring schools--when he said, "the PISA scores released this past Tuesday were "a massive wake-up call," because the scores show American students holding relatively steady in the middle of the pack of the developed nations taking the international exam.

There is, however, someone who recognizes that the data is being misinterpreted. NEAToday published remarks from National Association of Secondary School Principals Executive Director, Dr. Gerald N. Tirozzi, that have taken "a closer look at how the U.S. reading scores on PISA compared with the rest of the world’s, overlaying it with the statistics on how many of the tested students are in the government’s free and reduced lunch program for students below the poverty line." Tirozzi pointed out, “Once again, we’re reminded that students in poverty require intensive supports to break past a condition that formal schooling alone cannot overcome.” Tirozzi demonstrates the correlation between socio-economic status and reading by presenting the PISA scores in terms of individual American schools and poverty.  While the overall PISA rankings ignore such differences in the tested schools, when groupings based on the rate of free and reduced lunch are created, a direct relationship is established.

Free and Reduced Meal Rate	PISA Score
Schools with < 10%	551
Schools with 10-24.9%	527
Schools with 25-49.9%	502
Schools with 49.9-74.9%	471
Schools with >75%	446
U.S. average	500
OECD average	493

With strong evidence that increased poverty results in lower PISA scores the next question to be asked is what are the poverty rates of the countries being tested? (Listed below are the countries that were tested by PISA along with available poverty rates. Some nations like Korea do not report poverty rates.)

Country	Poverty Rate	PISA Score
Denmark	2.4%	495
Finland	3.4%	536
Norway	3.6%	503
Belgium	6.7%	506
Switzerland	6.8%	501
Czech Republic	7.2%	478
France	7.3%	496
Netherlands	9.0%	508
Germany	10.9%	497
Australia	11.6%	515
Greece	12.4%	483
Hungary	13.1%	494
Austria	13.3%	471
Canada	13.6%	524
Japan	14.3%	520
Poland	14.5%	500
Portugal	15.6%	489
Ireland	15.7%	496
Italy	15.7%	486
United Kingdom	16.2%	494
New Zealand	16.3%	521
United States	21.7%	500

Leveling the playing field

A more accurate assessment of the performance of U.S. students would be obtained by comparing the scores of American schools with comparable poverty rates to those of other countries.

Schools in the United States with less than a 10% poverty rate had a PISA score of 551.  When compared to the ten countries with similar poverty numbers, that score ranked first. 

Country	Poverty Rate	PISA Score
United States	<10%	551
Finland	3.4%	536
Netherlands	9.0%	508
Belgium	6.7%	506
Norway	3.6%	503
Switzerland	6.8%	501
France	7.3%	496
Denmark	2.4%	495
Czech Republic	7.2%	478

In the next category (10-24.9%) the U.S. average of 527 placed first out of the ten comparable nations. 

Country	Poverty Rate	PISA Score
United States	10%-24.9%	527
Canada	13.6%	524
New Zealand	16.3%	521
Japan	14.3%	520
Australia	11.6%	515
Poland	14.5%	500
Germany	10.9%	497
Ireland	15.7%	496
Hungary	13.1%	494
United Kingdom	16.2%	494
Portugal	15.6%	489
Italy	15.7%	486
Greece	12.4%	483
Austria	13.3%	471

For the remaining U.S. schools, their poverty rates over 25% far exceed any other country tested.  However, when the U.S. average of 502 for poverty rates between 25-49.9% is compared with other countries it is still in the upper half of the scores.

Mathematically Speaking

The results of the latest PISA testing should raise serious concerns.  However, the overall ranking of 14th in reading is not the reason to be concerned. The problem is not as much with our educational system as it is with our high poverty rates. The real crisis is the level of poverty in too many of our schools and the relationship between poverty and student achievement. Our lowest achieving schools are the most under-resourced schools with the highest number of disadvantaged students. We cannot treat these schools in the same way that we would schools in more advantaged neighborhoods or we will continue to get the same results. The PISA results point out that the U.S. is not alone in facing the challenge of raising the performance of disadvantaged students.

U.S. % Poverty	Other Countries	PISA Score
U.S. (<10%)		551
	Korea	539
	Finland	536
U.S. (10-24.9%)		527
	Canada	524
	New Zealand	521
	Japan	520
	Australia	515
	Netherlands	508
	Belgium	506
	Norway	503
U.S. (25-49.9%)		502
	Estonia	501
	Switzerland	501
	Poland	500
	Iceland	500
U.S. (Average)		500
	Sweden	497
	Germany	497
	Ireland	496
	France	496
	Denmark	495
	United Kingdom	494
	Hungary	494
	Portugal	489
	Italy	486
	Slovenia	483
	Greece	483
	Spain	481
	Czech Republic	478
	Slovak Republic	477
	Israel	474
	Luxembourg	472
U.S. (50-74.9%)		471
	Austria	471
	Turkey	464
	Chile	449
U.S. (over 75%)		446
	Mexico	425

Additional observations from PISA results:

·      Shanghai, China topped the list with 556 but is not included in this analysis because Shanghai is a city not a country and because only 35% of Chinese students ever enter high school and because "when you spend all your time preparing for tests, and when students are selected based on their test-taking abilities, you get outstanding test scores."

·      Of all the nations participating in the PISA assessment, the U.S. has, by far, the largest number of students living in poverty--21.7%. The next closest nations in terms of poverty levels are the United Kingdom and New Zealand have poverty rates that are 75% of ours.

·      U.S. students in schools with 10% or less poverty are number one country in the world.

·      U.S. students in schools with 10-24.9% poverty are third behind Korea, and Finland.

·      U.S. students in schools with 25-50% poverty are tenth in the world.

·      U.S. students in schools with greater than 50% poverty are near the bottom.

·      There were other surprises. Germany with less than half our poverty, scored below the U.S. as did France with less than a third our poverty and Sweden with a low 3.6% poverty rate.

·      Having recently listened to Sir Michael Barber talk about the amazing progress of the reforms in the United Kingdom, I was absolutely shocked to see that the UK, with 25% less poverty, scored below the U.S. average.

The Real Meaning of PISA: It's Poverty Not Stupid

If the so-called experts would have honestly and responsibly reported the PISA results, we might now be on the road to responsible school improvement instead continuing down the road of "reform de jour."

President Bill Clinton is famous for his campaign slogan, "It's the economy stupid!" When it comes to student achievement and school improvement, it's poverty not stupid! Researchers report that perhaps the only true linear relationship in the social sciences is the relationship between poverty and student performance. While there is no relationship between poverty and ability, the relationship between poverty and achievement is almost foolproof. To deny that poverty is a factor to be overcome as opposed to an excuse is to deny the reality that all educators, human services workers, law enforcement officers, medical professionals and religious clergy know and have known for years.

PISA reports average scores. The problem is that the U.S. is not average. While the U.S. is the top country in global competitiveness, we also have the highest percentage of students living in poverty and, regretfully, poverty impacts test scores.

To Secretary Duncan, poverty is not an issue that educators must address. At least he won't admit it in public. Apparently, he wants to take away all the excuses from teachers and principals. When I met with Duncan, I asked him if he had read the book or seen the movie, Blind Side. He indicated that he had and that he had enjoyed it very much. I reminded him of the pride and sense of accomplishment felt by the teachers in the private school attended by Michael Oher. In their minds, they had performed a miracle. I pointed out that, in high-poverty schools, a Michael Oher is the average student. In schools like ours we have hundreds of students like Michael Oher who depend on our school for everything including food, clothing, and emotional support.

The Bottom Line

School improvement is not an event. It is an ongoing process that has no end. As a principal, parents and community members would repeatedly ask me, "When can we stop our comprehensive school-wide literacy initiative? I would answer, "We will stop emphasizing reading, writing, thinking and speaking when our parents repeatedly complain that their children are reading too fast with comprehension that is too high and when our students' writing skills are so superior that they are regularly winning Pulitzers and other literary awards." Smiles would erupt throughout the audience. They got it. They understood that literacy skills can always be improved and so can our schools.

There are three compelling reasons why we must improve our schools:

1. We have a moral and ethical obligation to provide every student with the best education, the kind of education that we would want for our own children.
2. In a knowledge economy, the country with the best-educated populace will have the highest standard of living.
3. Every dropout as well as every graduate who is not prepared for at least some post-secondary education and training is and will continue to be an economic and social burden on their local community and on this nation for their entire life.

The challenge of ensuring that each and every student is a life-long learner prepared to contribute in a global community is daunting enough. We don't need more hyperbole, particularly from those education insiders who should know better. For those of us who are deeply committed to improving the performance of every student, this rhetoric is counterproductive because it seriously erodes our ability to hire teachers, obtain resources, and gain the confidence and support of our communities.

We count on our leaders to provide focus and direction. Sadly, our education leaders don't trust us enough to tell us the truth. The problem is that we will never solve a problem that our leaders refuse to admit even exists. The comparison of PISA scores by poverty clearly identifies our strengths and challenges as a nation. Our schools with less than 50% poverty) are some of the best in the world. Our extremely high-poverty schools, with over 50% poverty, are among the poorest performing internationally.

Instead of labeling all schools as failing, we must find a way to raise the performance of our students in under-resourced schools. Instead of looking to low-poverty countries like Finland for direction, we should be looking to take what we already know about educating students in high-performing, high-poverty schools like our Breakthrough Schools and scaling up their successes across the nation. We continually look for gold in other countries when, all along, we are sitting on Acres of Diamonds.

Truthfully, you and I know all too well that Secretary Duncan, who led schools in Chicago, is aware of the relationship between poverty and student achievement, but he doesn't trust us enough to tell us the truth. He is afraid that we will use poverty as an excuse and that we will forget about our disadvantaged students. Ironically, by not acknowledging poverty as a challenge to be overcome, Duncan is forgetting about our disadvantaged students. Duncan needs to deliver the message that all our students deserve not only access to an education, but access to an excellent education. He needs to repeatedly remind us that, when it comes to school improvement, it's poverty not stupid.

Posted by Mel Riddile on December 15, 2010 12:13 PM 

“Well, Duh!” -- Ten Obvious Truths That We Shouldn’t Be Ignoring

AMERICAN SCHOOL BOARD JOURNAL April 2011 By Alfie Kohn The field of education bubbles over with controversies. It’s not unusual for intelligent people of good will to disagree passionately about what should happen in schools. But there are certain precepts that aren’t debatable, that just about anyone would have to acknowledge are true. While many such statements are banal, some are worth noticing because in our school practices and policies we tend to ignore the implications that follow from them. It’s both intellectually interesting and practically important to explore such contradictions: If we all agree that a given principle is true, then why in the world do our schools still function as if it weren’t? Here are 10 examples. 1. Much of the material students are required to memorize is soon forgotten The truth of this statement will be conceded (either willingly or reluctantly) by just about everyone who has spent time in school -- in other words, all of us. A few months, or sometimes even just a few days, after having committed a list of facts, dates, or definitions to memory, we couldn’t recall most of them if our lives depended on it. Everyone knows this, yet a substantial part of schooling – particularly in the most traditional schools – continues to consist of stuffing facts into students’ short-term memories. The more closely we inspect this model of teaching and testing, the more problematic it reveals itself to be. First, there’s the question of what students are made to learn, which often is more oriented to factual material than to a deep understanding of ideas. (See item 2, below.) Second, there’s the question of howstudents are taught, with a focus on passive absorption: listening to lectures, reading summaries in textbooks, and rehearsing material immediately before being required to cough it back up. Third, there’s the question of why a student has learned something: Knowledge is less likely to be retained if it has been acquired so that one will perform well on a test, as opposed to learning in the context of pursuing projects and solving problems that are personally meaningful. Even without these layers of deficiencies with the status quo, and even if we grant that remembering some things can be useful, the fundamental question echoes like a shout down an endless school corridor: Why are kids still being forced to memorize so much stuff that we know they won’t remember? Corollary 1A: Since this appears to be true for adults, too, why do most professional development events for teachers resemble the least impressive classrooms, with experts disgorging facts about how to educate? 2. Just knowing a lot of facts doesn’t mean you’re smart Even students who do manage to remember some of the material they were taught are not necessarily able to make sense of those bits of knowledge, to understand connections among them, or to apply them in inventive and persuasive ways to real-life problems. In fact, the cognitive scientist Lauren Resnick goes even further: It’s not just that knowing (or having been taught) facts doesn’t in itself make you smart. A mostly fact-oriented education may actually interfere with your becoming smart. “Thinking skills tend to be driven out of the curriculum by ever-growing demands for teaching larger and larger bodies of knowledge,” she writes. Yet schools continue to treat students as empty glasses into which information can be poured -- and public officials continue to judge schools on the basis of how efficiently and determinedly they pour. 3. Students are more likely to learn what they find interesting There’s no shortage of evidence for this claim if you really need it. One of many examples: A group of researchers found that children’s level of interest in a passage they were reading was 30 times more useful than its difficulty level for predicting how much of it they would later remember. But this should be obvious, if only because of what we know about ourselves. It’s the tasks that intrigue us, that tap our curiosity and connect to the things we care about, that we tend to keep doing -- and get better at doing. So, too, for kids. Conversely, students are less likely to benefit from doing what they hate. Psychology has come a long way from the days when theorists tried to reduce everything to simple stimulus-response pairings. We know now that people aren’t machines, such that an input (listening to a lecture, reading a textbook, filling out a worksheet) will reliably yield an output (learning). What matters is how people experience what they do, what meaning they ascribe to it, what their attitudes and goals are. Thus, if students find an academic task stressful or boring, they’re far less likely to understand, or even remember, the content. And if they’re uninterested in a whole category of academic tasks -- say, those they’re assigned to do when they get home after having just spent a whole day at school -- then they aren’t likely to benefit much from doing them. No wonder research finds little, if any, advantage to assigning homework, particularly in elementary or middle school. 4. Students are less interested in whatever they’re forced to do and more enthusiastic when they have some say Once again, studies confirm what we already know from experience. The nearly universal negative reaction to compulsion, like the positive response to choice, is a function of our psychological makeup. Now combine this point with the preceding one: If choice is related to interest, and interest is related to achievement, then it’s not much of a stretch to suggest that the learning environments in which kids get to make decisions about what they’re doing are likely to be the most effective, all else being equal. Yet such learning environments continue to be vastly outnumbered by those where kids spend most of their time just following directions. 5. Just because doing x raises standardized test scores doesn’t mean x should be done At the very least, we would need evidence that the test in question is a source of useful information about whether our teaching and learning goals are being met. Many educators have argued that the tests being used in our schools are unsatisfactory for several reasons. First, there are numerous limitations with specific tests. Second, most tests share certain problematic features, such as being timed (which places more of a premium on speed than on thoughtfulness), norm-referenced (which means the tests are designed to tell us who’s beating whom, not how well students have learned or teachers have taught), and consisting largely of multiple-choice questions (which don’t permit students to generate or even explain their answers). The third reason is the problems inherent to all tests that are standardized and created by people far away from the classroom -- as opposed to assessing the actual learning taking place there on an on-going basis. This is not the place to explain in detail why standardized tests measure what matters least. Here, I want only to make the simpler -- and, once again, I think, indisputable -- point that anyone who regards high or rising test scores as good news has an obligation to show that the tests themselves are good. If a test result can’t be convincingly shown to be both valid and meaningful, then whatever we did to achieve that result -- say, a new curriculum or instructional strategy -- may well have no merit whatsoever. It may even prove to be destructive when assessed by better criteria. Indeed, a school or district might be getting worse even as its test scores rise. So how is it that articles in newspapers and education journals, as well as pronouncements by public officials and think tanks, seem to accept on faith that better scores on any test necessarily constitute good news, and that whatever produced those scores can be described as “effective”? Parents should be encouraged to ask, “How much time was sacrificed from real learning just so our kids could get better at taking the [name of test]?” 6. Students are more likely to succeed in a place where they feel known and cared about I realize there are people whose impulse is to sneer when talk turns to how kids feel, and who dismiss as “soft” or “faddish” anything other than old-fashioned instruction of academic skills. But even these hard-liners, when pressed, are unable to deny the relationship between feeling and thinking, between a child’s comfort level and his or her capacity to learn. Here, too, there are loads of supporting data. As one group of researchers put it, “In order to promote students’ academic performance in the classroom, educators should also promote their social and emotional adjustment.” And yet, broadly speaking, we don’t. Teachers and schools are evaluated almost exclusively on academic achievement measures (which, to make matters worse, mostly consist of standardized test scores). If we took seriously the need for kids to feel known and cared about, our discussions about the distinguishing features of a “good school” would sound very different. Likewise, our view of discipline and classroom management would be turned inside-out, seeing as how the primary goals of most such strategies are obedience and order, often with the result that kids feel less cared about -- or even bullied -- by adults. 7. We want children to develop in many ways, not just academically Even mainstream education groups have embraced the idea of teaching the “whole child.” It’s a safe position, really, because just about every parent or educator will tell you that we should be supporting children’s physical, emotional, social, moral, and artistic growth as well as their intellectual growth. Moreover, it’s obvious to most people that the schools can and should play a key role in promoting many different forms of development. If we acknowledge that academics is just one facet of a good education, why do so few conversations about improving our schools deal with -- and why are so few resources devoted to -- non-academic issues? And why do we assign children still more academic tasks after the school day is over, even when those tasks cut into the time children have to pursue interests that will help them develop in other ways? Corollary 7a: Students “learn best when they are happy,” as educator Nel Noddings reminded us, but that doesn’t mean they’re especially likely to be happy (or psychologically healthy) just because they’re academically successful. And millions aren’t. Imagine how high schools would have to be changed if we were to take this realization seriously. 8. Just because a lesson (or book, or class, or test) is harder doesn't mean it's better First, if it’s pointless to give students things to do that are too easy, it’s also counterproductive to give them things that they experience as too hard. Second, and more important, this criterion overlooks a variety of considerations other than difficulty level by which educational quality might be evaluated. We know this, yet we continue to worship at the altar of “rigor.” I’ve seen lessons that aren’t unduly challenging yet are deeply engaging and intellectually valuable. Conversely, I’ve seen courses -- and whole schools -- that are indisputably rigorous . . . and appallingly bad. 9. Kids aren’t just short adults Over the past hundred years, developmental psychologists have labored to describe what makes children distinctive and what they can understand at certain ages. There are limits, after all, to what even a precocious younger child can grasp (e.g., the way metaphors function, the significance of making a promise) or do (e.g., keep still for an extended period). Likewise, there are certain things children require for optimal development, including opportunities to play and explore, alone and with others. Research fills in -- and keeps fine-tuning -- the details, but the fundamental implication isn’t hard to grasp: How we educate kids should follow from what defines them as kids. Somehow, though, developmentally inappropriate education has become the norm, as kindergarten (literally, the “children’s garden”) now tends to resemble a first- or second-grade classroom -- in fact, a bad first- or second-grade classroom, where discovery, creativity, and social interaction are replaced by a repetitive regimen focused on narrowly defined academic skills. More generally, premature exposure to sit-still-and-listen instruction, homework, grades, tests, and competition -- practices that are clearly a bad match for younger children and of questionable value at any age -- is rationalized by invoking a notion I’ve called BGUTI: Better Get Used To It. The logic here is that we have to prepare you for the bad things that are going to be done to you later . . . by doing them to you now. When articulated explicitly, that principle sounds exactly as ridiculous as it is. Nevertheless, it’s the engine that continues to drive an awful lot of nonsense. The obvious premise that we should respect what makes children children can be amended to include a related principle that is less obvious to some people: Learning something earlier isn’t necessarily better. Deborah Meier, whose experience as a celebrated educator ranges from kindergarten to high school, put it bluntly: "The earlier [that schools try] to inculcate so-called 'academic' skills, the deeper the damage and the more permanent the 'achievement' gap." That is exactly what a passel of ambitious research projects has found: A traditional skills-based approach to teaching young children -- particularly those from low-income families -- not only offers no lasting benefits but appears to be harmful. Corollary 9A: Kids aren’t just future adults. They are that, of course, but they aren’t only that, because children’s needs and perspectives are worth attending to in their own right. We violate this precept -- and do a disservice to children -- whenever wetalk about schooling in economic terms, treating students mostly as future employees. 10. Substance matters more than labels A skunk cabbage by any other name would smell just as putrid. But in education, as in other domains, we’re often seduced by appealing names when we should be demanding to know exactly what lies behind them. Most of us, for example, favor a sense of community, prefer that a job be done by professionals, and want to promote learning. So should we sign on to the work being done in the name of “Professional Learning Communities”? Not if it turns out that PLCs have less to do with helping children to think deeply about questions that matter than with boosting standardized test scores. The same caution is appropriate when it comes to “Positive Behavior Support,” a jaunty moniker for a program of crude Skinnerian manipulation in which students are essentially bribed to do whatever they’re told. More broadly, even the label “school reform” doesn’t necessarily signify improvement; these days, it’s more likely to mean “something that skillful and caring teachers wouldn’t be inclined to do unless coerced,” as educational psychologist Bruce Marlowe put it. In fact, the corporate-style version of “school reform” that’s uncritically endorsed these days by politicians, journalists, and billionaires consists of a series of debatable tactics -- many of them amounting to bribes and threats to force educators to jack up test scores. Just as worrisome, though, is that these reformers often overlook, or simply violate, a number of propositions that aren’t debatable, including many of those listed here. _________________________________________ This essay is an abridged version of the introduction to Feel-Bad Education…And Other Contrarian Essays on Children and Schooling (Beacon Press, 2011)

Copyright © 2011 by Alfie Kohn. This article may be downloaded, reproduced, and distributed without permission as long as each copy includes this notice along with citation information (i.e., name of the periodical in which it originally appeared, date of publication, and author's name). Permission must be obtained in order to reprint this article in a published work or in order to offer it for sale in any form. We can be reached through the Contact Us page.