Why it’s OK for this school choice program to underperform on standardized tests

By Lindsey Burke | The Daily Signal

K-12 education in the nation’s capital has struggled for decades. Yet since 2003, the D.C. Opportunity Scholarship Program — a voucher program open to low-income children in the District of Columbia — has been a bright spot for educational opportunity.

But a new report released by the Institute of Education Sciences at the U.S. Department of Education, released Tuesday, found that students who participate in the D.C. scholarship program had worse math scores after two years than a control group of students who applied for a scholarship but did not receive one.


The D.C. Opportunity Scholarship Program is producing two critical outcomes for participants: significantly higher graduation rates and safer students — and at a fraction of the money spent on D.C. Public Schools.

The scientifically rigorous evaluation found that scholarship recipients were doing worse in math after two years, but that the program had increased parent and student perceptions of school safety.

Study authors Mark Dynarski, Ning Rui, Ann Webber, and Babette Gutmann found that the D.C. Opportunity Scholarship Program had a statistically significant negative impact on scholarship users’ math achievement after two years of participation. Students with a scholarship had math scores that were 10 percentile points lower than students who applied for, but were not awarded, a scholarship.

Reading achievement scores were three percentile points lower for scholarship students, but the difference was not significant.

The D.C. scholarship program appears to be demonstrating in real time what the University of Arkansas’s Jay Greene suggested recently in a widely cited post:

Schools and programs that increase test scores quite often do not yield higher high school graduation or college attendance rates. Conversely, schools and programs that fail to produce greater gains in test scores sometimes produce impressive improvements in high school graduation and college attendance rates, college completion rates, and even higher employment and earnings.

Indeed, an experimental evaluation of the scholarship program found that participating students graduated at a rate 21 percentage points higher than their peers who applied for, but did not receive, a scholarship. That is, the impact of voucher use in the program resulted in a 91 percent graduation rate compared to 70 percent in the control group.

So, while the D.C. Opportunity Scholarship Program might not be having an immediate positive effect on the standardized test scores prized by researchers and policy bean counters, it is having a potentially more meaningful, long-term impact on participants’ academic attainment and future life outcomes.

Moreover, the most extensive study to date of the intersection of test scores and later life outcomes demonstrated a weak relationship between the two, finding that a school choice program’s effect on student test scores was a weak predictor of longer-term outcomes such as graduation and college attendance.

The study’s authors, Collin Hitt, Michael Q. McShane, and Patrick J. Wolf, found that “achievement impact estimates appear to be almost entirely uncorrelated with attainment impacts.” The authors urge humility on the part of policymakers when it comes to “what they believe that test scores tell them about the performance of schools of choice.”

It’s also worth noting that while researchers often look to test scores to determine school quality, parents do so to a far lesser extent. Parents prioritize school safety, while student performance on standardized tests is one of the least important factors parents cite.

It is notable, then, that the D.C. Opportunity Scholarship Program had a considerable impact on parent and student perceptions of school safety. The new evaluation found a 19.5 percentage point difference between parents of scholarship users and the control group in their perception of their school as being very safe — a statistically significant difference.

Fully 74.2 percent of parents of scholarship users rated their child’s school as “very safe,” compared with 54.7 percent of parents of students who applied for, but were not awarded, a scholarship.

Students themselves also felt safer in their new schools: 59.2 percent of students who used a D.C. scholarship to attend a private school of choice rated their schools as “very safe,” compared to 43.7 percent of students who applied for, but did not receive, a scholarship–a statistically significant difference.

In a city that has long struggled with school safety, it is no wonder that parents would prioritize finding a school in which they feel their children are safe. The D.C. Opportunity Scholarship Program has enabled them to do that.

The bottom line is this: The new evaluation shows the program did not move the needle on something parents care little about anyway (standardized test outcomes), but had a significant impact on something parents prioritize highly (the safety of the school to which they entrust their children for the day).

The D.C. Opportunity Scholarship Program is producing two critical outcomes for participants: significantly higher graduation rates and safer students. And the fact that this program does so for fractions on the dollar spent in D.C. Public Schools is just icing on the cake of school choice.

Image courtesy of SchoolChoiceWeek.com

One thought on “Why it’s OK for this school choice program to underperform on standardized tests

  1. There is a more comprehensive report on the recent Washington D.C. Opportunity Scholarship Program.

    There’s a lot of information at play if anyone is inclined to make comparisons and this report is from one assessment perspective, the NAEP. There are others.

    For one thing, this Washington D.C. report data covers only two years of participation.Keep in mind too that NAEP data is from testing fewer than 5% of the students in 4th & 8th grades and its tests are not mandatory. For example, while Vermont NAEP scores are higher than Washington D.C., D.C. scores are holding steady or improving over the last 20 years, Vermont’s are now declining.

    A more comprehensive assessment of the Washington D.C. education system is the The Partnership for Assessment of Readiness for College and Careers, or PARCC. https://wamu.org/story/17/08/17/d-c-public-school-students-show-modest-test-score-gains-english-math/

    Washington D.C. is in the top 20 most dangerous cities in the U.S. so it’s understandable that safety is a more important issue for parents.

    Graduation rates must be taken with a grain of salt. In Vermont, for example, while only half its students meet grade level standards and Vermont has a 90% graduation rate. D.C.’s graduation rates must be suspect too.

    Interestingly, Washington D.C. charter schools scored 8 points higher than non-charter schools in eighth-grade math, having achieved better results at significantly lower cost. And as education scholar Matthew Ladner noted, the charter schools are at a disadvantage, since transfer students often “experience a temporary academic setback.” Vermont has no charter schools.

    School Choice is not a guaranty of success. Some parents will make good choices, some not so good. Importantly, however, with School Choice we can learn from our failures. Without School Choice, we never know.

Comments are closed.