When it comes to judging the academic preparedness of Vermont’s student population, especially those moving up to the next grade level or planning to attend college, how effective are the results of the latest multi-grade test instrument of the Smarter Balanced Assessment Consortium?
SBAC testing results are now recognized by over 200 colleges and universities around the U.S. In fact, SBAC officials describe their Los Angeles-based firm’s assessment program as “a valid, fair, and reliable approach to student assessment that provides educators, students and parents meaningful results with actionable data to help students succeed.”
Based on the 2017 results, SBAC reported that the top performing Vermont schools are, in order, as follows: Charlotte Central School (Charlotte), Fayston Elementary School (Fayston), White River School (Hartford), Marion Cross School (Norwich), Monkton Central School (Monkton), Burke Town School (Burke), Newton School (Strafford), Barstow Memorial School (Chittenden), and Stowe High School (Stowe).
But not all Vermont’s education experts find standardized test results (or school rankings) a useful barometer when it comes to recording the rise and fall of a student’s academic performance.
William J. Mathis, managing director of the National Education Policy Center and vice chair of the Vermont Board of Education, has written that standard test scores have grown more cryptic over the decades, akin to the impenetrable black alien monolith of filmmaker Stanley Kubrick’s iconic 1968 sci-fi movie “2001: A Space Odyssey”.
On his NEPC blog, Mathis writes:
Our standardized test scores float untethered in space, free of the very things they are supposed to measure; yet having great power. They claim to measure ‘college and career readiness’. Yet, it takes no particular insight to know that being ready for the forestry program at the community college is not the same as astrophysics at MIT. Likewise, ‘career ready’ means many different things depending upon whether you are a health care provider, a convenience store clerk, or a road foreman. The fundamental flaw is pretending that we can measure an educated person with one narrow set of tests. There is no one universal knowledge base for all colleges and careers. This mistake is fatal to the test-based reform theory.
In an interview with True North Reports, Mathis said the SBAC test isn’t good when it comes to comparing student performance between the New England states and elsewhere.
“The only reliable and comparable tests for comparing states is the National Assessment of Education Progress (NAEP),” he said.
“They don’t rank-order states, but others take the scores and rank them. The comparable scores over the long term are the biennial fourth and eighth grade reading and math scores. Vermont generally falls in the top five depending on the year. Sometimes, we are first and one test dropped to about eighth for one year. A few years ago, some researchers at American Institutes for Research (AIR, a reputable group) equated the NAEP scores to the international tests. Thus, you could see how Vermont would score, compared to the world, if it was an independent country. We tied for eighth.”
Vermont Secretary of Education Rebecca Holcombe, when she reported the 2017 SBAC test results last September, also downplayed the usefulness of the tests.
“Tests don’t measure everything that matters to a happy and successful life, including our ability to participate in democratic life, but there is no path to prosperity for students who don’t master reading, writing and mathematics,” she said.
“We were disappointed to see … score declines. The achievement gaps between our vulnerable youth and students with greater privilege remain, and in some cases were narrowed, but this was largely a result of score declines for more privileged groups.
Critics of current standardized testing and how Vermont reported its 2017 SBAC results cite murky terminology and difficult state-to-state comparisons, especially when it comes to explaining the data to parents and other taxpayers.
John McClaughry, vice president of the Ethan Allen Institute, a free-market think tank, and James Callahan of Callahan Associates, a Middlebury-based education consulting and mathematics tutoring firm, claim that the Vermont Agency of Education is all too willing to say it’s too difficult to compare student proficiency levels of one state with another.
McClaughry said that while the Agency of Education goes to great pains to emphasize that it’s not possible to compare the student proficiency levels of one state with another, “don’t leap to the unwarranted conclusion that public education is getting better results in state A over state B.”
McClaughry said that the average percentage of students achieving proficiency in Vermont this year was 48.4 percent, down from 50.92 percent last year.
Alongside McClaughry’s analysis, Callahan has been railing against state testing results for years. Callahan was the principal of Middlebury’s Mary Hogan Elementary School and a past member of the Middlebury School Board.
“When it comes to educational testing, words can be very tricky,” Callahan said. “You find terms such as ‘proficient’ and ‘proficient with distinction.’ This started under No Child Left Behind.”
According to Callahan, the old test had cut off achievement scores with the lowest being termed ‘significantly below proficient.’ Despite the new test prepared by Smarter Balanced, Callahan is still troubled by the scoring criteria which declares a student either proficient or otherwise.
“So if a student fell into the ‘significantly below proficient’ level, it meant the he/she passed only 20 percent of the test material,” he said. “This suggests to me that the student may have guessed at most of the answers.”
Callahan noted that the test score level of ‘proficient’ sounds good enough to parents, but it’s deceptive: such a test score meant the student passed only 40 percent of the test.
“Then we have the ‘proficient with distinction’ test score. What if all that a ‘proficient with distinction’ commercial airline pilot had to do was land at the correct airport 75 percent of the time?” Callahan asks. “That’s exactly how Vermont grades student test takers score: A highly proficient with distinction score means you got 75 percent of the test answers right. And this doesn’t even tell us what the student did to get to get to the answers.”
Today, the Common Core-linked achievement test levels are defined by what are called achievement level descriptors, the specifications for what knowledge and skills students display at levels 1 through 4. But because each Smarter Balanced test member state refers to them in different ways — such as “novice, developing, proficient and advanced” — it’s hard to make easy comparisons among the states.
According to the Smarter Balanced website, students performing at levels 3 and 4 “are considered on track to demonstrating the knowledge and skills necessary for college and career readiness. These achievement level descriptors were written by teachers and college faculty.”
Regarding McClaughry’s interpretation of the 2017 test data, half of Vermont public school students finishing their respective school years are not ready to advance to the next grade. “That’s measured by Vermont’s own proficiency standards, which may be too high or too low — who’s to say?” he said.
He added that when it comes to English and math, the number of Vermont students who scored ‘proficient’ or above came in lower than in 2016.
According to McClaughry, achievement testing in Vermont is much like the 1993 movie “Groundhog Day,” where the same actions are repeated day after day with little or no change.
“We have gone through half a dozen assessment regimes/fads in my 50 years,” he said. “Not long after our kids don’t do so well, the current assessment is quietly shelved and an exciting new model is rolled out. For a billion and a half dollars every year, it seems to me that we ought to be getting better than 50 percent proficiency.”
Lou Varricchio is a freelance reporter for True North Reports. Send him news tips at firstname.lastname@example.org.