Critics: Vermont student test scores involve tricky terms, a fundamental flaw

When it comes to judging the academic preparedness of Vermont’s student population, especially those moving up to the next grade level or planning to attend college, how effective are the results of the latest multi-grade test instrument of the Smarter Balanced Assessment Consortium?

Lou Varricchio/TNR

Vermont classroom math consultant and former school principal James Callahan: “What if all that a ‘proficient with distinction’ commercial airline pilot had to do was land at the correct airport 75 percent of the time?”

SBAC testing results are now recognized by over 200 colleges and universities around the U.S. In fact, SBAC officials describe their Los Angeles-based firm’s assessment program as “a valid, fair, and reliable approach to student assessment that provides educators, students and parents meaningful results with actionable data to help students succeed.”

Based on the 2017 results, SBAC reported that the top performing Vermont schools are, in order, as follows: Charlotte Central School (Charlotte), Fayston Elementary School (Fayston), White River School (Hartford), Marion Cross School (Norwich), Monkton Central School (Monkton), Burke Town School (Burke),  Newton School (Strafford), Barstow Memorial School (Chittenden), and Stowe High School (Stowe).

But not all Vermont’s education experts find standardized test results (or school rankings) a useful barometer when it comes to recording the rise and fall of a student’s academic performance.

William J. Mathis, managing director of the National Education Policy Center and vice chair of the Vermont Board of Education, has written that standard test scores have grown more cryptic over the decades, akin to the impenetrable black alien monolith of filmmaker Stanley Kubrick’s iconic 1968 sci-fi movie “2001: A Space Odyssey”.

On his NEPC blog, Mathis writes:

Our standardized test scores float untethered in space, free of the very things they are supposed to measure; yet having great power. They claim to measure ‘college and career readiness’. Yet, it takes no particular insight to know that being ready for the forestry program at the community college is not the same as astrophysics at MIT. Likewise, ‘career ready’ means many different things depending upon whether you are a health care provider, a convenience store clerk, or a road foreman. The fundamental flaw is pretending that we can measure an educated person with one narrow set of tests. There is no one universal knowledge base for all colleges and careers. This mistake is fatal to the test-based reform theory.

In an interview with True North Reports, Mathis said the SBAC test isn’t good when it comes to comparing student performance between the New England states and elsewhere.

“The only reliable and comparable tests for comparing states is the National Assessment of Education Progress (NAEP),” he said.

“They don’t rank-order states, but others take the scores and rank them. The comparable scores over the long term are the biennial fourth and eighth grade reading and math scores. Vermont generally falls in the top five depending on the year. Sometimes, we are first and one test dropped to about eighth for one year. A few years ago, some researchers at American Institutes for Research (AIR, a reputable group) equated the NAEP scores to the international tests. Thus, you could see how Vermont would score, compared to the world, if it was an independent country. We tied for eighth.”

Vermont Secretary of Education Rebecca Holcombe, when she reported the 2017 SBAC test results last September, also downplayed the usefulness of the tests.

“Tests don’t measure everything that matters to a happy and successful life, including our ability to participate in democratic life, but there is no path to prosperity for students who don’t master reading, writing and mathematics,” she said.

“We were disappointed to see … score declines. The achievement gaps between our vulnerable youth and students with greater privilege remain, and in some cases were narrowed, but this was largely a result of score declines for more privileged groups.

Critics of current standardized testing and how Vermont reported its 2017 SBAC results cite murky terminology and difficult state-to-state comparisons, especially when it comes to explaining the data to parents and other taxpayers.

Bruce Parker/TNR

John McClaughry is vice president of the Ethan Allen Institute

John McClaughry, vice president of the Ethan Allen Institute, a free-market think tank, and James Callahan of Callahan Associates, a Middlebury-based education consulting and mathematics tutoring firm, claim that the Vermont Agency of Education is all too willing to say it’s too difficult to compare student proficiency levels of one state with another.

McClaughry said that while the Agency of Education goes to great pains to emphasize that it’s not possible to compare the student proficiency levels of one state with another, “don’t leap to the unwarranted conclusion that public education is getting better results in state A over state B.”

McClaughry said that the average percentage of students achieving proficiency in Vermont this year was 48.4 percent, down from 50.92 percent last year.

Alongside McClaughry’s analysis, Callahan has been railing against state testing results for years. Callahan was the principal of Middlebury’s Mary Hogan Elementary School and a past member of the Middlebury School Board.

“When it comes to educational testing, words can be very tricky,” Callahan said. “You find terms such as ‘proficient’ and ‘proficient with distinction.’ This started under No Child Left Behind.”

According to Callahan, the old test had cut off achievement scores with the lowest being termed ‘significantly below proficient.’ Despite the new test prepared by Smarter Balanced, Callahan is still troubled by the scoring criteria which declares a student either proficient or otherwise.

“So if a student fell into the ‘significantly below proficient’ level, it meant the he/she passed only 20 percent of the test material,” he said. “This suggests to me that the student may have guessed at most of the answers.”

Callahan noted that the test score level of ‘proficient’ sounds good enough to parents, but it’s deceptive: such a test score meant the student passed only 40 percent of the test.

“Then we have the ‘proficient with distinction’ test score. What if all that a ‘proficient with distinction’ commercial airline pilot had to do was land at the correct airport 75 percent of the time?” Callahan asks. “That’s exactly how Vermont grades student test takers score: A highly proficient with distinction score means you got 75 percent of the test answers right. And this doesn’t even tell us what the student did to get to get to the answers.”

Today, the Common Core-linked achievement test levels are defined by what are called achievement level descriptors, the specifications for what knowledge and skills students display at levels 1 through 4. But because each Smarter Balanced test member state refers to them in different ways — such as “novice, developing, proficient and advanced” — it’s hard to make easy comparisons among the states.

According to the Smarter Balanced website, students performing at levels 3 and 4 “are considered on track to demonstrating the knowledge and skills necessary for college and career readiness. These achievement level descriptors were written by teachers and college faculty.”

Regarding McClaughry’s interpretation of the 2017 test data, half of Vermont public school students finishing their respective school years are not ready to advance to the next grade. “That’s measured by Vermont’s own proficiency standards, which may be too high or too low — who’s to say?” he said.

He added that when it comes to English and math, the number of Vermont students who scored ‘proficient’ or above came in lower than in 2016.

According to McClaughry, achievement testing in Vermont is much like the 1993 movie “Groundhog Day,” where the same actions are repeated day after day with little or no change.

“We have gone through half a dozen assessment regimes/fads in my 50 years,” he said. “Not long after our kids don’t do so well, the current assessment is quietly shelved and an exciting new model is rolled out.  For a billion and a half dollars every year, it seems to me that we ought to be getting better than 50 percent proficiency.”

Lou Varricchio is a freelance reporter for True North Reports. Send him news tips at

Images courtesy of Lou Varricchio/TNR and Bruce Parker/TNR

12 thoughts on “Critics: Vermont student test scores involve tricky terms, a fundamental flaw

  1. The big public education monopoly is very “proficient” they are proficient at getting people elected and appointed who care more about union jobs then our children.

  2. Mathis chose an interesting comparison, between forestry and astrophysics, and I agree there is a hard distinction. Astrophysics has a lot of higher math, with clear “right and wrong” answers. Once you calculate gravity and trajectory, you either hit a lunar orbit, or you miss and go off into space. Forestry is far more complex, with 50 different species in a living system and variables of soils, weather, pathogens and more. Forestry involves a wide range of thinking skills applied to a wide range of knowledge. I am a forester, and have a son that decided not to go into forestry, but chose astrophysics.

  3. With all due respect to Lou Verrocchio and True North, what’s the point behind this article?

    “When it comes to judging the academic preparedness of Vermont’s student population, especially those moving up to the next grade level or planning to attend college, how effective are the results of the latest multi-grade test instrument of the Smarter Balanced Assessment Consortium?”

    Answer: Judging test results that are used by at least 21 states is more effective than judging a ouija board or the opinion of an appointed education oracle like Bill Mathis.

    It’s no surprise that Mr. Mathis criticizes certain standardized testing; especially when the test data indicates a continuing systemic failure in Vermont’s student academic performance. Mr. Mathis is, after all, a public school monopoly lobbyist masquerading as a ‘test expert’.

    Why, for example, does Mr. Mathis prefer the National Assessment of Education Progress (NAEP) testing to Vermont’s Smarter Balance assessments? For one thing, the NAEP tests aren’t mandatory. While schools may claim to make an effort to test a representative cross section of students, there aren’t any guarantees they do so. In the first place, fewer than 5% of Vermont’s students take the NAEP tests each year. Meanwhile, all Vermont students are required to take the SBAC tests.

    And why doesn’t Mr. Verrocchio or Mr. Mathis or any of the other education soothsayers criticizing student academic assessments provide a specific example of an SBAC test question shortfall? What’s wrong with this question?

    “Based on the graph, estimate to the nearest $50 the average rate of change in value of the antique for the following time intervals:”


    ” Write an equation to represent the relationship between the cost, y, in dollars, and the number of pages, x, for each book size. Be sure to place each equation next to the appropriate book size. Assume that x is at least 20 pages.”


    “In the passage, Dr. Mortimer speaks several times of a legend surrounding the Baskerville family. Explain how the reader can tell that the legend suggests that a frightening hound haunts the family. Support your answer using details from the text.”

    Saul Alinsky best describes the political hypocrisy in these test assessment criticisms. “Whenever possible go outside the expertise of the enemy.” Unfortunately, most Vermonters are long past paying attention to education gobbledigook because they can’t do anything about it anyway. They can’t choose the school that best meets the needs of their children so why should they bother caring about tests? Parents don’t care and their children don’t care.

    Consider the last time Mr. Mathis addressed Vermont’s declining academic performance when he said there’s “… some systematic reason here, but what it is, I don’t know. It is a mystery to me.”

    Oh…. well okay then.

  4. We are ALWAYS sold a bill of goods—and the results are always the same lame, pathetic garbage of woeful percentages of proficiency. Since when, in our educational history, did 40 or 50% proficiency in anything translate to anything but FAILURE?

    All these test results have been manipulated for years and no educator with even tbe remotest degree of credibility can justify them—it is truly sad.

    • Right on! And these days our children are less educated than in the “archaic” school systems of the 1950s and 1960s because progressives have got hold of our education system and the unions are involved only to make themselves richer. When I was in school we had classes of 35 to 40 students. There was discipline in the classroom and if we made the effort, we all got a good education. The idea of smaller classes came from the unions so that they could hire more teachers (note I don’t say better) and get more money in their coffers. In addition, even good teachers are stuck with teaching to these abominable tests. My son could take these tests very well, but he wasn’t learning a thing. He came home once with academic excellence in science on his standardized test report. When we got his report card he was not doing well at all in science. Why? Because he didn’t do his homework like he should and wouldn’t study for regular tests. But he could take those standardized tests like a whiz. This told me back then (1980s and 1990s) those tests were absolutely useless to a child’s educational experience.

      • You’re absolutely right Barbara, the big public education monopoly does in fact teach to the test. They “teach” kids how to memorize information so they can do well on these tests. Vermont union jobs, more important than Vermont children.

  5. Willie Mathis is always very predictable. It appears to me his conclusion is that union jobs are more important than children. He creatively blames independent schools for the lower test scores in Vermonts big public education monopoly. It’s little surprise that Willie Mathis is a mouthpiece for union jobs. Willie Mathis serves as second in command on the Vermont education board (but undoubtedly controls the board) He crafts rules and regulations that cause children to attend union staffed schools, he does this AT THE VERY SAME TIME HE SERVES AS THE COLORADO BASED, TEACHERS UNION FUNDED, NATIONAL EDUCATION POLICY CENTER. That’s right, Willie Mathis is making regulations in Vermont that favor the teachers union while he is being paid by an out of state teachers union funded group. Vermont public education officials have become known for another sad tactic, they aggressively over diagnose children as “disabled,” once kids are diagnosed they don’t count against Vermonts poor test results, the public education monopoly also receives federal money for these “disabled” children, this federal money funds more big union jobs. I wish the success and well being of Vermont children and families was the biggest motivation for these people but they have consistently shown me otherwise.

  6. Drives me fuzzy, the insistence that even though little Suzy/Johnny got the correct answer on the test ,somehow ,someone needs to know how they came to that answer. The answer of a sheep was correctly made. Suzy saw them on her sheets and pillowcases. Johnny recalls tending them on the farm before coming to school. How they came to that understanding of a picture of a sheep really matters? Not in my day. Master the basics and people skills, you’ll be fine.

  7. Simple, do what I did. Move to a state that doesn’t teach common core. Your kids will get a better education that costs you SIGNIFICANTLY less. It’s a win win, and guess what:

    No one ever mentions “CARBON TAX”. Such a relief to businesses that they can continue to INVEST in the economy without having to worry they are going to get fleeced the next year.

    • What state is that? I live in Texas and some of our school districts (not mine fortunately) use Common Core. Or what they call Common Core. It’s more like an offshoot of the Communist Manifesto. We’ll teach you what we want you to learn but please, please, please don’t share any of what you learn in school with your parents. They wouldn’t understand. Progressives all of them and they are dangerous people, especially for our children.

      • South Carolina. It’s my understanding that Texas doesn’t use Common core either. South Carolina initially adopted Common Core (2014) and quickly (within a year) jumped ship after the department of education realized what a sham it was. When I was looking at schools down here, meeting with teachers, all were very thankful that they don’t teach common core.

        It’s also nice to know they still teach cursive, when I lived in Vermont the local school no longer taught kids cursive, so I resorted to doing it myself. You would be amazed how many high school kids can’t sign their name in cursive in Vermont.

        We have great charter schools, full school choice, and schools for gifted children. My daughter is in one currently. She would of never had that option in Vermont. I’m very pleased with the school system down here for the most part. Although, you can certainly pick poor performing schools, but that is certainly the case in any state. And to live is a district with great schools is VERY affordable unlike in Vermont.

  8. When everyone was rejection “Common Core,” Vermonters were assured that the Smarter Balanced Assessment Consortium testing had nothing to do with the flawed curriculum. Two years later, SBAC testing is a “Common Core-linked achievement test” – Sounds like we were sold a bill of goods !

Comments are closed.