All 98,000 Hawai'i State Assessment tests administered last spring will have to be reviewed after teachers discovered errors in how the tests were scored.

The review initially will focus on a group of more than 1,600 tests because some of them were incorrectly given a score even though the students apparently did not take the test.

By Dec. 15, every test will be scored again.

The problem was discovered this summer when some Hawai'i teachers saw scores for students who did not even take the math and reading tests, but had a blank test book submitted under their name, as required, to the testing company, American Institutes for Research.

"Obviously, a situation like this does cause concern," said Dave Rolf, executive director of the Hawai'i Automobile Dealers Association, who has criticized the current version of the test, which was administered for the first time this spring.

The Hawaii State Assessment math and reading tests are administered to third-, fourth-, fifth-, sixth-, seventh-, eighth- and 10th-graders on all Islands as part of the federally mandated No Child Left Behind Act. The results are used to determine whether schools meet the act's requirement for Adequate Yearly Progress.

The Department of Education says the new test has become grade-specific, allowing what school officials say is a more accurate measure of how students are doing. But officials are fending off criticism that the improved scores are a result of "dumbing-down" the test.

"We're in an era of high-stakes testing," said Board of Education chairwoman Karen Knudsen. "So we have to make sure and assure the public that we take it seriously. Any error ... is not acceptable."

She stressed that the test content, which has drawn previous criticism, and the scoring errors are two different issues.

"The current issue deals only with a mechanical glitch," she said. "It is a totally separate conversation."

It's not fair to question the validity of the test based on the technical glitches in the scanning machines, Knudsen said.

The latest problems occurred even as education officials switched testing companies last year after problems with Harcourt Assessment, the predecessor to American Institutes for Research.

The DOE switched to AIR after finding "significant errors" in testing materials in spring 2004 and distribution problems in spring 2005.

Because some blank tests received scores, AIR will now pay all of the costs to have the 1,682 tests under question reviewed by hand and to pay a new sub-contractor to rescan the remaining 96,094 tests.

COMPANY TO FOOT BILL

The company also will pay the costs to have Department of Education officials monitor both the hand scoring and rescanning at its Columbus, Ohio, headquarters.

"This industry has no tolerance for error," said Jon Cohen, vice president, chief statistician and director of assessment for AIR. "We have not had something like this happen before. ... We really are sorry that this has happened. We take it very seriously."

The problem with the latest batch of Hawaii State Assessment tests seems to rest with the scanning abilities of an AIR sub-contractor, Champaign, Ill.-based MetriTech, Cohen said.

"The scanners picked up marks when in fact there were not any," Cohen said. "There was a glitch in the process."

MetriTech officials could not be reached for comment in Illinois following yesterday's afternoon news conference at the DOE.

In July, when school opened, teachers began noticing scores for some students who did not take the test, Hamamoto said. The DOE then notified AIR about the concerns in September.

"It took a while to pull the book and check to find that something really was going on here," Cohen said. "Scanning errors are rare (but) they are not unheard of. It does happen."

Most of the tests involved booklets requiring written answers that have to be reviewed by a person as well as multiple-choice answers that can be scored by a scanner, Cohen said.

Some of the tests received scores even though there were no answers for either the written or multiple-choice portions of the test, he said.

Hamamoto said: "We scan all books regardless whether they were written in by the student or not to ensure that whatever was distributed was returned for test security. If you ship 1,000 books to the school, 1,000 books must be returned to the center and that's the integrity of the test."

AIR officials then searched the scanners' databases for tests that had marks in the multiple choice section but no written answers and estimated that number at 1,682, Cohen said.

"There was a problem with the scanners that caused, on some occasions, blank multiple-choice items to be read as having a response."

About half of the tests under review came from 10th-graders, who notoriously do not answer the written portion of the test, he said.

"It may well be that there are no greater number of misscanned booklets in the 10th grade, but rather the kids just didn't answer" the written portions, Cohen said.

HAND SCORES PLANNED

The 1,682 tests under question all will be hand scored and the results will be reported to the DOE by Nov. 1, he said.

"A lot of the problems are probably going to turn out to be valid scores," he said. "The final number may be significantly smaller than the 1,682 tests under review."

They represent 1.7 percent of all of the 97,776 tests that were administered.

AIR also has hired a new scanning subcontractor, Pearson Educational Measurement, which will rescan all of the remaining tests and have results on them to the DOE by Dec. 15.

No matter what the review reveals, no schools will see lower scores when it comes to their Adequate Yearly Progress under the federal No Child Left Behind Act, Hamamoto said.

But the review could result in test scores being raised for certain schools, she said.

DOE spokesman Greg Knudsen said state policy prevents school scores from dropping as the result of "systemic problems."

Hamamoto said that AIR "must correct the errors and put additional quality-control measures into place to ensure that these types of mistakes are not repeated."

She characterized her confidence in AIR as "cautious."

"I believe we were also cautious when we first entered into the contract," Hamamoto said. "We've experienced some unfortunate situations so we're very aware and very vigilant and we are watching to see how it progresses."

Cohen did not disclose the cost of the review to his company, but said Hawai'i taxpayers will not have "to pay a dime."

DOE attorneys, however, are looking at the $27 million, three-year contract with AIR, which expires on June 30, 2009.

A clause in the contract calls for penalties of $10,000 per day for the first 12 days after AIR has been notified of a problem, increasing to $20,000 per day after, Hamamoto said.

She could not be more specific about potential monetary penalties against AIR, saying the issue depends on the review by DOE attorneys.

With the change in tests, 182 Hawai'i schools, or 64.5 percent, achieved the goals mandated by No Child Left Behind.

NO LOWER SCORES

Last year just 35.5 percent of Hawai'i's 282 tested public schools met the goals. Education and school officials called the results the best they've seen since No Child testing began in 2002.

DOE spokesman Greg Knudsen said school officials will announce the results of both reviews after they are reported in November and December, even though they will not lower a school's Adequate Yearly Progress standing.

He said it's likely that schools' scores will go up, rather than down.

But some schools that barely met the threshold for AYP could lose ground in student participation from the review if some scored test booklets were submitted on behalf of students who did not take the test.

"The logic is that the scores will go up," Knudsen said. "The only thing that could go down is participation."

"No school will be lowered as a result of this review," Knudsen said. "No school that has achieved AYP will see their score lowered. But if a school right now has not been able to achieve AYP would be moved up as a result of this review and is found to have made AYP, they should be able to claim that distinction."

Reach Dan Nakaso at dnakaso@honoluluadvertiser.com.