Contact: Greg Knudsen
Date: October 15, 2007


Scoring Error Likely to Have Limited Effect on Schools and AYP


A scoring error by the test contractor on last spring's Hawaii State Assessment is not expected to significantly affect the results, and will not lower any school's No Child Left Behind determination of Adequate Yearly Progress for the current year. Once corrections are made, it is possible that Hawaii's proficiency levels will improve.

The Department of Education has directed the test contractor, The American Institutes for Research (AIR), to correct the errors, to reaffirm the reliability of the entire test, and to adopt greater quality control procedures. The DOE is also reviewing the contract’s provisions for financial damages.

"The responsibility for these errors rests with us," said Jon Cohen, AIR vice president for assessment, who is currently in Hawaii to address the concerns. "We're working to correct the problems and have implemented safeguards to prevent those errors from happening again. The American Institutes for Research regrets the situation, and we will fix this matter at no cost to the State of Hawaii."

"Test score validity and reliability are essential," said Superintendent Patricia Hamamoto. "Any time there are test score errors, it is a serious matter of concern. The test contractor must correct the errors and put additional quality control measures into place to ensure that these types of mistakes are not repeated."

Following distribution of the test scores at the beginning of the school year, some schools reported to the DOE that they'd received test reports for students who had not taken the tests. The DOE reported the irregularities to AIR in early September.

Upon investigating the problem, AIR discovered a scanning error that mistakenly recorded response marks for multiple choice questions that were actually blank. This resulted in scores being given for blank answer booklets. By looking for similar patterns in the test data files, AIR estimates that as many as 1,700 test booklets may have been misread, out of a total of nearly 98,000 booklets. AIR informed the DOE about the problem on October 5.

To correct the current errors, AIR and its subcontractor will hand-score each booklet that follows the pattern of those incorrectly scanned. Those corrections will be completed by November 1.

To ensure scoring accuracy of the entire spring 2007 test, AIR will rescan every Hawaii test booklet, which is to be completed by December 15.

DOE officials, at AIR's expense, will be present to observe both the hand scoring and the rescanning procedures.

Among measures to ensure future accuracy, AIR will be using a different scanning subcontractor, Pearson Educational Measurement, starting with the fall 2007 testing cycle. It will also improve scanner calibration methods and increase test score monitoring.

The Hawaii State Assessment measures student proficiency in math and reading as part of the federally mandated No Child Left Behind Act. Hawaii State Assessment results are used to determine student proficiency levels and whether a school is meeting the NCLB requirement for Adequate Yearly Progress (AYP).

According to AIR's Cohen, the blank test sheets that were incorrectly scanned very likely produced low, non-proficient scores. Mixing those low scores in with a school's valid scores would have the effect of lowering the school's overall proficiency average. Therefore, the school’s true average could be higher.

There is some possibility that removal of the blank booklets might affect a school's participation level, which is also a factor in determining AYP. However, no school will have its current AYP status lowered as a result of this error, since DOE guidelines hold the schools harmless on changes affected by systemic problems.

Final revised Hawaii State Assessment and AYP results will be available by mid-January 2008.


Source contacts:
DOE: Greg Knudsen, communications director, (808) 586-3230
AIR: Larry McQuillan, communications director, (202) 403-5119

-- DOE --