Resources

A Review and Empirical Comparison of Approaches for Improving the Reliability of Objective Level Scores

This paper reports on an evaluation of the psychometric issues associated with estimating objective level scores, often referred to as 'subscores.' Methods for increasing the reliability of subscore estimates that have been suggested in literature are reviewed. Based on this review, an empirical study comparing some of the more promising procedures was conducted using data from four different statewide testing programs, representing three different subject areas and various testing conditions. The comparison of subscore augmentation approaches found that generally all methods were very successful in dramatically increasing the reliability of subscore estimates. However, this increase was accompanied by near-perfect correlations among the subscore estimates. This finding called into question the validity of the resultant subscores, and therefore the usefulness of subscore augmentation process. Implications for practice are discussed.

Related Resources

• ProgramTechnical Issues in Large Scale Assessment (TILSA)
• WebsiteOnline Computer-Based (Testing) Decision Making Tool
• PublicationInterim Assessment Practices and Avenues for State Involvement
• PublicationInterim Assessment Practices and Avenues for State Involvement – Appendices
• PublicationVertical Scaling in Standards-Based Educational Assessment and Accountability Systems
• PublicationQuality Control Checklist for Processing, Scoring, and Reporting
• PublicationQuality Control Checklist for Item Development and Test Form Construction
• PublicationBuilding an Interim Assessment System: A Workbook for School Districts
• PublicationAddressing Two Commonly Unrecognized Sources of Score Instability in Annual State Assessments
• ProgramPublicationVertical Scaling in Standards-Based Educational Assessment and Accountability Systems