Overview: What Is a Comprehensive Data System?
Data are used primarily in two ways: to make accountability or “summative” decisions or to make continuous improvement or “formative” decisions related to system improvement and student achievement. Both types of data are necessary to simultaneously hold the system accountable while leaders strive for improvements. State longitudinal data systems are important to organize data and combine it with other data to answer complex policy and improvement questions – which can be done at all levels of the system.
A comprehensive state-level data system collects data and information primarily for accountability, reporting and decisionmaking purposes. Typically states are responsible for student assessments, accountability systems, financial reporting and reporting teaching quality measures. State policymakers use data to make policy decisions, foster school improvement, allocate resources, identify and share best practices and hold schools and districts accountable for student learning. To meet NCLB requirements, states are primarily responsible for developing and implementing student assessment and accountability systems and teacher quality measures. Results are reported by grade, subject and subgroups, noting when schools do or do not meet adequate yearly progress (AYP). States may also require high school exit examinations and/or end-of-course tests for core high school classes. States may also provide funding to districts to administer the ACT exam and/or Advanced Placement tests to help ensure students are ready for college. States typically require school accreditation and encourage best practice audits or school improvement indicators that districts can use to collect their own data.
States may also encourage districts to develop and use balanced scorecards or the Baldrige Education Criteria. The Balanced Scorecard is a tool that helps organizations monitor the implementation of organizational strategy. The Baldrige Criteria for Performance Excellence provide a systems perspective for understanding performance management. Both the Scorecard and Baldrige Criteria reflect validated, leading-edge management practices against which an organization can measure itself.
1. Essential Elements
Comprehensive data systems at the state level include the following essential data elements:
- Student assessment results on state academic standards by subgroups
- Accountability reports of schools and districts meeting AYP
- School report cards (NCLB requirement)
- Highly qualified teacher information on the school report cards
- Student promotion, graduation and drop-out rates
- Measures to assess progress on state goals
- School accreditation results
- Higher education program accreditation results
Optional Data Elements:
- Student results on high school exit examinations and or/end-of-course examinations
- Provision of a database of benchmark assessments districts can access linked to state standards
- Implementation and report of state-district-school balanced scorecard (i.e., Delaware)
- Results on college readiness measures such as the ACT (sometimes embedded in state assessments)
- Results on predictive college readiness measures such as PSAT and EPAS
- Results on school improvement indicators
- Results on college admission/placement tests
- Results on leader performance measures as feedback to preparation programs
Delaware is one of a handful of states that includes all of the Data Quality Campaign’s 10 essential elements in its longitudinal data system. The state assessment system includes a growth model to assess what students learn each year, and to report their proficiency level on state academic standards. A new system -- the Delaware Educator Data System (DEEDS) -- records information on educator licensure/certification and credential maintenance. Delaware has a professional licensure certificate and DEEDS records educators’ professional learning courses toward meeting this licensure. Because both students and teachers have unique identifiers in the longitudinal data system, it is possible to link student achievement to teacher’s experience and training.
A very new Delaware initiative -- the Education Success Planning and Evaluation System (ESPES) -- begins with the development of a district consolidated application process for federal and state funds. A balanced scorecard is now required by the state to set district targets for improvement and integrate them with school improvement planning processes. Districts are expected to report on the state-required measures and are encouraged to include their own measures – as well as the strategies they will use to improve student achievement. What is unique in Delaware is that districts can link to the Decision Support Architecture Consortium to learn of district best practices related to the teaching and learning, as well as operations (such as transportation, human resources, facilities and food services). Once benchmarks and best practices are identified, Delaware has developed a project management process that guides districts through the action steps to implement best practices based on their strategies.
Kentucky has a robust accountability, data use and school improvement system. The state assessment includes a growth model as well as student reporting by standards and school reporting by AYP. Items are released from annual assessments as guides for the types of items on the state exam. District and building assessment coordinators are trained in assessment administration and data use. Kentucky has a set of standards and indicators for school improvement. The school improvement process is supported by training, guides for roles and responsibilities for improvement planning and assessing the impact of school improvement. The state has also developed a set of Scholastic Audits (rubrics) for districts and schools linked to the school improvement standards. A summary of Kentucky’s efforts appears in the tools section, called Kentucky Data Use.
3. Critical Questions
- Does the state have a comprehensive data system that includes the essential data elements listed above?
- Which optional data elements does the state have? How are these being supported and used?
- Do states know how to distinguish and use different types of measures – those for accountability and those for improvement?
- What progress is being made to complete a longitudinal data system that includes the ten essential elements of the Data Quality Campaign?
- What progress has the state made to integrate multiple data bases?
- What data are collected to measure progress toward state goals? What progress is being made and do results suggest new state policies or programs?
- How are data being used to support schools in need of improvement?
- How are data being used to identify and share best practices that raise student achievement?
- How are states using predictive measures of students’ readiness for college or work to reform the education system?
- What is the state role in training districts in the use of quality measures such as the balanced scorecard or the Baldrige education criteria?
4. State Resources
Association for Supervision and Curriculum Development. (2003, February). Using Data to Improve Student Achievement. Educational Leadership, 60(5). Summary: This issue addresses how teachers and schools use data to make instructional changes and to assess the effectiveness of school programs. Article topics include classroom assessments to improve student learning, a reader's guide to scientifically based research, how to conduct data collection and analysis, the effects of testing on student motivation and learning, and data warehousing and information management systems.
Data Quality Campaign. (2006). Creating a Longitudinal Data System: Using Data to Improvement Student Achievement. Summary: This report describes the 10 essential elements that all schools should include in their longitudinal data system – and provides the policy and practice questions well-designed systems can answer.
Goldring, E. Porter, A., Murphy, J. Elliott, S. and Cravens, X. (2007). Assessing Learning Centered Leadership: Connections to Research, Professional Standards, and Current Practice. Wallace Foundation. Summary: This report presents the research base and conceptual framework for the VAL-Ed leader assessment tool. Leaders are assessed on two dimensions, the core components of school performance and the key processes of leadership.
Johnstone, C.J., Altman, J., & Thurlow, M. (2006). A state guide to the development of universally designed assessments. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Summary: This guide provides strategies to support states in designing and examining educational assessments to ensure accessibility for all users. The goal of the universal design approach to educational assessment is to create tests that accurately measure the skills and knowledge of a wide range of students, including those with disabilities and English language learners. Elements of universal design include precise constructs; clear and readable text, format, instructions, and visuals; respect for and inclusion of diverse test populations; and a test format that can be updated and/or changed without major consequences to measures. The authors contend that universal design considerations should be made from the beginning of test design through the entire assessment process.
Palaich, R.M., Good, D.G., & Van Der Ploeg, A. (2004, June). State education data systems that increase learning and improve accountability. North Central Regional Educational Laboratory. Summary: This paper reviews uses of education data and considers why state education leaders should build and maintain education data systems.
Sirotnik, K.A., & Kimball, K. (1999, November). Standards for standards-based accountability systems. Phi Delta Kappan, 81(3), 209-214. Summary: This article discusses 11 standards educators and policy makers should consider in evaluating standards-based accountability systems. The authors raise questions and issues to foster discussion about how to operationalize each standard in order to determine proficiency of assessment and accountability systems. These standards support legislators and education policy makers’ efforts to critically and responsibly examine assessment and accountability systems.
Stiggins, R. (2006). Balanced assessment systems: Redefining excellence in assessment. Princeton, NJ: Educational Testing Service. Summary: This paper describes a vision of the future of assessment that informs instructional decisions and encourages students to learn.