Overview: What Is a Comprehensive Data System?
Data are primarily used in two ways – to make accountability or “summative” decisions or to make continuous improvement or “formative” decisions related to system improvement and student achievement. Both types of data are necessary to simultaneously hold the system accountable while leaders strive for improvements. State longitudinal data systems are important to organize data and combine it with other data to answer complex policy and improvement questions – at all levels of the system.
District leaders use data to monitor student achievement, school success, the effectiveness of programs and allocate resources to ensure student needs are met. District summative measures include state assessment results, and often results on high school exit exams or end-of-course tests. Districts may use continuous improvement measures such as periodic benchmark assessments that are aligned to state and district standards. Districts may also use indicators or measures of school improvement, such as the results of the state school improvement planning process, indicators for school improvement or comparisons to best practice audits. How leader effectiveness is assessed varies by district but often it is used for goal setting and continuous improvement.
1. Essential Elements
Comprehensive data systems at the district level include the following essential elements:
- Student assessment results on state academic standards by subgroups
- Accountability reports of schools and districts meeting AYP
- School report cards (NCLB requirement)
- Highly qualified teacher information on the school report cards
- Student promotion, graduation and drop-out rates
- Measures to assess progress on state goals
- School accreditation results
- Results on district administered interim or benchmark assessments
Optional Data Elements:
- Results of high school exit examinations and or/end-of-course examinations
- Provision of a database of benchmark assessments districts can access linked to state standards
- Implementation and report of state-district-school balanced scorecard (i.e., Delaware)
- Results on college readiness measures such as the ACT (sometimes embedded in state assessments)
- Results on predictive college readiness measures such as PSAT and EPAS
- Results on school improvement indicators
- Results on college admission/placement tests
- School improvement indicators
- Other district administered assessments such as practice tests for state graduation exams or diagnostic and placement assessments for students entering the district
- Assessments of leader effectiveness
2. Promising Practices
Superintendent Linda Paul of the Aztec School District in New Mexico found understanding the state assessment results sent to her district frustratingly incomprehensible. Until recently, she received a big box of paper printouts and a CD of raw data. Printouts included a data line for each of the 3,500 students in her district, but there was no summary data for each of the six schools, no tabulations of grade level, subject matter, gender or ethnicity. New Mexico’s Office of Education Accountability set out to make the data more timely and useful using Wallace Foundation funds.
When schools analyze data, they often begin in the fall with state assessment results. Even though these results often come late in the year (and classroom rosters have changed) they provide information on how well students are meeting state standards, and if a school is meeting AYP. New Mexico solved the problem of late assessment data by requiring the assessment vendor to report assessment results by a certain date and then creating software (pivot tables on a CD) sent to districts that allowed them to quickly re-sort state student achievement data by classrooms. These are simple point and click tables that can break down student performance by grade and subject matter, teachers, students and particular standards.
Linda Paul and several other New Mexico districts have also added “short-cycle” assessments that are administered three times a year. Student results on these predict student performance on the state assessment. Because results on short-cycle assessments are returned within two weeks, teachers can address student learning needs. And teachers’ learning can be addressed as well. Principals can identify teachers who need help and send coaches into their classrooms to model new lesson plans and share teaching tips.
A set of promising data practices comes from the Long Beach Unified School District (LBUSD) that won the Broad Urban Prize for Education in 2003. State accountability data is collected and reported through the STAR system, California Standards Test (CST-CAPA), the California Achievement Test (CAT/6) and the Spanish Assessment of Basic Education (SABE/2). California also administers the English Language Development Test (CELDT) and the California High School Exist Exam (CAHSEE).
LBUSD also administers and reports its own district assessments and indicators, including: basic math facts, grades 2-8 and reading benchmarks in grades K-8. The district also administer end-of-course exams in art, English language arts, English language development, health education, history and science, mathematics, and Spanish. Practice or “mock” high school exit exams are administered as predictive measures that allow interventions to assure all students pass the high school exit examination.
LBUSD achievement monitoring strategies at the district level included:
- Principals Accountability and Evaluation Process
- Key Results Walkthroughs, K-12
- Principals professional development
Although they had a number of strategies to improve teaching and learning, district monitoring strategies for principals included:
- Ensuring instructional practices reflect learning
- Observing classrooms daily and provide feedback to teachers
- Requiring ongoing assessment as basis for instructional decisionmaking
- Using assessment data to determine student interventions
- Providing time for teacher collaboration
- Analyzing data by grade distributions by subject, class, and student and involve teachers in the same process
- Submitting student data to central office staff every 6-8 weeks
- Recommending support for teachers through Peer Assistance and Review
3. Critical Questions
Does the district have a comprehensive data system that includes the essential elements listed above?
Which optional elements related to data use does the district have? How are these being supported and used?
What progress is being made to complete a longitudinal data system that includes the 10 essential elements of the Data Quality Campaign?
What progress has the state made to integrate multiple data bases?
Does the district have a data warehouse with easy access by school leaders and teachers?
What data are collected to measure progress toward state goals? What progress is being made and do results suggest new state policies or programs? How are these decisions made?
How are data being used to support schools in need of improvement?
How are data being used to identify and share best practices that raise student achievement?
What role does the district play in assessing leader effectiveness and supporting their professional growth?
4. District Resources
Bernhardt, V. (2006). Using data to improve student learning in school districts. Larchmont, NY, Eye on Education. Summary: A how-to guide on selecting data and using a continuous improvement planning model. Sample district results are used and organized by: Where are we now? What are the gaps? Where do we want to be? And how can we get there? Continuous improvement continuums are included.
Celio, M. B. & Harvey, J. (2005). Buried treasure: Developing a management guide from mountains of school data. Seattle, WA: University of Washington. Summary: This report provides useful information on developing a school district management guide as well as an actual guide focused on seven evidence-based indicators: achievement, elimination of the achievement gap, student attraction to the school, student engagement with the school, student retention and completion, teacher attraction and retention, and funding equity. The report also includes several implications from the analyses of the role of indicators in the management system described. The authors conclude that indicator development encourages educators to think of new ways to assess accountability and to move beyond bottom-line assessment systems.
Copland, M. (2003). Leadership of inquiry: Building and sustaining capacity for school improvement. Educational Evaluation and Policy Analysis 25(Winter), 375-395. Summary:This article reviews a longitudinal mixed-methods study of leadership in the context of a region-wide school reform effort entitled the Bay Area Reform Collaborative (BASRC). Data analysis suggests the use of an inquiry process is paramount for building capacity for school improvement and developing leadership.
Corcoran, T., Fuhrman, S. H., Belcher, C. L. (2001). The district role in instructional improvement. Phi Delta Kappan 83 (September), 78-84. Summary: The researchers examined the roles played by district staff members in shaping and supporting instructional classroom reforms using evidence-based decision making in three large urban school districts.
Dailey, D., Fleischman, S., Gil, L., Holtzman, D., O'Day, J., & Vosmer, C. (2005). Toward more effective school districts: A review of the knowledge base. Washington, DC: American Institutes for Research.
Data Quality Campaign. (2006) Creating a Longitudinal Data System: Using Data to Improvement Student Achievement. Summary: This report describes the 10 essential elements that all schools should include in their longitudinal data system, and provides the policy and practice questions well-designed systems can answer.
Goldring, E. Porter, A., Murphy, J. Elliott, S. and Cravens, X. (2007). Assessing Learning Centered Leadership: Connections to Research, Professional Standards, and Current Practice. Wallace Foundation. Summary: This report presents the research base and conceptual framework for the VAL-Ed leader assessment tool. Leaders are assessed on two dimensions, the core components of school performance and the key processes of leadership.
Knapp, Michael S., Swinnerton, J. A., Copland, M. A., et al. (2006). Data-informed leadership in education. Center for the Study of Teaching and Policy, University of Washington. Summary: This article synthesizes and interprets ideas, frameworks, beliefs, and activities regarding the use of data in educational decision making. The concept of data-informed leadership relates to the availability, quality and use of data among school leaders in order to improve teaching and learning. The broader focus on leadership, rather than just data-based decision making, captures a wide range of purposes data can serve for leaders. This article reviews common practices and emerging strategies that support leaders’ use of data on the state, district, and school levels. Specifically, it highlights how data are used and what kinds of data are implied for specific types of leadership activities. The article concludes with noting unanswered questions that warrant further research and the enduring dilemmas in data-informed practice.
Stiggins, R. (2006). Balanced assessment systems: Redefining excellence in assessment. Princeton, NJ: Educational Testing Service. Summary: This paper describes a vision of the future of assessment that informs instructional decisions and encourages students to learn.