DID Resource Kit for States, Districts and Schools

Overview: Using Data Effectively

Put simply, data inform decisions. In the current educational context of accountability and school reform, data-informed decision-making is increasingly seen as an essential part of the educational leader’s toolkit. Because NCLB requires research-based practices, and high performance standards, timely, pertinent, accessible and understandable data is crucial to leaders as they make decisions that will result in increased student achievement. And as leaders consider strategies for school improvement, data are the cornerstone of making decisions related to equity and excellence.

State policymakers use data to foster school improvement; allocate resources; identify and share best practices; and, hold schools and districts accountable for student learning. To meet NCLB requirements, states are primarily responsible for developing and implementing student assessment and accountability systems and teacher quality measures. Results are reported by grade, subject and subgroups, noting when schools do or do not meet adequate yearly progress. State policymakers use accountability data to identify how well students are meeting standards, recognize achievement, provide rewards or sanctions, and provide assistance to schools or districts in need of improvement. When longitudinal data are available, state leaders can evaluate the effectiveness of schools and programs in improving student achievement, identify consistently high performing schools so that educators can learn about promising practices, and determine how well districts are preparing students for college or work.


 1. Essential Elements

According to Data Informed Leadership in Education (Knapp et al., 2006), data, appropriately interpreted, help leaders understand what is happening in educational organizations and take appropriate action. The policy and community environments in which educational leaders work are likely to prompt a variety of uses of data, by:

  • Demanding information from the educational system about its performance (as in accountability systems) or the effectiveness of particular programs (as in the evaluation requirements accompanying categorical program funding).
  • Offering sources of data or help by assembling, displaying or interpreting data (as technical assistance centers, universities, or vendors may do).
  • Creating opportunities for inquiry (as when an influx of new immigrant children raise questions about appropriate educational programs, school assignments, and so on).
  • Providing public images of the educational system’s functioning (as in media accounts that beg for response, clarification, or refutation).
  • Raising questions about the school system’s policies or responsiveness to particular constituencies or needs (as in legislative debate about support for teacher induction or school board debate about school closures).

According to a RAND study (Marsh et al., 2006), data are used to make decisions related to:

  • Setting and assessing progress toward goals
  • Addressing  individual or group needs
  • Evaluating the effectiveness of practices
  • Assessing whether client needs are being met
  • Reallocating resources based on outcomes
  • Enhancing processes to improve outcomes

According to the same RAND study, data are used when:

  • Data are readily accessible
  • Education leaders believe the data accurately reflect student achievement
  • There is a motivation to use data
  • The data come in time to make important decisions
  • Data users have the training and skill to analyze data and make appropriate adjustments.
  • There is strong system or school support to use data and create a culture of data use.


2. Promising Practices 

New Mexico’s standards-based assessment test (NMSBA) determines which schools are making the adequate yearly progress (AYP) required to comply with the federal No Child Left Behind (NCLB) law. Each year, the state raises the percentage of students who must be proficient in math and reading to achieve AYP. Districts that fail to make AYP are designated in need of improvement. If they fail to measure up for several years in a row, schools come under more scrutiny and, eventually, under direct control of the state. The state measures percentage improvement in each subject area separately for eight groups of students: Hispanic, Native American, white, African-American, Asian, English language learners, students with disabilities (including special education students), and low-income students. Until recently, the state sent data CDs to district personnel that they did not know how to download or use.

To make the data more usable, the New Mexico Office of Educational Accountability developed pivot tables for every school in New Mexico. (Pivot tables automatically sort, count and total data stored in a spreadsheet and create a second table displaying the organized data.) The state hired data experts to work in the six Wallace-funded demonstration districts to help the superintendent use the state assessment data. The result has been a simple point-and-click set of tables that can break down student performance not just by grade and subject matter, but by particular teachers, individual students, even by the individual topics on the test. These topics or “benchmarks” are the smallest unit of testable knowledge on the NMSBA for which New Mexico releases student scores. A benchmark may be tested with a half dozen questions or more.

By 2006, several districts had created a benchmark profile for all of their students. Every student’s score on each benchmark was mapped not only against the total points available in that benchmark, but also against the score of the average proficient student. The next step will be to create a historical record for each child with respect to specific benchmarks. After three years of working with pivot tables, it has become possible to do some longitudinal analyses. Coming up with the right question may be the challenge.

Western Michigan University researchers administered a survey to public school principals to determine how principals use data to inform decision-making about improving student achievement. (The survey is an adaption of the survey developed by the Wallace sponsored Data-informed Leadership group and appears in the tools section of this Web site.) The Michigan version of the survey investigated the gap between what data principals deemed important versus the data they actually use to improve student achievement.  The researchers found that principals are overwhelmed with data. The two most challenging questions that continue to face principals are: (1) What is the most appropriate data that I need? and, (2) Once I get the data, what do I do with it?

This survey also investigated the level of importance and actual data use in relationship to the 11 high impact strategies identified by Marzano (2003) that were positively correlated with student achievement. Consistently, analysis of the survey indicated a statistically significant gap between importance of data they wanted to use vs. the data they actually used. The survey reported that, “Findings raised serious issues regarding principal and teacher knowledge in data analysis, connecting data to curriculum and instruction, the inability to examine data through multiple lens and the time for teachers and administrators to review data and to make program improvements based on data.” Survey results led to the development of two tools, a data guidebook and a principals’ data use survey (both in the tools section of this resource kit). These tools have been disseminated through the Michigan Department of Education. The Michigan researchers also developed course materials for university educational leadership programs.

3. Critical Questions
  • Does your state provide standards-based student achievement measures?  In a timely manner to districts and schools?
  • How are accountability results used?
  • How are data being used to:
    • Set and assess progress toward goals?
    • Address individual or group needs?
    • Evaluate the effectiveness of practices?
    • Assess whether client needs are being met?
    • Reallocate resources based on outcomes?
    • Enhance processes to improve outcomes?
  • Do state education leaders have data literacy? How do they acquire it?
  • Does your state have a longitudinal data system that includes all ten elements advocated by the Data Quality Campaign?
  • Does your state provide training in the use of state assessment results?
  • What is your state’s role in fostering cultures of inquiry in districts and schools?
  • What is your state’s role in fostering the connection between data use and resource allocation?

4. State Resources

Anderson, S., Fowler, D., & Klein, S., et al. (2005). Judging student achievement: Why getting the right data matters. Washington, DC: MPR/NCEA. Summary: This policy brief stresses the importance of using good data to develop data information management systems and make judgments of student and school performance.

Association for Supervision and Curriculum Development. (2003, February). Using Data to Improve Student Achievement. Educational Leadership, 60(5). Summary: This issue addresses how teachers and schools use data to make instructional changes and to assess the effectiveness of school programs. Article topics include classroom assessments to improve student learning, a reader's guide to scientifically based research, how to conduct data collection and analysis, the effects of testing on student motivation and learning, and data warehousing and information management systems.

Cooley, V.E., J. Shen, D.S. Miller, P.N. Winograd, J.M. Rainey, W. Yuan, and L. Ryan. (2006). Increasing Leaders' Capacity in Data-based Decision-making: State Level Initiatives in Ohio, New Mexico and Michigan. Educational Horizons, 85 (1), 57-64. Summary: This report highlights three state-level initiatives around data-based decisionmaking.

Data Quality Campaign (2008). Tapping into the Power of Longitudinal Data: A Guide for School Leaders. Summary: This short document provides an overview of the power of longitudinal data, making the distinctions between snapshot and longitudinal data. The last section provides action steps for school leaders to build and use longitudinal data systems.

Feemster, Ron (2007). Making state accountability count: How New Mexico supports principals with data and tools. New York: The Wallace Foundation. Summary: This is a journalistic account of how the New Mexico Office of education Accountability streamlined and made more flexible the use of state assessment data by districts and schools.

Johnson, R. S. (2002) Using data to close the achievement gap: How to measure equity in our schools. Thousand Oaks, CA: Corwin Press, Inc. 978-0761945093. Summary: This book provides helpful information on how educators, policy makers and parents can use data to set achievement goals and to measure school progress towards these goals. This book addresses the issue of using data to understand and solve the problem of disparate levels of achievement among different groups of students and includes many useful tools, templates and examples from schools.

Knapp, M. S., J.A. Swinnerton, M. A. Copland, et al. (2006). Data-informed leadership in education. Center for the Study of Teaching and Policy, University of Washington. Summary: This article synthesizes and interprets ideas, frameworks, beliefs, and activities regarding the use of data in educational decision making. The concept of data-informed leadership relates to the availability, quality and use of data among school leaders in order to improve teaching and learning. The broader focus on leadership, rather than just data-based decision making, captures a wide range of purposes data can serve for leaders. This article reviews common practices and emerging strategies that support leaders’ use of data on the state, district, and school levels. Specifically, it highlights how data are used and what kinds of data are implied for specific types of leadership activities. The article concludes with noting unanswered questions that warrant further research and the enduring dilemmas in data-informed practice.

Marsh, J. A., J. F. Pane, & L.S. Hamilton (2006). Making sense of data-driven decision making in education. RAND Corporation. Summary: This article addresses many unanswered questions about the use of data to inform decisions and the effects on student achievement.

Marzano, R.,Waters, T., and McNulty, B. (2003). School leadership that works: From research to results. Denver: McREL. Summary: Drawing from 35 years of studies, the authors explain critical leadership principles that every administrator needs to know including the 21 responsibilities that have a significant effect on student learning and the correlation of each to academic achievement gains. It also includes 11 factors and 39 actions that help you take a site-specific approach to improving student achievement  and a five-step plan for effective school leadership that includes a strong team, distributed responsibilities, and 31 team action steps.

Palaich, R.M., Good, D.G., & Van Der Ploeg, A. (2004, June). State education data systems that increase learning and improve accountability. North Central Regional Educational Laboratory. Summary: This paper reviews uses of education data and considers why state education leaders should build and maintain education data systems.

Sirotnik, K.A., & Kimball, K. (1999, November). Standards for standards-based accountability systems. Phi Delta Kappan, 81(3), 209-214. Summary: This article discusses 11 standards educators and policy makers should consider in evaluating standards-based accountability systems. The authors raise questions and issues to foster discussion about how to operationalize each standard in order to determine proficiency of assessment and accountability systems. These standards support legislators and education policymakers’ efforts to critically and responsibly examine assessment and accountability systems.