Meta-Analysis Study of the Effects of Teacher Professional Development Improving Evaluation of Professional Development  Final Report Effects of Teacher Professional Development on Gains in Student Achievement: How Meta Analysis Provides Scientific Evidence Useful to Education Leaders This report is based on a meta analysis focused on completed studies on the effects of professional development for K-12 teachers of science and mathematics. Specifically, the study and its report addressed two questions important to education leaders: What are the effects of content-focused professional development for on improving teacher knowledge and skills and on improving student achievement? What professional development program characteristics explain the degree of effectiveness, and are the findings consistent with existing research on effective professional development, e.g., content focus, duration, coherence, active learning, and collective participation of teachers? The meta analysis found that 16 studies reported significant effects of teacher development on improving student achievement. These studies reported effect sizes for student achievement gains for a treatment group as compared to a control group and the studies provided adequate data and documentation for the CCSSO research team to compute or re-analyze effect sizes. The large majority (12 of 16) studies were focused on analyzing mathematics teacher professional development and effects on student achievement in mathematics. CCSSO also reviewed the professional development program designs and learning goals documented in the 16 studies. Several common patterns emerged including program designs with strong emphasis on teachers learning specific subject content as well as pedagogical content for how to teach the content to students, multiple activities to provide follow-up reinforcement of learning, assistance with implementation, and support for teachers and duration of professional development activities lasting six months or more. Based on the study, CCSSO have several recommendations for how the results and processes from the meta-analysis can be useful to researchers, evaluators, and state education leaders: The meta-analysis design and procedures employed by CCSSO proved to be effective in identifying a set of common findings regarding effects of teacher professional development on student achievement, and the procedures proved useful to determine which studies and their results met high standards for scientific validity and reliability. A scientific research design can be efficiently employed to evaluate teacher professional development, and a design to measure effects of teacher development on subsequent student achievement should be strongly considered for each funded program for teacher and teaching improvement. The use of research designs involving treatment and control groups should become a regular practice and built into the plan and organization for professional development and other initiatives. Measures of implementation of professional development are critical to evaluation design in order to document and measure activities to reinforce and extend learning for teachers in their school setting. Multiple measures of student achievement should be included in the research design if possible to provide for different types of assessments of learning and analysis of subject content learned. State and local education data systems can be accessed by providers of professional development and evaluators and regular statewide or district-wide assessment instruments can be effective measures of outcomes. State leaders should ensure that data systems are structured so that data on teacher development initiatives can be linked to student achievement measures, and these data can be effective for evaluation even where individual identifiers are removed. Procedures for meta analysis modeled in this study provide a consistent, quantified methodology for application and use in other studies, including initial identification, multiple coding and validation of reviews. comparison of research design with established criteria, and consistent procedures for effect size analysis and coding of treatment variables. Meta Analysis Coding Forms The document review process for the meta analysis study is aided by coding forms that coders and reconcilers complete in order to record systematically how they determined a document is included into the pool of studies to be analyzed. The systematic review is conducted with at least two coders in mind, with a third person as a reconciler. The coding forms are Excel file documents composed of multiple spreadsheets that 1) assist in determining whether a document in question is a candidate to be included in the analysis and 2) aid in the extraction of key data needed for the analysis. Each coder completes one form per document independently from his/her partner coder. The reconciler completes another similar form that combines the information from both coders by bringing their entered information side-by-side. Below are two types of coding forms: the coding form and the reconciler form. Meta Analysis Coder Form Meta Analysis Reconciler Form The reconciler form is very similar to the coder form but with the added feature of having side-by-side sections for coders’ responses. Both forms are organized in the same manner. The first spreadsheet presents the scaffolded guide that assists a coder through the process of determining the viability of a document for entry into the pool of studies to analyze. At certain decision junctures, the coder is forced to consider whether the document should continue to the next round of reviews or should be rejected. A document could be rejected any of the decision points of the review process. Coders and the reconciler are asked to complete the subsequent spreadsheets to record the following: Table 1a to record student outcome measures and constructs to determine the validity and reliability of these instruments · Table 1b to record teacher outcome measures and constructs to determine the validity and reliability of these instruments · Figure 1a to document and calculate the number of teachers participating in the study, by treatment and control groups · Figure 1b to document and calculate the number of students participating in the study by treatment and control groups · Table 2 to log number of teachers matching various teacher characteristics such as education background, teaching experience, and results from outcome measures mentioned in Table 1b, by treatment and control groups · Table 3 to log number of students matching various student characteristics including grade level, gender, disability, poverty level, race/ethnicity, urbanicity, classroom type, achievement level and results from outcome measures mentioned in Table 1a, by treatment and control groups · Table 4 to record characteristics of the professional development initiatives, and · Tables 5a, b, c, and/or d to record and calculate estimates of treatment effects (effect sizes) by student outcomes using standard formulas. Note that cases arise where non-standard formulas not provided in these tables must be used to calculate effect sizes. The meta analysis coder and reconciler forms are modified from the coding form and reconciliation forms designed by Kwang Suk Yoon  of the American Institutes for Research AIR in a systematic document review they conducted on how teacher professional development affects student achievement.

Council of Chief State School Officers
One Massachusetts Avenue, NW · Suite 700
Washington, DC 20001-1431
voice: 202.336.7000 · fax: 202.408.8072

Legal Conditions · Feedback · Sitemap

document last updated 7/1/2009