The curriculum surveys and data analysis services being offered were developed throughCCSSO’s State Collaborative on Assessment and Student Standards (SCASS) on Surveys of Enacted Curriculum, and through research and development conducted by Porter and Smithson at the Wisconsin Center. The surveys are based on state and national standards for content and teaching, as well as prior well-tested survey instruments, including TIMSS, NAEP, and National Survey of Science and Math Education (Horizon Research). Survey instruments were thoroughly field tested to ensure reliability and validity of the data.

A large field study was conducted with schools and teachers in 11 states to test and improve both the survey instruments and the data reporting methods and formats. More than 600 teachers completed self-report surveys that covered the subject content they taught and the instructional practices they used in their classes. From 2001-04, a CCSSO-led team conducted an experimental design study with 40 urban middle schools to develop and test a professional development model for improving instruction in math and science through use of Data on Enacted Curriculum (Blank, 2004).

Teachers participating in the Surveys report on a full school-year of teaching in English language arts, science or math. The data are analyzed and reported using pre-designed and tested charts and graphs found to be accessible and useful to educators as well as researchers. The survey method was validated by comparing teacher survey results, which require recall of instructional content and practices for a full school-year, with teacher reports using daily logs, independent observation, and teacher survey reports. This study found that data reported about curriculum content in teacher surveys covering a whole year were highly correlated with the data from daily logs of instructional content (Smithson and Porter, 1994).

Findings from Research and Development

Results from existing studies with Surveys of Enacted Curriculum show that educators can use the surveys and methods of analyzing data to address several kinds of problems and needs (Blank, et al., 2001; Porter, 2002; Blank, 2004). Specifically, the research studies show that SEC instruments and reports do address educators’ needs for comparable, reliable data and analyses of math and science instruction. The results show that

• the data demonstrated a high degree of variation in instructional time, practices and content of math and science from state to state, from school to school, and within schools
• curriculum taught in math and science differed according to amount of standards-based professional development in schools and level of implementation of state reform initiatives
• the method of surveying teachers using a “content matrix” based on standards, and then analyzing content of assessments using the same tool, showed strong potential for application in districts and states for measuring progress of standards-based math and science
• the comprehensive design of the surveys proved useful in analyzing the relationship of curriculum content and pedagogy to level of preparation of teachers, professional development, and school conditions for teaching

The following papers and reports address specific questions about survey reliability and validity. 

How valid are the self-report survey data provided by teachers? (Yes, survey responses on SEC have been analyzed in comparison to observational studies, as well as validated through interviews, focus groups, and student surveys.)   

• Blank, Rolf; Porter, Andrew; & Smithson, John. New Tools for Analyzing Teaching, Curriculum and Standards in Mathematics & Science: Results from Survey of Enacted Curriculum Project, 2001.
• Porter, Andrew, "Measuring the Content of Instruction: Uses in Research and Practice," AERA Presidential Address, Educational Researcher, October 2002.
• Porter, Andrew and Smithson, John,  "Defining, Developing, and Using Curriculum Indicators," Consortium for Policy Research in Education, December 2001.
• Blank, Rolf; Smithson, John; Jennifer, Local superintendent reports on use of SEC data, February 2008

What are the reliability of survey items and alignment analyses? (A high level of reliabiltiy has been found in test-retest statistical analyses and inter-rater reliability analyses of alignment content scoring.)

• Porter, Andrew and Smithson, John, "Alignment of Assessments, Standards, and Instruction Using Curriculum Indicator Data," Address, NCME Annual Meeting, 2002.
• Blank, Rolf, "Using Surveys of Enacted Curriculum to Advance Evaluation of Instruction in Relation to Standards," Peabody Journal of Education, Fall 2002.
• Smithson, John,  "Exploring Relationships between Professional Development, Teacher Practice, and State Content Standards," Address, American Institutes for Research, 2005. 
• Blank, et.al., Using Data on Enacted Curriculum in Mathematics & Science: Sample Results from a Study of Classroom Practices and Subject Content, 2000.

  Garet, Mike, Yoon, Kwang S. and Porter, Andrew,  "Measuring Differences in the Quality of Professional Development,” Address,  American Institutes for Research, 2005.

What is the predictive validity of the Survey responses? (The alignment between teacher reports of curriculum content taught and assessment content explain over 50% of variance in student achievement scores.)

• Porter, Andrew, "Measuring the Content of Instruction: Uses in Research and Practice," AERA Presidential Address, Educational Researcher, October 2002.
• Gamoran, Adam; Porter, Andrew; Smithson, John; and White, P.A., "Upgrading High School Mathematics Instruction: Improving Learning Opportunities for Low-Achieving, Low-Income Youth," Educational Evaluation and Policy Analysis 19(4), Winter 1997.  [Validation research with SEC--effects of aligned instruction on achievement]

What is validity of Survey data for use by educators in improving instruction?

• Blank, Rolf, Data on Enacted Curriculum Study: Summary of Findings, 2004. 
• Smithson, John, "Analyzing Instructional Content," WCER Research Highlights, Spring 2004.
• Nunnaley, Diana, "Test Scores: What Can They Tell Us?" Hands-On, TERC, V.27, N.1, Spring 2004.