The Council has been active in developing, testing, and applying two models in conducting alignment analyses:
SEC Model: Produces alignment analyses of standards, assessments, and instructional content by use of a content matrix or template for core academic subjects that allows comparison across schools, districts or states. The Council worked with Andrew Porter and John Smithson in testing and implementing the model with states.
Webb Model: Provides analysis of the degree of intersection of state assessments and content standards, which combines qualitative expert judgments and quantified coding and statistical analysis. The Council worked with Norman Webb in developing the model.
SEC Model
The Surveys of Enacted Curriculum (SEC) alignment methodology has been field tested and demonstrated with 11 states and four large urban districts. The model was developed by Andrew Porter (now at Vanderbilt University) and John Smithson from Wisconsin Center for Education Research. Development and application of the model was supported by CCSSO through grants from the National Science Foundation and through a state collaborative project. The SEC model produces alignment analyses of standards, assessments, and instruction by use of a common content matrix or template that allows comparison across schools, districts or states. The content matrix and language has two dimensions for categorizing subject content: Content Topics and Cognitive Demand (or expectations for student performance).
In this approach State Standards or assessments, as well as survey data on instruction, are systematically categorized according to a common framework of content topics by cognitive demand. The dimension of cognitive demands has five categories for mathematics: (a) memorize, (b) perform procedures, (c) communicate understanding, (d) generalize/prove, and (e) solve non-routine problems. For standards or assessment documents, four educators in the content area code each assessment item or benchmark into the two-dimensional matrix. To produce data on instructional content, surveys are conducted with teachers who report the time spent teaching content in their class over a one-year period using the same content matrix.
Highly accessible content maps and graphs are used to visually portray differences and similarities in content from instruction to standards to assessments, and written interpretations of the content charts are provided. Also, statistics of alignment for each grade and subject are computed. A new, Internet web-based survey, analysis, and reporting system will reduce time and costs and greatly increase accessibility to a variety of users. Alignment studies have focused on math and science, and, by early 2003, language arts/reading analysis will be available.
For further information on the SEC model or to review completed state alignment analyses, please contact Andrew Porter or John Smithson at WCER johns@education.wisc.edu, (608/263-4200 or 263-4354), or Rolf Blank, Rolfb@ccsso.org. Please refer to these websites for further details:
Webb Model
The model developed by Norman Webb from the University of Wisconsin, provides a reliable set of procedures and criteria for conducting alignment analysis studies, which combine qualitative expert judgments and quantified coding and analysis of standards and assessments. The product of the analysis is a set of statistics for each standard and grade on the degree of intersection, or alignment, between the content embedded in state content standards and the content in state assessments. The Webb model has been used in alignment studies with more than 10 states, partly through SCASS collaborative projects managed by CCSSO. The model has been used for language arts, mathematics, science, and social studies.
Following the training process, four to six reviewers, including teachers and content specialists, individually identify the content standard objectives that match each assessment item. They first determine the 'depth of knowledge' required by each objective or benchmark of the content standards being analyzed, and code each using one of four levels of knowledge: (a) recall, (b) skill/concept, (c) strategic thinking, (d) extended thinking. Operational definitions and labels vary somewhat by subject. Second, reviewers determine the objective or benchmark represented by each item or task on the state assessment being reviewed, and they rate the level of knowledge necessary for a student to successfully complete the item or task.
The results for each reviewer are entered into a spreadsheet by tracking the corresponding objectives for each item and if the level of knowledge of the item is below, at, or above the level of knowledge of the corresponding objective. The content ratings and codes are statistically analyzed across the reviewers to produce statistics and tabular reports on four criteria of alignment for each standard: 1) categorical concurrence, 2) depth-of-knowledge consistency, 3) range of knowledge correspondence, and 4) balance of representation. Reliability among reviewers in assigning levels of knowledge to items has been relatively high, generally ranging from .6 to .9. The criteria and measures were developed by a multi-disciplinary panel selected and convened by Webb and CCSSO.
For further information on the model, contact Norman Webb (608/263-4287), email nlwebb@factstaff.wisc.edu, or to review the report, see AlignmentPaper.pdf.
Achieve Model
The Achieve model provides an in-depth qualitative and quantitative analysis on the alignment of assessments to state standards. A number of states have worked with Achieve using this approach. In the Achieve model (protocol), a panel of content experts judges the degree of alignment between assessment items and standards using five criteria: (a) content centrality, (b) performance centrality, (c) challenge, (d) balance, (e) range.
To determine content and performance centrality, reviewers compare, in turn, the content and the performance, required by an individual assessment item to that of the related standard and assign each item to one of four categories for both content and performance centrality based on the degree of alignment. To evaluate challenge, reviewers analyze each item for source of challenge to ensure that the item is "fair," and for level of demand-rating the item on a scale of 1 to 4. They then examine all the items that relate to a particular standard, such as algebra, and judge the overall level of challenge of the item set. To evaluate balance, reviewers compare the extent to which the content delineated in the standards receives the same emphasis on the related item set and if that emphasis is appropriate.
Reviewers also compute the range (the proportion of objectives explicating a standard that are assessed by at least one item) of each item set as a simple check on coverage. Before reviewers apply the protocol they confirm the test developer's blueprint, verifying that each item generally corresponds to at least one standard. Achieve provides quantitative data on the test blueprint, content and performance centrality, source of challenge and level of demand, as well as written commentary on overall patterns, including the level of challenge and balance for each standard and for the test as a whole.
Achieve also provides four additional services: Standards Benchmarking in which content experts compare a state's standards to exemplary state and international standards (This service is often coupled with alignment studies so states receive a comprehensive review of their standards and assessment system); Augmentation Analysis in which experts compare "off-the-shelf," norm-referenced tests to a state's standards and recommend ways to customize the test to improve alignment and conform to NCLB; Professional Development for state educators to build their capacity to conduct their own alignment studies; and Policy Audits in which an expert panel examines state documents and conducts a site visit to determine the effectiveness of a state's reform efforts, identifying strengths and areas for improvement. Future policy reviews will also focus on helping states maintain the quality of their reforms, while implementing NCLB.
For further information and examples of standards benchmarking, alignment analysis and policy studies, refer to the Achieve website, http://www.achieve.org, or contact Jean Slattery, Director Benchmarking Initiative, jslattery@achieve.org.
