This report reflects the contributions of many institutions and individuals. We would like to first thank the study funders. The Institute of Education Sciences of the U.S. Department of Education and the Smith Richardson Foundation funded the evaluation component of the study. Funders of the interventions included the Heinz Endowments, the W.K. Kellogg Foundation, the Grable Foundation, the Institute of Education Sciences, the Ambrose Monell Foundation, Barksdale Reading Institute, the Haan Foundation for Children, the Richard King Mellon Foundation, the Raymond Foundation, and the Rockefeller Foundation. We also thank the Rockefeller Brothers Fund for the opportunity to hold a meeting of the Scientific Advisory Panel and research team at their facilities in 2004.
We gratefully acknowledge Audrey Pendleton of the Institute of Education Sciences for her support and encouragement throughout the study. Many individuals at Mathematica Policy Research contributed to the writing of this report. In particular, Mark Dynarski provided critical comments and review of the report. Micki Morris and Daryl Hall were instrumental in editing and producing the document, with assistance from Donna Dorsey and Alfreda Holmes.
Important contributions to the study were received from several others. At Mathematica, Nancy Carey, Valerie Williams, Jessica Taylor, Season Bedell-Boyle, and Shelby Pollack assisted with data collection, and Mahesh Sundaram managed the programming effort. At the Allegheny Intermediate Unit (AIU), Jessica Lapinski served as the liaison between the evaluators and AIU school staff. At AIR, Marian Eaton and Mary Holte made major contributions to the design and execution of the implementation study, while Terry Salinger, Sousan Arafeh, and Sarah Shain made additional contributions to the video analysis. Paul William and Charles Blankenship were responsible for the programming effort, while Freya Makris and Sandra Smith helped to manage and compile the data. We also thank Anne Stretch, a reading specialist and independent consultant, for leading the training on test administration.
Finally, we would particularly like to acknowledge the assistance and cooperation of the teachers and principals in the Allegheny Intermediate Unit, without whom this study would not have been possible.
F. TIME-BY-INSTRUCTIONAL-ACTIVITY ANALYSES 46
G. TEACHER REPORTS OF STUDENTS’ HOURS OF READING
IV Impact Analysis 53
A. ESTIMATION METHOD 53
B. INTERPRETATION OF IMPACTS 58
C. CONTEXT OF THE IMPACTS 59
D. IMPACTS FOR THIRD-GRADE STUDENTS 61
E. IMPACTS FOR FIFTH-GRADE STUDENTS 63
F. IMPACTS FOR SUBGROUPS OF THIRD AND FIFTH GRADERS 63
G. DO THE INTERVENTIONS CLOSE THE READING GAP? 67
A: DETAILS OF STUDY DESIGN AND IMPLEMENTATION A-1
B: DATA COLLECTION B-1
C: WEIGHTING ADJUSTMENTS AND MISSING DATA C-1
D: DETAILS OF STATISTICAL METHODS D-1
E: INTERVENTION IMPACTS ON SPELLING AND CALCULATION E-1
F: INSTRUCTIONAL GROUP CLUSTERING F-1
G: PARENT SURVEY G-1
H: TEACHER SURVEY AND BEHAVIORAL RATING FORMS H-1
I: INSTRUCTIONAL GROUP VIDEOTAPE ANALYSIS I-1
J: VIDEOTAPE CODING GUIDELINES FOR EACH READING
K: SUPPORTING TABLES K-1
L: SAMPLE TEST ITEMS L-1
M: IMPACT ESTIMATE STANDARD ERRORS AND P-VALUES M-1
N: ASSOCIATION BETWEEN INSTRUCTIONAL GROUP
HETEROGENEITY AND THE OUTCOME N-1
O: TEACHER RATING FORM O-1
P: SCHOOL SURVEY P-1
Q: SCIENTIFIC ADVISORY BOARD Q-1
According to the National Assessment of Educational Progress (U.S. Department of Education 2003), nearly 4 in 10 fourth graders read below the basic level. Unfortunately, these literacy problems get worse as students advance through school and are exposed to progressively more complex concepts and courses. Historically, nearly three-quarters of these students never attain average levels of reading skill. While schools are often able to provide some literacy intervention, many lack the resourcesteachers skilled in literacy development and appropriate learning materialsto help older students in elementary school reach grade level standards in reading.
The consequences of this problem are life changing. Young people entering high school in the bottom quartile of achievement are substantially more likely than students in the top quartile to drop out of school, setting in motion a host of negative social and economic outcomes for students and their families.
For their part, the nation’s 16,000 school districts are spending hundreds of millions of dollars on often untested educational products and services developed by textbook publishers, commercial providers, and nonprofit organizations. Yet we know little about the effectiveness of these interventions. Which ones work best, and for whom? Under what conditions are they most effective? Do these programs have the potential to close the reading gap?
To help answer these questions, we initiated an evaluation of either parts or all of four widely used programs for elementary school students with reading problems. The programs are Corrective Reading, Failure Free Reading, Spell Read P.A.T., and Wilson Reading, all of which are expected to be more intensive and skillfully delivered than the programs typically provided in public schools.1 The programs incorporate explicit and systematic instruction in the basic reading skills in which struggling readers are frequently deficient. Corrective Reading, Spell Read P.A.T., and Wilson Reading were implemented to provide word-level instruction, whereas Failure Free Reading focused on building reading comprehension and vocabulary in addition to word-level skills. Recent reports from small-scale research and clinical studies provide some evidence that the reading skills of students with severe reading difficulties in late elementary school can be substantially improved by providing, for a sustained period of time, the kinds of skillful, systematic, and explicit instruction that these programs offer (Torgesen 2005).