Study of Instructional Improvement

Study Sample & Components

The Study Sample

The Study of Instructional Improvement sought to recruit 120 schools into the study:  30 schools from each of the three school reform programs (SFA, AC, & ASP) as well as an additional sample of 30 control schools. To obtain a sample schools, SII used a variety of data, including complete lists of schools participating in each of the CSR programs under study as of the 2000-2001 school year.  Additional data on schools from this list and potential “control group” schools was then obtained through the Quality Education Data (QED) database, a commercially available database.This data base was used as the primary source for the identification of control schools in the study population. QED data for both treatment and control schools were also matched to the National Center for Education Statistics (NCES) Common Core Database (CCD) for district level data, the 1990 Census for socio-economic data, and to school listings provided by school reform programs to identify the student population. The school-level data are available for download here.

Schools were selected for the study in four steps. First, a list was compiled of all U.S. public elementary schools that had begun their affiliation with ASP, AC, or SFA in the 1998-1999, 1999-2000, or 2000-2001 school years. Initial inspection of this list indicated that schools participating in these programs were widely dispersed across the country. For cost purposes, it was necessary to identify geographic regions around the country which contained concentrations of schools in the three programs, thus minimizing data collection travel.

In the second step, a set of 17 geographic regions was selected from which to sample schools. Regions were selected using ArcView®, a geographic information systems (GIS) program, to plot intervention schools on maps. Geographic regions were identified by drawing one hundred mile radii around zip codes containing program schools and by visually inspecting maps on which these radii and the program schools contained within them were plotted (most of the study regions roughly correspond with U.S. Census Bureau standard metropolitan statistical areas).

In the third step, intervention schools from the 17 geographical regions were selected. We attempted to balance the samples of schools from the intervention programs in two ways. First, an attempt was made to equalize the samples with respect to the length of time sample schools had been affiliated with the three programs. The study targeted equal numbers of schools from each program for each initial year of program affiliation, 1998-99, 1999-00 and 2000-01. An attempt was also made to “equate” selected schools from the three programs with respect to socioeconomic disadvantage. This was done by first classifying schools on a three-point index of socioeconomic disadvantage (described below), and then targeting equal numbers of schools from each program from each category of the index.

In the third step, intervention schools from the 17 geographical regions were selected. We attempted to balance the samples of schools from the intervention programs in two ways. First, an attempt was made to equalize the samples with respect to the length of time sample schools had been affiliated with the three programs. The study targeted equal numbers of schools from each program for each initial year of program affiliation, 1998-99, 1999-00 and 2000-01. An attempt was also made to “equate” selected schools from the three programs with respect to socioeconomic disadvantage. This was done by first classifying schools on a three-point index of socioeconomic disadvantage (described below), and then targeting equal numbers of schools from each program from each category of the index.

In the final step, a set of “comparison” schools was chosen from within the 17 geographical regions. In addition to coming from the same geographical areas as selected intervention program schools, comparison schools were also selected so that their distribution on the three-point disadvantage index matched that of selected intervention program schools. Our sampling efforts yielded 115 schools located in 45 different school districts, in 15 different states, and in 17 different metropolitan areas. Overall, 31 AC schools, 30 SFA schools, 28 ASP schools, and 26 Comparison schools participated.  The schools were chosen to balance the sample, as much as possible, in terms of geographic location, school demographic characteristics, and years working with the CSR program, as well as to achieve a representative sample of schools participating in each CSR program. By design, however, the final sample over-represented schools in the highest quartile of socio-economically disadvantaged schools in order to study instructional improvement in high-poverty settings.

The reader may also view Table 1, to see how the SII sample compared to the nationally representative sample of the Early Childhood Longitudinal Study (ECLS). The data show that the SII sample included a substantially higher percentage of African-American children (49.8%) compared to ECLS (15.7%). The largest demographic group in the ECLS sample is White elementary-aged children (57.3%), while the SII sample includes less than half that percentage (23.1%) of Whites. Table 2 also shows that the SII sample included a higher percentage of children whose mothers did not complete high school (22.9% vs. 15.1%). Moreover, 21.7% of mothers in the ECLS sample reported obtaining a bachelor’s degree or higher, while only 9.9% of mothers in the SII sample obtained a comparable level of education. Table 2 also shows differences in family structure between the ECLS and SII sample. In ECLS, 64.3% of respondents indicated that both a mother and a father were present in the household, while only 40.8% of SII parents reported this traditional structure. Moreover, the SII parent survey respondents indicated that 41.9% of the sampled children come from single mother households, while only 21.3% of the ECLS children lived in this household arrangement. Disparities between the SII and ECLS samples are also evident in reported total family income. In every income category ranging from $0 to $39,999, the SII students were represented in somewhat larger percentages compared to the ECLS sampled students. And in the higher income categories ranging from $40,000 through $200,000+, the SII students were represented in consistently lower percentages compared to the ECLS student sample.  The SII family background information in Table 2 comes mostly from the Parent Questionnaire data (available here) which was gathered using the Parent Survey protocol (available here).

Propensity Score Stratification Methods

Although SII was designed as a quasi-experiment with three “treatment” groups (one for each CSR program) and a matched control group, Table 2 showed that, after the SII sample was constructed, subsamples of treatment and comparison schools were not perfectly matched.  For this reason, we recommend that statistical analyses oriented to establishing the effects of “treatment” (CSR) participation on outcomes using “control” schools as the counterfactual be conducted using some form of propensity score matching.  An excellent discussion of this approach to causal analysis can be found in Rosenbaum and Rubin (1983).

In SII, a multi-step process was used to develop a propensity score model for these kinds of data analyses. An example of that approach is provided here and comes from our analyses of CSR program effects on student achievement.  The data file used to conduct the propensity score analysis can be found here. Readers wishing to conduct secondary analyses of SII data are welcome to use this data file and the propensity scores contained in it to conduct such analyses.

The Survey Components

The Teacher Questionnaire (TQ) (available here) was a 28-page booklet and respondents were expected to take approximately one hour to complete it. Questions were primarily closed-ended. The TQ asked questions about the teacher’s perspective of the school and its faculty. In addition, teachers who taught language arts or math as part of their assignment were asked to complete language arts and / or math sections, respectively. The subject sections asked questions about teaching practices and priorities, as well as pedagogical content knowledge (PCK) questions. The pedagogical content knowledge questions, sometimes referred to as content knowledge for teaching (CKT) measures by SII researchers, are described in more detail here. Teachers were also asked questions about their experiences with school improvement efforts, professional development opportunities, demographic information, and their professional background. Over the course of four years, data was gathered from over 5,300 teachers. The TQ data are available for download here.

The School Leader Questionnaire (SLQ) (available here) was a 20-page booklet. School leaders were expected to take approximately 45 minutes to complete the questionnaire. The school leaders surveyed included principals, assistant principals, subject area coaches, and program coordinators. The SLQ was designed to capture information about the school and instructional improvement programs adopted by the participating sites. Questions asked about the programs the school was participating in, formal leadership roles of the respondent, respondent priorities, as well as the respondent’s assessment of the availability of resources, satisfaction levels among students and staff, and school priorities. School leaders were also asked to evaluate the state of the language arts and math programs in the school. This first set of domains encompasses how principals understand their role, what responsibilities fall to their charge, how they prioritize these responsibilities, and what they actually do (i.e. what their practice is). Over the course of the study, data was gathered from over 800 leaders. The SLQ data are available for download here.

Data on literacy and mathematics instruction were gathered from separate logs for Language Arts and Mathematics (available here)that were administered to all teachers of cohort students. The log is a survey instrument containing roughly 100 items that teachers used to record information about a single day of instruction for a single student.  The opening section of the log asked teachers to report on the amount of emphasis given to major topics. In language arts: (1) word analysis, (2) concepts of print, (3) comprehension, (4) reading fluency, (5) vocabulary, (6) writing, (7) grammar, (8) spelling, and (9) research strategies. In mathematics: (1) number concepts, (2) operations, (3) patterns, functions, or algebra, and (5) other mathematical content. If teachers checked a major emphasis topic for a student on a given day, they then completed additional items about the specific content that was taught in any checked domain, the methods used to teach that content, and the tasks and materials the focal student used that day. To assure that log reports were representative of days of the school year and all cohort students in a classroom, teachers were asked to participate in three extended logging periods spaced evenly over the academic year.  During each logging period, teachers rotated daily log reports across the sample of cohort students in their class.  If students changed teachers during the course of the year (as many SFA students did), their new teachers completed logs. The log data are available for download here.

The Parent Survey (available here) component consists of interviews with parents whose children were active participants of the SII. The survey includes questions about each child’s home environment and activities, the child’s experiences with school, services provided, basic demographic information and questions about the family’s access to basic needs. Most of the interviews were conducted by telephone, but where necessary, interviewers in the field were conducted face-to-face interview sessions. Most parent interviews were conducted in the second year of the study. However, additional interviews were conducted in the third and fourth years of the study to capture information for new students moving into a sample school and to obtain data from parents not reached in year two. Approximately 6,700 interviews were completed and the data are available for download here. Although not a primary section of the report, the Parent Survey file was central in the development of demographic controls used in our analytic work. Readers may find SII examples for coding a socioeconomic status composite measure, determining home structure from the family roster, and procedures used for assigning parent occupational codes here.

As kindergarten students (Cohort A) began the study they took the Letter/Word identification and applied problems sections of the Woodcock-Johnson Tests of Achievement - Revised. It was expected for these sections to take approximately 15 minutes to complete. In the spring, these same kindergarten students completed the TerraNova Level 10 assessment. The TerraNova is a nationally recognized assessment instrument. These assessments were given one-on-one to first grade students and conducted in a group with second through fifth grade students. It was expected that the TerraNova would take approximately 45-60 minutes to complete in one-on-one sessions. Each fall and spring, Cohort B (3rd grade through 5th grade) participating students completed the TerraNova assessment for their grade or performance level. Due to copyright limitations, SII cannot provide copies of the test batteries but the assessment data files are available for download here.

The Student Motivation Form (SMF) (available here) is designed to provide information from the student on how the student perceives him/herself in academic interests or skills. The SMF form, or self-description is administered each spring as part of the assessment. This SMF asked children to report on how much they enjoy reading and mathematics, how easy or hard reading and mathematics are for them, and any behaviors with which they might struggle that may also interfere with their learning. The SMF was administered individually to kindergarten through 2nd graders in an easel format, and an assessor recorded the students’ answers on a single-sided scan answer form. The form was administered in small groups to 3rd through 5th graders. In this case, students each had their own 2-sided scan answer sheet. An assessor read the instructions from a card and the students filled in their own answers. Both individual and group administrations take approximately 10 minutes.

Teachers were asked to complete a Student Rating Form (SRF) (available here) for each student for whom they filled out an instructional log. The SRF instrument gathers information on a student’s academic engagement, approaches to learning, and problem behaviors (if any). The form also contains several Yes/No questions about the student’s participation in Title 1 programs and other school services. Unlike the instructional logs, the SRF did not need to be completed on specific days for the specific target students. Data files for both the SMF and SRF are available here. Although not a major section of the report, example measures developed by SII researcher may be viewed here.

The Study of Instructional Improvement used multiple sources to assemble school-level information. SII used the Quality Education Data (QED) database (a commercially available database) the NCES Common Core Database (CCD), the School Characteristics Inventory (available here) , and Parent Survey data responses aggregated to the school level. The school-level data are available for download here. We recommend using the SII school composite data file for most purposes.