500-600 words Article Summary. Please see the article and the instructions that attached. Due tomorrow. No Plagiarism.    

239

American Economic Review: Papers & Proceedings 100 (May 2010): 239–243 http://www.aeaweb.org/articles.php?doi=10.1257/aer.100.2.239

The charter school landscape includes a vari- ety of organizational models and a few national franchises. The nation’s largest network of char- ter schools is the Knowledge is Power Program (KIPP), with 80 schools operating or slated to open soon. KIPP schools target low income and minor- ity students and subscribe to an approach some have called No Excuses (Abigail Thernstrom and Stephen Thernstrom 2003). No Excuses schools feature a long school day and year, selective teacher hiring, strict behavior norms, and encour- age a strong student work ethic.

KIPP schools have often been central in the debate over whether schools alone can sub- stantially reduce racial achievement gaps. Descriptive accounts of KIPP suggest positive achievement effects (see, e.g., Jay Mathews 2009), but critics argue that the apparent KIPP advantage reflects differences between stu- dents who attend traditional public schools and students that choose to attend KIPP schools (see, e.g., Martin Carnoy, Rebecca Jacobsen, Lawrence Mishel, and Richard Rothstein 2005). There are few well-controlled studies of KIPP schools that might help sort out these competing claims, and none that focus on KIPP.1

This paper reports on a quasi-experimental evaluation of the only KIPP school in New England, KIPP Academy Lynn. KIPP Lynn opened in the fall of 2004 and is the only charter school in Lynn, Mass., a low income city north of

1 See, for example, the studies linked at http://www. kipp.org/01/independentreports.cfm. Caroline M. Hoxby and Sonali Murarka’s (2009) quasi-experimental study of New York City charters includes four KIPP schools. Will Dobbie and Roland G. Fryer Jr. (2009) use quasi-experi- mental methods to study charter schools in Harlem.

Inputs and Impacts in Charter Schools: KIPP Lynn

By Joshua D. Angrist, Susan M. Dynarski, Thomas J. Kane, Parag A. Pathak, and Christopher R. Walters*

Boston. KIPP Lynn is a middle school that serves about 300 students in grades 5–8. Like most other Massachusetts charter schools, KIPP Lynn is funded primarily through tuition paid by students’ home districts. Tuition is typically set to match sending districts’ average per pupil expenditure, though this is offset by state subsidies to the send- ing district when a student first transfers.

Each KIPP school sets its own curriculum, but KIPP Lynn shares many features with other KIPP schools. KIPP Lynn has a long school year, starting in August and including some Saturdays, and a long school day, running from 7:30 in the morning to 5:00 in the afternoon. Students are expected to conform to a behavioral code and receive “paychecks,” points awarded for good work that can be spent on field trips and other perks. Parents or guardians, students, and teachers are asked to sign a “Commitment to Excellence,” a promise to come to school on time and work hard, among other things.

As at other KIPP schools, KIPP Lynn’s non- union teachers work long hours and are expected to be on call to assist their students in the evening. Many KIPP Lynn teachers are graduates of the national Teach For America (TFA) internship pro- gram. KIPP Lynn teachers are much younger than middle school teachers in the Lynn Public Schools (LPS), and more likely to be African American.

Most KIPP Lynn students live in Lynn and would otherwise attend an LPS school. The pop- ulation of Lynn is more than two-thirds white, but 70 percent of the 13,000 students in LPS are nonwhite. This is documented in Table 1, the first two columns of which show average character- istics of fifth graders, the entry grade for KIPP Lynn. (Some applicants who have already com- pleted fifth grade enter KIPP Lynn by repeating a grade, as do a small number of sixth graders.) LPS fifth graders are 42 percent Hispanic, 17 percent black, and 11 percent Asian. KIPP fifth graders are 57 percent Hispanic and 24 percent black, with few Asians. Roughly 19 percent of students are classified as special education at both LPS and KIPP. LPS has a moderately

* Angrist, Pathak and Walters: MIT and NBER, (e-mail: angrist@mit.edu, ppathak@mit.edu, crwalt@mit.edu). Dynarski: University of Michigan and NBER (e-mail: dynarski@umich.edu). Kane: Harvard University and NBER (e-mail: kaneto@gse.harvard.edu). The authors are grate- ful to Josh Zoia and Esther Vargas at KIPP Academy Lynn, and to Carrie Conaway at the Massachusetts Department of Elementary and Secondary Education for data and assis- tance. We also thank Eric Bettinger for helpful comments.

MAY 2010240 AEA PAPERS AND PROCEEDINGS

higher proportion classified as having limited English proficiency (LEP), 22 percent versus 17 percent at KIPP. Many charter schools in the Northeast are disproportionately nonwhite, but KIPP Lynn is unusual in enrolling such a high proportion of Hispanic, LEP, and special educa- tion students. This is significant, because critics have charged that charter schools ignore or have been unsuccessful with these populations.2

Statewide regulations require Massachusetts charter schools to use a lottery when oversub- scribed. KIPP Lynn was undersubscribed when it opened in the fall of 2004 and only marginally oversubscribed in 2005. More recently, however, more than 200 students have applied for about 90 seats. The 2005-2008 admissions lotteries are used here to develop a quasi-experimental research design. These randomized lotteries allow us to estimate the causal effect of KIPP Lynn on

2 For example, this United Federation of Teachers report: http://www.uft.org/news/issues/uft_report-sepa- rate_and_unequal.pdf

achievement, solving the problem of selection bias that plagues most studies of school effectiveness.

I.  Data and Empirical Framework

A. Data

We started with lists of applicants to KIPP Lynn for 2005 through 2008. These four appli- cant cohorts were matched to the Massachusetts Student Information Management System (SIMS), a database with demographic and other information for all public school students in the state. We found 91 percent of KIPP applicants in the SIMS database, though winners are some- what more likely to be matched than losers.3 Applicants’ SIMS records were then matched

3 Controlling for year of application and whether the applicant comes from LPS, the match differential is about ten percent. Looking at LPS applicants only, however, the differential falls to 0.029 (SE = 0.027). An analysis using only applicants from LPS generates results almost identical to those reported here.

Table 1—Descriptive Statistics and Covariate Balance

Means for 2005–2008 Balance

regression Lynn 5th KIPP Lynn KIPP Lynn Winners graders 5th graders applicants vs. Losers

(1) (2) (3) (4) Hispanic 0.418 0.565 0.538 −0.052

(0.053) Black 0.173 0.235 0.254 0.027

(0.044) Asian 0.108 0.021 0.022 0.026*

(0.015) Female 0.480 0.474 0.484 −0.010

(0.054) Free/reduced price lunch 0.770 0.842 0.825 −0.030

(0.041) Special education 0.185 0.189 0.197 −0.013

(0.042) Limited English proficient 0.221 0.172 0.206 −0.075

(0.047) Baseline (4th grade) math score −0.307 −0.336 −0.390 0.097

(0.114) Baseline (4th grade) ELA score −0.356 −0.399 −0.438 0.054

(0.118) Fourth grade applicant 0.768 0.056

(0.046) p-value from joint F-test 0.671

Notes: Column (4) reports coefficients from regressions of the variable indicated in each row on an indicator variable equal to one if the student won an admissions lottery. Covariates include dummies for application year interacted with a contempo- raneous sibling applicant dummy as well as application grade. The regression sample is restricted to randomized applicants with baseline demographics.

VOL. 100 NO. 2 241INPutS AND IMPACtS IN ChARtER SChOOLS: KIPP LYNN

to fifth–eighth grade scores on Massachusetts Comprehensive Assessment System (MCAS) tests given in the Spring of 2006–2009. We stan- dardized math and English language arts (ELA) scores to a statewide reference population by subject, grade, and year. Excluding applicants with siblings already enrolled in KIPP (who are guaranteed a slot), late applicants (who miss the lotteries), unmatched applicants, and a few oth- ers leaves 457 in the lottery analysis sample.4 Of these, 69 percent were offered a spot at KIPP Lynn, and 54 percent enrolled.

Older cohorts contribute more follow-up data to our analysis than do more recent cohorts. For example, the 2005 applicant cohort was tested in fifth through eighth grades, while the 2008 cohort was tested in fifth grade only. The 2005 cohort is the only one observed in eighth grade. Lottery winners spent an average of about 1.85 years at KIPP Lynn in our sample period.

Table 1 reports descriptive information for samples of fifth graders attending LPS and KIPP Lynn during our study period, and for our sample of KIPP Lynn lottery applicants. LPS students are poor and mostly nonwhite, but students at KIPP Lynn tend to be poorer and are even more likely to be nonwhite. LPS students had low fourth grade math and ELA scores, about a third of a standard deviation below the state average. KIPP Lynn applicants and enrolled students were even further behind.

Lottery winners and losers are similar, a fact documented in column (4) of Table 1, which reports differences in demographic characteristics and baseline scores between winners and losers.5 By virtue of random assignment, we should expect these differences to be small. In practice, however, our reconstruction of the KIPP lotteries is based on spreadsheets with some missing information. We corrected most of the entries by reviewing the lottery records with school staff but may not have fixed the problem entirely. It is therefore especially encouraging that most pre-treatment

4 We look at first time applicants only. We also exclude applicants from sixth or seventh grade as few of these end up attending KIPP.

5 These are from regressions that control for year and grade of application, applicants’ sibling status (siblings who apply together are more likely to get in, since a winning sibling improves the loser’s position on the waiting list), and interaction of sibling status with year of application. Note applicants with siblings already enrolled are excluded from our sample.

characteristics come out balanced. Only one vari- able (proportion Asian) shows a marginally signif- icant difference between winners and losers. The F-statistic for a joint test of balance on all charac- teristics gives little cause for concern.

Although lottery winners and losers look similar at the time the lotteries were held, subse- quent attrition may lead to important differences in the follow-up sample if the attrition process is nonrandom (attrition here means exit to a private school or a move out of state). Of particular con- cern is whether we are more likely to have test score outcomes for winners than for losers. We observe math and ELA scores for about 85 per- cent of charter lottery losers. Lottery winners are about five points more likely to have follow-up scores, but these differences shrink and become insignificant with controls. These small follow- up differentials seem unlikely to impart substan- tial selection bias in our impact analysis.

B. 2SLS Strategy

We’re interested in the causal effect of KIPP Lynn on MCAS scores, where this effect is mod- eled as a function of time spent in KIPP. The equation of interest is

(1) yigt = αt + βg + ∑ j

δ j dij + γ ′ Xi

+ ρsigt + ϵigt,

where yigt denotes the scores of student i tested in year t in grade g. The variable sigt records years spent at KIPP Lynn as of the test date, counting any repeated grades. The (average) causal effect of interest is ρ. The terms αt and βg are year-of- test and grade-of-test effects, while Xi is a vector of demographic controls with coefficient γ, and ϵigt is an error term that captures random fluc- tuation in test scores. The dummies dij indicate three of the four KIPP Lynn application cohorts, indexed by j.

The first stage uses the randomly assigned lot- tery offers as an instrument for sigt and takes the form:

(2) sigt = λt + κg + ∑ j

μ j dij + Γ′Xi

+ πZi + ηigt,

where λt and κg are year-of-test and grade effects. The excluded instrument is the lottery offer

MAY 2010242 AEA PAPERS AND PROCEEDINGS

dummy Zi, with first stage effect π. The reduced form generated by this system comes from substi- tuting (2) for sigt in (1). The reduced form effect is the coefficient on Zi in a regression of yigt on Zi with the same controls and data structure as in equations (1) and (2). Because the model is just- identified, 2SLS estimates of ρ are given by the ratio of reduced form to first stage coefficients.

II.  Results

A. Lottery Results

The lottery first stage coefficient is a little over 1.2 years, as can be seen in the first column of Table 2. In other words, by the time they were tested, lottery winners had spent an average of 1.2 years more at KIPP than lottery losers. The addition of demographic variables and baseline scores has almost no effect on the first stage estimates.6 In a world with perfect lottery com- pliance, no late entry or grade repetition, and no loss to follow-up, the first stage in our sample would be 1.75, but this is reduced by the fact that some winners never enroll in KIPP or leave before finishing, and some losers end up in KIPP later.7 On the other hand, although KIPP schools

6 We report separate first stages for math and ELA because samples differ slightly by subject.

7 The 2005 cohort contributes one score after one year (in fifth grade), one after two years (in sixth grade), one after three years (in seventh grade), and one after four years (in eighth grade) for an average of 2.5 years in KIPP across

are sometimes said to encourage weaker stu- dents to leave (see, e.g., the discussion in chap- ter 46 of Mathews, 2009), we found that lottery winners were about as likely as losers to change schools in grades 6–8, a result discussed more fully in our working paper (Joshua D. Angrist, Susan M. Dynarski, Thomas J. Kane, Parag A. Pathak and Christopher R. Walters 2010).

Lottery winners score about 0.4 standard devi- ations (hereafter, σ) higher than losers on math, a robust, reduced-form result reported in column (2) in the top half of Table 2. The reduced forms for ELA, reported in the bottom half of the table, are more variable across specifications, ranging from 0.15–0.18σ as the set of controls varies. This variability probably reflects the modest imbal- ance between winners and losers in the propor- tion LEP documented in Table 1. The estimated effect on ELA is marginally significant in models with demographic and baseline score controls.

Because the first stage coefficients are larger than one, the 2SLS estimates are smaller than the reduced-form estimates, though they also have a different interpretation. The 2SLS esti- mates imply that math scores increase by about 0.35σ for each year at KIPP Lynn. The 2SLS estimates for ELA show per year gains on the order of 0.12–0.15σ. The most precise of these is 0.12σ, estimated in models with demographic and baseline score controls (standard error = 0.058). These effects are remarkably similar to the mid- dle school results in Atila Abdulkadiro  g lu et al. (2009) estimated using lotteries from a sample of charter schools in Boston. Measured against Lynn’s Hispanic-white score gaps of about 0.5σ in math and 0.6σ in ELA, both the math and ELA effects are substantial.

B. hispanic Subgroups and Interactions

KIPP Lynn serves more Hispanic students than is typical of charter schools in the Northeast. Table 3 looks at effects in the Hispanic and non- Hispanic subgroups. Concerns about access for Hispanics notwithstanding, effects are reasonably similar for Hispanic and non-Hispanic students, though ELA estimates for the latter are imprecise.

grades. A similar calculation for the other cohorts, which are seen in fewer grades, produces 2.0 potential years in KIPP for the 2006 cohort, 1.5 potential years in KIPP for the 2007 cohort, and 1 year of potential years in KIPP for the 2008 cohort. The average of these is 1.75.

Table 2: Lottery Results

First stage

Reduced form 2SLS

Controls (1) (2) (3) Panel A. Math (N=856 w/baseline scores) Basic 1.222*** 0.431*** 0.353***

(0.063) (0.116) (0.095) Demographics & 1.228*** 0.425*** 0.346*** baseline scores (0.066) (0.066) (0.052) Panel B. ELA (N=856 w/baseline scores) Basic 1.223*** 0.183 0.150

(0.063) (0.117) (0.094) Demographics & 1.234*** 0.149** 0.120** baseline scores (0.066) (0.073) (0.058)

Notes: Grades are pooled. Basic controls include those from the regressions in Table 1 as well as year-of-test and grade-of-test dummies. Robust standard errors (clustered at the student level) are reported in parentheses.

VOL. 100 NO. 2 243INPutS AND IMPACtS IN ChARtER SChOOLS: KIPP LYNN

Charter skeptics have sometimes argued that while relatively motivated and able students may benefit from charter school attendance, weaker students lose out. We briefly explored this type of treatment effect heterogeneity by estimating a model that adds an interaction between appli- cants’ baseline (fourth grade) scores and years at KIPP Lynn, normalized so that the main effect of years at KIPP Lynn is evaluated at the mean of the baseline score distribution. Panel B of Table 3 shows that KIPP Lynn raises achievement more for weaker students. Children with baseline scores half a standard deviation below the appli- cant mean appear to get a rough 0.06σ additional achievement boost from each year they spend at KIPP Lynn. This finding echoes a similar nega- tive interaction with baseline scores reported in our Boston study (Abdulkadiro  g lu et al., 2009).

III.  Concluding Comments

Our estimates suggest that KIPP Lynn gener- ated substantial score gains for lottery winners, with effects on the order of 0.35σ for math and 0.12σ for ELA. The characteristics of lot- tery winners and losers are similar and lottery based 2SLS estimates are insensitive to controls for demographic variables and baseline scores.

KIPP Lynn benefits Hispanic and non-Hispanic applicants about equally, with larger gains for students with low baseline scores. Our esti- mates for KIPP Lynn are also remarkably similar to those reported for Boston charters in Abdulkadiro  g lu et al. (2009). The results reported here are for a single school, but KIPP runs many similar schools across the country. The key elements of KIPP’s No Excuses model feature in other charter schools, such as those in our Boston sample. Our results suggest the major elements of this model combine to pro- duce noteworthy achievement gains, at least as measured on statewide standardized tests.

REFERENCES

Abdulkadiro  ̆     g lu, Atila, Joshua D. Angrist, Susan  M.  Dynarski,  Thomas  J.  Kane,  and  Parag  A.  Pathak.  2009. “Accountability and Flexibil- ity in Public Schools: Evidence from Boston’s Charters and Pilots.” National Bureau of Eco- nomic Research Working Paper 15549.

Angrist, Joshua D., Susan M. Dynarski, Thomas  J. Kane, Parag A. Pathak, and Christopher R.  Walters.  2010. “Who Benefits from KIPP.” National Bureau of Economic Research Work- ing Paper 15740.

Carnoy,  Martin,  Rebecca  Jacobsen,  Lawrence  Mishel,  and  Richard  Rothstein.  2005. the Charter School Dust-up: Examining the Evi- dence on Enrollment and Achievement. New York: Teachers College Press.

Dobbie,  Will,  and  Roland  G.  Fryer,  Jr.  2009. “Are High Quality Schools Enough to Close the Achievement Gap? Evidence from a Social Experiment in Harlem.” National Bureau of Economic Research Working Paper 15473.

Hoxby, Caroline M., and Sonali Murarka. 2009. “Charter Schools in New York City: Who Enrolls and How They Affect Their Students’ Achievement.” National Bureau of Economic Research Working Paper 14852.

Mathews, Jay. 2009. Work hard. Be Nice. Chapel Hill: Algonquin Books.

Thernstrom,  Abigail,  and  Stephen  Thernstrom.  2003. No Excuses: Closing the Racial Gap in Learning. New York: Simon & Schuster.

Table 3: Subgroups and Interactions

Math ELA (1) (2)

Panel A. hispanic Subgroups Hispanic 0.346*** 0.121

(0.074) (0.075) Non-Hispanic 0.331*** 0.086

(0.076) (0.099) Panel B. Baseline Score Interactions Main effect 0.367*** 0.139**

(0.054) (0.057) Interaction −0.106*** −0.157***

(0.041) (0.045)

Notes: Estimates are 2SLS coefficients analogous to those in Table 2. Panel B instruments years at KIPP Lynn and its interaction with baseline score using the lottery offer and offer × baseline score. Covariates include demographics and baseline scores.

Copyright of American Economic Review is the property of American Economic Association and its content

may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder’s express

written permission. However, users may print, download, or email articles for individual use.