The Tests Used Since 2013 Are Designed to Be More Challenging Than Previous State Tests

The State Education Department has consistently emphasized that the tests introduced in 2013 are designed to hold students to a much higher level of proficiency than the previous testing program. A recent report in Education Next cited New York State among the top 10 states to set its proficiency bar at a level that is equivalent to or tougher than the standard set by the National Assessment of Educational Progress (NAEP)—long considered the gold standard of assessment of student achievement across the country. The higher standards imposed in 2013 need to be considered when parsing both state and city proficiency results.

It is useful to compare the results of the state test to those of the New York State results on the 2015 NAEP—administered in grades 4 and 8. In English Language Arts (ELA), proficiency rates on the state grade 4 ELA were 3 points lower than on the NAEP Reading; in grade 8, the state test results were 2 points higher than NAEP. In math, the state test results were 8 points higher in grade 4. The results of the state’s grade 8 math scores are not comparable to NAEP because the state test offers an exemption to high achievers who take a Regents exam instead.

Controlling for Differences in Student Characteristics, NYC Schools—Traditional Public and Charter—Are Doing Better Than Those in the Rest of the State

Based on the 2015 New York State test results for grades 3 through 8 for each traditional public and charter school in the state, New York City appeared to lag behind schools in the rest of the state in terms of raw proficiency rates for ELA and math. In ELA, 31.2 percent of students in the city’s traditional public and charter schools tested proficient, whereas 32.5 percent of students in the rest of the state tested proficient. In math the gap was greater: 37.1 percent of students in New York City schools tested proficient compared with 40.5 percent of students in the rest of the state. Comparing these proficiency rates, test results for the city’s students had proficiency rates that were 1.3 percentage points to 3.4 percentage points below those for students in the rest of the state.

These raw differences, however, do not account for the fact that test results for schools are strongly related to the demographic characteristics of their students. By controlling for these differences statistically, we can estimate school performance in light of the demographic differences across schools or groups of schools. This allowed us to compare the relative performance of New York City’s schools, including the Department of Education’s traditional public schools and charter schools, to those in the rest of the state. The results showed that New York City schools are actually doing better than schools in the rest of the state.

IBO analyzed the ELA test results for all 3,553 public schools (district and charter) across New York State, where we had complete demographic data; for our analysis of math test results the sample was slightly smaller, consisting of 3,546 public schools. The demographic variables we controlled for were: the share of students who are economically disadvantaged; the share of students with disabilities; ethnicity, measured as the share of Asian, black, and Hispanic students relative to whites; and gender, measured as the share of females.

With these controls in place, the model was able to account for 68.1 percent of the variation in school ELA proficiency rates and 62.4 percent of the variation in math proficiency rates. All seven variables had a statistically significant effect on ELA and math proficiency rates, although the percent female was only statistically significant at the 10 percent level for math proficiency rates (see table below).

The impact of being a school in New York City was both considerable and statistically significant. After controlling for demographics, we found a 14.1 percentage point advantage in ELA and a 15.2 percentage point advantage in math for the average New York City public school, including charter schools, over the average school in the rest of the state. While the direction of the effects for the demographic variables was similar for ELA and math (positive versus negative), the effects were larger in magnitude in math for all demographic groups except for females (which was only marginally significant).

Charter Schools in New York City Do Better Than Traditional Public Schools

Once again controlling for student characteristics, IBO also identified the New York City advantage based on whether students attended a charter school or a traditional public school. Again, performance was gauged against schools in the rest of New York State (all schools outside of the city). After controlling for demographics, traditional public schools in New York City have a 13.1 percentage point advantage over the rest of the state on ELA and a 12.5 percentage point advantage on math proficiency. The gaps are larger for charter schools in the city: 18.8 percentage points on ELA and 30.1 percentage points on math.

As is the case with traditional public schools, there is a wide range of performance across charter schools. Among the 20 charter schools in new York City with ELA proficiency rates above 50 percent, 12 are part of the Success Academy network, four are part of the ICAHN network, three other schools are each part of different networks (Family Life Academy Charters, Classical Charters, and Uncommon Schools), and one is an independent charter (The Neighborhood Charter School of Harlem). The 20 New York City charter schools with math proficiency rates above 70 percent are mostly the same schools as those with high ELA rates. Among the other highest-achieving charter schools in math are Children’s Aid College Prep and Leadership Prep Ocean Hill.

Schools at the Very Lowest Achievement Levels Are Found in Every Part of the State and Include Traditional Public and Charter Schools

A considerable number of schools remain at the lowest levels of achievement. Across the state, 329 schools have ELA proficiency rates below 10 percent. Of these, 161 are traditional public schools in New York City. District-run schools in the rest of the state include 154 such schools. Finally, 14 charter schools are among these low-achievement schools, including 7 located in New York City.

NAEP Trends Also Indicate That New York City Has Narrowed the Gap With New York State, Especially for Eighth Graders

Despite a dip in fourth grade NAEP proficiency rates for New York City schools in 2015, the gap between New York City and New York State has narrowed over time, most significantly for eighth graders. Note that for NAEP results, we are not able to control for demographic differences, so these results are as reported. The gap in eighth grade reading proficiency rates has been halved over 12 years—from 13 percentage points to 6 percentage points. The gap in eighth grade math proficiency rates has shrunk even more—from 12 percentage points to 4 percentage points. The gaps for fourth graders shrank by much less—2 percentage points in reading and 3 percentage points in math.

Importantly, New York State is found to be at or around the national average on the NAEP. Thus, while the New York City proficiency rates are low when compared with where the state has decided students need to be, it is also true that on the ELA test, city students largely match the results for New York State, and the NAEP tells us that New York State is performing one point above the national average in grade 4 ELA and at the national average in grade 8. For math, the city performs slightly below the state, and the state’s NAEP results are 4 points below the national average in grade 4 and 1 point below in grade 8.

Conclusion

Comparing the proficiency rates for students in New York City with those of students in all of New York State over time shows that the city has considerably narrowed the gap with the state, although the city still trails. This trend is evident when looking at both the state tests and on the NAEP. When we control for demographic differences across districts in New York State, however, we find that the city is actually doing better than the rest of the state on the state tests.

Disaggregating the effects for New York City traditional public schools from charter schools, we find that charter schools have a greater advantage than district-run schools, relative to the rest of the state, particularly in math proficiency. Among both charter schools and traditional public schools, it is important to note that performance on the state tests varies widely within those groups. This variability, in terms of both performance and the demographics of students they serve, are important considerations in efforts to replicate and scale up high-achieving schools in both the Department of Education and charter sectors.