My school has a strong data driven culture. We use data not just to identify grade level or class wide trends, but trends with individual students. Because it’s not enough for me to know that the entire class scored, say, an 80 percent on a specific assessment. I need to be able to identify high, middle and low performers on each assessment and reflect on what has made them successful. This is a wonderful practice because it helps me identify what the strong performers are doing well and consider how I can use this information to help the weaker performers improve.

I think this offers a perfect analogy for what we need to be doing with charter schools here in Tennessee.

Last week I wrote a profile of charter schools in Tennessee with the purpose of establishing that collectively charters in Tennessee are in fact the real deal. They are doing very well in the aggregate compared to traditional public schools. From a public policy perspective this is a vital question to answer because it informs our view about where best to spend our limited public resources.

But its not enough just to know that collectively we’re doing well. Just like in my classroom, it’s important to be able to identify who is doing well so that we can study and potentially replicate their success, especially when it comes to serving students from a low income background.

This piece looks at charter schools compared to three different groups of schools to identify which individual schools are best serving this student population. When we look closely at charters, particularly high schools here in Memphis, it turns out that we have a lot to be excited about.

Summary of Methods (a more detailed explanation can be found at the end)

To create a valid measure, I first decided to compare individual Memphis charter high schools to traditional high public schools using data from the Tennessee Department of Education’s 2013 Report Card. The report card allows anyone to separate out achievement data by category, in this case of economically disadvantaged students only (economically disadvantaged is defined as students living at 185 percent or below of the poverty level).

I can then compared each charter school performance to that of traditional public schools using three measures:

1) A group of peer schools with similar poverty rates average for each subject

2) The legacy MCS average for each subject

3) The average of a group of the best optional schools in Memphis (schools with our version of an honors program) for each subject

To determine how each charter high school performs in relation to each measure I simply subtracted the average of each comparison group from each charters achievement data. The resulting number is the difference. Lists of the schools in each group can be found at the end of this piece.

Before going into the data, I want to add a couple of notes. First, I want to fully acknowledge that teach in one of the schools in question, The Soulsville Charter School, but that this post is in no way affiliated with the school nor has it been commissioned by them. It is entirely my independent creation.

Second, I chose achievement data rather than growth data for this first comparison because a charge often levied against charters is that while they grow students they don’t cut it when compared on an absolute achievement scale. As such I think this is an important point to examine and to my knowledge it hasn’t been undertaken by any official studies here in Tennessee.

Third, my goal is not to advocate for an expansion of charter schools nor call for shutting down any individual ones based on this data, especially given that I’ve only used achievement data for one year. All I’m trying to identify is who is doing well educating students in poverty using one measure from one year to identify whom we should be studying.

The Data

You can see all data shown in the table below (green cells = higher than average achievement, red = lower) followed in the next section by analysis:

The Analysis by Comparison Group:

In analyzing the data I decided to break it up further by comparison measure. To achieve this I placed all six schools together on a graph for each measure. For each graph, a bar above zero indicates higher than average scores while a bar below zero indicates lower than average scores for the comparison group.

Comparison #1, Peer School: this compared each charter school to the average of a group of schools serving a similar percentage of economically disadvantaged students. Three schools stand out as having higher than average scores; Memphis Academy of Health Sciences (MAHS), Power Center Academy (PCA) and The Soulsville Charter School. These schools displayed above average achievement in all three accountability subjects. Notably all three of these schools scored 18 percentage points higher than the peer average in Algebra II (a very difficult test), Biology and English I. Two other schools perform at or near the peer school average, Memphis Academy of Science and Engineering (MASE) and Memphis Business Academy (MBA). In both cases these schools scored higher than the comparison group average in at least two subjects.  One school, City University, scored below the peer average in all five EOC subjects.

I should note that there has been some discussion about the most appropriate measure for comparison, free and reduced price lunch or just free lunch. Some people contend that the reduced lunch population can vary enough within schools that they produce completely different environments. Just to be safe I also ran the same analysis using schools with only similar populations of free lunch students, but the results were negligibly different, only 2-3% in most cases. I decided to keep free and reduced lunch peer comparisons rather than just free because of this and because the sample size was larger for all comparison groups.

Comparison #2, MCS Average: this compared each charter school to the entire MCS system’s achievement average for its economically disadvantaged students. The data looks almost identical to the peer school comparison in its trends but with different orders of magnitude. Soulsville and PCA continue to perform well, achieving even higher outcomes than they did in the peer school comparison. MAHS and MBA perform at a similar level to their peer school comparison levels with MBA doing slightly worse on English I. City University still performs below the average on three of five subjects, but the difference shrinks noticeably from the peer average to the MCS average.

Comparison #3, Optional Schools: 

Beyond looking at the comparison to peer schools and MCS in general, I wanted to see how charter schools in Memphis fare when compared to the best our city high schools have to offer. In Memphis, this means our optional program. Optional programs are essentially honors programs nested within traditional schools for which students can apply. They do not encompass all students in the schools, but the very presence of the optional program creates a different educational climate in those schools. This shows up in that these schools typically have higher achievement than the general city-wide average.

With that in mind, I wanted to see how our charter schools fare when stacked up against the best our city has to offer. Keep in mind that this isn’t compared to students in the optional program, but to economically disadvantaged students in general, some of which do participate in the optional program and some of which do not. You can find a list of the schools used in the comparison at the end of the piece.

Optional Program Comparison: Two charter high schools in the city still perform at a higher level than the optional average: PCA and Soulsville with PCA doing better than Soulsville overall across subjects but Soulsville doing better in mathematics than either PCA or the optional programs. MAHS performs at a slightly higher level for three subjects, but slightly lower in Algebra I and English II. MBA performs at a similar or higher level to optional programs for Algebra II and Biology but notably lower in all other subjects.  MASE and City University perform significantly lower than the optional average in all subjects with the exception of English II for MASE, where it performs only slightly below the average.

Summary of Findings:

Soulsville, MAHS, PCA and MBA all meet or exceed the average achievement score for economically disadvantaged students when compared to peer schools and the city average. Charter high school’s lowest score comes when compared to the city’s optional high schools, but two schools, Soulsville and PCA, still have above average achievement levels, while achievement at MAHS and MBA drop off slightly but still perform well in comparison in some subjects.

MASE does about average when compared to its peer schools and the MCS average achievement in terms of poverty but we see a noticeable drop off when compared to optional schools.

City University performs lower than the average on all three comparison measures except for English I when compared to the MCS average and notably lower when compared to the city’s optional programs.

The Interpretation/Take-Away:

When we break down this data we have four Memphis charter high schools out of six that are doing a significantly above average job when it comes to student achievement with economically disadvantaged students and two that perform at or above the optional program schools. Soulsville and PCA do the best, followed by MAHS and MBA. We also have two charter high schools that, despite the overall positive performance of charters, perform as well or worse than traditional public schools by almost all measures in MASE and City University.

So what can we take away from this?

First, we have some great schools that are serving our economically disadvantaged student population at the high school level when it comes to absolute achievement, and those schools should be lauded for their outcomes. Three of the six schools in question perform significantly above the average of their peer schools and two more perform at or above in individual subjects. Only one school (City University) performs lower than the average for its peer and MCS comparison.

Second, three of the six are delivering outcomes on par or better than with our city’s best high schools. We can debate whether or not these schools are fully closing the national achievement gap, but at the very least they’ve closed the gap with those students that they do serve between themselves and similar students at optional schools.

Third, achievement on five test subjects is impressive, but its simply one measure of a school’s performance, not everything. We should also be studying these schools to learn how students are doing when it comes to areas such as critical thinking, problem solving and imparting a life long love of learning. We should also be looking at other measures such as ACT, graduation rate, AP scores and, perhaps most importantly, success in college after high school. The data I’ve presented here shows us something valuable about these schools, but its only one piece of the puzzle that must be examined.

Fourth, it’s important to note that not all charters are equal in this comparison. Some do better than traditional public schools, some meet the bar and some fall below it. This should inform our decisions as we identify what charters we should be studying and whose methods we should potentially be replicating. This is just one measure and more work needs to be done to determine exactly what each school in question is doing to achieve these outcomes, positive or negative. But by first identifying the achievement level of our city charter schools compared to their peers, my hope is that it will lead to improvements of both charter schools and our traditional public schools here in Memphis and Shelby County.

Additionally, my goal is to do a similar comparison analysis with growth data for high schools and then also middle schools. Any constructive suggestions for future iterations is welcome and appreciated.

By Jon Alfuth

Follow Bluff City Education on Twitter @bluffcityed and look for the hashtag #iteachiam and #TNedu to find more of our stories.  Please also like our page on facebook. The views expressed in this piece are solely those of the author and do not represent those of any affiliated organizations.

 

 

 

 

Detailed Summary of background and methods

Why do this Comparison?

First and foremost, why do this comparison? I’ve decided to undertake it because to my knowledge nobody has ever done a straight up comparison of charters vs traditional public schools and how they do serving economically disadvantaged students using straight up achievement data. The Stanford CREDO study looked at overall achievement in 2012, but I can’t find comparative data to see the magnitude of the difference. Hence this comparison.

Once again, I should note that I work in one of the charters in question (Soulsville).

What charters are Included (and excluded)?

We have eight charter high schools operating in Memphis: Power Center Academy (PCA), City University School of Liberal Arts, Soulsville Charter School, Memphis Business Academy, Memphis Academy of Science and Engineering, Memphis Academy of Health Sciences, KIPP Collegiate High and Freedom Prep. KIPP’s data was not on the TN state report card and Freedom Prep only goes through 9^th, so I’ve left those two out of the analysis. Comparison data uses all other high schools operating in the legacy Memphis City Schools system in 2013.

Why did you use achievement data for just MCS and not Shelby county?

In 2013, Memphis City Schools was still that, Memphis City Schools. The merger had not yet happened and we were not yet Shelby County Schools, hence using this data only.

Why did you only use one year’s worth of data (2013)?

The state report card this year is set up in such a way that they’ve made it easy to do this type of comparison with economically disadvantaged students for all five comparison tests (Algebra I, Algebra II, Bio, English I and English I). The website also has a large amount of data available for download from previous years, but the data is incomplete for the comparison I want. For example, the aggregate data file only includes Algebra I and English II. In the base accountability file, which does include all five tests, they do not include scores for economically disadvantaged students only for Biology, English I or Algebra II. The only year that allowed me to do that comparison I wanted with all five subjects was 2013.

What are the actual levels of economically disadvantaged students in each school in 2013?

You can see the graph below for numbers with charters highlighted and the legacy MCS average noted in red:

Which Schools Were Used as Poverty Peer Schools?

I selected all schools for each charter within 10% of a similar population of economically disadvantaged students both above and below the measure. For example, if a school had a 70% economically disadvantaged all schools at or between 60% and 80% were included in the peer group.

PCA – Central High, Ridgeway High, Power Center (included) Middle College High, City University, Sheffield, Whitehaven

City University – Central high, Ridgway High, Middle College High, City University (included), Sheffield, Whitehaven

Soulsville – City University, Sheffield, Whitehaven, Hollis F. Price Middle College, Overton, Soulsville (included), East, Kirby, Craigmont, MBA, Wooddale, Carver, MASE, Mitchell, Kingsburry, Westwood, Hillcrest, Manassas, MAHS

Memphis Business Academy – Sheffield, Whitehaven, Holls F. Price Middle College, Overton, Soulsville, East, Kirby, Craigmont, MBA (included), Wooddale, Carver, MASE< Mitchell, Kingsburry, Westwood, Hillcrest, Manassas, MAHS, Douglass, Fairley, Frayser, Melrose, Hamilton, BT Washington, Trezvant

Memphis Academy of Science and Engineering – Holls F. Price Middle College, Overton, Soulsville, East, Kirby, Craigmont, MBA (included), Wooddale, Carver, MASE< Mitchell, Kingsburry, Westwood, Hillcrest, Manassas, MAHS, Douglass, Fairley, Frayser, Melrose, Hamilton, BT Washington, Trezvant

Memphis Academy of Health Sciences – Holls F. Price Middle College, Overton, Soulsville, East, Kirby, Craigmont, MBA (included), Wooddale, Carver, MASE< Mitchell, Kingsburry, Westwood, Hillcrest, Manassas, MAHS, Douglass, Fairley, Frayser, Melrose, Hamilton, BT Washington, Trezvant

Comparison schools by zip code:

Which Schools were use as optional schools?

Central, Middle College, Whitehaven, white station, Overton, Ridgeway, Craigmont

[update July 1st, 9:51 AM]

Why No KIPP Collegiate High?

for some reason KIPP has achievement data available through the report card but you can’t pull out achievement within the economically disadvantaged subgroup the way you can with the other six schools, except for world history.  Here’s what we see when we look at their economically disadvantaged data (highlighting added by me):

 

Given the interest nationally in KIPP, however, I may go ahead and do a straight comparison regardless in a future post. Thanks to commentor Meg for asking this question!