Allegheny Institute: Is the eighth grade PSSA reading exam too easy?

A close perusal of 2007-2008 PSSA (Pennsylvania System of Schools Assessment) reading test scores uncovers a serious question. What is going on with the 8th grade test that such high percentages of these test takers score at the advanced level compared to the other grades tested? A sample of 220 school districts geographically dispersed across the state was examined. Of that number all but seven districts reported a higher percentage of 8th graders achieving advanced scores than any other grade tested (3rd, 4th, 5th, 6th, 7th, and 11th). In most cases the 8th grade advanced percentage was well above the percentage achieved by the other grades. It did not matter whether the district is among the elite performers or among the mediocre performers.

Why is this finding important? Because of all grades tested the biggest improvement in reading scores statewide since 2002 were posted by 8th graders, climbing from 58.8 percent proficient or higher in 2002 to 78.3 percent proficient or advanced in 2008. Indeed, in the Department of Education’s recent press release in which the Department reports glowingly on academic progress, they noted that "The percent of students scoring on grade level in 8th grade has increased by almost 40 percent in reading and math."

But here’s the more important point, 8th grade reading scores have been far and away the biggest contributor to the appearance of meaningful improvement in statewide PSSA reading scores for grades 5, 8 and 11 grades (the original set of tested grades) since 2004-05. In fact, the 5th and 11th grades had virtually no increases in the percentage of students scoring proficient or better.

Meanwhile, from 2004-05 to 2007-08, 8th graders scoring advanced on the reading exam rose 61.5 percent from 33.5 to 54.1 percent. Contrast these results with the 5th grade experience where the advanced share essentially remained unchanged from 2004-05 to 2007-08 (22.9 to 23.5 percent). In a similar vein, the 11th grade advanced scoring percentage actually slipping a bit from 33.6 to 31.8 percent. Thus, the increase in the 8th grade share since 2004-05 is nothing short of remarkable as the other grades have shown little or no gain. Finally, note that in 2003 and 2004 8th graders did not have the highest share scoring at the advanced level.

To get a handle on just how wide the spread between 8th and other grades is in the latest year of data that has been released, the percentage of 8th graders scoring at the advanced level was compared to the average percentage of all other grades tested for seven counties and 90 school districts. For example, the six school districts in Adams County had 8th grade advanced scores ranging from 51.3 percent to 54.9 percent for an average of 53.4. The other six grades tested had an average 27 percent scoring at the advanced level for the six districts (the district spread was 22.8 to 30.4 percent). Thus the ratio of advanced eighth graders to advanced other grades is 1.98 (53.4/27.0) or said another way, eighth graders in Adams County had double the percentage of advanced performers compared to the other grades.

For all Allegheny County districts combined, the ratio of 8th grade advanced to other grades was 1.75 meaning 75 percent more 8th graders were advanced than the average for the other six grades tested. In Allegheny County, the percent of 8th grade advanced for the 43 districts ranged from a low of 14 to a high of 87.4.

Other counties analyzed are summarized in the table below.

2007-2008 Reading Test Score Results

County 8th grade advanced Other grades advanced Ratio eighth to other grades
Bedford 51.5 24.8 2.07
Bucks 65.9 40.1 1.64
Berks 58.5 31.0 1.89
Centre 56.6 35.3 1.60
Cumberland 61.9 33.6 1.84

The results in the table illustrate the magnitude of the advanced scoring gap between 8th graders and other grades on the reading exams. These geographically dispersed counties are closely representative of the state as whole. Statewide 54.1 percent of 8th graders scored at the advanced level while the other grades averaged 29.8 percent. The ratio of 8th grade advanced to other grades advanced is 1.81, right in line with the range of values determined for the seven counties examined in this study.

If the test has not been made easier, do the results indicate that 8th graders have suddenly become very smart compared to other grades and then lose those smarts by the time they are tested in the 11th grade? Not if the math scores are any indication. The percentage of 8th graders statewide scoring at the advanced level on the math portion of the PSSA tests (42.6) is the lowest of all grades tested except 3rd (42.5) and 11th (25.9) and is almost the same as the all grade average of 42.3. If there has been a sudden increase in the relative abilities for 8th graders or their instructors the surge doesn’t extend to math.

Or is it possible that 8th grade instruction experienced a quantum leap in effectiveness from 2005 to 2008 while other instruction in other grades failed to make substantial effectiveness gains? Common sense would argue these are not reasonable explanations.

A more plausible third explanation would be that the scoring system for the test was changed in a way to make the 8th grade results look stronger than they really are.

Another way to look at the puzzling issue is to ask where the 2002 to 2008 jump of 33.7 percentage points in advanced PSSA reading test takers comes from? Note that there are three groups of scoring, advanced, proficient and below proficient. Over this period, the proficient score share for 8th graders fell just over 14 points and presumably most of that group moved to the advanced category since the below proficient score dropped 19.5 points from 41.2 to 21.7 percent. At the same time, the below proficient share of 5th grade test takers fell a paltry 4.5 points from 43 to 38.5 percent and the 11th grade below proficient share fell a mere 5.8 points from 41 to 35.2 percent.

Remarkably, the 2008 below proficient share of 8th graders at 21.7 percent is 17 points below the 5th grade share and 13.5 points below the 11th grade level whereas in 2004-05, the below proficient share for 8th graders at 36.0 percent was actually slightly above the percentage scoring below proficient for all grades taking the PSSA reading test; 3rd grade (32.0), 5th grade (35.8) and 11th grade (34.9).

These data reinforce the notion that the scoring results for the 8th grade reading exam have been inflated in some way compared to other grades, especially since the 2004-05 school year. There is no other compelling explanation for why so many below proficient performers in grades 3rd through 7th would suddenly become proficient or advanced when they entered the 8th grade. This is not to say the test was deliberately made easier but something dramatic has happened to the test or scoring the test. How else explain the far larger than average jump in the share of advanced and the far larger drop in the share of those scoring below proficient?

One might argue that tests for other grades were made harder while the 8th grade test stayed the same and in fact all grades are doing much better than in earlier years. Unfortunately for that argument there is no corroborating evidence of large improvements in achievement as indicated, for example, by flat SAT scores.

Because the improvement in 8th grade PSSA performance is such an important part of the Department of Education of student achievement improvement across the state, it is important to understand how there could have been such enormous gains relative to other grades and why those gains are not sustained through the 11th grade when the students are tested again. If it is because of improved 8th grade reading instruction that should be examined to see how those methods can be used in other subjects and in other grades. If it turns out that the 8th grade test is in fact less rigorous relative to other grades that must be recognized and dealt with appropriately. Finally, if the 8th grade mystery has been caused by a change in the way results are scored and has inadvertently inflated the 8th graders performance, which must be figured out and fixed.

The Department can learn a lot by analyzing this conundrum. In the meantime, their current story of the wonderful improvement in 8th grade achievement is suspect and almost certainly overstates the statewide progress on achievement tests.
________________________________________
Jake Haulk, Ph.D., President
________________________________________

For updates and commentary on daily issues please visit our blog at alleghenyinstitute.org/blog.
If you have enjoyed reading this Policy Brief and would like to send it to a friend, please feel free to forward it to them.
For more information on this and other topics, please visit our web site: alleghenyinstitute.org

If you wish to support our efforts please consider becoming a donor to the Allegheny Institute. The Allegheny Institute is a 501(c)(3) non-profit organization and all contributions are tax deductible. Please mail your contribution to:

The Allegheny Institute
305 Mt. Lebanon Boulevard
Suite 208
Pittsburgh, PA 15234