Cities Ranking: Flawed, Misleading
(August 4, 2016)-An organization called WalletHub recently released, to some local fanfare, a report ranking 150 U.S. cities on how well they are run. Pittsburgh was one of the cities included in the Best Run Cities ranking. Out of 150 cities, it ranked 103rd overall. The organization describes the rankings as being based on "six key categories" (financial stability, education, health, safety, economy, and infrastructure/pollution) and used 25 separate metrics under those categories. As will be shown, this explanation is far from the whole story. Complete ranking results and a methodology explanation are provided on the WalletHub website.
Pittsburgh officials were quick to dismiss the ranking as flawed methodologically. Of course when the City placed high on a number of rankings in recent years, City officials were just as quick to beam proudly about the proof of how well the City compares to others with never a word about the methodology of the ranking. On many such occasions the Allegheny Institute wrote cautionary Policy Briefs explaining that the methodology of the rankings was severely lacking in terms of offering any useful gauge of how the City compared to others. However, on this occasion the Institute must join the Pittsburgh officials in their assessment of the ranking as being flawed. Indeed, the WalletHub ranking system is deeply flawed.
This is not to imply that the City actually compares more favorably or less favorably than WalletHub says it does in regards to how to well it is run. The Institute’s work has focused on comparing the City to benchmark measures using data on several financial and management performance indicators from several similarly situated cities. In some measures Pittsburgh might compare favorably to benchmarks in others not so well. No effort is made to produce an overall ranking of the cities that requires artificially weighting results on disparate measures.
Here are the most troubling aspects of the WalletHub ranking scheme. On the six "key categories" that are lumped together by WalletHub and called "city services" Pittsburgh ranks 50th out of the 150 cities in the sample. This despite ranking 144th on financial stability as gauged by Moody’s bond rating and debt per capita. However, Pittsburgh stood very high in education (15), health (19), and safety (36) and fairly strong on infrastructure and pollution (48), and mediocre on the economy (76). But as mentioned above it had an overall ranking of 103. Why that seemingly inexplicable overall rating happened when the six categories rating was 50 is explained below.
Interestingly, on the one metric over which City officials have the most direct control, "financial stability", Pittsburgh stacks up very poorly. On metrics where City control is loose or non-existent the City does well or very well. Incredibly, the City gets a 15 ranking (the highest individual factor rank) in education. Two problems. The City has no authority over the schools. They are a separate creation of the state and have their own governing body with taxing and spending authority. Secondly, the spokesperson for the WalletHub ranking said Pittsburgh ranked so highly on education because of its 90 percent graduation rate. Not even close. It is in the 70s at best with a couple of magnet schools doing better but with some schools at less than 60 percent. And what about the Great Schools rating that makes up the other half of the education measurement? Pittsburgh rates a 3 out of 10. So unless the rest of the school districts in the 150 cities are total duds, 3 of 10 is a dreadful score. So the ranking of 15 on education—over which City officials have no control—boosts the City’s ranking above what it would be if education were left out.
Ditto the health ranking. The City has little to do directly in managing the City’s hospitals or determining how many beds they offer, a key gauge in the WalletHub ranking. Nor does the life expectancy at birth in the City have much to do with City government, except perhaps in City provided emergency medical services that saves lives. What percent of people who are born in the City will live all of their lives in the City? Unless high murder rates or high infant mortality are present and skewing the life expectancy downward, the stat means little.
Pittsburgh scores well on infrastructure and pollution. But on several measurements used by WalletHub in this category, the City has little control including commute time for instance and transit access. Commute time is determined by roads, bridges, and tunnels the City neither builds nor maintains and transit access is the province of a County authority. Nor does the City government control pollution that flows into the City. Indeed, for the most part pollution control is a state and county function. Again, the City’s ranking is boosted or marked down by factors it does not have primary authority over.
In short, the methodology fails to get at how well the City is actually being managed through the metrics the ranking organization has chosen to use.
But the story is not finished, for then comes the truly stunning flaw in the ranking scheme. After ranking the cities by the key six categories of statistics they so laboriously assembled, WalletHub mysteriously adds another ranking measure not explained in the methodology. To wit, the cities’ per capita government spending. And this factor has great weight in the overall ranking. So great in fact it is hard to know why it was not the principal measurement to begin with. It would have saved a lot of data gathering.
Three examples will make the point. Oklahoma City was ranked 81 on its performance in the six key categories, but gets an 8 ranking on per capita spending that amazingly lifts its overall ranking to 11. Similarly, Albuquerque placed 72 in the six key categories ranking but a 10 ranking on per capita spending boosted the City to 15 overall. Thus, it is now obvious how after placing 50 in the six key categories ranking, Pittsburgh suddenly drops to 103 overall. It is because the outsized weighting assigned to its per capita spending ranking of 117 swamped the "city’s service" ranking of 50.
To conclude, it must be said that this attempt at a city ranking scheme is the worst to come out in a long time. The overall ranking is basically driven by the per capita spending metric. But that is never stated in the release nor on the WalletHub website discussion. The authors would have saved themselves a lot of embarrassment by just looking at that metric. Why try to appear scientific with the six key categories and the dozens of measures within them, many of which have little to do with city governance and then make a mockery of all that mishmash by coming up with another metric and giving it overwhelming weight in the final composite ranking?
WalletHub took it in the wallet on this one. They got little for their expenditure, whatever it was.
Jake Haulk, Ph.D., President
Eric Montarti, Senior Policy Analyst
If you wish to support our efforts please consider becoming a donor to the Allegheny Institute.The Allegheny Institute is a 501(c)(3) non-profit organization and all contributions are tax deductible.Please mail your contribution to:
The Allegheny Institute
305 Mt. Lebanon Boulevard
Suite 208
Pittsburgh, PA15234
For more on this topic click here
Link to Allegheny Institute Website
Forward this Brief to a friend
Tel: (412) 440-0079
Fax: (412) 440-0085
Email: [email protected]
