How To Think About Rankings

Each year with the help of Xavier University I publish two annual rankings.  The first is the American Dream States Report, and the second is the American Dream Cities Report.  Both are based on resident sentiment regarding the degree to which they are achieving their American Dream.  The data I use is from a robust and validated quantitative research study into the American Dream conducted by Xavier University.  To the best of my ability, I try and keep my conclusions within the limits of the data.  My P&G training has taught me to be fairly rigorous in market research analysis.  As a consequence, I tend to over disclose.

With that as background, it should come as no surprise that when my friend Malcolm Allan shared a recently published article entitled The Top 6 Reasons to Be Wary of City Rankings, Ranked, I read it with the intent of ensuring my reports were not violating any good practice guidelines.

What I found was excellent counsel on how to read market research.  The key theme was before drawing any conclusions about a ranking understand what the research actually measures.  This means understand the demographics of the research respondents since conclusions cannot be projected on people who are not represented by the group.  It also means you need to understand what is actually being measured, particularly if you are trying to make comparisons with other study outcomes.  Third, the article suggests it is important to understand the study sponsor’s motivation so you can be on the look out for interpretive bias.

Hopefully you will find that my ranking reports adhere to the guidance provided in the article.  Perhaps the only missing piece is disclosing my motivation.  In short, I publish the reports because 1) I am trying to change the dialogue in economic development from job creation to better enabling residents to achieve their American Dream, and 2) I believe a quantitative understanding of the American Dream will result in better decisions around local asset creation, infrastructure investment, public policies and support of public programs that provide a hand-up.  In full disclosure, I provide the national reports for FREE and offer an opportunity for people to buy specific comparison reports.  The fee for the comparison simply covers the cost of pulling the data and preparing a customized report.  In my mind, the $50 cost represents a breakeven price point.

Let Me Know What You Think

1 Comment  |    Forward this to a friend   |   Number of emails sent: 545

Category Measures

You can follow any responses to this entry through the RSS 2.0 feed.