Luck is Not a Problem-Solving Technique
With the summer drawing to a close, another academic year is ramping up. One of the earliest indicators of the madness that is to come is the release of the US News and World Report college rankings.
There are lots of problems with these rankings, but they're very popular with students and parents, so colleges take them very seriously. Attempts to improve in the rankings come in many forms, from the sincerely well-intentioned to the astonishingly cynical, but any college or university official who tells you they're not acutely aware of their institution's standing is flat-out lying to you. (My current employer jumped up four spots this year, and there was much rejoicing.)
Of course, any successful scheme inspires imitators, and there are now dozens of copycat rankings. Some of these are merely an attempt to cash in, while others attempt to highlight some perceived deficiency in the US News ratings. The latest entry in the latter category is the Washington Monthly's College Rankings (via Kevin Drum, of course). Like the good squishy liberal types that they are, they have the best of intentions:
How much more important, then, is it for taxpayers to know that their money—in the form of billions of dollars of research grants and student aid—is being put to good use? These are institutions, after all, that produce most of the country's cutting-edge scientific research and are therefore indirectly responsible for much of our national wealth and prosperity. They are the path to the American dream, the surest route for hard-working poor kids to achieve a better life in a changing economy. And they shape, in profound and subtle ways, students' ideas about American society and their place in it. It seemed obvious to us that these heavily subsidized institutions ought to be graded on how well they perform in these roles, so we set out to create the first annual Washington Monthly College Rankings. While other guides ask what colleges can do for students, we ask what colleges are doing for the country.
While these are admirable goals, they end up with a ranking scheme that is, if anything, even sillier than the US News rankings, at least when it comes to liberal arts colleges.
For the sake of full disclosure, I should note that I'm writing this largely because my alma mater takes a big hit in the rankings. As they note:
And the traditional prestige schools didn't all benefit from the Washington Monthly ranking system. Williams, which U.S. News ranks as the top liberal arts school in the country, wound up at #14 on our list, one slot below Presbyterian, largely because of its weak service numbers.
But let's take a look at how they get those numbers:
We determined the Community Service score by measuring each school's performance in three different areas: the percentage of their students enrolled in the Army or Navy Reserve Officer Training Corps; the percentage of their students who are currently serving in the Peace Corps; and the percentage of their federal work-study grants devoted to community service projects.
Now, this represents a good-faith effort to construct something useful out of publicly avaiable data. The end result, though, is pretty heavily biased against private schools in rural areas. Two out of the three measures are inherently more difficult for a rural school to do well on (the Peace Corps being the exception).
Take the ROTC component. Very few liberal arts colleges have enough participating students to support their own ROTC organization, so students who are interested in that are obliged to piggy-back on other institutions. For example, I've had a couple of students in class who were taking part in ROTC programs, and they've all needed to go to either Albany or Troy (~20 min drive) for the required drills. That's not too big a deal if you're in an urban area, but from Williams, that would require a drive of around an hour, over some roads that get pretty treacherous during the winter. In principle, ROTC is an option, but in practice, it'd be awfully difficult to manage.
And then there's the community service issue. Community service is a good thing (though somewhat incidental to the actual purpose of a college or university), but by its nature, it's highly dependent on the nature of the local community. If you're in New York or Boston, there are endless opportunities for meaningful community service projects. If you're in Berkshire County, the pickings are slimmer, just because there aren't anywhere near as many people to serve.
That's compounded by the normalization method-- they measure "the percentage of their federal work-study grants devoted to community service projects." Not only is the numerator of that fraction necessarily smaller for rural colleges, the denominator is also bigger for the elite schools. The higher tuition charged by the top-rank private colleges is going to require a larger amount of work-study funding, pushing the percentage down (you can see this in the university rankings, too, where 6.5 of the top ten instituions are state schools (I'm counting Penn as private, and Cornell as half-and-half)).
If you look at the whole list, you can see the systematic effects. Look at the top five schools: Wellesley is in Boston, Wesleyan is between Hartford and New Haven, Bryr Mawr is in Philly, Harvey Mudd is in LA, and Fisk University is in Nashville. Look at the prominent schools that that are in the middle of nowhere: Williams dropped 13 spots, Middlebury 9, Grinnell 10, Hamilton 14, and Carleton 25. The only anomaly is Colby College in Maine, which jumped up 11 spots despite being way out in the sticks (and ought to be commended for it).
The irony here is that I don't actually disagree with their conclusions. Is Williams a service-oriented place? Not really, no. I don't think I knew anybody who would've opted for ROTC if only it had been less of a drive, and the bulk of the community service work I was aware of was court-ordered.
But as I tell my students, getting the right answer by the wrong method doesn't get you full credit.
(A couple of disclaimers, here. First, the Washington Monthly people are not completely oblivious to the crappiness of their metrics-- they provide a list of data that they would prefer to use, but that isn't publicly available. As they put it, "...we have tried to abide by the best principles of social science and used the best data available to generate the closest possible measures of the qualities we value."
(Also, it's entirely possible that I'm way off base in my analysis, and it's the Peace Corps thing that's sinking Williams's ranking, not any of the factors related to its location. I have no way to tell, as the "best principles of social science" evidently don't include making the component data available on the Web.)
Posted at 8:38 AM | link | follow-ups |