Uncertain Principles

With the summer drawing to a close, another academic year is ramping up. One of the earliest indicators of the madness that is to come is the release of the US News and World Report college rankings.

There are lots of problems with these rankings, but they're very popular with students and parents, so colleges take them very seriously. Attempts to improve in the rankings come in many forms, from the sincerely well-intentioned to the astonishingly cynical, but any college or university official who tells you they're not acutely aware of their institution's standing is flat-out lying to you. (My current employer jumped up four spots this year, and there was much rejoicing.)

Of course, any successful scheme inspires imitators, and there are now dozens of copycat rankings. Some of these are merely an attempt to cash in, while others attempt to highlight some perceived deficiency in the US News ratings. The latest entry in the latter category is the Washington Monthly's College Rankings (via Kevin Drum, of course). Like the good squishy liberal types that they are, they have the best of intentions:

How much more important, then, is it for taxpayers to know that their money—in the form of billions of dollars of research grants and student aid—is being put to good use? These are institutions, after all, that produce most of the country's cutting-edge scientific research and are therefore indirectly responsible for much of our national wealth and prosperity. They are the path to the American dream, the surest route for hard-working poor kids to achieve a better life in a changing economy. And they shape, in profound and subtle ways, students' ideas about American society and their place in it. It seemed obvious to us that these heavily subsidized institutions ought to be graded on how well they perform in these roles, so we set out to create the first annual Washington Monthly College Rankings. While other guides ask what colleges can do for students, we ask what colleges are doing for the country.

While these are admirable goals, they end up with a ranking scheme that is, if anything, even sillier than the US News rankings, at least when it comes to liberal arts colleges.

For the sake of full disclosure, I should note that I'm writing this largely because my alma mater takes a big hit in the rankings. As they note:

And the traditional prestige schools didn't all benefit from the Washington Monthly ranking system. Williams, which U.S. News ranks as the top liberal arts school in the country, wound up at #14 on our list, one slot below Presbyterian, largely because of its weak service numbers.

But let's take a look at how they get those numbers:

We determined the Community Service score by measuring each school's performance in three different areas: the percentage of their students enrolled in the Army or Navy Reserve Officer Training Corps; the percentage of their students who are currently serving in the Peace Corps; and the percentage of their federal work-study grants devoted to community service projects.

Now, this represents a good-faith effort to construct something useful out of publicly avaiable data. The end result, though, is pretty heavily biased against private schools in rural areas. Two out of the three measures are inherently more difficult for a rural school to do well on (the Peace Corps being the exception).

Take the ROTC component. Very few liberal arts colleges have enough participating students to support their own ROTC organization, so students who are interested in that are obliged to piggy-back on other institutions. For example, I've had a couple of students in class who were taking part in ROTC programs, and they've all needed to go to either Albany or Troy (~20 min drive) for the required drills. That's not too big a deal if you're in an urban area, but from Williams, that would require a drive of around an hour, over some roads that get pretty treacherous during the winter. In principle, ROTC is an option, but in practice, it'd be awfully difficult to manage.

And then there's the community service issue. Community service is a good thing (though somewhat incidental to the actual purpose of a college or university), but by its nature, it's highly dependent on the nature of the local community. If you're in New York or Boston, there are endless opportunities for meaningful community service projects. If you're in Berkshire County, the pickings are slimmer, just because there aren't anywhere near as many people to serve.

That's compounded by the normalization method-- they measure "the percentage of their federal work-study grants devoted to community service projects." Not only is the numerator of that fraction necessarily smaller for rural colleges, the denominator is also bigger for the elite schools. The higher tuition charged by the top-rank private colleges is going to require a larger amount of work-study funding, pushing the percentage down (you can see this in the university rankings, too, where 6.5 of the top ten instituions are state schools (I'm counting Penn as private, and Cornell as half-and-half)).

If you look at the whole list, you can see the systematic effects. Look at the top five schools: Wellesley is in Boston, Wesleyan is between Hartford and New Haven, Bryr Mawr is in Philly, Harvey Mudd is in LA, and Fisk University is in Nashville. Look at the prominent schools that that are in the middle of nowhere: Williams dropped 13 spots, Middlebury 9, Grinnell 10, Hamilton 14, and Carleton 25. The only anomaly is Colby College in Maine, which jumped up 11 spots despite being way out in the sticks (and ought to be commended for it).

The irony here is that I don't actually disagree with their conclusions. Is Williams a service-oriented place? Not really, no. I don't think I knew anybody who would've opted for ROTC if only it had been less of a drive, and the bulk of the community service work I was aware of was court-ordered.

But as I tell my students, getting the right answer by the wrong method doesn't get you full credit.

(A couple of disclaimers, here. First, the Washington Monthly people are not completely oblivious to the crappiness of their metrics-- they provide a list of data that they would prefer to use, but that isn't publicly available. As they put it, "...we have tried to abide by the best principles of social science and used the best data available to generate the closest possible measures of the qualities we value."

(Also, it's entirely possible that I'm way off base in my analysis, and it's the Peace Corps thing that's sinking Williams's ranking, not any of the factors related to its location. I have no way to tell, as the "best principles of social science" evidently don't include making the component data available on the Web.)

Posted at 8:38 AM | link | follow-ups | 3 comments

We've hit that point in the summer-- finally-- when my research students have left, and I can actually take a little bit of a vacation. Yes, I know, one of the advantages of my job is supposed to be getting the summers off, but, you see, I don't have tenure. Thus, the summer is almost exactly as hectic as the academic year.

My nerves are stretched to just about the breaking point, though, so I'm taking this week off. Later in the week, Kate and I are heading up to Quebec City for a few days (suggestions of must-see attractions and activities are welcome in the comments), but I'm not going to work at all this week, letting me finally deal with some of the jobs that have been going undone around the house.

One of the before-vacation tasks is to clean out those Bloglines feeds that I've marked as things to comment on. So, here's a list of miscellaneous stuff:

First, Arcane Gazebo reports some nifty results ina fairly comprehensible manner. It's already been linked by some other people, but it's worth pointing out as a fairly readable explanation of some fundamental quantum-mechanical research. More like this, please.

On the political side, I think I'm going to stop posting comments on current events, and just link to Timothy Burke when he posts. He keeps saying eminently sensible things that are basically what I want to say, but am too clumsy to actually get down in text. He's got good pieces recently on "Intelligent Design" and political theatrics-- now, if only he had an RSS feed.

For another side of blogging, sennoma points to the blog Tricks of the Trade, offering, well, little tricks that experts in some field know, that might be useful to ordinary people. It offers no less than five different ways to start a fire in the woods, along with tricks to keep people from stealing your pens, and a possible way to make the elevator go faster. There's some neat stuff.

Speaking of tricks of the trade, props to Matt McIrvin for his Really Basic Photo Editing With GIMP 2.0. I've been assigned to be the department webmaster this year, and his tips have helped a great deal with cleaning up some of the digital pictures we have on the page. I've also been doing some cargo-cult JavaScript, with limited success. When I get the whole thing finished, I'll post a link.

Matt also linked to streaming video of Feynman giving some well-known lectures. I haven't watched it yet, as Kate's torrents of bits have dragged the connection speed down under 1 kB/s, but I'm posting this here to remind me to look at the damn thing.

On the multimedia front, Nanopolitan has a link to famous people talking about Einstein. Technorati is a wonderful thing when it works.

Finally, something silly: the proprietor of Conrdoggerel is offering an interpreting service:

Given that I am a male and that most of my readers are females, I thought I might offer my insights into the male psyche as a service to women in the blogosphere who are confused by a man or men in their lives. Let me list my qualifications:

I have played: football, basketball, baseball, golf, bowling, horseshoes, poker, quarters.

I can match or exceed most of his qualifications (add to the above "I have played" list: "rugby," "beer pong," and "with fire"), but it'd probably be a bad idea to go through the whole list. I don't have tenure, but I would like to have tenure someday.

Maybe then, I'll get a real summer vacation.

Posted at 10:23 AM | link | follow-ups | 7 comments