vorosmccracken.com

Bill James’s remarks on the BCS are making all sorts of rounds on the internets, and when you do what I do it’s awfully hard not to pay attention to whatever Bill writes. This particular subject is important to me because I find the process of using statistical rankings to rank sports teams fascinating (in a variety of sports).

I think Bill is mostly spot on. One area I strongly disagree with involves getting Congress involved. As much as the NCAA bugs me on all sorts of levels, it hardly seems like Congress is likely to improve anything it touches, especially something like this. He is however bang on the money when he talks about the “goals of the method.” This is really critical as I will demonstrate. Whether we want to strictly evaluate the quality of teams or whether we want to reward teams for success, the end results could have significant differences.

What I find odd is that after making that point, in the next part he seems to fall away from that position. I say this because using margin of victory lies pretty much at the crux of that issue. I am in complete agreement with Bill that margin of victory has substantial effects on our best estimates of overall team quality. A ban from the BCS on using such numbers I think would conclusively hurt those estimates over any significant period of time. If you want to know which team is better, MOV (margin of victory) matters by a significant amount.

But if you start to hand out awards based on ratings (either National Championships or Bowl Bids or slots for conferences) you run into difficulty by subverting the actual goal of any football game a team plays: winning the damned game. Bill certainly mentions that, but I feel he’s giving it less importance than it deserves. Ultimately, when you use MOV you inevitably run into a situation where the difference in effects on the ratings between a win by one point and a loss by one point is going to be relatively small. Here’s an example from this year’s college football season:

Boise State finished their season at 11-1 with a 17-16 loss to a strong TCU team at a neutral site. Utah played TCU on November 6th and beat TCU 13-10 in Utah and finished their season at 12-0. The difference between the two teams versus TCU was a mere four points and Utah had the advantage of playing the game at home. Most MOV methods I know would treat Boise State’s result as if it were at least almost as good as Utah’s (and some would treat it as good or better). There’s much sense in doing so if you’re trying to accurately evaluate the strength of the two teams.

But that’s also a lousy way to run a football competition. When you start handing out awards, an equally convincing argument can be made that it’s important that Utah reaps the full benefits of their win, while Boise State suffers the full consequences of their loss. Do otherwise and you run the very significant risk of altering the way team’s approach a football game as its being played. How you play in the 3rd and 4th quarters when all that matters is winning, and how you play when margin of victory is important can quite clearly be two different things. You run a risk of having the systems drive the results instead of the results driving the systems.This matters.

Porting my soccer and college baseball rating systems to this year’s college football season, here’s the top 10 current teams when margin of victory is ignored:

1. Utah (12 – 0)
2. Oklahoma (11 – 1)
3. Texas (12 – 1)
4. Florida (11 – 1)
5. Southern Cal (12 – 1)
6. Alabama (12 – 2)
7. TCU (10 – 2)
8. Texas Tech (9 – 2)
9. Boise St (11 – 1)
10. Penn State (10-2)

And here’s the list when an algorithm where the more you win by, the better your ratings, but with a diminishing return effect where every additional point you win by is worth less than the previous (you stop effectively gaining anything once you get to a 21 point win):

1. Oklahoma (11 – 1)
2. Florida (11 – 1)
3. Texas (12 – 1)
4. Southern Cal (12 – 1)
5. Penn State (10 – 2)
6. Boise St (11 – 1)
7. Texas Tech (9 – 2)
8. TCU (10 – 2)
9. Alabama (12 – 2)
10. Utah (12 – 0)

And here you see the difference when a team like Utah has a series of very close wins over the season (sometimes against not all that great teams). Boise State vaults ahead of them based on the fact that Boise State only played two close games all year: a 5 point win over a very good Oregon team and of course the 1 point bowl loss to TCU. They pretty much trounced everyone else they played. Utah on the other had five games decided by a touchdown or less and won all of them. Three of those games were to mediocrities Michigan, Air Force and New Mexico.

But they won those games nonetheless, and I fear that treating those wins differently for the purpose of handing out bowl bids sets up some dangerous incentives. You could further ratchet up the MOV algorithm so that even close wins have huge benefits over close losses, but then ultimately you just wind up with a system that differs very little from one that doesn’t use MOV at all.

Ultimately I think the BCS is correct in not using MOV in the computer rankings, not because it necessarily makes the rankings more accurate (I don’t believe it does), but because it makes the rankings more in line with what ought to be the goal of any football game: winning it. But the BCS system does tend to cause a whole lot of other problems that Bill correctly points out. I do think a playoff of 16 teams is too many games, and even 8 seems like too many. But I would like to see at least a final four. Under the no MOV system on December 7th, that would have been Oklahoma, Utah, Texas Tech and Texas.

A note. While the MOV rankings interestingly mirror tonight’s matchup of #1 Oklahoma and #2 Florida, when the BCS bowls were announced on December 7th, Florida was 4th behind Texas and Texas Tech respectively. Texas Tech were upset by Mississippi in their bowl, while Texas snuck by Ohio State on a last second score. Those two results allowed Florida to creep past without the Gators actually playing any games.

Here is an excel file that has the ratings for the two different systems on December 7th, 2008 and today. The actual numbers on the ratings can be used to estimate the win% of a matchup between the two teams. IE, Team A win% against Team B = Team A Rating/(Team A Rating + Team B Rating).