I looked at the source data from which the ratings are derived (6,972 international matches from October 12, 2001 to October 11, 2009) and compared the predicted results in those games to the actual results.
Now to fully test the system, you’d have to test it against a subset of data not used in the source data (for obvious reasons). But I’m not trying to do that here. What I’m really looking for is a built in bias that has nothing to do with how well it will predict what will happen, but rather how well it matches up with what has already happened.
The following table might make it more clear:
| Favorite Pred Win% Range | Total Adjusted Games | Predicted | Actual | |||||
|---|---|---|---|---|---|---|---|---|
| Win% | GF | GA | Win% | GF | GA | Poisson Pred | ||
| 50-59% | 667 | 55.0% | 1.26 | 1.05 | 56.2% | 1.31 | 1.06 | 55.9% |
| 60-69% | 558 | 65.1% | 1.48 | 0.86 | 65.5% | 1.52 | 0.87 | 65.5% |
| 70-79% | 491 | 74.9% | 1.84 | 0.72 | 79.4% | 1.85 | 0.68 | 76.2% |
| 80-89% | 393 | 84.7% | 2.32 | 0.56 | 87.8% | 2.32 | 0.52 | 85.7% |
| 90-94% | 148 | 92.3% | 3.00 | 0.44 | 96.0% | 3.02 | 0.37 | 93.3% |
| 95%+ | 160 | 97.9% | 4.88 | 0.31 | 98.2% | 4.74 | 0.28 | 98.7% |
| 90-100% | 308 | 95.2% | 3.98 | 0.37 | 97.2% | 3.91 | 0.32 | 97.1% |
| All Favorites | 2,417 | 71.3% | 1.95 | 0.77 | 73.4% | 1.96 | 0.76 | 76.1% |
Win% = (wins + (draws/2)) / games; Total Adjusted Games = games adjusted for recentness and match importance, GF and GA = goals for and against the favorite.
What you see here is a test on whether the predicted outcomes are in any way biased against favored teams, such that the sims I’ve been doing might give a team like San Marino more of a chance against Slovenia than they actually have. And I’m not entirely sure what to make of this other than it looks a little like the poisson distribution might be a little off. It’s hard to say how much the predicted goals are off, but it seems clear to me the differences there are smaller than the differences between poisson’s projected win% and actual.
I’m not entirely convinced of the final column’s mathematical applicability (this column takes the average actual goals scored and allowed and runs that through poisson to get an expected win%). I put it there to get a rough idea of how much goal differences were affecting the differences in win%, and how much poisson was.
For now all I can do is chew on this and experiment to see if I can dial in a few improvements.
One final note, the predicted Slovenia at San Marino yields this:
Slovenia = 2.65
San Marino = 0.17
Slovenia predicted chances of winning = 89.7%, Slovenia predicted chance of a draw = 8.9%
So to compare with the table above, you’d place Slovenia’s “win%” at 94.2%. If the numbers above are a real effect and not just randomness, the chances of Slovenia winning would be maybe 1 to 3 percentage points better than predicted by the current model. Obviously that lowers the Czech Republic’s chances of making the playoff by a similar (though slightly smaller) amount.
26 responses so far ↓
1 dorian // Oct 13, 2009 at 6:20 pm
Voros,
Thanks again for the insights!
Sure you saw this, but in the 90-94% row, where the Predicted Win% is 92.3%, means the Predicted Loss% is 7.7%. Meanwhile, the Actual Loss% is 4.0%. So the Predicted Loss% divided by Actual Loss% is just under 2. Does this mean the deep underdog is getting a Predicted/Actual ratio of about 2?
(Also, in the 95%+ row, and somewhat in the 80-89% row, the Predicted/Actual ratio is much greater than 1.)
Would you want to put a non-linear parameter into the mix to cut the Predicted Loss% in half as it approaches 0%, but keep the Predicted Loss% mostly unchanged as the Predicted Loss% gets north of 15-20%? Such an adjustment would reduce the probability of a San Marino draw by about half.
2 Voros // Oct 13, 2009 at 7:14 pm
Dorian,
There’s a lot of things I can do. Ultimately they all mean abandoning poisson and coming up with something a little more accurate for our purposes. With modern computers I have the means to try any number of strange things to get a close approximation.
What I first need to address is to whether the system’s goal prediction algorithm is as close as I can hope for, or whether that needs to be tweaked. It looks close with the only number missing by more than a tenth of a goal per game is for the favorite in the 95%+ bracket (which makes a lot of intuitive sense).
So maybe I can tweak that slightly, and then devise a different goal prediction method than poisson.
I think for the time being, it’s worth saying that San Marino might have a better chance (being at home) than many might think, but probably not quite as good as the sims currently say. Your estimate of about “half” seems like a close enough estimate.
I really want to get this all sorted out in time for the World Cup simulations, though it’s possibly less of a concern (though that big gap in the 70-79% bracket is very interesting/alarming).
3 dorian // Oct 13, 2009 at 8:35 pm
Got it. Thanks.
Meant to also note how very accurate the existing Prediction to Actual numbers truly are.
4 Voros // Oct 13, 2009 at 8:43 pm
Well that’s not such a big deal. It’s using the same source data after all. The point was simply look for structural biases as a means to maybe correct things.
Actual predictive ability would need to use a separate data set (though that tends to turn out pretty well too when I’ve done it).
5 Mitz // Oct 14, 2009 at 1:51 am
It’s a very long time since I studied statistics in any meaningful way, and while I do understand basically what a Poisson distribution is I’m going to steer clear from commenting directly on its efficacy in this situation – I’ll leave that to the experts…!
However, I can look at some history and point out some facts:
Since they started playing international football in 1990, San Marino have played 97 matches. Here is their record to date:
Won – 0
Drawn – 3
Lost – 94
Goals scored – 16 (per game – 0.16)
Goals conceded – 414 (per game – 4.27)
Only once have they managed to score 2 goals in a game – in a 2-2 draw with Liechtenstein. The other draws were a 1-1 with Latvia, and their only clean sheet ever, a 0-0 against Turkey. That result came during their campaign to qualify for the 1994 World Cup when Turkey, far from the half decent contenders they have become latterly, were authentic minnows themselves. Back in Ankara, Turkey won 4-1.
San Marino’s favourite opposition in one respect is Belgium – they have played them 6 times and on three of those occasions they have scored a goal! The down side is that the final results in those three games were 1-2, 1-4 and 1-10. Their heaviest defeat was at home – Germany came to town on September 6th 2006 and mercilessly put them to the sword 13-0.
Yes, very funny, ha ha, they once scored a goal inside 10 seconds against England. England won that game 7-1 so you can all shut up and go away now.
The point is that one day, against very weak opposition (say Andorra maybe, or if there is some bizarre Euro-Oceania tournament and they get to play American Samoa), San Marino might win a game. But it simply is not ever going to happen against even moderately competent opposition like (to pick a name not at all at random) Slovenia. And any system that gives them more than a 1% chance of doing so, or even a 0.1% chance, needs a little bit of tinkering.
6 Voros // Oct 14, 2009 at 2:12 am
San Marino does in fact have a win. A 1-0 win over Liechtenstein in a 2004 friendly. I have them with 1 win and four draws in a little under 100 matches lifetime which would be right at around a 3% win%.
One of the draws was a 1-1 draw in 2001 against Latvia in Latvia. That’s a result that is probably at least as good as getting a draw against Slovenia at home. Roughly two years ago in Euro qualifying they fell by a score of only 2-1 to the Welsh. They apparently were threatening an equalizer when they got a man sent off in the 85th.
Less then five months after Liechtenstein became San Marino’s only victim to date, they drew in a World Cup qualifier with Portugal (who had just taken 2nd in the Euros at the time).
Odd results happen from time to time. The Czechs don’t _need_ them to win, a 0-0 would be splendid. It’s possible, I suppose, but very unlikely.
7 Voros // Oct 14, 2009 at 2:27 am
One other note, in the four home games they’ve played so far this qualifying group, they’ve scored 1 goal and allowed 11.
Interesting because that rate is very close to the projected scoreline for this game. 2.75-0.25 to the projected score of the upcoming game of 2.65-0.17. On the road San Marino of course went 0 goals for and 28 against versus those teams.
I mention poisson because poisson gives the following result percentages for the two scores:
2.75-0.25 = 1.9% win, 9.0% draw
2.65-0.17 = 1.3% win, 8.9% draw
So I think if the system is overprojecting San Marino’s chances, I think Poisson is where the problem lies. The goals prediction doesn’t really look that bad.
8 Mitz // Oct 14, 2009 at 3:35 am
Bugger me, you’re right about their win against Lichtenstein! I’m really annoyed with myself for missing that.
Can only find three draws though…
Even though my argument has been shot out of the water in one fell swoop, I would utterly refute your assertion that a 1-1 in Latvia in 2001 is “at least as good as” the idea of them getting a draw in Serravalle against Slovenia tonight. Later on in 2001 Latvia lost to Scotland for goodness sake!
Seriously though, back in 2001 Latvia were nowhere near as good as Slovenia are today, and in that game qualification for the World Cup Finals was absolutely not up for grabs…
9 Brett // Oct 14, 2009 at 7:28 pm
My guess is that the independence assumption of the Poisson process is violated. More specifically, the Possion assumes that the numbers of occurrences counted in disjoint intervals are independent from each other.
This is perhaps not true. As an example, when a giant plays a minnow, ad-hoc observation suggests that they generally don’t play as hard once the score is 4-0. If this is true (and we could have a good crack at testing it) then the true goals scored will be lower than if the team was playing to its full potential.
As a result, the number of goals in the data understate true potential of giants versus minnows and the Poisson predicts more draws (and wins by minnows) than it should.
This whole argument is predicated on the assumption that the losing time still tries as hard as possible, that is, it doesn’t given up even though it is losing 4-0.
As potential evidence, notice that “predicted goals for” is less than “actual goals for” for all but the 95%+ favorites where it then over predicts by what appears to be a substantial amount.
Possible solutions? You could try and use a right truncated Poisson. In order to match the same mean this forces the distribution to put more weight on lower outcomes that still have a substantial number of goals and less on goals scored of 0 and 1. So you get less draws and more wins for the giant. Take the San Marino – Slovenia example, based on current ratings there is a 5.6% chance that Slovenia will score 6 or more goals. I would like to see how many games that has happened to San Marino at home.
Just my two cents.
10 Voros // Oct 14, 2009 at 8:42 pm
Brett,
The independence issue is my assumption as well, and there’s a number of ways I can approach identifying and solving that problem. Easiest way would be to have the underdog’s goals scored simulation work as normal poisson, and then readjust the expected goals of the favorite based on how many the underdog scores. Ultimately if I find such an approach to be valid, it may also affect how the rankings are eventually calculated as well.
So other than running the national team rankings and finishing up with the playoffs, I’ll now have a little bit of time to make some tinkering happen before we start running World Cup sims. I’ll have to do some data crunching.
But I suppose it could be something else
11 california viola // Oct 14, 2009 at 10:52 pm
This has nothing to do with this topic, but UEFA groups are completed and play-off are set and I want to do just some quick math to show how cruel are the qualifications on some European nations that should be in South Africa but will watch the show on TV.
Five European teams in the top 32 are out of the World Cup: Croatia, Czech Republic, Bulgaria, Israel, and Romania. Croatia is currently ranked #7, well ahead of the top team in four confederations (Africa, Asia, CONCACAF, and Oceania).
Six European teams ranked in the top 32 will fight in the November play-off. These include three teams in the top 13 (France, Portugal, and Russia) plus Greece, Ukraine, and Ireland. Of these six teams, a minimum of two will be eliminated next month.
12 Rest Area on the Road to South Africa // Oct 15, 2009 at 12:23 am
[…] RSS ← A Quick Update on the “Big Favorite” Problem (e.g. Portugal vs. Malta, Slovenia vs. San … […]
13 Mitz // Oct 15, 2009 at 1:08 am
Hi Viola,
I know this is a bit of a crusade for you, to get more places for Europe, and there is a large dollop of logic in your argument. It is flawed, however by including teams such as Croatia, Czech Republic and Bulgaria – all three finished third in their groups, behind teams significantly lower than them in the rankings. They have nobody to blame for their failure to qualify except for themselves!
14 Mitz // Oct 15, 2009 at 1:15 am
Hi Brett,
Definitely merit in what you say about all goals within a game affecting how the respective teams play in the remainder of a match. As you say, you often see a team take their foot off the pedal when they get a certain distance in front.
Conversely, to take Ukraine v Andorra yesterday as an example, the minnow can often hold out for a significant length of time. Only when Ukraine got their second goal after an hour of play did they relax and show their quality – four more goals were scored in a rush as the tension dissipated.
There are so many variables in a game of football that trying to create an absolutely perfect simulation system is nigh on impossible. The fact that Voros’ system is already so accurate is a massive testament to him, and his desire to make further refinements in an attempt to get closer to perfection demands massive respect.
(Hope you’re not blushing too much Voros…)
15 Voros // Oct 15, 2009 at 1:46 am
“(Hope you’re not blushing too much Voros…)”
Just a little.
Thanks for the praise. It really is about my own intellectual curiosity on the subject. I post it here because others may share my interest and maybe sometimes have suggestions that might help me.
I appreciate all the feedback I get and thanks much for paying attention to my odd hobby.
16 california viola // Oct 15, 2009 at 9:33 am
Mitz, many European teams don’t make it to the finals because competition is so strong in Europe. If you check the UEFA groups some had three teams ranked in the top 32 with only 1.44 spots available to qualify for the world cup. In one of the Asian groups there was not even a single team in the top 32 and 2.25 spots available for the WC.
17 dorian // Oct 15, 2009 at 1:37 pm
Hi Voros,
Regarding your comment (#2) about keeping the poisson distribution, but maybe tweaking the goal prediction algorithm, here’s an idea (and maybe it’s worthless), but what if you keep your GF algorithm in place, but tweak your GA algorithm (meaning, the underdog’s goal production is modified). Currently, your table above seems to predicted a slightly higher Predicted than Actual goals for the underdog — but what if that were reversed — what if the Predicted was less than the Actual? I suppose there is a trade-off between fitting the win percentage versus fitting goals scored, but if the win percentage could be fit better at the expense of fitting goals scored, maybe you would like this trade-off.
One idea for adjusting Predicted goals for only the underdog would be to add one more step by multiplying the current underdog goal prediction by a function of the ratio of the predicted goals.
For example, here’s a linear function of the ratio: let’s say Adjusted Goals for the Underdog (AGu) is:
AGu = Gu * ( x-1 + Gu/Gf) / x
where Gf is the predicted Goals for the Favorite. You could play around with x; if x is 2 then AGu would be at least half of Gu; if x is 4, then AGu would be at least three-quarters of Gu. (Note that Gu/Gf is always between zero and one.)
So, for similarly matched teams, let’s say your Germany-Russia example, with Russia the favorite (Gf = 1.474) and Germany the underdog (Gu = 1.339), then AGu (using x=3) would be:
AGu = 1.339 * ( 3-1 + 1.339/1.474) /3 = 1.298
For Slovenia (Gf = 2.65) and San Marino (Gu = 0.17), then AGu (using x=3) would be:
AGu = 0.17 * (3-1 + 0.17/2.65) / 3 = 0.117
By reducing the San Marino predicted goals by just under one-third, I don’t know how the win-loss probabilities would change (other than directionally), but this may be one way to tweak the predicted goal algorithm while keeping the poisson process in place. Call it the Underdog Progressive Adjustor
And if it turns out this linear function impacts the close matches (e.g 50-59% and 60-69%) too much, maybe use the same Gu/Gf ratio in a polynomial or exponential function.
Have fun if you choose to pursue!
18 Voros // Oct 15, 2009 at 2:38 pm
Dorian,
Looking at the table I think you may be right. Maybe I simply reverse the process I stated above: start off predicting the goals scored for the favorite and have the goals scored by the underdog be conditional on that number. I think that’s slightly less intuitive, but does seem to fit the data better.
I have no idea whether that will really solve the underlying problems with poisson. I’ll know more after I run a few more regressions.
19 dorian // Oct 15, 2009 at 3:04 pm
Possible intuition? Maybe underdogs are more likely to park the bus in front of the goal (and thus score fewer goals than they might otherwise)?
20 Voros // Oct 15, 2009 at 4:16 pm
Looking at some regression analysis right now (and cooking dinner). There’s a few interesting nuggets in there, but I have to think about how I want to approach this. More to come on this issue.
21 Amir // Oct 16, 2009 at 3:40 am
The problem with the dorian’s suggestion, is that you lose the continuity:
a 3.01-3 advantage suddenly becomes about 3.01-2 with x=2 in the above suggestion. Of course it doesn’t make any sense…
22 dorian // Oct 16, 2009 at 8:22 am
@Amir
Apologies if I misunderstand your comment, or my original wasn’t clear. If the favorite has a predicted goals of 3.010 and the underdog has a predicted goals of 3.000, then if x=2, the adjusted predicted goals for the underdog would be:
AGu = 3.000 * ( 3-1 + 3.000/3.010) /3 = 2.997
Thus, not very much different for evenly matched teams.
23 Amir // Oct 16, 2009 at 9:44 am
Oh sorry, i was wrong in my calculation…
Actually your formula could be written in a simple form of AGu = Gu + ( Gu/Gf-1) *c where c=1/x
So the continuity is preserved…
Maybe it would be a nice idea, but the constant c must be part of the iterations, which is making the Predicted win% equal the Actual win%.
24 Amir // Oct 16, 2009 at 9:49 am
I mean it would add an equation to the iterations,
which is Predicted win%=Actual win%.
I still don’t like the asymmetry between the favorites and underdogs though…
25 Joe M // Nov 18, 2009 at 9:32 am
From your table, the historic bias of over-favoring the favorites seems reasonably across the board vs just the case for big favorites.
Perhaps one could compare the actual distribution of victory margins in a particular bucket versus the predicted distribution. My guess would be that “good” teams are better able to get that decisive goal when behind and prevent it when ahead in close games. If so, one would expect to see more +1 results than expected and less -1 or +3.
In that case, perhaps a two-part predictor that had 80-90% of current model + the balance in a digital “clutch goal” predictor would yield higher win% prediction than currently.
26 Voros // Nov 18, 2009 at 7:21 pm
Yeah, Joe, my biggest concern is to continue to have the system logically consistent. IOW, if I make an adjustment in the simulations, if that adjustment is equally as applicable to the ratings, it should be made there too.
I tried making an adjustment that didn’t assume independence of goal scoring (IE, Team A goals scored depending at least a little on how many goals their opponent, Team B, has). The problems I ran into were:
a) This adjustment didn’t affect the simulations very much at all. As best as I can tell, the problem has more to do with the Poisson model specifically than independence. Even if actual goal scoring isn’t independent (and since teams change tactics based on score it certainly isn’t), an independence assumption could still wind up being accurate as a model.
b) The bigger problem is that when I apply a non-independent factor to the rankings themselves, I get caught in sort of a circular loop where the numbers stop converging. That’s a technical problem that there’s probably a solution to, but I have yet to work that solution out.
For the immediate future, I suspect that I will indeed scrap poisson and go to some other format that better fits the data, even if it’s sort of a kludge.
Leave a Comment