This article is reasonable in so much as it is a rational argument against Nate Silver’s work at 538 (rather than the ridiculous nonsense he had to endure from folks who are totally clueless – UnSkewedPolls.com, etc.). However, it is ultimately unsuccessful.
“Nate Silver didn’t nail it; the pollsters did.” Not true. They both got it correct (or as accurate as can be when there is only 1 event that is being modeled).
“To be fair, the art of averaging isn’t simple.” Well it is not just averaging. Pure averaging is totally stupid. This is weighting and it is non-trivial because you need to build a notion of how much signal vs. noise to assign to each {pollster, time point combo}. Some of these polling outfits are totally disreputable and some have historic “house effects” (see e.g. Rasmussen). With respect to time – the question is how much of the past is useful for predicting the future – so you need some sort of decay function to phase out the impact of prior data points (prior polls) on your current prediction.
It is correct to say that Nate Silver’s model cannot be validated in a traditional sense – he uses simulation – because on every day other than election day – there is no way to execute a direct test of the accuracy of the model. Simulation is basically as good as we can do in an environment where there is only one event and it is perfectly valid as a scientific endeavor. If folks want to complain and actually be taken seriously – they can come up with their own positive approach. The scientific community can engage the competing claims. For example, the Princeton Election Consortium is a good example of a challenge to the 538 methodology.
No matter what 538 is a hell of a lot better than the status quo practices previously existed to its founding in early 2008. The level of jealously directed toward Nate Silver is completely transparent. If you want to get all Popperian – go right ahead but then you have little or nothing to say about this or most other prediction problems. This is what happened in quantitative finance / algo trading and the arbitrage went to those who were not worried about whether what they were doing was science or just engineering [insert Sheldon Cooper quote here] .
One thing we can hope comes out of all this is that all of the data free speculation that was undertaken prior to the election can be put to bed. I talking about you Dick Morris, Karl Rove, etc. – perhaps you guys should consider retirement and leave the arguments to the serious quants.
