Analysing the Crowd’s predictions

Analysing the Crowd’s predictions

We have always enjoyed posting out the good news when the Crowd’s predictions went right and so here comes the difficult time of presenting the Crowd’s predictions as it missed the mark in the Lok Sabha Election.

Unlike the past elections, we used 3 methods to determine the results. We had ordinary results in two, and above ordinary results in one.

Our final NDA prediction was 276 in the polling as well as direct systems. This missed the mark by 27%.

In the prediction market, we sold massive units at above 255 for BJP and above 350 for NDA. We also sold large amount of units at the lower end of the scale as well. Narendra Modi as PM was the most sold prediction.

The only good news is that the final BJP and Congress predictions were also the highest chosen predictions on our platform.

Highest Chosen prediction for BJP = 286 to 325, % who correctly predicted BJP’s performance : 39%

Highest Chosen prediction for Congress = 51 to 65, % who correctly predicted Congress’s performance : 16%

Highest Chosen NDA Prediction = > 300 seats, % who correctly predicted it: 37%

So overall, while we got the direction right and most chosen prediction was indeed the correct one, it did not constitute the majority. In many ways, we are beginning to face the same problems of Exit Polls until recently. 92% of exit polls got the direction right but not the values.

Lot of people criticise us for sampling issues and so on but in our view those are not the real issues. For example, Urban India saw a massive swing for BJP. The Crowd saw a 3% swing and not the 6% that happened. But remember, the whole concept of crowd is not about representativeness, it is about knowledge, bias and risk. Let me illustrate the point by this.

Before April 2019, average BJP supporter thought BJP will win 286 seats. By May second week, it had come down to 271 seats.

Hypothesis: After the assembly election results, BJP supporters became little pessimistic about their forecasts while Congress supporters became more confident creating an imbalance in the predictions

The other thing is all the wisdom of the crowd method fails – Chhattisgarh, Telangana, All India and Andhra. In 3 cases, the direction was right but not the numbers. The Crowd is consistently failing to catch landslides. In AP, our team was hyper confident it would be a landslide but there isn’t much we could do as even Jagan supporters were aiming at 125 and not the 150+

One way we try to warn the crowd is by constantly producing data that seems to suggest a landslide. Example: In Telangana and Andhra it was clear there was going to be a landslide. When we published Praveen Patil’s report about 370 seats, many scoffed. But we know it is critical for predictors to know the data on the ground. However, we also published 3rd party reports that said exactly the opposite. The challenge appears to be that the wisdom of the crowd model is likely to be increasingly being impacted by the echo chamber effect and people will believe not what they see around them but what they WANT to believe

Either way, we owe everyone a responsibility to improve our results and we shall do that. Our methodology is now 18 months old (and one of the first of its kind in the world). We have avoided interfering in the data from the crowd as that would be wrong but going forward we will produce platform’s own predictions using advanced analytics. We have tied up with one top tier academic institution and hope to get analytics inputs for the next election.

2 thoughts on “Analysing the Crowd’s predictions

  1. Because of the imbalanced sample, I believe that the Crowd’s judgement should only be trusted to the extent of directionality. Further, the stock market-like gamification & weekly prices don’t help the cause either, as people purchase predictions that they feel will fetch them more points, and no what they really believe.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: