As the dust settles on one of the most opinion-dividing and controversial election results in history many people were left wondering what the consequences will be with Donald Trump taking over from Barack Obama in the New Year. Social media sites were awash with people adding their two cents on the results and what it means not only for the USA, but also for the rest of the world.

I am no expert on politics or economics, so I am not about to weigh in with my thoughts on Trump, Clinton or the US electoral system. What is fascinating to me is how so many Data Analysts and Polling firms got the result wrong. I was recently involved in launching a new training course on Big Data Science so the analysis after the election on poll predictions, and where the data had gone wrong was of great interest to me. The Daily Telegraph reported that only ten out of ninety-three polls published on RealClearPolitics over the last ten months had Trump in the ascendency [1]. Of those ten, nine were from the LA Times, so in fact only two unique sources had put Trump in front during this time. Big Data and Data Analytics are big business today and the market is growing. The Democrats have a team of over fifty people dedicated to Campaign Data Analytics and even Trump, who had earlier rejected the need for Data Analytics in his campaign, retained the services of Cambridge Analytica following confirmation of his nomination as the primary Republican candidate back in February[2]. Outside of the US election, organisations continue to invest heavily in Analytics, Data Science and Big Data initiatives with Gartner positioning Big Data at the peak of the expectation curve in their 2016 Hype cycle for EA [3].


So with all this investment and advances in Data Analytics, how did so many people get it wrong? I read that one agency was using over 400 data points to form their calculations on the polls [4]. So how then were so many of the predicted outcomes wide of the mark? Online articles published in the aftermath of the election results cited varying reasons for the inaccuracy of their predictions. The common thread in many of these articles is that we are trying to predict human behaviour in a very rare life event. Elections are not an everyday occurrence so finding trends and making reliable predictions is a tricky task, even with the advancements we have seen in recent years in the Data Analytics space. If you consider other major votes around the world in the past 12 months, the shock US election result is not the only major surprise we have seen. There have been two other surprising results with the Leave vote in the Brexit referendum and the outcomes of the Israeli election [5].
An area that I foresee opportunities for further advances is how we link data models to behavioural models. For example feelings of instinct, sentiment, emotion and intent[6] can all have the same characteristic or choice but with very different outcomes, and the variance can be increased when you consider the different external environments in which a decision or opinion is made. To use the US election as an example, Voter A may have an instinct to vote for Clinton (which is what would be reflected in a Poll) but when it comes to the action of voting, they may end up voting for Trump. When you consider the impact of society and the external environment there is potentially even more chance of a change in the outcome as the Voter can make their decision anonymously without any influence of what society (or social media) thinks is the ‘right’ way to vote.


Statista [7] research shows that spending in the Big Data space isn’t going to abate, so what do organisations need to do to maximise their investments in Data Analytics and to gain competitive advantage over their competition? Matt Ockowski, Director of Product for Cambridge Analytica, the analytics firm engaged by Trump certainly thinks Data is ‘alive and kicking’ [2]. His view is that the value is in how you understand, interpret and use your data that will dictate the success of your Data initiatives. Gaining actionable insights and a new and deeper understanding of your business to help shape business planning and strategy is likely to be the primary raison d’être for Data Science and Analytics professionals for years to come. We may never see Data Analytics with a 0% margin of error, but the organisations that can get closest to truly understanding the needs and wants of their customer are likely to be better poised for success than those who are not fully leveraging the full spectrum of information that is available to them.
 
References:
[1] – http://www.telegraph.co.uk/news/2016/11/09/how-wrong-were-the-polls-in-predicting-the-us-election/
[2] – https://www.wired.com/2016/11/trump-polling-data/
[3] – Gartner Inc. – Hype Cycle for Enterprise Architecture, 2016
[4] – http://www.eweek.com/database/big-data-analytics-plays-big-role-in-2016-election-campaigns.html
[5] – http://www.computerworld.com/article/3140052/data-analytics/election-night-whoa-was-i-was-wrong.html
[6] – Hwaiyu Geng – Internet of Things and Data Analytics Handbook
[7] – https://www.statista.com/statistics/254266/global-big-data-market-forecast/

 

Author
Danny Graham