Big Data – What Awaits Us?

In this super visible world of exabytes and petabytes of data, we have obtained the power to see what we had never seen before. It is said that everything is quantifiable including human emotions. Now, with the microscope like instrument to analyze what was there all along, what awaits us? A fading bloom or a major revolution?

Considered to be the holy grail of the 21st century, Big Data is believed to be one of the extremely important tool for the future. Data is not a new term and probably it happened to be coined to create a hype. Data existed always from the time humans started their communication through words. We stored unchangeable data through stones, letters during stone ages. Next, we advanced to computer disks and pen drives giving more storage capability, more processing and re usability. In this era of social media and internet of things, the data has gone beyond the realm of storing in a static device. Data has transformed from a static storage form to a more fluid flowing real time form. This rapid flowing, huge amount of real time data is termed as Big Data. In simple terms, Big Data are those large datasets that are challenging to store and analyze using traditional relational data bases and queries. And it’s not new, specialists in the field of banking, telecommunications have been struggling with Big Data for decades. What is new however is that, now technologies have emerged that offers options to exploit this Big Data.

According to IBM, we create 2.5 quintilian bytes of data every day from various sources including social media posts, finance, GPS locations and more. And it is also observed that, in the last two years, we have generated nine-tenths of data that exists now and all the data processing done in last 2 years is more than all that was done in 3000 years. The more information we get the more difficult problem we solve.

What can we do with Big Data?

To understand and gain insights from Big Data, there is a whole set of technologies, one undeniably is Hadoop ecosystem and other ETL concepts. These technologies can handle unstructured complex data with utmost scalability. Now, let’s look at some of the scenarios where big data is very helpful in certain industries.

• Sentiment Analysis – A focus on millions of customers reveals insights on how they think or feel about a certain product or a topic. For e.g. Big data helped in understanding the power of advertising spots in Super Bowl based on the social media data.

• Predictive Modelling – it is one of the important skills of big data. Forecasts and prediction are extensively used in Banking industry. For e.g. an airline can make maximum price predictions for customers and develop a pricing strategy to increase revenue.

• Recommendations – Using Big data, we can promote recommendations, product bundles using customer history of purchases, and make customized offers through online ads etc., This can enhance customer loyalty and increase sales. For example, Netflix custom made movie recommendations using past behavior and preferences.

What cannot we do with Big Data?

Even though Big data can be used to do certainly amazing things, there are various scenarios where Big Data fails and it is not applicable, yet. Let’s look at some cases where applying Big Data will end up in a failure.

• Predict a certain future – With all those predictive modelling techniques, at a maximum, we can achieve only 90% accuracy. We could never reach a 100% accuracy even with more enhanced machine learning concepts and hence predicting a definite future is highly impossible for now.

• Read your mind – Big data technologies are machines which work on commands and we cannot expect these algorithms to read our mind. It is used in understanding the customer behavior as closely as possible but it cannot know certainly what a customer will purchase or how he will behave.

• Solve Non-Quantifiable problems – A biggest use case of Big data is understanding customer behavior but still one cannot quantify a human behavior. A group of individuals can be analyzed and observed for trends but it is statistically impossible for now to give a score to a human behavior.

Big Data – A Marketing Perspective

John Wanamaker, an American retailer, quoted “Half the money I spend on advertising is wasted; the trouble is I don’t know which half.”

With significant initial investment, there are nearly endless opportunities for a marketer to derive insights and gain better than ever. From an analysis by eMarketer recently, it is observed that the spending trends on “data-driven marketing” has increased and marketers are learning to leverage the data to benefit their approach. As per the report on Data Driven Advertising and Marketing tactics used by U.S B2B marketers, cross-channel, cross-device and lookalike targeting are the three most popular data driven tactics for 2017. For example, analytics will help to gain insights of which lead will turn into revenue and save from spending huge money on low quality leads.

Winners of the Data Creativity Category of Marketing New thinking awards 2016. Inspired by their 20 years’ worth of customer information about their journeys, the marketing team at EasyJet decided to create emotional stories as dynamic email customized for every individual telling a story from their first flight to recommending future flights with some facts about their travel behavior. They had a successful open rate of more than 100% from previous time and 25% higher click rates

These are some of the cases which took data marketing to another level. And the huge success of this part of story telling is the customer feels connected and feels special and noticed. This might be a beginning to establish a life-term relationship. Marketers should start thinking about using data effectively to build a customized experience in order to engage the customers and drive retention. Going forward, marketers will need to cater to the needs of various sources of data, not just social media but from virtual reality to sensors. I believe that it is the future of big data marketing where nothing is being said or shared. Everything is just being observed and learned.

It’s all good?

Well, big data is not all amazing and good, it sure has its own drawbacks. I would like to take reference here to generally highlight what major threat big data will bring to the world.

False Prediction – The movie, Minority Report, set in 2054, handled the concepts of machine learning and analyzing large data sets to predict human crimes. Pre-cogs, the super data scientists, can predict future crimes like murder along with the intricate details of who, when and where. The movie then revolves around a wrong prediction over its own head officer and one should watch the movie to understand the impact of the same.

Security – If that explains how the future might bring to us, at present we are facing an alarming increase of cyber crimes. With more and more data, mostly personal information, how well is our data protected and encrypted. Given that most of the data handling software are open source like Hadoop, which is not secured, it is important to ensure the security of the data itself.

Facts are stubborn – In reference to the John Adams quote, “Facts are stubborn things”, It is now known that, analytics of data cannot understand creative thinking or as an extreme free will. It might affect largely when a generation emerges with only logical thinkers who values analytics that in turn values only given set of metrics and under performs values such as emotions, creativity.

Humans vs Machines –  Sooner or later, we will be officially on a war of machines vs humans, where we already know that automation and machine learning is taking up most of the jobs from us. This has the potential to do what the white collar jobs did to the blue collars in 1950s. Consider a lab technician, who analyzes thousands of data of a gnome or cancer cells. He/She wil take weeks to gather data, analyze and finally produce results. But now this is being done in days by machine learning algorithms. Gone were the days, when banks had 15 to 20 representatives such as cashier, accountant, cheque handler. Now computer have taken over all the jobs and the same will re happen to take away the jobs of people handling the computers since now the computers can handle themselves.

Now, we have come to this breaking question we are trying to understand from the beginning. Is Big Data a fading bloom or evidently can create a major revolution. Having touched upon, the definition of big data, its capabilities, its impact in my field of interest, and also slightly brushed on the threats of big data, I believe that Big Data is here to stay. Interestingly, recently I was faced with the question regarding the impacts of all this data analytics on a regular human life. I was asked to imagine as an under age 18, pregnant girl, who has access to all these internet, social media. Imagine how would she feel when a retailer could predict her pregnancy and send personalized baby care products brochure to her home and her dad sees it. I was asked how would she react and how would she feel about this. My mind instantly freaked out and said that’s creepy. Apparently, that was not an imagination, but an incident happened in US where the giant retailer Target knew the girl was pregnant before her own father.

Yes, it is creepy. That’s why there is an increase in the number of adblockers which has become a huge hurdle for online advertising. But when I think of it in a broader sense, the concept of advertising has been there all along. When you try to correlate the initial stages of marketing and advertisements through a mass medium, how it opened a whole new gate to communicate with consumers and understand the likes and dislikes to the current data driven marketing, you end up seeing the similarities. It would have considered to be annoying when the tv ads were introduced but eventually we have grown past it and are able to embrace and even love the ads for what they bring to the table. Similarly, the more we get hold of the customer data and get clear and more sensible insights, online advertising would become more targeted and personalized and interesting. Hence, I believe there will come a time where online ads would be as praised and looked forward like in super bowls.

With over a decade passed since we started using Big Data, I personally, do not perceive any fading whatsoever. In fact, we have stated a new term this year as Fast Data, data that is real time and more rapid in nature.  There is no stop for this, as I could see, there are more products keep popping out of that ecosystem and various job descriptions are being created that involves data analytics. Properly handled data might eliminate most of the shortcomings I mentioned above. But there will be job loss to some extent but it is also observed that there will be need of human touch. International Data Corporation predicts more people will be needed with deep analytics skills in next two years. Analytics has reached all levels of an enterprise and possibly touched different work environments, businesses are starting to embrace data science and there are cars that drive themselves. Big data will continue to get bigger, we just have to learn to embrace the journey