#VDCConnect2016 – Dialogue on Big Data Analytics
By Charles W. Stryker
On October 6, 2016 Time Inc. provided a venue for some of the leading American Big Data and Analytics players to meet and exchange views on the current state and future challenges around the growing Big Data and Analytics ecosphere. When we attempt to assess the state of the Big Data Analytics ecology today, we can’t help but be amazed at the accomplishments of some companies. The leaders of these companies have taken their enterprises into the forefront of the Big Data world. In so doing, they have experienced challenges and provided solutions that can serve to benefit us all.
Where are we in the evolution of Big Data and Analytics? In some ways, in the day-to-day work we do, it gets frustrating that we can’t seem to make more progress. There’s so much to do with the many new, spectacular data sources and remarkable analytics tools that have become available recently. There have been big advances in data visualization and insight development and it becomes challenging to figure out the best way to take full advantage of these powerful new tools.
As we attempt to define the current state and future prospects for Big Data Analytics, we need to realize that 45 years ago there was no computer-generated data and 15 years ago there were no mobile devices. This story starts with remembering that, before 1970, there really were no electronic databases. As hard as it is to imagine, only 45 years ago, Big Data and Analytics was done with paper on 3 by 5 file cards. The analysis would be to read through documents and take pieces of paper and check off yeses or no’s based on what they see and try to draw analytical conclusions.
The birth of the Internet can be traced back to when the first browser was shipped in April of 1993. That certainly represented a seminal point in the evolution of the Big Data and Analytics practice. The Internet enabled digital activity that created the first, seemingly infinite supply of data. From 1970 to 1993, the world of electronic data was able to create about one billion gigabytes of data that could be used for decisioning. So it took us roughly 25 years to go from 0 to 1 billion gigabytes in total. Driven heavily from 1993 by the development of the Internet, today there’s about 25 billion gigabytes per day being captured and stored on servers around the world.
Since we went from a total of 1 billion gigabytes for decisioning in 1993 and we’re at 25 billion gigabytes a day of decisioning information, it is little wonder that we’re finding it very difficult to keep up. We also need to consider what we’ve accomplished in such a relatively short period of time. Looking to the future, that 25 billion gigabytes a day is going to look like a drop in the bucket. With the advancement of the Internet of Things and the introduction of digital devices around the world, the traditional pool of compiled data is remaining stagnant while this new world of digital data is growing and will continue to grow at an exponential rate.
The next question is, where does the data come from? The way we look at the world, there are six sources or so where we break out those 25 billion gigabytes and we’ll be adding new sources for sure. Some of the sources of the current supply of data for decisioning comes from published material from around the world. Then we should consider all of the search data created digitally around the world. Next, consider all the social interactions that take place digitally around the world. Then, add in mined data, both off the open and deep web, plus the web infrastructure. In addition, within this very short history of Big Data, there are about 40 million people around the world who do customized, task-based data entry to create databases for data needs that can’t be serviced through use of automated data collection methods. Then there’s traditional transaction data which is the fastest growing segment of the Big Data ecosystem as companies that have data as a by-product of what they do see a significant revenue opportunity to translate the data that they own into dollars and slowly but surely, a global business data market exchange is being created.
There are about 700 companies that get the data they use as a by-product of their core business. They package it and deliver it into decisioning systems around the world. These companies include cell phone service providers, credit card companies, ad serving companies, mobile app developers and many more.
The result of all this is the current state of Big Data. What has come of this development is that there are hundreds of companies that have developed remarkable capabilities to work with this new Big Data ecosphere.
So we are at the second or third inning of what is going to play out to be a very exciting game: massive availability of new data. That’s the 25 billion gigabytes a day growing exponentially. Remarkable advances have transpired in data hosting. Advances have been made in the field of analytics to identify insights. And, the ability is now there to deliver insights economically to the points of use. Think of Salesforce.com, for example.
So all of us who care about this topic have great reason to be excited and interested in how to more rapidly advance because the ability of the technology to outpace our ability to use it is going to continue for a long period of time. Another way to think about this is the impact of this Big Data and Analytics revolution will likely have the same impact on all of our business and personal lives as the advent of the Internet did from 1993 to today.
A way to look at it is to go back to pre-1993 and ask what do I do today that I do differently than I did in 1993. I deal with my financial needs differently whether it’s business or consumer. I interact with my friends differently than I did. I shop differently. There’s virtually no aspect of my life or my business life that I do the same today that I did before 1993. And if we do this right and we look forward 10 years, we’re going to be making the same comparisons.
We’re going to be dealing with disease differently because of Big Data. We’re going to deal with our personal lives differently; we’re going to plan trips differently; we’re going to bank differently. All of those things we did differently that were enabled by the Internet, we’re going to do even more differently, enabled by Big Data and Analytics.
The machine is going to make a lot of decisions for us, not just to provide us data. And we can already see the beginning of driver-less cars. We’re going to potentially hear from IBM about their system that’s advancing the cure for cancer with their data.
In the next few blog installments we will highlight some seminal views on Big Data Analytics by leaders in industry. We are at the beginning of a very interesting time.