McKinsey & Company estimates that the US faces a huge shortage of people who have the skills to understand and make decisions based on the analysis and modeling of Big Data*.
Plus, in today’s environment, companies are using less than one-tenth of one percent of the data they could be using. This is primarily due to a lack of understanding about how to approach the monumental volume of data being generated every day.
Structured databases are fading away as the standard means to house business data, and storage is now a commodity, so that untold amounts of data can be retained for trending and analysis. Further, with the advent of more and different data formats, including video files, web logs, and social networking, new capabilities in staff and skills are required to exploit Big Data’s potential.
New thinking is now required of your staff so your organization can move beyond structured / linear data fields to a more conceptual approach to database marketing. As more data is created, newer ways to capture, process, and cultivate it to work for your businesses will be required. That thinking requires a holistic view of how to cultivate the richness to be derived from social media sites like Twitter, Facebook, and LinkedIn.
New thinking will also be demanded of your staff regarding modeling / analytics. In the legacy approach to data modeling, numerous tests on subsets of variables were run iteratively until a model emerged. The processing power and storage capacity available today has the potential to reduce the cost of building models by a factor of 10 and, at the same time, improve the predictive power of the model. All of this can be accomplished through the application of Big Data.
In order to optimize the full value of Big Data, you’ll need to ensure your staff has the skills required to collect, analyze and apply the new insights to your organization’s specific strategy. Big Data sources require new thinking in how to evaluate new insights for the benefit of your business’ success. In addition, the availability of new data formats requires new methods for processing and implementing the information. However, when approached successfully, the new data environment allows for improved modeling and reduced cost, greatly contributing to your organization’s success.
*Big Data: The next frontier for innovation, competition, and productivity.
McKinsey Global Institute May 2011