It sounds like a drive through meal, why should I care?
If you join the ranks of those entering the world of the technology religion…er, I mean industry, or technological exploration and invention, or are just a hobbyist there seem to be two prerequisites:
- You must love, love, love, love, and dream in acronyms
- If you name something, make it so generic as to leave it completely open to any conversation, application or ability.
“Big Data” falls into that latter to the proportion of the “smart phone” (what the heck does that mean???). This inability of the technology industry to intuitively name anything that self-describes in this case is tragic because of the wide-ranging implications and potentialities of Big Data. In fairness, none of us were English majors! What is Big Data in simple terms? Here is a non-PhD explanation of Big Data and why it is truly one of the last exciting frontiers, albeit not in a completely physical sense.
In our infancy of exploration of the scientific method, business, fitness, marketing, and almost any arena big or small, we started with siloed data. In other words, a business had accounting numbers; marketing had demographic data; and science had measurements beyond my comprehension. The goal was simple, to bring this information specific to a process or entity and make sense of it to make decisions, assumptions, valuations, or whatever was the pertinent decision point. We humans, in our inherent need to organize things, created data organized in columns and rows which brought the data together into “information” which we in a moment of utter lack of a comprehensive vocabulary, called reports. By sorting the data in in linear and even multi-dimensional formats, these reports enabled measurement of profitability, scalability, probability, etc.… Today this is still the widest used decision-making technique from household budgets to probabilities of orbital change affecting global temperature.
After years, actually decades, of analyzing reports to assist in decision-making someone asserted that this process of looking at formatted data that gave insightful information and then making decision missed a quintessential question: “what if?” After all, that leaves a great deal to interpretation and “gut feel.” By tracing both good and bad causal effects across information, modern computing could provide models by which we could look for predictive trends. In essence, by breaking down the silos of data, and joining them with other silos, technology could produce more accurate, and more actionable, information. In addition, models could then be manipulated to see where and how it impacted business, health, molecular generation, oil prices…you get the idea. The advent of relational databases made this sort of data storage, configuration, indexing and results available. Hence, the oxymoron term “business intelligence” (BI) was born because technology could now relate data to other data, with very minute variables in common.
The scientific industries and big business fueled the growth of what is now a several billion dollar a year business of BI. Technological advances with computing power and data storage and retrieval continued to make this type of analysis more accessible. Once the professional industries (consultants) joined in with their specific theories and understandings, beautiful modeling techniques such as Key Performance Indicators (KPI’s), benchmarks, acceptable tolerances, sprouted up across every industry. In the simplest forms, many organizations use Excel with pivot tables to accomplish this task at a high level. Yet even in 2016, only a small portion of private organizations take full advantage of these capabilities. The major hurdles, which are slowly coming down, are the cost to deploy technologically, the expertise to ensure the models are actually sound (and not biased to the desired results), and the cost to adapt to each individual data set and configuration of decision trees which is a costly venture in consulting fees alone. Frankly, it has become a battle between the cost savings of proven models versus the innovation of out of the box thinking that brings an entire new set of variables into the mix. Undoubtedly, BI is today one of the most powerful analysis tools available, and it is just now starting to be deployed outside fo the fortune 5,000 and academia.
So, what is left? And, why Big Data? We broke down the data silos very intentionally, joining data with commonalities so technology could give us some predictive results on the question of “what if.” What is next, and more appropriately, why is it big? Simply, Big Data can look at tremendously vast data sets across seemingly unrelated data sets—more than the Library of Congress to the third power of ten hypothetically. This is only possible because computing power has exponentially risen and the cost of storage diminished. There are challenges as well, mind you: cooperation, collaboration, anonymity to name a few. A final frontier for technologists—programmers, analysts, subject-matter experts, and the experiences of past data joining—is coming together to create programs that systematically look for the “joins” instead of the answer in a universe of unlimited data. In short, we are tasking the massive computing power to actually find the relevant causal effects, no matter how distant they may be. Tell me: 42 year old humans living between the 40-43rd parallel of the earth with chromosome 19 at 1,450 genes…what are material circumstances, outcomes, effects, trends, etc.… Except in this case, subject matter experts are only giving potential variables and the computing power takes over the rest, looking for trends, and other material variables. Imagine finding the correlation between potable water elements of a region and the effects on scientific exploration from those hailing from that region. The trends and how things may relate are limitless.
This is important because it does not give us all the answers. Rather, we are now finding and identifying connections between things we would have never imagined. None of this answers “why am I here.” Nor, will it create immediate change or cures. What it will do is expand our thinking—computers are just machines keep in mind…Artificial Intelligence (AI) is another giant technological leap. Big Data can deliver significant insights into the relevance and relation of organisms, people, ideas, environments, outcomes, tendencies to each other. Those results will lead to more “ologies” than we already know, and certainly to a new era of thought, philosophy, understanding, and connection.
In short to me, the only way to truly describe Big Data’s potential is taking a giant step towards technological wisdom. Out of that will sprout the next generation of entrepreneurial ventures and innovations that touch every aspect of our lives.
Here are a few interesting resources:
- http://www.thewindowsclub.com/what-is-big-data
- The Seven ‘Simple’ Steps To Big Data – Forbes
- Beyond Volume, Variety and Velocity is the Issue of Big Data Veracity
- Getting Personal With Big Data