In this episode, Cormac Walsh helps us to better understand what big data is, why you should care about big data, and how to get started.
Head of Data Analysis at Fonacta Enterprise Solutions
Cormac Walsh is the head of data analysis at Fonecta Enterprise Solutions where he helps organizations by producing quality and actionable insights. Fonecta Enterprise Solutions is a leading provider of data driven insight. They provide the right answers to industry, allowing them to make better decisions. Cormac frequently writes and speaks on big data topics.
After listening to this episode, you'll understand:
- What big data really is
- The difference between a Data Scientist and an Analyst
- How to get started using big data
What makes data big data? Big data characterized by a diversity of sources and a correlation and connection of different data sources together. It doesn’t matter how much data you have; if there isn’t diversity of sources, it’s not big.
As you add an increasing number of diverse sources that make sense together and the data quality is sufficient, then you have the opportunity to explain past events. Of course, what people really want is the ability to predict what’s going to happen in the future. With sophisticated modeling of data, you have the opportunity to start predicting the future for short to medium horizons. Similar to predicting the weather, the farther out into the future you attempt to predict, the higher the risk that the predictions will be inaccurate.
Big data can be used for everything from ice cream sales and positioning of gas stations to predicting crime and reducing credit card fraud.
The first step is to understand what problem you’re trying to solve. From there, you can begin to determine what data is available internally to help address the problem. If needed, it’s fairly easy to add data from external sources to improve the quality and depth of the data you have.
Once we find the data that we have, we bring it into a secure and coherent environment and then work on how to use the data to achieve the desired results. It’s not just the data; just giving people numbers doesn’t work. Insight doesn’t come from interacting with a spreadsheet. Information needs to be presented in a way that promotes a playful interaction with the data.
What You Need to Know About Big Data
A word of caution . . . In any project or experiment with big data, the client must be made aware that too often, these types of projects turn into huge efforts without seeing a benefits for a long time. This is usually due to far too ambitious and aggressive scope at the beginning of the project.
To address this, understand at the start of the project what the end point should be and then distill the requirements down to find the simplest thing that we can measure to which we can very quickly access. After you achieve the first small win, build the platform on the small, incremental benefits. Every iteration should depend on the success and the delivery of insight from the previous iteration.
Data Scientist vs Data Analyst
The term Data Scientist implies that you will use the scientific method. It helps if you have a scientific and mathematic background. A data scientist does not bring an opinion; they let the facts speak for themselves. They are skilled at and comfortable with manipulating and working with data.
Analysts such as web analysts understand at their core how things work and trends. They bring their opinion into it as they use their past experience and judgment to make some decisions. This applies whenever an analyst has domain expertise.
Big data is a very powerful technology opening capabilities and opportunities which, if done properly, look like magic to your competition. In any successful implementation, it’s likely to have been built incrementally and started with small goals and promises. In any learning system, it will start from basic and obvious ideas and then test, measure, and adapt. It’s an evolutionary system.
- Don’t try to do it internally or by yourself. In many organizations, the data is shielded by vested interests (“It’s my data”). The vested interests of the data holders may override the interest of the company.
- Try to control ambitions. Start small and incrementally build on results.
- If someone tells you that a big data system will be able to flawlessly predict the future of your business, don’t believe them.
Thank you for listening to the program
To get more valuable content to enhance your skills and advance your career, you can subscribe on iTunes.