Data storage used to be really, really expensive.
In the 1990s, it wasn’t uncommon to forecast the expected growth of your databases as part of the annual budgeting process. Because every megabyte, gigabyte, and terabyte could have a material impact on the bottom line (based on the size of your business), it was important to project just how many more customers, and just how many more orders, and just how many more PDFs and images you’d collect. You didn’t ever want to be caught, disks full, with an expensive emergency procurement request… that might take weeks or even months to get and implement. Business could suffer, or even grind to a halt.
As costs have decreased over the past two decades, the risk of not having affordable storage at the ready has become mostly obsolete. In response, many organizations started adopting a new approach: storing all the data they might ever possibly need or want in the future.
This data gluttony led to two problems: vast archives containing data of dubious quality and value, and dark data. Many organizations that adopted Big Data strategies focused on collecting and warehousing data efficiently, and ultimately failed to realize the business value and impact that were initially projected to result from using that data. (Torres et al. 2018)
Dark data is “data stored… [but] no longer in use — or nearly not in use — by any user, whether this means the original user that created the data, or new users who might be interested in reusing the data. It is ‘dark’ because this data becomes invisible by means of not being easily found or accessible by any user or administrator. Instead, it remains unused in the storage servers, allocating resources that could be redirected to more useful ends.” (Shembera & Duran, 2020)
The best way to avoid the unrealized potential of dark data is to be deliberate about collecting it and managing its metadata. But that doesn’t sound as exciting as “leveraging the potential power of Big Data”. While good data always contributes to operational wins, Big Data tends to win in the boardroom. Big Data sounds more exciting, more limitless - like you’ll be capturing some unique competitive advantage that your competitors won’t. In practice, few organizations get this far.
Fortunately, researchers have figured out why. Organizations need to prioritize sensemaking, because “business value can only be created from Big Data analytics when companies successfully manage to make sense of their data…” and “social components such as the… system’s human factors, including human resources and management capabilities, contribute more to business value creation compared to technological factors.” (Oesterreich et al., 2022)
What’s the winning strategy? Collect less data, collect it mindfully and deliberately, and make it explainable and understandable. Make sense of where it comes from, why it’s part of your ecosystem, and what went into measuring it or making it. Capture the business context, and seek to understand who is driving value from your data, how they’re doing it, and why their process matters.
Over the past three years, Ultranauts has even developed a problem solving approach to make this kind of sensemaking more methodical and manageable. Using the AIADV conceptual model, that tracks Arriving data, how it’s Integrated and Analyzed, how it’s Delivered to users and stakeholders, and how they drive business Value, we help clients create Data Stories that make data flows meaningful.
With effective Data Stories in place, you’ll be able to better determine where and how to apply quality controls. This will help you get better data that has lasting value. Your data scientists will thank you! (Actually, so will everyone else in your organization. What’s not to like?)
Ultranauts helps companies establish and continually improve data quality through efficient, effective data governance frameworks and other aspects of quality management for data and analytics, especially high impact data value audits. If you want to make your organization’s data management practices more sustainable, or if you need to design quality into your ESG program, Ultranauts can quickly help you identify opportunities for improvement that will drive value, reduce costs, and increase impact.
Additional Reading:
Oesterreich, T. D., Anton, E., Teuteberg, F., & Dwivedi, Y. K. (2022). The role of the social and technical factors in creating business value from big data analytics: A meta-analysis. Journal of Business Research, 153, 128-149.
Torres, R., Sidorova, A., & Jones, M. C. (2018). Enabling firm performance through business intelligence and analytics: A dynamic capabilities perspective. Information & Management, 55(7), 822-839.
Schembera, B., & Durán, J. M. (2020). Dark data as the new challenge for big data science and the introduction of the scientific data officer. Philosophy & Technology, 33(1), 93-115.
