I spent the end of last week at the 5th International Conference on Quality Engineering and Management, where I got to hear from many current and emerging leaders in quality and digital transformation. Many of them came from manufacturing, where problems and challenges associated with data management tend to surface earlier than in other industries.
One of the speakers challenged us to look at waste, rework, and technical debt a little differently. In the planning and improvement stages of projects, he said, “maybe we shouldn’t solely focus on maximizing value. Maybe we should also spend time reflecting on how we can minimize regret.” *
What kinds of regrets have I heard senior leaders and Chief Data Officers talk about?
- We’ve purchased too many products for data management, and we’re concerned that some are not being used (or used to their potential).
- We’re spending millions on data management products, and do not seem to be getting the value out of them that we would like.
- Our engineers say that the data quality is good, but our business leaders still don’t trust it or use it consistently.
- We’ve hired 10-20 people to manage data and ensure data quality, and I can’t clearly demonstrate the value they’re bringing to the business.
Why do these regrets happen? First, it’s difficult to build institutional knowledge when retention is an issue, and even in the best of cases, engineers tend to change jobs frequently. The average tenure of a data engineer or software engineer is only 1.1 years (if you work at a place like Google, or are in the San Francisco Bay area), 2 years (for those who work at midsize to large companies), and 4.2 for others. (Stack Overflow, 2022) That’s not a long time, especially when you consider that it takes 9-12 months for most engineers to build a solid conceptual model of the technology architecture they are working within.
Second, the bigger the company, the more difficult it can be to build a shared conceptual model of how data flows through the company’s systems to generate business value. And even more concerning, rarely do organizations prioritize this kind of work. Engineers are trained and onboarded by their engineering departments, left to discover ways in which their work impacts the business on their own. By the time they’ve sifted through high-level architecture documents (many of which are incomplete, inaccurate, and unmaintained) and mountains of code, they’re ready to move on to a different company.
“Like the data-driven exercise demons obsessing over their activity trackers, nutrition trackers and other life hacks to the (possible) detriment of other pathways towards wellbeing, researchers and managers pursuing sustainability run the risk of being fixated on a set of data-rich internal indicators while remaining oblivious to broader, systemic deterioration in their environment.” (Etzion & Aragon-Correa, 2016)".
Taking a business value perspective to managing your data, and focusing on only those data flows that are the most critical to the business, can be the single greatest way for your organization to minimize regret. Not only can you reveal and resist systematic deterioration in your data management environment, but you can also help your engineers and leaders connect on the business value of the work that they do.
(*) = Unfortunately, I can’t remember which presenter shared this, so I can’t cite him or her directly.
Ultranauts helps companies establish and continually improve data quality through efficient, effective data governance frameworks and other aspects of data quality management systems (DQMS), especially high impact data value audits. If you need to improve data understanding at your organization, Ultranauts can quickly help you identify opportunities for improvement that will drive value, reduce costs, and increase revenue.
Additional Reading:
Etzion, D., & Aragon-Correa, J. A. (2016). Big data, management, and sustainability: Strategic opportunities ahead. Organization & Environment, 29(2), 147-155.
Stack Overflow (2022). Stack Overflow Annual Developer Survey. Available at https://survey.stackoverflow.co/2022/
