“The world’s most valuable resource is no longer oil, but data.” — The Economist
Organizations are challenged daily to come up with innovative business models, use data at the core of the business, and make more data-driven decisions.
I’ve seen a few analytics maturity matrices and curves over the years. Most talk about what types of analytics organizations do along the maturity curve. I want to build on those models, include the kinds of tools and methods that go along with those maturity steps, and introduce the concept of the Chasm — essentially a bump or gap in the journey of analytics maturity that takes a little more than usual effort to cross.
Level 1: Data Awareness
This is the first phase and typically not too long in duration. Organizations realize the value of data and acknowledge the need to use data to make decisions. Most established organizations have already crossed this level. A lot of work happens in Excel and people are wrapping their heads around the use of data. Most data management processes are non-existent and there is no formal team managing analytics.
Level 2: Descriptive
This is the most common spot of maturity — enterprises using data to make decisions. Companies are using formal Business Intelligence tools and trying to realize the value of investments made in data warehouses. IT organizations take months or years to build new capability and are very inflexible in keeping up with business demands.
Some organizations use advanced self-service BI tools such as Power BI and Tableau, which provide a lot of capability to end users.
The Chasm
Advanced self-service tools like Power BI and Tableau can provide a lot of value and are loaded with very advanced capabilities. However, organizations are not ready with the right plumbing to serve the right data at the right time to these tools — and hence the value outcome is not as anticipated. This is what I call the Chasm.
Investments are not returning value, and the tools which were supposed to change the game become just another BI tool. This is essentially a gap in moving from basic BI reporting to the next level of maturity. It is difficult to cross because it demands more investment and focus from the organization.
Level 3: Diagnostic
At Level 3, organizations have crossed the chasm and are ready to make the leap to complete data-driven decision making. The problems of Level 2 have been fixed and organizations use some type of Modern Data Platform Architecture, along with a Data Lake, to support the next level of analytics.
Level 4: Predictive
As customers get a strong foundation built in terms of data platform and data starts logically filling the lakes, doing forecasts and using historical data to predict the future with a certain confidence interval becomes easier. Use of machine learning techniques becomes unavoidable, and for most organizations, cloud provides the necessary compute.
This is the phase where business models start to change and data can become part of the core business. For example, insurance companies and automakers are using telematics data to predict automobile health or driving behavior.
Level 5: Prescriptive
Still machine learning and similar techniques, but the use becomes different. The focus is on improving business processes and achieving efficiency through optimization. Some techniques involve genetic algorithms and other advanced ML technologies. The goal is no longer to predict with certain confidence, but to find the most optimal outcome given the constraints.
Level 6: Scenario Modeling
This phase is not listed in most maturity curves. It’s more of a perception and organizational attitude toward embracing data culture. Scenario modeling refers to understanding and getting ready for all possible outcomes of running the business. Getting ready to handle unknown-unknowns and having the agility and power to face those scenarios using the power of data is the ultimate maturity.
In Part 2, I cover the key steps organizations can take to cross the chasm.