Tools alone do not make a data-driven organization. What separates those who aspire to be data-driven from those who actually are comes down to how quickly decision-makers can get the answers they need to make informed decisions.
Today, there's been an explosion of innovation focused on reducing data latency and giving companies access to information in near real time. But despite advances in the availability, access, and analysis of data, enabling analysts to generate insights faster than ever, companies still struggle to be truly data-driven in their decision making.
The true measure of a data-driven organization is in its decision latency. With more data available than ever before, and more ways to understand and analyze it quickly, the bottleneck is no longer with the infrastructure — decision latency is buried within the business process itself.
Decision latency: What it means and what it costs
Decision latency is the amount of time it takes for a team to make a decision in response to a business change. For even the most sophisticated teams, the time between noticing a troubling change, formulating questions, reaching an answer, and taking action can be days, and in some cases weeks. Analysts and business leads must find and process the relevant information, generate numerous queries to get to an answer, and present the insights in a clear, actionable format.
Decision latency should be a charter metric that leaders track. The companies that thrive in today’s economy, especially in periods of extended uncertainty, use data as a competitive advantage to adapt rapidly to changing conditions and continuously optimize their business. Unfortunately, with few exceptions, decision latency often increases as companies grow. The longer it takes to get the answers and make truly data-driven decisions, the more missed opportunities can pile up. This latency can be the cause of increased risk in the business, with critical errors in product, targeting, or engagement going unnoticed under a pile of unanswered questions and complex data— resulting in lost revenue, disappointed customers, missed first-mover advantages, and more.
Most damaging to a data-driven culture, a delay between business requests and actionable insights forces business teams to continue as usual, making decisions based on best guesses and instincts, and undermining the goal of building a truly data-driven culture.
Why problems of decision latency persist, despite advances in technology
In theory, the growing complexity, dimensionality, and speed of data available to businesses today should make decision-making easier. But these major advances in data engineering, storage, and compute are both the poison and the cure for decision latency.
With more data available than ever before, data teams find themselves facing a new problem: knowing where to look in the data and where to focus their attention. While much has been invested in building faster databases and infrastructure to solve this dilemma, an incrementally faster database will not meaningfully reduce decision latency.
The problem is, the rate at which an analyst can point and click through today’s analytics tools is a snail’s pace compared to how quickly new data is generated. Analytics tools are overly complex, requiring decision makers to rely on analysts and data scientists to deliver answers and context from their data. And even with new tools designed to enable self-service analytics, it’s still on the data organization to prepare and maintain the data, which often requires custom efforts for each new metric or project.
Finding the root causes of decision latency
To start understanding where the delays are in your teams, you need to examine where in the business process you’re experiencing friction. This means inspecting four main areas: data availability, observability, analysis, and action.
To assess data availability latency, determine if you have the right data easily available to make everyday decisions about your metrics. Does your analytics team know where to look within the data for answers? Do they have to waste time with manual, tedious data prep for every question?
When metrics are stable, it’s easy for teams to deprioritize the close monitoring of dashboards, causing observability latency. To root out problems in observation latency, ask yourself when something does change in your key metrics, how long does it take for someone to notice?
Analysis latency is exacerbated by data complexity. And if analysts are relying on legacy BI tools to understand what is changing, they’ll spend hours slicing and dicing the data — leaving little time to answer why the data is changing and to come up with a recommendation. When the business asks follow-up questions on the analysis, analysts are forced to go back to the drawing board, prepping the data again to test a new hypothesis, and re-running the analysis, slowing down the time to decisions even more. To combat analysis latency, you need to make more of each analyst's time by automating the rote, time-consuming pieces of analysis, and cut down on the time analysts spend finding, prepping, and exploring data manually. This will free them to test every possible factor that may impact their metrics, and deliver detailed analysis on both what changes in metrics and why.
With fast access to comprehensive analysis, the latency between data and decisions will decrease. When a business team has access to the answers they need, in a format they can understand, they’ll be empowered to act confidently based on their data. Action latency can occur when that “last mile” translation is delayed by unclear requirements, poor communication, or a lack of business context with the data teams. Here, trust and practice between data teams and decision-makers is critical. When analysts engage with their stakeholders consistently and clearly, the pace of decision-making overall increases.
Understanding your decision latency is the key to a data-driven culture
Building a truly data-driven culture can never happen until you reduce or eliminate your decision latency. To get there, data organizations need to begin identifying the hidden causes of latency within their business processes and address the problems head on if they ever hope to truly ask, answer, and act on data in real time.