As your organizations catches analytics fever—and as vendors hype analytics seemingly without restraint—it’s good to remember that information is not inherently valuable. In fact, without the requisite thoughtfulness and wisdom, analytics can actually drive people to do exactly what they shouldn’t do.
Here are four analytics-related errors that have become especially common and pernicious:
Analytic error No. 1: Bad metrics that drive bad behaviors
Some data merely appears useful, but isn’t. A classic example is call center talk-time. Call switching systems capture lots of data about caller and agent behaviors. Unfortunately, this has led some call center managers to obsess about talk-times as an indicator of staff productivity.
This can be a fatal metric. When you incentivize operators to wrap up calls quickly, you also incentivize them to deliver bad customer experiences by ending calls prematurely. The result: Customers leave you for your more helpful competitors, and your productivity goes down as operators take second and third calls about issues that should have been resolved the first time.
Similar mistakes are now being made in DevOps environments, where managers are obsessing over metrics such as re-work. But re-work can be good if it’s driven by customer feedback—and if it happens sooner, rather than later.
Takeaway: Make sure your analytics are truly designed to drive the right business behaviors.
Analytic error No. 2: Assuming that good insights drive good actions
Analytics teams often make the mistake of assuming that analytics themselves will inherently make the correct course of action apparent, especially if the data nicely visualized.
But consider this classic thought experiment. A fleet operator has an equal number of vehicles that get 10 miles per gallon and 20 miles per gallon, respectively. All vehicles travel the same 10,000-mile distance annually. The fleet manager has enough budget to either convert the 10 miles per gallon vehicles into 20 miles per gallon vehicles—or, for the same money, convert the 20 miles per gallon vehicles into 50 miles per gallon vehicles. Which is the smarter move?
Looking at the typical bar-chart visualization, the 30 miles per gallon savings will look more much impressive. However, the 10-to-20 miles per gallon conversion saves 500 gallons per vehicles—while the 20-to-50 miles per gallon conversion only saves 300 gallons.
Do the math if you’re still scratching your head over this one. But the point is this: Accurate analytics and intuitive visualization are often insufficient to drive the right action. Clear analytics can even lead decision-makers down the wrong path.
Takeaway: Provide decision-makers with “what-if” tools to project the consequences of their actions.
Analytic error No. 3: Delivering good insights in bad context
Even with great data and sophisticated analytics, you can still wind up misleading your decision-makers by failing to properly contextualize your presentation layer.
Consider stock prices vs. genetic profiles. Charts of a stock’s price over time often use a truncated Y axis that does not go below the stock’s lowest price during the period presented. This is done for graphic efficiency. But it’s also deceptive, because a stock that only fluctuates between $130 and $140 can appear to have been careening wildly. A non-truncated Y axis provides a much more accurate view of fluctuation.
Genetic profiles are exactly the opposite. Since 99 percent of our DNA is common, a chart with a non-truncated Y axis will show almost zero difference between cases, so it’s logical to truncate the Y axis.
Customer sat scores offer another example. Say you design a dashboard to generate alerts whenever a score is below a certain threshold. Seems reasonable. But some sat scores may be more important than others. Sometimes your biggest customers are especially important. And sometimes they’re not. For example, if you already have 90 percent of a customer’s business, they’re probably pretty committed to you—so there’s not much more incremental business to be won. A customer who isn’t giving you much business today, on the other hand, may have huge upside potential. So those are the customers who you should watch like a hawk.
Takeaway: Contextualize all analytic insight in ways that relate to business/P&L objectives.
Analytic error No. 4: Quant insights negated or offset by non-quant factors
Numbers are extremely important in business, but they don’t always tell the whole story. Companies that lease equipment, for example, can now get telemetry that provides rich insight into how that equipment is being used. Some of that information can be useful to customers and should be passed along to them as a value-add. But some of it is best left alone.
In some cases, inaction is advisable because it could be politically offensive for the leasing company to start critiquing the behavior of a customer’s operators in the field, which isn’t likely to be remedied anyway. In other cases, wear and tear on the equipment may generate higher service revenues for the leasing company. Either way, there is no business case for action on the analytic results, even though a data scientist without a full understanding of the business might think there is.
Takeaway: Temper analytics delivery with hands-on expertise about the business and its customers.
