How to get business users to ask better questions from their data

On October 5, Katie Wagner asked this question on Twitter. It made me realize that I’ve been meaning to write a blog on this topic for months! After all, I’ve been working within teams of smart people to get to the root of this question for 20 years.

In the early 2000s, I had the privilege of leading the product management function for the release of Cognos Metrics Manager (CMM).  The promise of CMM was to provide business users with a small set of clear business metrics.

In this role, I met thousands of business users. They ranged from senior executives to individual contributors. It was inevitable that they would complain that they had hundreds to thousands of important metrics so where would they start? 

With a single question - “with those metrics, what happens when the light goes red?” - I could get the list of endless metrics down below 10 almost every time. This is a partial answer to Katie’s question and the title of this post. That is, asking a variation of “what happens if you uncover something in this chart” will likely lead to more impactful dashboards.

The second part of the answer goes to the insights needed when the light does go red! For each metric, there were answers to 5 questions:

• Is this a trend?

• Is someone else (I trust) already on a solution?

• What are the underlying details that make up this miss in performance?

• What other factors could have led to this miss and what might happen if we don’t get on top of this now?

• Do I trust this number?

Over the last 20 years, it has been universal that everyone I speak to agrees to those five questions in one form or another. That is, they cover everything people need to get to the root of most operational problems. 

One of the things hindsight taught me that we got wrong with CMM is that:

• we forced business people into too many clicks (too many pages)

• we didn't let business users pick the appropriate visualization to answer the five questions

• we always answered the questions in the order presented above - even though many people had a different thought process to get through them

(We were ahead of our time on some other things though - hovering on metric would bring up a trend sparkline… viz-in-tooltip 15 years earlier!)

This brings me to the second part of the answer. What if we designed all dashboards with those five questions in mind?

A change in conventional wisdom from this:

To this:

In my experience, the first dashboard almost always leads people to download to Excel. Other times, it leads to vague questions for an already overtaxed data science team. Either way, answers come too slow and governance is out the window.

The second dashboard incorporates visual best practices to call attention to important insights. It should also incorporate statistics and machine learning to help with predictions and forecasting when appropriate. The questions don't have to always be in the same order and the visuals vary based on the data. It can incorporate the flexibility that we didn’t have with CMM.

What about the answer to “do I trust this number?” Modern analytics platforms tend to solve this problem with data catalogs, lineage, and impact analysis. Tableau’s Data Management offering is an example. 

Most modern analytics platforms also handle "Is anyone else working on it?” natively as well.

Finally, the second dashboard also works well with alerting. That is, bring back management by exception - why stare at a dashboard when everything is going ok? I think the recent Salesforce acquisition of Slack and subsequent product announcement point to enterprise software getting serious in this direction too.

This approach works well for known questions. What about the ad hoc questions business users have? That answer is for another blog post!

PS I don't see the second visual as a "dashboard" but that too could be a post on its own.