We recently hosted a webcast conversation between Mark and Marilyn focused on the importance of metrics and how visualizing data can help foster better learning and more improvement for your organization.
Don’t have time for the full webcast now? Catch the webcast highlights and tips from their conversation in our companion blog below.
If you’d like to read the full transcript of Marilyn Gorman’s conversation with Mark Graban, you may download it.
The Noise in our Numbers
For the past twenty years, Mark Graban, author of the book Measures of Success: React Less, Lead Better, Improve More and senior advisor to the software company KaiNexus, has been paying attention to how companies and startups use and react to metrics. “Everybody’s got goals and objectives and targets and the action or reaction — or maybe overreaction — that occurs every time there’s an up and down in metrics,” he says, “people feel like they’re taking action, but it might not always be the right action.”
Mark believes that it’s important for companies of all sizes — from startups to big corporations — to take a step back and evaluate how they look at metrics. It can be easy for a company to look at the numbers and react to every uptick, downturn or minor change — something Mark likes to call the “noise” in a metric. But he thinks it’s important for companies and individuals to determine whether or not that noise is a meaningful signal or just a standard fluctuation around an average that doesn’t need to be addressed.
But, Mark emphasizes, it doesn’t have to be complicated. A lot of companies use metrics to hyper-focus on growth, but what they should be using them for is better learning.
Stop Overanalyzing and Start Visualizing
That’s not to say you need to start obsessing over the numbers on a page. In fact, it’s just the opposite. Mark warns that searching for a cause or reason in every dip or increase ends up consuming a lot of time. Not only that, it can distract from the bigger picture of determining whether or not that fluctuation is statistically significant. Or worse, it can make you find false answers or try to find someone or something to blame. Instead of placing blame, “we need to look at things from a systems perspective,” Mark says.
That’s why Mark suggests you should first work on visualizing the data — something he likes to call “plotting the dots” — by drawing a chart to help provide insight into whether your metrics are indicative of long-term trends or just short-term variations. Once your numbers are plotted, you can add three lines to the chart, marking the baseline average and the upper and lower natural limits depicted on the chart.
A chart like this is what he calls, “the voice of the process” and it helps you determine a range of the expected variation in your metrics. Once this is done, it’s a lot easier to determine whether or not there’s a signal in the metrics that needs to be addressed.
To do this, there are three rules he typically uses:
- Look for a single data point above or below the lines that demark the upper and lower limits.
- Look for eight consecutive data points that are either above or below the baseline average.
- Look for a cluster of three out of four data points that are closer to the upper or lower limits than they are to the baseline average.
By systematically visualizing the data this way, it helps leaders to take a breath, filter out the noise and focus on the signals worth reacting to. “Instead of overreacting to every up and down,” Mark says, “try to do things that shift the average level of performance up or maybe….shrink variation in performance.”
You Don’t Have to be a Numbers Person to Use Data Well
While this methodology is incredibly effective at strengthening cause and effect thinking, Mark cautions to take an honest approach when analyzing your data and avoiding what he calls “improvement theater.”
“There are times when… we can massage the data or cherry-pick numbers to make it look like we’re improving,” he points out, so it’s important to determine the improvement story the data is actually telling and not the one that you want it to tell. So it’s important to avoid reacting to every single thing that you see from week to week and to trust the voice of the process.
Even though this is a data-driven process, it’s something everyone can do — you don’t need to be a math whiz to figure it out. “I think a lot of times people get scared off by the real hardcore statistical analysis that we were taught in a university setting,” Mark says. But his methodology is very visual, and was cultivated to be applicable and valid for use in the real world, by real people — even if you’re not a numbers person.
Ultimately, by taking the time to read your metrics better, you can learn how to distinguish between noise and signal to have better learning and more improvement in your growth and processes. And since data is king, it can only help us to have the patience and discipline to let the metrics and data do their thing.
If you enjoyed this blog post, you can sign up to receive more of Lean Startup Co.’s stories.