I’ve never been a big fan of detailed measurement and data tracking.
Not because I don’t think it’s useful. It can be incredibly useful. But it is by far the easiest way to be deceived. Bringing about a desired end in a complex world of autonomous individuals requires the ability to recognize patterns. Patterns in motivations, words, behaviors, actions, and reactions.
Gathering data does not reveal patterns. Analyzing it rarely does either. But it’s almost impossible to not think you see patterns from the data. Data tends to make people draw conclusions and most of the time they aren’t warranted.
At its best, measurement is done based on a pattern already spotted by some other, more direct and less aggregated means. To measure the veracity of the pattern, or check its conditions, data is gathered and assessed. Ideally, the data is used to falsify a hypothesis. It works better at falsification than verification.
Case in point: I had a strong hunch recently that users of Crash were dropping off because of where the signup page was in the product flow. But we looked at the numbers and what I thought was the biggest roadblock was stopping almost no one, and they trailed off later in the flow. The data was only useful once I had a specific – falsifiable with data – hypothesis. Note the data did not tell us whether we needed to improve the signup page. It can’t tell us that. It only revealed that my assumption about the signup page being the most frequent hurdle was incorrect. The signup process may be flawed in myriad ways, and no data can reveal exactly how and why.
Data can work well as a way to narrow in on insights, as long as the data gathering is a genuine effort to increase understanding and not just a way to slap numbers on a decision you’ve already made.
It’s exceedingly rare to be collecting data for no particular reason, scanning it with no particular question, and discover a genuine and valuable insight. But we all kind of pretend that happens, which is why data can be dangerous.
It’s important to be able to recognize and admit when the data don’t provide any clear patterns or insight. This is most of the time. Just because you have numbers doesn’t mean they tell you something and you need to act on it.