I recently spoke at the Tech in Asia Product Development Conference in Jakarta about being data-driven versus data-informed. Post-conference I was approached a few times to discuss more and speak at other events about the topic. Since people who attended my talk found it useful, I thought I’d share it with the global community.

The term data-driven has become a buzzword, adding on to the list of other cool, techie jargon. This includes disruptive, pivoting, design thinking, and many more. So, let me first attempt to get us all on the same page.

What’s Data-Driven?

Being data-driven means letting data be the center of a team and company’s decision-making process. Data then plays an essential part of your company. Decision makers reply mostly and sometimes solely on data. Some decisions can be made without human being involved.

Some of the conversations that came up from a data-driven company are:

• “Let’s A/B test this, and implement the version with the better result”
• “Don’t talk to me unless you have have data”
• “We’ll just follow what the data tells us”

The data-driven organization will implement this process across teams and functions. This means that every team such as product, marketing, customer success, and operations uses data intensively for all their decisions.

What are the benefits?

It eliminates a lot of human bias involved in decision making

In fact, a mentor of mine Misha Chellam created flashcards to help you memorize them. Data helps get the team members and stakeholders on the same page with less of their own judgement.

It is less time-consuming because we take human out of the equation

How many of you have been in a 3-hour long meeting that people argue back and forth with no sign of compromise? Even better if you work in at a startup. Sometimes we discuss and argue for the sake of doing it, then the HiPPOmade the final decision. Yep, I know how you feel.

In fact, a study from MIT has proven the benefits of being data-driven. Prof. Erik Brynjolfsson and his colleagues studied 179 large publicly-traded companies and concluded that those companies are 5% more productive and profitable than their competitors.

However, being data-driven also has its drawbacks.

It requires a huge amount of data for the decisions to be accurate

You have to watch out for outlier distortion. Outliers are a data points which are different from your average data. In gaming, positive outliers are considered “whales.” Those are customers who either make big purchases or make them very often. They generate a significant amount of revenue for the company.

At my previous company, we even coined the term, “whale hunting.” In other industries, it might be those customers who use your product to solve a different problem than most of your customers.

Their behaviors are so different they should be excluded from your data to make decisions about your average customers. A few data points together with the outliers can lead your company toward the direction that you don’t want to go.

It requires someone with data science knowledge and a lot of resources to be beneficial

For startups and organizations that just started a Data Science department, it might be hard to be completely data-driven due to lack of capabilities and resources. Current employees might not have enough knowledge to build the infrastructure. Some might not even have anyone who has expertise with data.

At my previous company, we had 10 data scientists. In my current position at Kulina, we have 0.5 person (myself and our head of tech).

Related Read: Do We Actually Understand Data?

There is still a bias in the way we gather data

Most of times, what people say is not what they do. So if we gather data by asking them implicitly, we risk making decisions on the wrong information.

For example, during the scandalous time of Uber and the #deleteuber movement, most of my friends said they would never use Uber again. When Lyft (its main competitor in the U.S.) has a surcharge, they went back and ordered Uber for their regular rides!

Let’s look at an example from my experience

One great example is when we were trying to figure out which social media platform we should spend more time and money to engage with our users. We sent out a survey asking them, and the result was quite surprising. We saw Facebook on top of the list, as expected. Yet, what puzzled us was the fact that Google+ was second on the list (no hard feeling to the team at Google).

A few team members asked if I would agree with letting them spend more time building our Google+ presence. “Maybe we didn’t know our users well,” a team member mentioned.

Being data-driven, I would probably green-light this. But being stubborn as usual, I discussed with another product marketing manager on what could’ve gone wrong.

We came to a conclusion that our users might not understand the difference between Google and Google+. We looked deeper into the way the question was asked.

Instead of using the icon, we used the wording Google+ in the questionnaire. If you didn’t pay close attention to the survey, you may have thought we were asking if you use Google!

What is a data-informed decision?

Making data-informed decision takes data only as a factor that can be dismissed from time to time. This kind of decision making allows for other factors such as customer experience, gut feeling, brand consistency and the HiPPO to take the lead.

The benefit is that data is put to be challenged

Because we don’t purely make our decisions based on data, we can use it to create a hypothesis. No matter how good the data is, it has its limitations. It is only a snapshot of reality that doesn’t paint the full picture of our customer journey and behavior. We need to use other factors to help make better decisions, including our own judgement.

One example of this happened at my previous company. The nature of our gaming business was having great revenue on the weekend because that’s when our users played games. On a Monday, one of the new product directors approached me about a low weekend revenue, and he was very worried.

With some experience and guessing I suspected that our revenue was down because we ran a sale the prior weekend. The same effect occurred for the past five times we ran sales.

If we look at the same cycle before, our average was actually better. We figured out later that it was because of one of the marketing campaigns that we launched to encounter this specific problem.

Facebook’s Newsfeed version that moved the key metrics was actually done without being data-driven

Adam Mosseri and his team back then decided to create the newsfeed without data. In fact, it got a lot of negative reactions that users were angry and even created a Facebook group I Automatically Hate The New Facebook Homepage. Watch his talk here.

“At the end of the day, we have to have a gut to make bold decisions without data.” — Adam Mosseri

This doesn’t mean that data-informed decisions don’t have its drawback

It requires a lot of time to discuss and analyze, and conclusions could not be formed easily. But remember at the end of the day:

“All the data in the world won’t fix a fundamentally bad product.” — Andy Carvell

To be innovative and build the right product for our customers, we cannot just sit in the room looking at rows of data and charts.

Hint: Data doesn’t have to always be big data.

The key to building products is to:

• Gather qualitative feedback by talking to your customers
• Observe customers struggles through usability tests
• Figure out the jobs customers are hiring you for

So, my recommendations to you is

For smaller companies, you should always aim to be more data-driven. You may not utilize the data you collect, or even collect them at all. The more data you have, the better hypotheses you will generate.

For larger companies, make bold decisions that not only move the key metrics but also improve the customer experience. We should use data to help us make better decisions, but don’t rely on it a hundred percent!

Also Read: The Missing Link in your Market Data: Know Your Customers Better

(Disclaimer: This post was originally published in Medium and was reproduced with permission. Techstory is not responsible or liable for any content in this article.)

About The Author:

Casper Sermsuksan worked as a Product Marketing Manager for a social casino company in San Francisco. During that time, he led marketing operations and cross-functional team for one of the top grossing apps on Google Play, App Store and Facebook Games. He currently consults companies and start-ups on growth, retention and monetization strategies. His work doesn’t represent any of the companies he’s previously worked at before.