NPS — 9 reasons not to use it

There’s lots on the positives out there, here are some negatives. I believe we can do better.

Are you using NPS? A great deal of companies are using NPS nowadays. Yet, don’t let it’s popularity fool you. If your business relies on referrals, it might be one of the measures you should use; however, there are a good number reasons not to use it. To be successful, make sure you are using sound principles when choosing your metrics.

1. Recommendation is the wrong question

Most people use NPS with the admirable goal of creating products people love enough to recommend. However, there are some issues with asking about recommendation.

• Don’t want to recommend: in many cases customers don’t actually want to recommend your product (e.g. many B2B). In this case, it could be specific to your type of product, it could be that they don’t have friends/colleagues that are suitable, or it could be contextual (e.g. customers might perceived a competitive disadvantage in sharing with peers).
• I’m not someone who recommends: in many cases customers recommendations are something they do or don’t do individually (regardless of experience with the product) — am I someone who makes recommendations to friends/colleagues plays a big part.
• Wrong data: if you want to know about satisfaction, it’s better to ask about satisfaction directly (instead of asking about recommendations). We sacrifice accuracy for convenience of measuring an action. But, we still don’t measure the action, we measure the prediction of an action (we are highly inaccurate at predicting the future, see HBR’s article Where Net Promotor Score Goes Wrong).
• Confusing word choices: the customer needs to decipher what does it mean to “recommend” something. Does the recommendation have to be initiated by the customer or if they are asked by their friend or colleague? What is the customer recommendation based on: the last interaction, overall, or some part of the product in between?

2. Tempting to improve NPS without achieving actual goals

When we set targets for NPS, we are hoping that it will align our organization around improving the lives of our customers.

However, NPS does not make a good target metric because it’s easy to manipulate without changing what’s actually important. If we solely focus on NPS, we can improve it by asking it at only the right times. True of any single measure, but specifically, a “recommendation to a friend/colleague” is something that can be asked at better times in the customer journey to game the metric. There are countless ways to manipulate NPS without improving what actually is important.

One way around this issue is that you can choose to measure NPS without targeting it.

3. It doesn’t help with what to do

Once most companies have collected NPS, they are hoping to improve it. “How can we use these data to improve the livers of our customers?” is an admirable question to ask.

The problem is we don’t know why they aren’t recommending to a friend. Is it the product’s UX, bugs, features, or maybe it’s the company’s reputation, status, brand? Unfortunately, NPS doesn’t help us with what to do next. Most often, teams are told to just do “something” to raise it… and you can just hear the whole team groaning… “grrrrreeaaaatttt, thanks Mr/Mrs Manager for that”.

For example, even if we use the more specific version of “Based on your last interaction with _____, how likely are you to recommend…”. We still wouldn’t know what went wrong. Was it a bad Customer Support agent, bad policy, bad brand, etc.? This gives us two bad options: ask this question far too frequently to ask about every interaction or focus on one area and ignore the bigger picture.

With NPS, you’ll always need better data to help make decisions. Every person or company I’ve seen selling and advocating for using NPS still require sections of their business or promotions on “what to do with detractors”. And, in those sections they advocate getting more/better data about the detractors. The same goes for getting more data and leveraging the promoters. So, if you want to do anything with NPS, the answer is go beyond NPS, which begs the question, why spend time on NPS in the first place?

4. Trailing metric

We usually find out about it too late. “Customers were unhappy last month”. A mix of predictive and trailing is ok. But the standard for trailing metrics should be much higher than what we accept with NPS.

5. Too simplified and too complicated at the same time

NPS starts out as a nice simple question for users.

Although the question itself is simple, in the mind of our customers, it becomes a bit more complicated for many. It can be unclear to customers why they are being asked about recommending. For many products, they wonder why would they recommend or even “is this a trick to get my name/email?”.

In addition, we often add on more questions or open comments (e.g. asking “why”), which ruins the simplicity. On the other hand, the “why” follow up question is probably the most valuable information one gets from NPS.

Finally, the scoring and calculations (although seemingly simple again) just muddy the data for most audiences (why 11 point scale, why ignore some scores? etc).

6. Often lacking baseline comparable

Having baselines from your industry could be a huge benefit of a popular metric. Whereas, custom metrics have no comparable. However, in every field I’ve tried to use NPS, we lacked a true baseline to compare to. We maybe had to reach for a comparable or had nothing at all.

Some industries have a published baseline one can use for comparison. These baselines are difficult to apply directly and usually leave room for questioning the baseline. In addition, if you have an industry baseline, it becomes dangerously tempting, to get back to the earlier point, to game the system to try to meet your competition — when did they ask? who did they exclude? “Let’s get this number up right away I don’t care what it takes”.

Further compounding problems, managers and executives stretch the boundaries of their “industry” in order to have a baseline (or a more favourable baseline) to compare to.

Baselines are especially important because the NPS number is an invention, it doesn’t really mean anything on it’s own. A fair question is “our NPS is 10, what does that mean for our industry, target market, product domain, etc.?”. All industries without a baseline are left wondering, what’s good or bad NPS for us? Allowing only relative comparison’s to our past numbers of getting better or worse, but never what is good or bad.

7. Misplaced trust

Trusting our data is critically important. As such, many investors want to see a number they are familiar with, which leads to lots of NPS requests.

Unfortunately, this trust in NPS is misguided. It’s over-indexing on that as an executive or investor I understand what the number is “suppose to represent” over understanding what it is actually measuring (which is usually more complex).

Similarly, within the organization, being data driven (data is central or only information used) vs data informed (data is one piece to aid in understanding) is already and will continue to be key as the world has more data. People, teams, and companies become overly reliant and data driven around NPS far too quickly. They put trust in it because it somehow “feels right” or it’s “popular”. These are not valid reasons to trust it.

It’s such a dangerous slope for all of us and all data, especially as we get more data; paramount here is the high level of unearned “trust” in NPS makes it more dangerous than other metrics. The only thing worse than not having data is misinterpreting data — you go from “unsure” (accurately assessing your situation) to “sure and wrong” (misleading assessment of your situation).

8. Falsely customer centric

Focusing on customers is one of the best recent developments for a companies looking to create successful, fulfilling, and impactful products over the long-term.

A customer centric company should focus on the customer — not themselves nor the competition. By the definition of “customer centric” these companies should focus on measuring and improving the value they are providing to their customers. Yet, many customer centric companies rely on NPS to measure their “customer focus” and NPS does not do that well.

As mentioned in the first point above, NPS is measuring reported “satisfaction” (remember this satisfaction is actually: the correlation of satisfaction with an interpretation of a 0–10 rating of a customer’s guess of recommendation likelihood that relies on human predictive abilities and individual interpretations of nebulous terms “likely” and “recommend”). This is not the same as measuring value or benefit to the customer. To be clear, NPS is measuring the customer’s reported “satisfaction” (which is not actually satisfaction) with the company’s current solution — the NPS question is solution/product focused, not customer focused.

Further, while this measure can be used to compare to our own past measures (are we better or worse), it often relies on comparison to competition baselines (what is good or bad). First, comparing to your past version of your product is not customer centric, it is again product or solution centric (it’s about you, not them). Second, NPS is most meaningful and useful when compared to competition. And, if it’s about the competition, it’s not customer centric.

9. Wrong decision makers

I’m fortunate enough that almost everyone I’ve ever worked with wants to create empowered teams (or I like to say “high trust teams”). One of the ways to create these types of teams is to trust that they can collect and analyze the right data at the team level. If we trust the team to do that, we do not need to have NPS as a higher level metric.

Many senior managers, executives, and board members like NPS because it gives them the perception that they understand what is going on, without going into the details — because it’s impossible for them to get into the details. Usually, this gets in the way of anyone actually getting better details. The focus is on getting the numbers for the high paid people. And, even if some of the team members get more detailed data (in addition to spending time on NPS), it’s extremely unlikely those details will enter into the decisions that are being made at the company. If the higher levels have data to act on, they will usually dictate a top down strategy to go with it. Resulting in less informed decisions being made.

It’s almost always insufficient to measure a single number. In my experience, requiring a single number means that the decisions are being made by people too far away from the situation. Therefore, one should, instead of requiring this simplification to one number, work on the other side to push the decision making down to the team who actually has the time and the information to make good decisions.

Maybe, one unintended good use of NPS is that… if you see NPS being used at a company, see it as a red flag of potential for more serious issues. :)

What do I recommend?

This is probably best summed up by “how to choose a good metric” which would be a whole other article. However, to leave here on a positive note, I think we can learn a few good principles from the above:

1. Limit variables that can influence the response. For example, ask about what actually happened — instead of “would you”.
2. Target metrics should only be achievable when achieving desired outcomes for customers.
3. Before collecting, make sure you will be able to do something differently based on the data.
4. Use a mix of leading and lagging metrics.
5. Keep metrics as easy to interpret as possible.
6. Make sure what you are measuring is what is important for your success.
7. Make sure what you are measuring is what you think you are measuring.
8. Make sure what you are measuring aligns with the values of your company and your customers.
9. Better decisions require multiple measurements and discussions at the level closest to the problem.