Building and maintaining the product roadmap is a central part of your role as a product manager. Yet there is surprisingly little consensus about product roadmaps across the product management community. Opinions vary wildly, for example, about what exactly a product roadmap is, how to structure one, what to include in it, and which tools you should use to develop it.

In this post I will offer a few guidelines for how to structure your product roadmap in ways that can lead to the development of a successful product. But before we dive into these suggestions, I would like to start with two fundamental points about roadmaps — points I hope will make the guidelines that follow much clearer.

The top-down approach to product development works best.

For successful product development, I recommend a top-down product strategy. Product roadmaps fit very strategically into this hierarchy. Here’s how it works.

Start with your product’s vision — which you’ll derive from your company’s larger strategic objectives. Then translate that high-level vision into actionable goals. Next, turn those product goals into your product roadmap. Finally, move from your roadmap to your backlog.

Starting at the highest level, and working your way strategically down into the details, is the best way to stay focused and on track toward your main objectives — and to avoid losing sight of why you’re doing what you’re doing and getting lost in the weeds.

And speaking of getting lost in the weeds…

A product roadmap is not a list of features.

A list of features is just that — a list of features. The product roadmap, on the other hand, is a strategic document that represents your high-level goals for the initiative along with an execution strategy to communicate how you plan to achieve those goals.

A roadmap might include features, of course, but the features themselves are only a part of the execution plan.

Five Guidelines for Structuring Your Product Roadmap

1. Product Roadmaps Should be Flexible

You’ve got to be okay with uncertainty. You’ve got to be willing to embark on your product plan, knowing that you can’t possibly know everything at the outset — that you will hit surprises, challenges, and opportunities along the way. Your roadmap needs to factor in these uncertainties — and needs to be flexible enough to allow you to change course quickly when necessary.

To avoid making promises you can’t keep, make sure your stakeholders understand that the roadmap is not an end-all, be-all document. One way to do so is to show more granularity in the short term while keeping your initiatives high-level and your dates approximate in the long term.

Thinking in themes is another effective way to keep your roadmap flexible. For example, maybe you and your stakeholders have agreed to focus on increasing conversions from a particular target persona in the upcoming quarter. This area of focus becomes your theme, and you retain the ability to reprioritize specific features and fixes within that theme while keeping the overarching goal intact. Likewise, if an opportunity arises that was not originally on your roadmap but furthers your shared goal, you’ll have an easier time making a case to include it.

Finally, a flexible roadmap can be a headache if changes are not communicated promptly and effectively. Ensure your stakeholders always have access to the most up-to-date version of the roadmap, whether that means storing it on a shared wiki or using cloud-based software that automatically updates. Transparency is key to building alignment on product strategy, no matter how frequently you need to reprioritize.

2. Product Roadmaps Should be Deeply Rooted in Well-Thought-Out Goals

This might sound like a contradiction to the previous guideline — but it’s not. You can frequently reprioritize your backlog while still staying true to well-thought-out, agreed-upon goals.

Your specific roadmap priorities might need adjusting in light of new information — in fact, they almost certainly will at one time or another. But you should be prepared to adjust focus in your roadmap, reallocate resources or otherwise change direction only if doing so is in alignment with your product’s high-level strategic goals and vision.

As a product manager, it’s your job to make sure that you’re making decisions about what to do, and what not to do, for the right reasons. Even if a sales executive is loudly demanding the immediate inclusion of a new feature to close a deal, you still need to weigh the request in light of your organization’s strategic goals. Quick wins do not necessarily spell long term success. If the feature isn’t in line with your product vision, it’s okay to say no. Indeed, if you’re doing your job well, you’ll find yourself saying no quite often.

3. Product Roadmaps Should be Developed with Plenty of Input

You don’t need to craft your roadmap yourself. You shouldn’t. Silos rarely work for any business function, and for roadmaps they can lead to ill-advised plans and products developed without critical knowledge.

To successfully bring a product to market, or to update an existing product in such a way that benefits the customer and your company, you’ll need plenty of input from experts across a variety of teams and departments. That includes, for example, engineering, customer support, sales, and marketing.

Sales, for example, will have valuable anecdotes about their most recent wins, or their most important ones. They can also tell you about the features prospects and customers are asking for.

As for your engineers, the more you involve them in the creation process, the more ownership and responsibility they will take for their role, and the more creativity and enthusiasm they’ll bring to the project. Collaborating with your engineers, and soliciting their help, will make them feel more like a part of the process, and less like order takers simply being told what to do by a product manager.

Prabhat Jha, CTO of the Net Promoter Score platform InMoment adds, “Be sure to solicit plenty of user input. Some years ago I learned this the hard way. The software enterprise where I worked did not have a rigorous system in place for listening to customers. We had collaborated internally on our roadmap priorities and thought we knew our customer’s needs. We missed key features, though, and as a result, there was very little adoption of the first version of the product. We ended up having to do a second release very quickly in order to get traction.”

4. Product Roadmaps Should be Visual

Ultimately, a product roadmap is a communication tool — an execution strategy you will use to convey your product plans and goals to a variety of constituencies. And the best way to communicate a complex initiative is visual. If your roadmap is simply a long list of features presented in a spreadsheet, people’s eyes will glaze over.

Visual roadmaps make it easy for everyone in your strategy meetings to quickly understand what you are proposing and hoping to accomplish quickly.

Another valuable reason to make your product roadmap visual is that it forces you to be ruthless about which initiatives to include and which to leave out. Venture capitalist Guy Kawasaki’s “10/20/30 Rule of PowerPoint” instructs that PowerPoint presentations should have 10 slides, last no longer than 20 minutes, and contain onscreen text no smaller than 30 points. In addition to making the presentation more digestible for an audience, this exercise forces the presenter to distill both the whole talk and each slide down to its most essential elements.

The same is true for product roadmaps. If you use a visual tool for creating your roadmaps — such as PowerPoint or a visual roadmap software application — that exercise will force you to distill your plan down to only those initiatives that serve your product’s strategic goals and vision.

Looking at this from another angle, if your roadmap contains 1,000 initiatives, that probably means you haven’t done a good job of prioritizing and strategizing what needs to be included in your product. Indeed, if you do have a list of 1,000 initiatives, your document could probably be more accurately called a backlog than a roadmap.

5. Tailor Your Roadmap Discussion to Your Audience

You are likely going to show your product roadmap to many different constituencies, in many different meetings, throughout the development cycle of your product. Each constituent group will have a unique focus and set of priorities, and each meeting will call for you to delve into different aspects of the product roadmap.

There are a couple of ways of accomplishing this. You can create separate roadmap documents for each constituency, or you can use a single, general roadmap and zoom in to what’s important for each group. The important thing is that you provide your stakeholders context and show them how your plans will further their unique goals.

For example, for your sales team, you might want to highlight the aspects of your roadmap that are designed to bring them better-qualified leads — for example, a trial version of your software that can help to prequalify interested prospects.

Customer-facing teams also likely had input into the features that made it onto your roadmap in the first place. Be sure to communicate to them where their requests fit into the plan (or didn’t) and why.

However, when presenting the roadmap to your executives, you’ll want to focus on how the choices you’ve made will lead to increased revenue or grow market share.

And when meeting with your engineering team, you’ll want to focus both on the high-level themes and specific feature details and discuss how your engineers can help to make those product goals a reality.

In other words, you want to keep your product roadmap flexible enough that it can help facilitate a productive meeting at any level of detail needed, with any constituency group you are presenting to.  

Conclusion: Whatever structure your product roadmap takes, its main job is to communicate your strategy.

A product roadmap needs to communicate your strategy. It is your job to create your roadmap in such a way that it lays out a high-level execution plan for the product’s successful development and eventual launch into the market. It’s also your job to make sure all relevant constituencies understand the goals of the roadmap — and work with you to achieve them.

New things will always come up — cool ideas for new features, requests from executives for a shift in priority,  urgent demands from sales reps, and so on. The challenge for the product manager is to view each one as an opportunity to evaluate through the lens of strategy and goals and help drive a sound decision-making process.

That’s why you need to start with your product vision, and from there derive specific goals — and only after you’ve developed those goals, build your product roadmap. If you haven’t fully fleshed out your vision and strategic goals for the product, it’s too early to start building your roadmap.

About the author:
Andre Theus is the Vice President of Marketing at ProductPlan. He works closely with customers and prospects to build better product roadmap software. Prior to ProductPlan, he was a member of marketing teams at RightScale, Sonos, and Citrix. Andre received a master’s in computer science from the Cologne University of Applied Science in Germany.

In this guest post, Nathan Lippi, Head of User Research at PandaDoc, shares a Pareto principle approach to getting the most from a B2B Net Promoter Score program. 

NPS. It’s debated, loved, and hated, but in the world of B2B SaaS it’s rarely used to its full potential.

At PandaDoc, we’ve become increasingly customer-obsessed since the introduction of our NPS program two years ago, but we feel as if we still have meaningful room for improvement.

We’ve found there isn’t much written about NPS, specifically for B2B companies, so in order to level up, we’ve gone straight to the experts. With their permission, we’re sharing some key findings here.

We hope this guide helps you to get the most out of your CX program.

Let’s get to it!

The main purpose of NPS is to drive action

NPS is an easy, trusted, and benchmark-able way to start driving customer-focused action at your company.

Many companies obsess too much about the number when they’re starting out.

However, the most successful companies never lose sight of the fact that the primary purpose of customer experience metrics is to drive customer-focused action.

Once you’re driving customer-focused action, you’ll start to actually reap the benefits of increased retention, expansion, and word of mouth.

One Oracle VP’s Three-Step Recipe for NPS Survey Success

Joshua Rossman is an NPS OG, having run NPS at eBay and McAfee, among other companies. He’s now Vice President, Customer Experience Strategy at Oracle.

Through his years of experience, Rossman has created a three-step system he uses to get the most out of customer experience surveys, including NPS. He’s been kind enough to give us permission to share it publicly.

Step 1: Ask an easy-to-answer anchor question first to improve response rates

This principle is standard for NPS, but powerful enough to use across other CX surveys.

Ask your broad question first, and get a quantitative rating. Making your first question easy to answer will improve your overall survey response rates.

Step 2: Get S-P-E-C-I-F-I-C with your open-ended ask

Rossman has found that the standard open-ended question, “Care to tell us why?” often leads to vague, inactionable responses (e.g. “It’s hard to use”).

He’s found that asking promoters for specific reasons they recommend — and non-promoters for specific ways to improve — leads to much more actionable feedback.

Here are the specific questions he recommends for brand-level NPS:

Promoters: “What is it that makes you most likely to recommend {{company}}?”

Non-promoters: “What is it we could do that would make you more likely to recommend us in the future?”

These questions ask more specific questions — and tend to get more specific answers.

Various platforms such as InMoment can help you automatically categorize your now-more-specific NPS verbatims!

Each company will want to tag their work in a way that makes the most sense to them, but Shaun Clowes, former Head of Growth at Atlassian, says that they used machine learning to tag their feedback into three categories: Reliability, Usability, and Functionality. They used the ratio of different complaints to understand, at a high level, where their product needed work.

Step 3: NPS’ Secret Third Question

Even with the more specific responses, you’ll hopefully get from the tweaks recommended in Step 2, not all B2B companies get such a high volume of responses that they can glean mathematically reliable responses from text alone.

One way to gain a deeper understanding of the factors that lead to an excellent (or poor) user experience is to follow questions about satisfaction with questions about various attributes of your brand. Ask a few extra questions with NPS and you can capture the overall sentiment for each area:

After you’ve captured these details you can then run a simple linear regression, which will tell you which factors most influence if a person is a promoter or a detractor.

Various versions of the linear regression technique were also mentioned by Allison Dickin, VP of User Research at UserLeap, and other experts.  Hearing them reinforce the power of this third question helps us get really excited about what we might do with it.

“Extra questions should be used judiciously,” counters Jessica Pfeifer, Chief Customer Officer at Wootric. “Think about it: When was the last time you responded thoughtfully to a multi-question survey?”

If you’re worried that such a long third step may lead to a negative user experience or lower response rates, a lighter option may be to ask the respondent to tell you what drove their score by selecting from a pick list of reasons.

If your follow-up question to detractors is, “What is the main thing we need to improve?” you could offer a picklist that includes product, support, training, and value.

Not only are you learning what’s driving your score overall, but you’re also generating groups of users to follow up based on their interests. For example, your customer support team can learn more by reaching out to detractors who cite “support” as an issue.

Drive Strategic Action with a Cross-Functional Cadence

You may have noticed that our first heading was about driving action on behalf of the customer.

We’re touching on it again because, ultimately, driving action on behalf of your customers should be the primary concern of an NPS program.

Driving tactical action on behalf of customers was something we were already doing well at PandaDoc, before talking to the experts. Getting NPS data into Slack and other systems has been a pillar of our NPS program — this helps us take immediate action on issues that surface in feedback. One example: reaching out to an unhappy detractor and quickly fixing the issue that her NPS feedback brought to our attention.

However, learning how many companies drive strategic action on behalf of the customer in the following way, was eye-opening:

• Collect customer feedback in a central repository (NPS, sales feedback, CS feedback, etc. — all combined together, somewhere like InMoment, UserVoice, or ProductBoard.
• Perform a 360° analysis of this data on a quarterly basis
• Set up a monthly cross-functional cadence to decide which action to drive, and to track progress and accountability on ongoing courses of action

Fictional Examples of Driving Strategic Action:

Product Team

Diagnosis: Self-serve onboarding is our most common NPS complaint.  People often come away without understanding our platform’s core concepts.

Initiative: Improve self-serve onboarding to teach core concepts of the platform.

Success Team 

Diagnosis: Feedback about CS indicates all roles except admins are quite happy. Admins specifically have trouble understanding how to set user permissions, and they’d rather avoid going through training to learn something so small.

Initiative: Create micro-videos that explain to admins on how to manage user permissions.

Support Team

Diagnosis: NPS feedback indicates enterprise customers are unhappy with the time it takes to resolve support interactions involving custom features. 

Initiative:  Route tickets from enterprise customers directly to senior agents who have the expertise and product knowledge to resolve their issues.

Marketing 

Diagnosis: Many of the leads we’re attracting cannot benefit from our core value proposition.

Initiative: Better align their SEM campaigns and landing pages with promises that the product can fulfill.

Your metrics should flow from your unique business strategy

NPS has been sold by some as the be-all / end-all metric of a customer-centricity program. But this approach can be harmful.

While NPS is often a great way to understand brand-level sentiment, it makes sense to layer on additional metrics as your CX program progresses.

Jessica Pfeifer at Wootric and Allison Dickin at UserLeap agree on the idea that your CX metrics should flow from what’s most critical to your business’ success.

“You’ll be able to benchmark and track trends over time when you complement NPS with established customer experience metrics like CSAT, PSAT, or Customer Effort Score at critical touchpoints in the customer journey,” says Pfeifer.

“For example, you might trigger a Customer Effort Score survey to gauge how easy it is for a user to achieve ‘first value.’ What is that critical milestone in your product? In PandaDoc’s case, it might be sending a document. Here at Wootric, it’s when a customer has live survey feedback flowing into their dashboard.”

Both took time to talk to us about questions that can be used in addition to (or as an alternative to) NPS. Here are some examples:

Example Non-NPS Questions

Takeaways

• NPS is a great way to get started with driving customer-centric action
• Use Josh Rossman’s three-part system to get the most out of your CX surveys, including NPS
• Use analysis and a cross-functional cadence to drive org-wide, customer-focused action
• As your business grows, layer on metrics that fit your specific business needs

This is just the tip of the iceberg for NPS, but we hope it will help your company squeeze the most out of your CX research program.

Hit me up on Twitter (@nathanlippi), and to let me know what’s worked well for you and your company!

So you’ve been reading up on Net Promoter Score. Your colleagues in the SaaS world tell you that it’s the best way to take your customers’ pulse. You’ve seen a few case studies claiming it’s the only number you need to measure.

It’s true that Net Promoter Score is a great way to engage with your customers and solicit tons of feedback. But it’s also true that there are quite a few nuances that result in a successful survey program.

As a SaaS company with SaaS customers like Zoom, DocuSign and Hubspot, we have a unique perspective on NPS in cloud software. To make the most of your time and energy, we’ve put together this list of things SaaS businesses should know before they dive into the NPS world. Read More…

Financial markets are sliding, a pandemic is spreading around the world, and every company is scrambling to respond to quickly changing circumstances. Planned investments that were intended to drive growth — like hiring, media spend and software purchases — are being reevaluated as business leaders are forced to triage what they need to do to weather the storm. We’re all in survival mode, but survival is about prioritizing what is most important.

And what is most important to a SaaS business at this moment?

It’s not toilet paper.

It’s our existing customers.

Now more than ever, customer experience is job #1. 

We think the SaaS businesses that focus on retaining customers and building loyalty are the ones that will survive and thrive in this uncertain climate. 

Of course, the question then becomes how do you retain customers and build loyalty?

Shift from a growth mindset to a retention mindset 

This may not hold true for every business we work with – Zoom, GrubHub, and the e-commerce toilet paper company Who Gives a Crap are having quite a moment. But most businesses are facing contraction because people don’t buy in a panic. Budgets are being trimmed everywhere, and customer success and renewal conversations must be deeply empathetic to this.

So, if a customer is achieving goals with your software, and you have other features and capabilities that will make them even more successful in 2020, then, by all means, paint a bold vision of an expanded partnership. But, if that isn’t the case, and they want to reduce or leave, don’t come across as tone-deaf. It’s likely that everyone in their company has been asked to find ways to trim spend.

That means that it’s even more important to know that your internal champion can confidently advocate for you – because you are delivering value. Step up your customer success initiatives. Make sure you and your clients are recording successes. And don’t be afraid to change the conversation from trying to get the customer to buy more, to showing him or her how the company can get more value from what they’ve already purchased.

Listen to your customers even more carefully – and respond

Even when you’re focusing on your existing customers, don’t make the mistake of thinking they’re the “same” customers – they’ve changed. We all have. Because our needs change in a downturn. Companies that are on the pulse of those changes by proactively listening are better poised to adapt, innovate, survive, and serve.  Make calls to key customers.  It’s even more important now than it usually is to listen and respond to concerns quickly. 

Take care of your people

Your employees, your teams, are the key to your customer relationships. They may be concerned about their health or the health of their parents, or grandparents. They may be struggling to find childcare options if their schools are shut down. They may be stressed about their 401K balance. Whatever it is, empathy and flexibility are going to be key – and so is prudent business planning. How can you plan to support your employees through these challenges?

Be a good citizen

For the collective good, and for the good of your brand, it’s so important right now to prioritize the good of the community and show conscientious, caring judgment. To do this, you may need to make some tough calls that hit your short-term profits, but protect people. We’re all making sacrifices – the New York Times dropped its paywall for coronavirus news and Zoom is giving K-12 schools free video conferencing. Is there a way your company can help people in need right now? You’ll be remembered for it. 

Establish a company policy of flexibility

Just as you have to be flexible with your employees and their quickly-changing challenges, you also have to be flexible as a company. For example, if a client calls customer support to request an extended payment plan, empower your support team to deviate from your standard policies and allow it. Be open to changing how you usually do things if it makes sense and shows compassion. You’ll likely prevent avoidable churn.

Jessica Pfeifer, Chief Customer Officer at Wootric, shared this recent story with our team:

“I just had a customer reach out about putting their subscription on hold. They operate in the hospitality sector which has been particularly hard hit. We offered to work out a plan to enable her to continue to get customer feedback during this critical time. Her response was ‘That would be amazing! Thank you!’ I know we’ve strengthened customer loyalty.”

Customers notice the companies that support them in difficult times, so be flexible when you can, and you’ll build loyalty for the future.

Close the loop with customers when they offer feedback

This may be built into your CX program already, but if not, now is the time to double-down on listening and responding to customers. Ensuring your customers feel heard and cared about in times of high stress carries more weight than when times are easy. So if a customer responds to one of your surveys, be sure to close the loop and let them know you value their time and will take appropriate action. 

Customer success managers can reach out one-on-one via email or phone, but that isn’t always practical. Closing the loop can be automated when you have your feedback readily available in systems like Intercom or Salesforce. Here’s a quick guide on how to automate closing the loop on customer feedback.

Improve customer experience at customer journey touchpoints

In SaaS, this often means using an NPS survey to gauge overall loyalty and surface any issues that may affect renewal. To get serious about retention, consider asking for feedback at critical SaaS journey points–after onboarding, support interactions, and during product/feature use.

This isn’t about quickly adding a slew of new surveys overnight; it’s about prioritizing improvements to moments that, if not successful, can sow the seeds of churn. Now is the time to double-down on understanding and improving the customer journey.

Also, remember to analyze the qualitative feedback from these surveys. A customer who is “satisfied”  but mentions a concern over price may now be at a higher risk of churn.

Focus product development on reducing friction for existing customers

In the software business, product experience is the all-important driver of customer experience. So, to foster customer loyalty, think about what you can do to create more ease for existing users.

Blake Barlett at OpenView says product-led growth is the key to success in the End User Era. In this era, end user annoyance spells opportunity. Think Slack vs. email, or Zoom vs. Hangouts.

In financially uncertain times, a product-led development philosophy can hold the key to faster end user adoption and increased retention. Tune into those day-to-day annoyances – they hold the key to retention.

Accelerate end user adoption

Happy end users make your application stickier, so if your champion is struggling to persuade others in their company to use your platform, you need to know why. You may need more in-app cues and guidance to make tasks easier. What is “so annoying” about your product? Ask your customers that, and you may find exactly what you need to reduce friction – which will pay off in retention.

Now is the time to deepen relationships and partnerships with promoters

Guneet Singh, Director of Customer Experience programs at Docusign, spoke about this in a recent Voice of the Customer webinar. He looks for champions among his promoters who have a common pain point, and then brings them together in councils that engage with DocuSign’s product team. Through this customer advocacy program, his customers learn from each other, get a first look at new product features, and provide valuable insights for the DocuSign product development roadmap.

How do you begin a customer advocacy program like this? Pay attention to customer requests and “start with small wins,” says Guneet. “If you complete the feature that a customer asks for, by listening and acting on their words, you’ve won that customer for life.” 

We couldn’t agree more!

The most valuable commitment we have is to our customers. And as much as we work to grow, to scale, to expand — it’s times like these where we have to remember to appreciate the people who already support us and show them support too. We’re all in this together.

What’s your biggest problem as a Product Dev professional? Too many demands and not enough time? Limited resources? Oddly enough, none of those topped the list for Hiten Shah’s crowd.

Hiten Shah (of KISSmetrics, Crazy Egg, and Quick Sprout fame) recently wrote in his newsletter that “the problems people have on Product teams fall into two main categories: Customer Feedback and Alignment.” This conclusion came after Hiten asked his readers to share their biggest product problems, and in more than 100 replies, those two themes emerged as the leaders.

Wootric helps customers gather, organize, categorize and analyze customer feedback – at volume – every day. And we’ve got a few insights into how Product teams can solve the issues that come with customer-centricity – while improving alignment at the same time.

Let’s go through the problems real Product professionals sent Hiten Shah point by point.

“Fast/Effective ways to quickly recap and synthesize qualitative research”

Qualitative data – ie. freeform responses versus ratings or multiple choice answers – are notoriously difficult to sift through and analyze. It’s only recently that, with advanced technology and machine learning, it’s become much easier to tag, sort, and assign sentiment to qualitative feedback at scale.

Tagging, in particular, is a huge time-saver when you switch from just manual tagging to auto-tagging. Tagging comments with their major themes is the first step towards conducting frequency analysis to identify trending topics – or find relevant feedback with a click.

Using an NPS survey with an open-ended comments section, for example, you might find that your ‘detractors’ (low scorers) comments tend to be tagged with “slow loading time” or you may see a specific feature request recurring.

Yep, modern customer feedback software should be able to deliver every comment with a feature request, for example, tagged and prioritized by frequency, from the highest-value customers, in about a second.

You can even use tags to route specifically tagged feedback straight to the appropriate department for follow-up. No need to hunt for bugs – the bugs will come to you! (Don’t they always?)

“It’s [customer feedback] very subjective and sometimes doesn’t have context, therefore I take it with a grain of salt, but engineers may not see it that way and want to address the feedback immediately.”

When your customer feedback comes primarily through surveys that *don’t* include open-ended responses (to gather all of that golden qualitative data), it’s impossible to get the context you need to evaluate the issue and possibly solve it.

But understanding the why behind NPS, CES and CSAT scores (to name a few) isn’t all the context you need to decide where to allot your time and resources.

You literally have to consider the source.

Is the feedback coming from a high-value, ideal client? Is your existing survey solution capable of identifying those markers?

Did you know that it’s even possible to target specific customer segments with survey campaigns?

And for even more context – you can target customer surveys based on product milestones. For example, you can set a CES survey to deploy after new feature use to find out how easy (or difficult) new customers think it is to use.

“Feedback overwhelm – how to prioritize what users want/need the most.”

An overwhelming number of customer comments can leave you feeling like you are trying to drink from a fire hydrant. It’s time to talk about the wonders of machine learning.

Historically, extracting insights from piles of unstructured feedback has been difficult, expensive and time-consuming. That is not the case today. When you need insight from feedback at scale, it is time to invest in text and sentiment analysis using software with natural language processing.

Machine learning has come a loooong way. Yes, algorithms must be trained to understand your company and customers, so chose a software vendor that will keep their team in the loop and ensure you’re getting good insights right off the bat. Then the software just gets better and better at telling you what is most important to your customers.

Wootric CXInsight™ combines natural language processing with sentiment analysis to categorize feedback based on what matters most for your customers. When you know why your customers love you — or don’t — prioritization becomes a much easier task.

“Having a regular cadence of customer interaction to develop insights and product intuition.”

Okay, there’s no excuse – this is so easily doable. You can set any CX survey you want to deploy on a regular basis, or, deploy after customers complete specific milestones. Having to go get customer feedback shouldn’t be something you have to think about. It should be automatic! Part of your daily, weekly, or monthly routine.

But, it’s only that easy if you’ve got software that makes it that easy – let’s be honest here. Modern customer feedback software can integrate with Slack, Intercom, or whatever you use, as well as deliver surveys to customers while they’re in your app, and deliver it to you tagged, sorted, and prioritized.

Regularly!

You can have your finger on the pulse of customer satisfaction and will know immediately if there’s any fluctuation. As an added bonus, give a pat on the back to whoever built an update or solution for customers so they can see the results in action!

“My main problem is to get to know our audience and talk directly to them.”

Surveys are great – we love them. But you know what? Even with a qualitative feedback field, a survey can’t take the place of a real, person-to-person conversation. And usually, the biggest barrier to having those conversations is making the time.

We can’t pick up the phone for you, but we can save you time. Enough time to schedule interviews with your customers and get even deeper insights that they may never tell you in writing.

“In Product we’re expected to be customer-centric. We’re supposed to get feedback and talk to customers all the time. It’s literally our day job. But that’s on top of making sure we’re focused on building the right things and helping our teams ship too.”

Here’s the thing, Product friends. You aren’t the only department that has to be “customer-centric” and talk to customers all the time and review steady streams of feedback. So to make this part of your job easier, you might have to reach out to other departments and make customer-centricity a multi-team effort.

If you have a Customer Success department, start there – you might find that the Customer Success Manager is your new BFF. They’re also talking to customers every day, and in many ways, they’re closer to the problems customers face than you are. Most CSMs would be delighted to build better relationships with their Product Dev departments, working together to answer the question “What can we do to help our customers achieve success?”

“It’s not easy and it isn’t getting easier. Customer feedback can come from anywhere: Customer support requests, live chats, social media, the sales team, customer reviews, competitor research, and more. Adding to the pile are the endless opinions about what to do with the feedback from people on our teams.”

It’s not easy – true. But it is getting easier to solve qualitative feedback issues with modern customer feedback software!

Sorry, we can’t help with the ‘too many cooks in the kitchen’ problem – that’s right up there with finding the cure for the common cold. We find that if you have to pick one source to guide product, NPS feedback is the going to be the most actionable.  That said, when it comes to gathering customer feedback from many sources into one, easily searchable place, modern technology comes to the rescue again.

What you want to look for is a customer feedback program that can pull all of customer comment sources together, like NPS or CSAT feedback, user interviews, support tickets, app store reviews, social and analyze those comments in a way that lets you see the big picture and slice & dice by theme, sentiment, survey date, and data source.

This guest post was written by Martin Ceisel, the lead Content Strategist at MindTouch. His hobbies include writing, writing, and writing some more. MindTouch is a self-service platform that helps companies improve support agent productivity, increase ticket deflection, and fuel self-service support.

A quick look at some Net Promoter Score benchmarks will quickly reveal a painful truth: bad NPS scores happen. It’s inevitable.

The worst response to your company’s detractors, though, is no response at all. So, how to best learn from bad Net Promoter Scores and use them to improve the customer experience?

Here are a few strategies to consider:

Do your research

Look at all of the support tickets your detractor customer has put in and read all the notes that your agents have written about these interactions. Review the goals they had when they initially became a customer. Check which help articles they may have read. This will give you important context when you close the loop with the customer.

Respond promptly and personally

Though the customers behind bad Net Promoter Scores might still be feeling the sting of their negative experience, receiving a prompt response to their NPS survey might help turn the tide. If nothing else, a personal response is an opportunity to take the NPS survey beyond a transactional call and response to an ongoing (and honest!) conversation. You’ll be surprised how much constructive feedback a simple “What can we do to improve your experience?” might unlock.

Segment response types

What customer group or business segment is driving the bulk of your bad Net Promoter Scores? One way to find out is to segment NPS scores to identify hotspots. You might find that a particular point in the customer journey, such as onboarding or renewal, is creating an inordinate number of detractors. Or maybe your NPS from product A is higher or lower than product B. Ask yourself why one group of customers is more successful than others. By categorizing responses, you can drill down and identify actionable takeaways. One way to see themes is to create reason codes, a method of categorizing responses so they can be organized and analyzed.

Don’t get tunnel vision

Remember that NPS is just one measure of customer sentiment. Don’t forget key metrics like customer effort (CES) and customer satisfaction score (CSAT). These, too, are important metrics that can lead you to the root cause of negative customer experiences. Regarding NPS specifically, consider trends in your industry. What are the NPS benchmarks you should be aiming for? This will help you decide how urgent an action to take—which bad Net Promoter Scores to prioritize first.

Because it’s about the whole customer experience

Tunnel vision makes for a good segue to my close: remember the reason we pay such close attention to customer sentiment. Perusing, parsing, and responding to bad Net Promoter Scores is about more than improving your company’s own internal metrics. It’s about improving the customer experience. If we can’t deliver low-effort customer experiences throughout the customer journey—if we don’t demonstrate a commitment to reading and responding to what our customers are telling us—we risk losing those customers entirely.

“What grade did you get?”

Do you remember getting asked that question in grade school? Or maybe you were the one asking it? Humans like to know how they’re doing compared to everyone else.

This carries over into customer experience as well. At Wootric, we advise companies on setting up an effective Net Promoter Score (NPS) program. We get asked questions about NPS industry benchmarks all the time.

In general, we believe focusing on an external NPS benchmark is not incredibly helpful.

The Net Promoter System is the quantification of customer loyalty and the process for improving it over time. The power of this system lies in the analysis of feedback and the action taken based on that analysis.

However, net promoter score benchmarks are still useful in certain cases, which is what this article is all about.

If you’re unfamiliar with NPS, here’s a quick rundown:

Net Promoter Score (NPS) is a customer loyalty metric between -100 and 100 that captures the propensity of a company’s customers to attract and refer new business or/and repeat business.

NPS also stands for the Net Promoter System®, which was built around the Net Promoter Score. It is a model that ties a corporation’s bottom line to customer happiness and loyalty.

In the NPS survey, customers rate their likelihood to recommend your company on a scale of 0-10. To get your Net Promoter Score, take the percentage of people who are happy and willing to recommend your product or service (those who respond with a 9 or 10) — “promoters”– and subtract the percentage of people who would not be willing to recommend your product or service — (score of 0-6) “detractors.”

For example, a +50 NPS means that the company has more than 50% promoters and less than 50% detractors, so generally an NPS score of +50 is, indeed, great! You may see scales out there that say +30 is a decent score, and that +80 or greater is the ultimate dream score.

To learn more about NPS, get the ebook, The Modern Guide to Winning Customers with Net Promoter Score, which teaches how to modernize your NPS program for growth and higher loyalty.

Net Promoter Score industry benchmarks

There are two different types of NPS: absolute and relative. Absolute NPS refers to the NPS in and of itself, and comparing the score with what is generally considered a “good” or “bad” score. Relative NPS is taking into account the average NPS within an industry, which takes into account the factors that could affect an average Net Promoter Score, and can change the NPS benchmarks you set.

While an absolute NPS goal is nice and simple, it can be helpful to take a look at what others in your industry have been able to achieve, since every industry is different and has unique relative NPS results. The relative Net Promoter Scores generally achieved in each industry help construct what are called the NPS industry benchmarks. NPS Industry benchmarks give you a way to evaluate your NPS relative to your competitors. They help control for factors that often create major differences in what is considered a good NPS score.

Oftentimes, other companies in your industry have established an average NPS for you to use as a net promoter score benchmark. If you make smartphones or other tech hardware, for example, companies like Apple have been tracking NPS for years.

To get averages and examples from your industry, try reports from the Fortune 500.

NPS Benchmark variance between industries

Let’s take a look at some examples of net promoter score benchmarks according to your industry.

Let’s say you have an NPS of +50. As we explained, that’s already pretty good! But if you’re a department store or specialty store, you are actually below the NPS benchmark (+62) for the industry.

Walmart pharmacies have an NPS score of +32. Considering the highest score is +100, you’d guess that they’d be lukewarm with this score, but I’m sure that the folks in charge of customer experience there are actually ecstatic. Walmart pharmacies have one of the highest NPS scores within the drug store & pharmacy industry.

Compare this number to the software industry, where +34 is the average. Becoming a leader in the software industry would mean having an NPS in the +60 range, like Salesforce (+66) and Adobe (+62).

Caveats for using NPS industry benchmarks

Unfortunately, NPS benchmark programs aren’t always as helpful as you’d hope. This comes down to the nature of surveying for feedback. There are so many contributing factors to an NPS benchmark, such as:

• Which channels you use to survey customers
• Demographics and habits of your customer base
• Customer tolerance levels
• The size of your competition
• The difficulty of building brand loyalty
• External circumstances (such as a global pandemic)
• When and how often you ask
• Whether you have enough data to be statistically significant or not

All of these factors can have varying effects on your overall NPS score. For example, your competitor may ask the NPS question within the context of a longer annual brand survey, while you survey using just the NPS question after a transaction. These will have different consequences for the feedback you gather. If you don’t have enough feedback coming in, your NPS may vary significantly from quarter to quarter or month to month.

Bear in mind, a ‘good NPS score’ doesn’t just depend on your industry, since it’s not difficult to game the system. It’s not always fair to compare your NPS score to another company’s NPS score because you don’t know their survey methods, or their employee compensation plans.

When competitive individuals are incentivized based on NPS score, things can get ugly.

A motivated person or company could improve their numbers by letting their customers know that positive feedback would mean a lot to them or by only showing the survey to customers who are positively inclined. They might offer incentives to customers to complete the survey. Clearly, the feedback received from these methods will lead to an inflated NPS score that is not a useful comparison for those using a more objective survey process.  

Setting an NPS goal if you don’t have a benchmark

If no Net Promoter Score benchmark exists for your industry, benchmark against yourself.

The great thing about NPS is that it is an actionable metric. It’s a number that you can rally the company around as a north star to guide improvement efforts.

“A good NPS score is one that is better than the last.”
– Jessica Pfeifer, CCO & Co-founder of Wootric

Remember, NPS isn’t just a score. It’s a system that’s meant to drive business improvement in product and customer experience. It helps you identify and close the loop with unhappy customers and solve their specific problems in real time.

Your goal is to boost customer loyalty and retention, and that happens by reading verbatim comments to understand the why behind the scores you receive. By making changes based on customer feedback, and responding quickly to detractors, you will naturally see your NPS improve. And gains in NPS correlate with revenue growth.

How to report NPS

After all this, you will want to report numbers to the rest of the team on a regular basis. NPS should be shared along with other monthly or quarterly metrics like revenue, new customers and customer churn.

We understand that, so here’s what we recommend:

• Instead of fixating on your score in the absolute sense, we recommend focusing on improving your score over time. Understand NPS as a trend over several periods, like if you were looking at a stock’s price.
• Determine the business goals of your NPS program, then report NPS in relation to the goals. For instance, if you are trying to improve retention, report NPS alongside churn data.
• Pay attention to trending topics in your verbatim responses. Reporting these topics will help everyone understand what’s important to your customers, and the pain points they experience. Share what customers love and what they don’t love about your company with internal stakeholders. Then you can work to make those points as frictionless as possible. 

Note: For startups, be sure to read and respond to every single comment. As you grow, you’ll start needing aggregate and to pull themes from customer comments. To automate that process, check out AI-powered text and sentiment analysis.

• Segment your Net Promoter Score by relevant customer groups. For example, this could be by user role (in the SaaS example above), geography, or size/frequency of purchase–whatever drives your business. This will help you pay close attention to groups that are critical to your business success. Learn more about segmentation here.
  
• If you want to compare your score to a competitor, choose a company in your industry that you admire and use their score as an aspirational benchmark. Many companies have volunteered their NPS scores to research and reports such as this one by the Fortune 500.

Measure NPS and work to improve it over time.  Dig into customer comments and close the loop with customers. You will learn their needs, and their pain points, and have plenty of guidance to make those improvements. Both your NPS and your customer retention rates are sure to improve. 

In case you missed it, customers’ expectations have changed.

The way we communicate with them has changed, and Intercom users are leading the way. Customers and prospects like communicating via Intercom chat. It’s efficient, modern, and conversational.

Integrating a customer feedback program with Intercom takes customer experience to the next level. Having all your customer experience data gathered, viewed, and managed in Intercom sets you up to build high-quality relationships with your customers.

Easy, breezy, and code-free option: Surveys in Intercom chat & email

Wootric gives Intercom users two code-free ways to survey customers – in Intercom chat and via email. The survey responses then flow back into Intercom records to view for follow-up.

Wootric Surveys is an Intercom Messenger app, available now in the Intercom App Store, that lets you send surveys within the Intercom Messenger chat bubble. From your customers’ point of view, sending Wootric surveys through Intercom makes sense — that is where they are used to getting communication from you.

Alternatively, you can survey your customers via email. Download the free Wootric email survey template (you’ll find it in Wootric settings) and upload it into Intercom.

Whether you are surveying in Intercom chat or via email, you can create auto-messages based on Intercom rules and attributes, or send a survey out to individual customers manually.

Here are the top 5 use cases for the Wootric-Intercom integration to unify your customer feedback program.

Target a specific customer segment with an NPS survey campaign

To better understand the sentiment for each your customer personas, set up a campaign-style auto message with a Net Promoter Score survey in Intercom.

For example, you may choose to target all of your enterprise customers, or all of your self-service customers. Perhaps you want insight into customers in the EU, Asia, or South America. You could also choose to survey customers who have been with you for 6 months.

With Intercom and Wootric, you can send a survey can be based on any group of customers that you can define in Intercom.

Target customer surveys based on a product or service milestone

Gather feedback after product or service milestones. The feedback you gather at customer journey points can help you prioritize improvements that will increase customer retention. Here are two popular examples:

• Deploy a Customer Effort Score (CES) survey after a product milestone is achieved

If you are trying to understand customer sentiment around installation processes, onboarding, or other product milestones, the Customer Effort Score is the survey for you. Measuring customer effort gives you insight into pain points or friction that customers may experience while using your product or service. Asking “how easy was it to __” will help you quantify ease and give you wide-ranging feedback. You may find that your processes (e.g. installation, getting started, etc.) are easy enough, but documentation is difficult to find. Customers may end up going to Customer Support because they are frustrated about finding answers and instructions on their own.

 As for when to send a CES survey, you may choose to send a survey via auto message for folks who have logged into your platform after completing onboarding. You could choose to send an email survey to users who have exported their data to another platform to see if the process is easy enough. Surveys can be triggered based on any event you are tracking in Intercom, giving you a plethora of options.

• Send a Customer Satisfaction (CSAT) survey to ask about product features

Customer Success and Support aren’t the only teams that will benefit from surveys via Intercom.

This channel is a great way to survey customers about different product features as they use them. For example, you can choose to deploy a CSAT or “PSAT” survey to all users who have used a feature X number of times or 30 days after upgrading to a new feature set.

The feedback you get from asking the Customer Satisfaction question can be valuable to your product development team and product managers.

Make sure everyone is covered

• Send a survey in a live conversation

“Oops, I didn’t mean to score you a 2!”  Sometimes a customer will make a mistake or get distracted and fail to click “submit” when they see your survey.

Now, you can manually insert a fresh survey into the conversation. This lets you fill in the gaps that automated campaign-style messages may leave.

• Hear from more customers by sending an in-app survey to people who don’t respond to email surveys

If you’ve been surveying your customers via email, you may find that a good portion of them go unopened. Other email surveys may be opened and the surveys don’t get filled out because folks think “Oh, I’ll get to that after I finish up this other thing!”, then get distracted and forget about your survey.

Cover your bases and reach customers where they’re already working by sending these users another survey via Intercom messenger.

The set & forget option: Wootric in-app surveys

One challenge with Intercom auto-messages is that they are not recurring. If you want to survey your customers say, every 90 days, the best way is to install the Wootric code snippet on your web application (or use the SDK for you mobile app). Then you can easily configure your sampling requirements and survey cadence in your Wootric settings.  The best part is that survey responses collected this way will still flow into Intercom records for follow-up action.

See survey feedback & respond to customers in Intercom

Consolidating all of your customer experience data in one platform isn’t just efficient. It provides your customer-facing agents with vital context as they interact with users without having to shift from one platform to another.

For example, your Customer Success Managers will be able to see each company’s survey responses, including user comments.

Your Support and Success agents can manage the follow up with customers in the same platform that they view users’ survey responses. The Wootric integration can be configured to automatically open a conversation when a survey response is received. This can prompt team members to reach out to personally to thank the customer or follow up on any issues. Or, set up auto-messages to response to customers based on their score. For example, trigger an email that asks happy customers to write a review of your product. 

With Intercom, customers know where to expect communication from you and know exactly where to reach out for help. Using Wootric and Intercom together is a convenient way to close the loop with customers, letting them know you value their feedback.

What is Empathy?

Empathy is the ability to understand and share the feelings of another. For Product Managers looking to improve customer experience (CX), that definition translates to doing more than understanding the user’s pain points, but also looking at the emotional landscape of what it’s like to use the product – when it is working, and when it isn’t working.

Empathetic Product Managers ask themselves:

• How does using the product make the customer feel?
• How does the customer want to feel when using your product? What would be the best possible emotional outcome for them?
• How do I ensure the product developers understand and take the customers’ needs into consideration in their process?

The answers to those questions affect every facet of business, from acquisition to retention. It’s how, through CX, you can generate rapid growth through word-of-mouth recommendations, and sustain your success with customers who never want to leave.

Tying Empathy into CX

Empathy is a soft skill, and while those are typically difficult to measure, the effects of empathetic product development can be seen in every CX metric: Customer satisfaction (CSAT), Customer Effort Score (CES) and Net Promoter Score (NPS).

Sinead Cochrane, Senior Product Researcher at Intercom, wrote “For product teams, empathy building activities such as observing research or doing customer support is often not considered ‘real work’. However, product teams that consistently keep customer needs in mind are able to maintain and evolve their products in ways that won’t negatively impact the user experience.”

For her, empathy for product departments means “When a customer tells you something is broken, you are able to imagine the impact it’s having on the job they’re trying to get done,” and, “you realize the emotional impact the problem is having on that person.”

But I think we can go further than just recognizing the emotional impact of problems. That’s scratching the surface of what having empathy for customers can mean for producing superior customer experience.

Because empathy shouldn’t be reduced to realizing customers feel bad when a product isn’t working for them. A whole new world opens up when you also consider how you can design your product, updates, and expansions to enhance positive emotions as well.

Here are the questions that lie at the heart of empathetic product management:

• How can you get more emotionally in sync with your customers?
• Which are the most important negative emotional outcomes to manage?
• Which emotions should you seek to heighten (and how?)

To answer these questions, try these empathy building exercises.

Empathy Building Exercises for Product Managers and their Teams

1. Listen actively to discover underlying needs and emotional motivations

Often relegated to customer service and customer success departments, ‘active listening’ to find out why your customers use your product and what they really want to achieve is very important. You can’t get the depth and honesty of answers by just sending out a survey – this works much better if you do phone, Zoom or in-person interviews. In fact, Roman Pichler recommends product managers meet real users on a regular basis. You may find that your assumptions of why customers use your product aren’t accurate, or don’t tell nearly enough of the story.

“At first, our assumption was that they wanted to make more money. That often was true, but frequently we heard something different. Many simply wanted to maintain the business but run it more efficiently so they could have more free time (we heard about golfing on Fridays more than once). Others wanted to build a sustainable business they could pass on to their son or daughter.” – Jim Semick, Founder & Chief Strategist at ProductPlan

To get down to customers’ real motivations, ask open-ended questions beginning with “why” and “how.” Then make sure to record their answers in their own words (you can hand those assets to your copywriters for later use).

2. Use your own product

Empathy is often described as ‘putting yourself in someone else’s shoes’ – and there’s no better way to do this for a product manager than to actually use the product, just like any user would. You’ll empathize with users’ frustrations as you experience your own product’s shortcomings and hopefully find moments where it’s possible to create more delight.

But always keep in mind – you are not the average user. You’ll still need to listen to your users to get a complete picture of how they feel, and what problems they perceive as being severely aggravating.

3. Share verbatim comments

Someone, somewhere, is tracking customer experience metrics, sending out surveys, and collecting the answers. That someone might even be you. When reading users’ written responses, don’t just look for problems to solve and ignore the positive comments. Read them for emotion and see what conclusions you can draw about what people are feeling, and want to feel.

Pick a few relevant verbatim comments to bring to the rest of the product team. Reading these comments often helps engineers and designers feel the same joy or frustration as their users. This new emotional understanding will help you evangelize CX as a priority with everyone.

4. Mine your qualitative data and quantify customer sentiment

Those open-ended response answers are a goldmine for user research that can alert you to problems – and give you hints into the customer’s emotional state of mind. However, once you are getting more than a hundred comments a month, seeing the forest for the trees can be a difficult exercise. Qualitative feedback is notoriously tough to quantify, but it is now possible and easy to quantify sentiment with the help of machine learning.

AI-powered platforms, like InMoment CXInsight™, automatically sort your customer comments into themes while simultaneously assigning positive or negative sentiment. This provides you with a big picture understanding of how customers feel about your product and why. Categories of feedback vary by business sector and business model–payment processes & delivery for e-commerce, perhaps, while UX and usability may surface for SaaS products. Quantifying the sentiment of what your customers are talking about can help you track emotional trends over time. Presenting this kind of data alongside verbatim comments connects customer emotion with real business consequence.  

5. Set empathy KPIs

What gets measured gets done, and adding empathy into your product development work is no different. The KPIs for empathy may look a little different than your typical performance indicators, but the good news is: They’re not difficult to get. You’ll find key performance indicators like NPS, CES and CSAT are a good start, and comments in the open-ended questions can give you insight into the metric. Start identifying what kinds of ratings and qualitative answers correlate to genuinely happy customers – and frustrated customers likely to churn.

6. Chart out an empathy map

You’ve done your user journey, but even though it’s part of the buyer persona building process, you may not have done an empathy map.

• What your user sees – on competitors’ websites, common visuals in their industries, maybe what they enjoy watching or reading
• What your user says – how they measure success, what they say they want, what they say about your product
• What your user hears – what their influencers are saying, not just about your product, but about their jobs and what constitutes success, what they enjoy, what they don’t like about their experiences with your competitors, etc.
• What your users think and feel – worries, aspirations, what they really want, what really annoys them

Notice how the empathy map includes business/industry-specific observations, but also branches out into the user’s personal life and larger environment. People are not their jobs – or even their ‘jobs to be done.’ For true empathy, you have to look at the whole person.

This is a great activity to get other teams involved in – consider hosting a meeting with Customer Success, Sales, Marketing and Customer Service for a wider scope of insights.

Activities involving multiple teams help to build a shared understanding of your customers’ experiences that can strengthen the whole company.

7. Add happy moments to your Customer Journey Map

You’ve probably mapped out your customer/user journey, but you probably didn’t include this: Happy moments. See if you can take your old customer journey map and mark the points where positive, fun, delightful things happen. Can’t think of any? Then you have some serious CX work to do!

And of course, also note points where you’ve observed friction, difficulties, and problems, and address those in the order of biggest impact + easiest to implement.

8. Work in Customer Support for an afternoon

Whether that means answering the live chat questions, picking up the phone, or monitoring your product’s customer Slack channel, try out being the Customer Support agent for an afternoon to and put yourself on the front lines! There’s no better way to find problems than to let customers tell you exactly – and in great detail – what they are. And they’ll likely throw in how frustrated it makes them feel too.

9. Build a prototype to test your emotional hypotheses

By now, you probably have a few ideas on how you can improve the customer experience, and it might be time to test those theories. Create a prototype for a select group of qualified users to try (and react to). And, if possible, have them test the prototype in a testing facility that allows you to observe their reactions as they use your product.

If there is a Golden Rule for empathy, it’s a simple one: Forget your assumptions and be genuinely interested and curious about what people are feeling (not just what they’re doing) while using your product. Empathy is a learned skill that needs practice so don’t forget to try out these empathy exercises on a frequent basis for enhanced customer experience.  

To quote Maya Angelou: “People will forget what you said, people will forget what you did, but people will never forget how you made them feel.”

Data is the beating pulse of business, but customer data is more like DNA. Customer data, if we’re using it right, directs how we grow and what we develop. But what happens if that customer data becomes corrupted by our own bias?

We can’t grow or develop in the ways we need to.

But what is bias exactly? Where does it come from?

The most prevalent bias is, perhaps, confirmation bias – seeking out data that confirms our existing beliefs.

In an early study of confirmation bias, young children were asked what features in a sports ball are important to the quality of a player’s serve. Some said size, others said material, some dismissed color as a factor – but once they’d made up their minds, they failed to acknowledge evidence that was contrary to their theory – or explained away evidence that didn’t fit.

But what’s worse, especially for those of us using data to steer our businesses, is that confirmation bias caused them to not generate alternate theories unless someone asked them to. They missed exploring and finding other possibilities.

There are other types of bias too, including:

Algorithmic bias – When the data used to teach an AI machine learning system reflects the implicit values of the humans involved in collecting, selecting and using that data. You might remember the 2015 uproar around Google’s image recognition AI algorithm that auto-tagged photos of black people as gorillas? Yes, that happened. And in 2009, Nikon’s image recognition algorithms consistently asked Asian users if they were blinking.

Survivorship bias – When the data analyzed only comes from success stories.

Sample bias – When the population you collect data from doesn’t accurately reflect the population you’re trying to learn about.

Avoiding bias when gathering, analyzing and acting on data is impossible. Bias creeps in with assumptions, instincts, guesses, and ‘logical’ conclusions – and mostly, we don’t even know they exist until someone without those particular biases point them out.

But, while we can’t escape biases, we can try our best to account for them when we collect, analyze and interpret data.

How to fight bias in your data

In Forrester’s The illusion of insights recording, Forrester Vice President and Research Director Sri Sridharan makes three recommendations to reduce bias in data.

She says to “triangulate insight” by using multiple methods of arriving at an insight, and cross-validation. For example, pairing behavioral data with feedback from customer surveys to see if you arrive at the same or similar answers.

Her second piece of advice is to create a “self-correcting system of insights” that connects customer data with an effective action to create a closed loop of action, learning, and optimization. Essentially, this means testing the data by taking action and iterating based on how well you succeed in addressing the issue.

Tracking a ‘North Star’ metric like NPS or CSAT over time can be very helpful in confirming whether the changes you make are having the desired effect.

Sri’s third piece of advice is to “show your work to build trust” both internally and with customers. Your customers will be quick to correct you if your insights don’t hold true for them – and you have the bonus of showing them how hard you’re working to make sure they have what they need to succeed.

But there is also the potential for bias to happen before any of these fixes can be made – especially in Customer Discovery.

Bias in Customer Discovery, Before You’ve Even Gotten to the Data

Bias in whom you ask

Who you survey, interview or meet with can bias your results. This is called “sample bias” – but it can also turn into confirmation bias. Sample bias happens when some members of the intended population are less likely to be included than others. Think of all of the different segments of users you have – what would happen if you only surveyed one of those segments? You would get responses that don’t work equally well for all of your customers.

This can slide into confirmation bias if the population you select is more likely to give you the answers you want to hear.

And, there’s also the risk of “survivorship bias,” if the people you’re surveying are the customers who are still with you, rather than the users who have churned. Current users are much easier to collect data from, and while they can give you important insights, they can’t tell you why your churned customers left.

Bias in how you ask

How you frame questions can have a dramatic effect on the responses. In fact, by the wording you use in a survey, or even your tone of voice in a phone interview or facial expressions in an in-person interview, you can effectively steer the conversation to deliver exactly the answers you’re hoping to hear. Many of the words we use have positive or negative associations that cause people to react accordingly.

Biased question: How much do you like the color blue? (This presupposes they like the color blue at all)

Unbiased question: How does the color blue make you feel? (A much more neutral phrasing)

Or, if you aren’t specific enough about the information you want, you risk confusing your respondent and getting answers that aren’t at all helpful. Unless you have a professional market researcher on staff, you may want to stick with established questions like NPS and CSAT.

Bias in what you ask first – and last

The order of the questions you ask can also bias your results, and you’ll need to review your question order carefully to make sure the sequence doesn’t cause biased responses. Typically, you should ask general questions before specific ones, ask positive questions before negative ones, and ask questions about behavior before questions about attitude.

Bias in when you ask

Holidays and the summer months, when families often take their vacations, can be problematic for both response rates and sample bias. For example, if you send a survey during religious holidays, you’ll likely get different responses rates from different groups of people, who may or may not be taking that time off.  Be aware of your timing, including if you’re sending surveys during deadline rushes, before or after holidays, or other significant patterns that may affect who responds and how they respond. To be on the safe side, don’t send your survey at the same time every year – send a few, at different times, to get the most accurate feedback.

Objective Data Leads to More Accurate, More Valuable Insight

Sherlock Holmes famously tells Dr. Watson that he never forms a theory before he gathers all of the facts. But he’s much better at disassociating himself from the results than most of us are (and he’s fictional). Bias has a way of seeping into our results, and how we view and react to our results. But, when we put bias-countering measures in place, like gathering data from different sources, using different types of data, and checking our work through a process of action and iteration, we can get to the truth in the end.

PART 1

At Wootric we collect hundreds of thousands of NPS, CSAT and CES  survey responses every week. We do this across different industries and product categories. Our customers then use our various integrations such as Salesforce, HubSpot, and Slack to route this feedback to relevant teams. Some of these customers like DocuSign and Grubhub have huge user base. This means even with a conservative sampling strategy they get hundreds of pieces of feedback daily.

The Challenge of Analyzing Qualitative Feedback

The quantitative aspects of feedback — NPS, CSAT scores, for example — are relatively easy to aggregate and analyze. It is the qualitative comments that provide rich insight into customer experience, but analysis of unstructured feedback is hard. Someone could read each piece of feedback one by one, but having a human read each comment obviously does not scale. If you can’t or don’t review what customers are telling you, then why have CX program to begin with? This is where machine learning saves the day.

Here is a concrete example of problem we needed to solve:

We set out to solve this multi-label classification and  topic sentiment analysis problem for our customers. We knew that doing it at scale would require some sort of automation using machine learning (ML) and natural language processing (NLP).  Machine learning requires lots of training data — in our case it meant we needed thousands of survey responses manually labeled into different categories. Luckily our customers have had the ability to manually categorize survey responses in our dashboard.  In the last 3+ years, they have categorized thousands and thousands of comments. Fortuitous, but sometimes we would like to think we saw it coming. 😉 At this point, we have significant volume of training data, now we just needed to find the right machine learning algorithms to take a stab at the problem.

Sentiment Analysis

To get our feet wet in machine learning ecosystem, we thought we would start with sentiment analysis because there has been tons of research on this problem over last few years and there are several open source solutions claiming to solve this. We started with Stanford NLP library because it was and is still actively worked on and lots of great research papers have come out of this group. It took us about a week to get our heads around concepts of Tokenization, Lemmatization, Named Entity Recognition (NER), PoS tagging, Dependency parsing, Coreference Resolution, word embeddings and finally to embed the library into a REST API framework. The results were okay but not great for our use case.  We **think** it’s because feedback comments belong to different domain — these are responses to questions as opposed to news article, blogs and tweets. Most of feedback is short, ranging from a couple of words to couple of sentences. This does not give enough “context” for algorithms to find the sentiment. An alternative solution would have been to train our own model using Stanford NLP library but we did not have engineering resources and bandwidth at that point in time. But we had something working. It was a baby step, but a step in the right direction.

Categorizing Feedback Comments

For feedback categorization, we also first looked into a hosted service and open source libraries. The most compelling, trainable and easy to get started solution at that point in time (mid 2016) was Google Cloud Prediction API where you upload a CSV of training data and in few minutes you get a model and REST API to make a prediction. This sounded like a tailor-made solution for us. After all, we had lots of training data that our customers had manually labeled. We were able to quickly format our responses and training labels in order to meet the Prediction API requirements, and saw some initial results.

Results were better for some sets of feedback but horrible for other sets. Dissecting further we realized that the customer feedback set where prediction was more accurate belonged to DevOps, Data Analytics and similar developer centric and data analytics.  There was less success with feedback from e-commerce or other consumer-centric SaaS products. This made sense because most of our customers who spent their time manually labeling feedback had offerings catered to developers — such as New Relic, Docker etc. The bigger downside of Prediction API was that it was a black box so we did not have any ability to tune algorithms. Ironically, Google decided to deprecate Prediction API in favor of their Cloud ML Engine, driving us to improve our internal, customizable prediction methods instead.

Our experiments with Stanford NLP libraries and Google Prediction API gave us a good understanding of the complexity of problem we were tackling, provided more awareness of the ML ecosystem — people and research labs to follow, research papers to read — and finally helped us better understand the nuances of building machine learning models. It’s not as simple as having some training data, copy pasting some code from open source libraries and voila you have a ML solution. There are lots of hype and noise around what ML can do and how to go about doing it right.

At this point, we concluded that there was no shortcut and that we had to invest time focusing on high quality training data, going through various research papers, trying the algorithms in research papers using our training data and have nobs ( i.e. hyperparameters) tuned for our use case.

Delivering insights to customers

In April 2018, after six months in beta, we launched v1 of our product CXInsight™. The platform enables our customers to import and analyze customer feedback from any source. To date, we have analyzed 200,000+ comments pertaining to a wide spectrum of product categories and types of feedback — PaaS, SaaS, E-commerce, Mobile App reviews, employee reviews, social media, etc.

Of note, most of the data we’ve analyzed so far was originally collected using our competitors’ survey platforms. This is sweet validation of our goal: Regardless of how and where you collect customer feedback, Wootric gives you the best analysis.   

In our next series of blog articles, we will talk about how we have used and are using:

• Bag of Words with Naive Bayes and SGD Classifier (Part II)
• word2vec with PCA, Logistic Regression, SVM and SGD Classifier
• Bidirectional LSTM
• CNN
• Custom word embeddings
• Productizing — DevOps around ML
• UX and UI
• Online learning and Human in the loop

Our goal has never been to build a generic text categorization tool. Rather, our focus is to build the best customer feedback analysis platform. We are also aware that our system is never going to be 100% correct so we have made it easy for our customers and our own team to be human in the loop.

I would be remiss not to thank Stanford NLP, Richard Socher, Lukas Biewald, Sebastian Ruder , Google ML research whose research papers, blogs, tutorials, videos and guidance have directly or indirectly helped us build CXInsight product.

Please stay tuned to our Engineering Blog for next series of articles on this topic.  The Hacker News discussion is here.