Author: David Ensing, Ph.D., Vice President, Solutions Strategy
Is a short customer experience survey as effective as a long one? Is it better? Do we need to ask questions with rating scales now that we can analyze comments using machine learning algorithms and other tools? These are questions that are being asked more and more frequently by professionals who are striving for Experience Improvement (XI). And, as is often the case when answering these types of questions, the answer is “it depends.”
The Purpose of XI Programs
The question of whether to use a short, comment-based survey depends on program type (VoC, VoE, etc) and purpose(s). Companies and their frontline outlets use experience information for a variety of purposes, including acquiring new customers, retaining existing ones, determining training needs, and recognizing effective employees.
We also need to remember that most XI programs have secondary goals, including monitoring whether stores and offices are behaving a certain way, assessing usage and customer experience with specific products and services, and asking about signage or marketing displays at outlets. All of these objectives determine which type of survey is best for organizations to use.
Which Purposes do Comment-Based Surveys Best Support?
My team and I analyzed hundreds of thousands of comments across multiple studies to determine how well a short, comment based surveys could support the goals we just talked about. We found that comment-based surveys can be a good way to identify the strongest customer sentiments as long as a large number of comments are available (generally a few thousand or more).
Additionally, brands can determine both the “absolute impact” of a sentiment category and its “relative impact” on an overall experience score by combining the frequency of category mentions with a single experience rating. Brands can calculate a category’s absolute impact simply by seeing how much it affects the overall experience score regardless of how often it’s mentioned. Relative impacts consider that factor in addition to how frequently it’s mentioned.
For example, in an automotive vehicle satisfaction survey, the category “transmission failure” would have a large absolute impact because it is a severe problem, but a small relative impact because it occurs very infrequently.
Comment-Based Surveys’ Shortcomings
A short survey consisting of one overall rating question and a comment is often a good way to both identify customer problems and to identify how a brand and its outlets are performing in various categories at both regional and national levels. However, this survey design often falls short when conducting analyses at lower levels, such as at a store, a bank branch, or with employees. This is because customers usually discuss only a small number of topics in their comments. Since fewer comments are available at these lower levels and because comment topics are spread over a large number of categories, only a few mentions for a few categories are often available here.
Gleaning actionable intelligence from comment-based surveys also becomes more complicated when you consider that a large percentage of feedback contains little information. Comments like “good job”, “it was terrible,” “thanks”, and, of course, the infamous “no comment” are unfortunately commonplace, and encouraging customers to be more specific is always a challenge.
Also, as I mentioned previously, many companies want to know if their outlets are performing specific behaviors (wearing name tags, introducing customers to the service department, etc.). Customers rarely mention these things on their own and must thus be asked specifically. Issues like these are why comment-based surveys are less ideal for diagnosing process effectiveness, employee coaching, assessing corporate initiatives, and determining whether stores are following corporate-mandated procedures.
How Comment-Based Surveys Can Be Better
To sum up, comment-based surveys’ limitations stem primarily from low information content and low comment specificity. Many customers also write comments into surveys via their smartphones, the difficulty of which can turn them off to the process altogether. This is a problem when you consider the dramatic increase in smartphone usage over the past decade. However, there may be a solution to this collection problem today, and that’s encouraging customers to provide multimedia feedback. Brands can accomplish this by wielding experience platforms that enable customers to respond via voice-to-text, image feedback, and other means of more personalized expression. These methods can both increase the effectiveness of comment-based surveys and help customers more easily express themselves. Brands can then collect more actionable intelligence with which to create a more meaningful experience.
You can download the full PoV as a PDF by clicking the button below!
