Wed 23 Apr 12:28pm CDT
Register | Login
Survey Error

Don’t be fooled by high response rate programs that measure very little.

When experts criticize their own industry, it tells you there is a real problem brewing that deserves attention. As a surveyor of customer satisfaction for over 20 years, I am going to share with you a disturbing trend that is leading many large-scale home builders seriously astray, causing them to lose millions each year.

In the last several years, phone marketers have converged on home builders with an intriguing value proposition – namely to provide 75% response rates for customer satisfaction surveys. Sounds good? So good that I was intrigued, and became tempted to join them in offering this service, until I did the research. Bottom line--this approach can seriously damage businesses.

Instead, I have been sharing the research studies of the world’s foremost experts from Harris Poll, Oxford Press, and the US Census, all of who warn against adopting such programs. (A summary of this article was published in the February 2012 issue of Builder magazine).

The Tantalizing Offer

What we hear is that telemarketing company sales reps who, with little or no formal research background, make statements like “your current program is error-prone because it’s not getting the 70-80% response rates that our surveys can get”. This “more is better” sales pitch is quite tantalizing given that most surveyors today would be happy with 20-50% response. When JD Power and Associates did their studies of home builder satisfaction their response rates averaged 20%. The average response rate for Avid Ratings' 30-Day Move-In Survey is 56%.

The fact is that, in all surveys, we must rely on sample sizes that are less than the total population to do sound research. 75-100% is simply not necessary. In order to test someone’s blood for disease, you don’t need to drain his or her entire body—Right? Randomization is the key principal that makes sample sizes work and, when it comes to customer surveys, decades of research have established that we have an equal number of happy vs. unhappy homeowners who fill out surveys. At Avid we have tested this assertion multiple times within the building industry. When we survey builders at a 50% response rate and compared the results after we drive the responses up to 75% (through multiple mailings)-- the results show virtually no change in the scores. Contrary to the erroneous assertion that the current survey methods are broken and higher response rates are needed, actually what is needed for improving the validity of survey results is properly controlling for other sources of error like survey length and social desirability. Simply put, telemarketing companies vying for your business fail to tell you the entire story about survey methodology.

The Facts: Detailed Surveys Deliver More

While on the surface it may seem better to get more surveys from customers, the reality is that these telemarketing surveys actually capture less information. Telemarketers must dramatically shorten their surveys in order to administer them over the phone. Often, 50-75% of the questions typically found on a detailed home buyer surveys are eliminated from the telemarketer’s survey, resulting in a net-loss of information, and not a gain as they claim.

Unfortunately, shorter surveys have been sold as a benefit by telemarketing companies who often quote the book The Ultimate Question by Fred Reichheld, as justification for eliminating questions. Unfortunately, Reichheld’s work, also known as Net Promoter Score (NPS), has come under considerable fire within the last five years. Morgan and Rego (2006) conducted an extensive study comparing the various survey methods to future business success and found that short “Net Promoter Surveys” have little relationship to business financial performance. Below is a summary of the results of their study that was summarized by the consulting firm The Cicero Group.

The Cicero Group concluded:

“Recent directives for companies to replace these traditional satisfaction metrics with recommendation or net promoter metrics are misguided. Companies wishing to maximize the benefits of customer satisfaction should measure at multiple customer lifecycle touchpoints. The multiple touchpoint approach allows companies to dig deeply into the customer experience while not sacrificing data reliability.” [Source: Quantifying the Impact of Customer Satisfaction on Buisness Performance, White Paper, The Cicero Group, Shumway and Wright]

Since telemarketers use shortened survey instruments to measure a complicated transaction such as home building, a comprehensive customer review is simply unattainable, rendering many builders blind to the realities of their business.

The Facts: Phone-Survey Bias

Research has found that phone-surveys are highly subject to Social Desirability Bias, which means respondents provide answers that are least controversial in order to avoid conflict with the real-life surveyor. For example, we see this every day in the restaurant industry where a majority of restaurant customers respond with "good" when the manager asks “how was everything?”, –even though the meal or service was below expectations when asked later about it. We have all done it because we want to avoid conflict with the person asking. Not surprisingly, this phenomenon goes beyond restaurant interactions to all aspects of life.

The renowned research group Harris Interactive studied this phenomenon and uncovered the pervasiveness of this problem. The graph below illustrates the difference between survey methodologies and the impact social desirability has on a number of commonly asked questions.


The bottom-line with high response rate, phone-survey programs is 1) less real information to diagnose problems, and 2) customer satisfaction results are skewed toward a positive score. Overall, this is not a good situation if your company doles out bonuses to its employees based on customer satisfaction, unless of course you are an employee of one of these companies (tongue firmly in cheek).

Unfortunately, we may have come to a point where home builders are valuing raw quantity of response rates over the quality of customer research, only to find it misguiding their companies by allowing substandard performance to go unnoticed, and in some cases even rewarded. If more builders are fooled by this folly, we could find many executives resembling emperor in the children’s story The Emperor’s New Clothes (in it, the emperor is told what he wants to hear rather than the truth). However, I am hopeful that factual reporting and open dialog on this issue will keep our industry from being bitten by this brand of snake oil salesman.

Paul Cardis is founder and CEO AVID Ratings, a provider of customer loyalty research and consulting to the home-building industry. Through the AVID system, home builders improve referrals, reduce warranty costs, and strengthen their brands. He can be reached at