More attributes lead to more noise in the data.

Perhaps no buzzword is bigger in research right now than “big data,” that magical mountain of insights just waiting to be mined. News stories and press releases come out almost daily extolling the virtues of big data for analyzing everything about customers and competitors.

Seemingly counter-intuitive to this massive data trend, we constantly find ourselves, as a consultative research agency, trying to convince our clients to ask fewer questions and test fewer attributes. If big data is the watch-word, why would we possibly want less data?

To think through this problem, it’s crucial to take yourself out of the frame of mind of a researcher or brand manager, and instead view the survey from the respondent’s perspective. Respondents qualify to take a survey because they meet our determined criteria, but that doesn’t mean they live and breathe our client’s product the way we do.

When that respondent is faced with a list of 15, 30, or even more attributes, they are expected to carefully consider each attribute, parse its difference from other attributes, and then decide whether it applies to a product/brand or whole group of brands. They may be the most honest, hard-working people in the world, but this task will fatigue anyone, making answers to this item and each subsequent item lower quality than they would have been otherwise.

More attributes also lead to more noise in the data.

When our lists expand too much, respondents simply don’t cluster as tightly.  We can parse and debate and determine that “high-quality” and “well made” are different attributes, but is a customer really going to differentiate? By including similar attributes in a single list, we are insuring that customers will disperse across those attributes instead of cluster within the close-enough attribute.  This dispersal may lead us to miss some aspects of brand identity while over-stating others, hurting the overall actionability of our results.

It’s also important to remember that the aspects of our brands and products are simply not as salient to our customers as they may seem in our planning meetings.

As consultants and employees, we are strongly incentivized to spend large portions of our day considering our products, brands, and categories.  For our respondents, however, our product/brand/category is just one small aspect of their life, something they may interact with only occasionally.  What seem like critical issues in our internal meetings may in fact be distant concerns or completely off the radar for our customers.

Obviously, price is also a factor – shorter surveys simply cost less to field.  But even if we have the budget, it may be worth considering that sometimes, when it comes to survey questions, less really is more.