Should Market Researchers Be Worried About Bias in Insight Communities?

By: Trisha Jacobs, Director of Customer Success, Vision Critical

A version of this article first appeared on the Vision Critical blog.

Are insight communities biased? 

In my role as customer success director, that’s a question I sometimes get from our new customers, particularly those who work in market research. Many researchers with traditional background are concerned that because community members are loyal customers, the results from insight communities won’t be as objective as other sources.

I have a background in research myself, so I understand this concern. In this blog post, I’d like to address it head on.

But first, a question…  

When was the last time you took a survey because you were promised a “chance to win”? Think hard, you might have to, because even though we are all bombarded with these types of emails every day, the response rates for these types of surveys are on a steep decline.

And if you did complete the survey, what kind of feedback did you give? Did you spend time completing it, or did you rush through to get your points or cash? Was your intention to truly make a difference?

Now, let’s think about this a bit differently. What if a brand that you truly cared about asked you to help make them better. Maybe it’s the restaurant chain you visit a few times a week for lunch, maybe the company that makes your child’s clothing, or perhaps your local health care system.

If you decided to participate in an ongoing conversation with this brand, you would get to make a difference in how that company serves you and others like you. And you would also get insider insight into how that company uses the information you give to shape crucial decisions. What kind of feedback would you give in this case? It would likely be well thought-out and very honest, because you want to see this brand succeed.

This is exactly why insight communities work. They engage customers who have agreed to give you feedback. Because they’re customers and they’ve opted in, they feel like they have a stake in your decision-making—and they provide better feedback.

What researchers say about insight communities and bias  

Okay, okay–I get you. You are a researcher. You need some proof, you need data, you need evidence.

John Lo, director of marketing intelligence at the University of British Columbia, addressed the bias question in a recent interview with Quirks. According to Lo, “In my experience, and occasionally through deliberate testing, I have found that data from insight communities are fully comparable to data from full population surveys.”

In fact, said Lo, key performance data (like a customer’s likelihood to recommend a brand to a friend or family member) from insight communities are “statistically identical to those collected under random response conditions.”

Similarly, two of my colleagues ran a project that dissected the effectiveness of insight communities. The study found that the quantitative differences in survey results with community members versus “gen pop” sample were comparable. If anything, community members were a bit more critical with their feedback. The researchers found that community members give richer and more honest feedback to these brands.

Some of our customers have held the same skepticism that you still might; after all, market research has a strong legacy of using gen pop sample from access panels and ad hoc emails spamming customers. As such, some have done their own studies comparing their community members to outside sample sources.

The result? Just like UBC’s Lo, these researchers found the same trends among their community members compared to traditional sample sources. A restaurant chain did research comparing market panel sample with results from their insight community, and found that the results were close. A food company found that outside sample matched the feedback they got from their insight community. A health care organization ran parallel studies and found that the results from an outside sample and an insight community were similar.

In a recent webinar, we asked Susan Frazier, director of custom research at PBS, about this topic, and she revealed that her team has ran similar studies to alleviate concerns about bias in insight communities. Frazier said that “the feedback we get from other sources is not directionally different from what we get within our insight community.” In fact, viewers in the organization’s community aren’t shy about providing critical feedback, according to Frazier. Today, many teams at PBS, including programming and marketing, rely on the community to get insight that improves campaigns and drives ratings.

Bias is a reality in research  

The fact is bias is unavoidable in research, regardless of the method you use. As market research expert Ray Poynter wrote in a blog post, “All research suffers from some form of bias” and every method has problems that need to be assessed and managed. “Trade-offs need to be made between speed, cost, relative accuracy and type of bias.”

In the end, insight communities provide directional insight that you can use for customer-centric decision-making. It provides a pulse on what your current customers and people like them think and feel. But, more crucially, studies show that communities provide results that are, in general, not directionally different from research conducted in the general population.

The bigger picture: customer relationships matter  

All interactions that you have with your customers impact their relationship with your brand. Research experience is customer experience. Communities allow you to get faster, higher quality insight, but they also help you build deeper relationships with your customers.

So, one last question to think about—would you rather that your research activity damage your customer’s relationship with you, or strengthen it? If you choose the latter, then an insight community is for you.