We recently published a special report titled 25 Common Field Mistakes To Avoid When Conducting Your Qualitative Market Research. At point n. 12, we suggest that researchers eliminate open-ended screening questions that aim to identify respondents who are outgoing and able to express themselves.
Here is the full text of point # 12:
“If your evaluator contains open-ended questions that are intended to generate expressive types of people, leave those questions. Questions like that do not work and unnecessarily lengthen your detection process. Yes, you want to exclude respondents who cannot or will not express themselves, but You don’t need an additional question to identify these people. Well-trained recruiters will eliminate them within the first few minutes of the screening. If you want more assurance that your respondents will be outgoing and communicative, go overboard recruiting and include a pre-discussion phone interview This would be done by the moderator, who would then select the appropriate respondents. “
Our suggestions generated a lot of comments. Some readers agreed with us, others did not. One of the most interesting comments we received came from a qualitative fieldwork manager at a major full-service research firm. It went like this: “I disagree with an item on your list about excluding raters from open-ended questions. Good recruiters can easily identify articulate respondents, but tired or distracted recruiters can sometimes get on autopilot … I think a little qualitative on a screener pays off. “
Do you agree with this reader’s comment? You should?
What is an articulation question?
The articulation questions measure the communication skills of a respondent. Articulation questions also judge respondents expected communicativeness in a focus group or interview.
Some synonyms for “communicative” include: outgoing, outspoken, communicative, talkative, unbridled, talkative. So who decides what is communicative? The recruiter? The hiring supervisor? The customer who reads your daily reports verbatim? And how much communication is enough? How much is too much?
Even the most seasoned recruiters cannot determine how outgoing, outspoken, communicative, talkative, unbridled, or chatty a respondent will be at a future point in time. That is a decision that recruiters are not qualified to make. But they can be counted on to spot respondents who have …
- language barriers
- Casual attitudes toward the recruiter, recruiter questions, or research.
- reservations about your ability to attend the investigation
- some problems communicating during the selection process
What to watch out for …
Respondents get tired or use autopilot when screening interviews take too long (10 minutes or more).
Articulation questions do not belong at the end of your filter. For whatever reason, the joint evaluation is almost always done at the end of the evaluation interview. But why is a question that is supposedly so important put at the bottom of the filter, when the chances of respondents being tired or distracted are higher? What are recruiters learning about respondents at this point in the process that they don’t already know?
The articulation questions should also not be in front of your evaluator. Well-trained recruiters immediately engage respondents in a conversation about the details of the research. It is during this prelude to the screening questions that recruiters address the questions and concerns of the respondents and make an assessment of the communication skills of the respondents.
Articulation questions are not magic formulas that ensure good participants in the focus groups. These questions simply ask recruiters to use their own biased judgment to decide whether a respondent can communicate clearly.
Articulation issues lengthen your screening. Remember this. The longer your screener, the higher your costs.
Respondents get anxious when asked questions outside the left field that are not related to the selection questions. Being asked, “What is a gazinkle?” or “How many different things can you do with a paper clip?” or “If you were a tree …?” it could baffle even the most eloquent respondent. Outlandish questions from recruiters confuse and frustrate respondents. This line of questioning is the territory of the moderator.
Of course, the group dynamics and the personalities of the respondents affect how open and receptive the respondents will be. For example, a person may talk to the recruiter on the phone, but feel intimidated if an aggressive personality dominates the group. Or, a respondent may not feel as comfortable with the research topic as they thought they would be and feel out of place, especially if the topic offered during recruitment was vague. How can recruiters know how respondents will perform in a variety of conditions? Handling reserved respondents is the moderator’s area of expertise.
In fact, moderators are better qualified to know what can and should be expected of respondents in terms of communicativeness and articulation. So it makes sense that, as we suggested in point # 12 of our special report, moderators should pre-interview respondents and select the right personalities for the investigation.
So what about articulating questions that are helpful in snapping distracted or unaware recruiters out of their daze (as our reader suggested)? Assuming a tired and distracted recruiter missed all red flags during selection, will the articulation question suddenly remind the recruiter that the respondent is not talkative? What Should You Do With Recruiters On “Auto Pilot”? Easy.
The investigator’s job is not to come up with questions that keep recruiters alert and focused. Tired or distracted recruiters are not an asset to your research. They don’t help you get good respondents. And neither are the articulation questions. Don’t use any of them.