Ah, surveys. They're used everywhere--election polls, censuses, major corporations, even "mom and pop" stores are doing them. It seems like everyone wants to ask other people some questions to get feedback. And everyone seems to think that anyone can write a survey. How hard could it be? Just write down some questions and see what kind of answers you get, right?The respondents are not as familiar with the subject matter at the beginning of the survey as they are at the end, and they are typically hesitant to answer difficult questions or share personal things immediately in the relationship they have with you, the researcher. So start out with more broad, easy questions to get them introduced to the topic. Court them. More difficult or lengthy questions like grids or multi-choice questions should be near the end. And demographics, being very personal in nature, should always be optional, and should (almost) always be at the end (exception--if a demographic is needed as a screener, but that's for another blog entry).
Having surveys online gives a lot more options that aren't possible otherwise, like using skip patterns and page breaks. Take advantage of these, and use them correctly! Spread the survey out over multiple pages because having the entire questionnaire on one page can be too daunting or overwhelming. If you want some people to answer a particular question but not other people, program it so that only those people see that question, rather than collecting data from everyone which may be confusing to some who find the question not applicable. Never let the respondent guess what they are supposed to do if you can avoid it!
It's tempting to assume everyone knows what you're talking about when you use certain words or phrases. But be aware that a) not everyone is as educated as you are, b) some of your words or terms may be specific to your industry, so even though you think everyone knows it, it may just seem that way from your perspective and c) some words or phrases may have different meanings to different people. When in doubt, always clarify what you mean.
Never give the slightest hint that you anticipate or guess how the respondent might answer. You must remain completely neutral! The way you phrase a question must indicate that any answer (and any opinion) is "correct." Bad idea: "So be honest, how drunk did you really get on Saturday night?" Good idea: "Which option best describes how many drinks you had on Saturday night?" The good question here not only asks in a non-leading way, but it also asks for a specific answer, and not a relative one--that means that one person will interpret it the same way as the next person. Which leads to the next commandment....
Without being specific, your respondents might all be answering different questions. If you ask, "do you like oatmeal?", your respondents might wonder if you want to know whether they like the taste, the texture, the affordability or value, the variety available, the health benefits, or something else. Never assume they know what you mean--always be specific and remove all doubt.
What does MECE mean? Write this down, because you're about to learn your fact of the day, and if you remember only one thing from this article, this should be it. MECE stands for "Mutually Exclusive, Collectively Exhaustive." Repeat that to yourself--I promise, it's actually important in areas outside of survey writing, too. This should be applied to both questions and answer choices. You can't ask "Do you think this event was affordable and informative?" Those are actually two different questions! It's also very important to be MECE in answer choices (and this is a common mistake I see all the time). Try to find the problem with the following question:
How many children do you have?
-0
-1 to 2
-2 to 3
-3 to 5
-5 or more
Please describe your area of specialization (choose one):
-Finance
-Marketing and Statistics
-Data analysis
-Graphic Design
The easy side of this is to say a question like, "How satisfied or dissatisfied were you with the service?" would have an equal number of responses ranging from strongly dissatisfied to strongly satisfied, with a neutral response in the middle. Easy, right? It gets trickier when you get to questions with weighted answers, like income. Consider the following question:
What was your total pre-tax household income in 2012?
-$0-$49,999
-$50,000-$99,999
-$100,000-$150,000
-$150,000-$200,000
What was your total pre-tax household income in 2012?
-Less than $30,000
-$30,000-$49,999
-$50,000-$69,999
-$70,000-$89,999
-$90,000-$109,999
-$110,000 or more
Questions like open-ended response, multi-choice grids, ratings or rankings etc. (apologies if you don't know what those are--maybe that will be another blog post), are laborious in that the respondent needs to think harder about their answers, and they take longer to complete. You should be careful to only include a reasonable number of these in the survey. If you over-burden the respondent, they might either quit the survey, or just fill in bubbles without even reading the question! That's not good...
It is VERY tempting to get greedy when we're collecting data. If we have someone's captive attention, we want to keep questioning them! Ask them all sorts of things! Just keep going and who knows, maybe something brilliant will come out of it! Whoa there, tiger, don't get too excited. More is definitely not better when it comes to surveys. The longer the survey gets, the more tired or annoyed the respondents become at the end, and the more likely they are to become fatigued and either drop out, or give faulty answers. You should have in mind exactly what kind of analysis you are going to perform, and what questions you need for that analysis. Ask those things--no more, no less.
I won't go into detail on this because I've already said it (see the irony?). But it bears repeating.



