The Ten Commandments of Survey Writing

Ah, surveys. They're used everywhere--election polls, censuses, major corporations, even "mom and pop" stores are doing them. It seems like everyone wants to ask other people some questions to get feedback. And everyone seems to think that anyone can write a survey. How hard could it be? Just write down some questions and see what kind of answers you get, right?

Sadly, that is not the case. Survey writing is actually a specific skill. The reason for that is, if you want the data you get in return to be accurate enough that you can actually do something with it, it is important to format the survey as a whole and all of its individual questions in the right way. Otherwise, you could run into problems like different folks interpreting questions/answers differently (giving you inconsistent results), becoming confused and quitting the survey (lowering your response rate), or becoming nervous and not answering truthfully (or completely misunderstanding the question and answering incorrectly). It's difficult to know when you get data back if they are correct or not (because you can't actually compare it to what's in people's heads), but you can reduce the potential for error by following some basic rules.

1) Thou shalt format the survey as a funnel.
The respondents are not as familiar with the subject matter at the beginning of the survey as they are at the end, and they are typically hesitant to answer difficult questions or share personal things immediately in the relationship they have with you, the researcher. So start out with more broad, easy questions to get them introduced to the topic. Court them. More difficult or lengthy questions like grids or multi-choice questions should be near the end. And demographics, being very personal in nature, should always be optional, and should (almost) always be at the end (exception--if a demographic is needed as a screener, but that's for another blog entry).

2) Thou shalt utilize skip patterns and page breaks correctly (for online surveys)
Having surveys online gives a lot more options that aren't possible otherwise, like using skip patterns and page breaks. Take advantage of these, and use them correctly! Spread the survey out over multiple pages because having the entire questionnaire on one page can be too daunting or overwhelming. If you want some people to answer a particular question but not other people, program it so that only those people see that question, rather than collecting data from everyone which may be confusing to some who find the question not applicable. Never let the respondent guess what they are supposed to do if you can avoid it!

3) Thou shalt use common or layman's language
It's tempting to assume everyone knows what you're talking about when you use certain words or phrases. But be aware that a) not everyone is as educated as you are, b) some of your words or terms may be specific to your industry, so even though you think everyone knows it, it may just seem that way from your perspective and c) some words or phrases may have different meanings to different people. When in doubt, always clarify what you mean.

4) Thou shalt avoid leading or loaded questions
Never give the slightest hint that you anticipate or guess how the respondent might answer. You must remain completely neutral! The way you phrase a question must indicate that any answer (and any opinion) is "correct." Bad idea: "So be honest, how drunk did you really get on Saturday night?" Good idea: "Which option best describes how many drinks you had on Saturday night?" The good question here not only asks in a non-leading way, but it also asks for a specific answer, and not a relative one--that means that one person will interpret it the same way as the next person. Which leads to the next commandment....

5) Thou shalt be as specific as possible
Without being specific, your respondents might all be answering different questions. If you ask, "do you like oatmeal?", your respondents might wonder if you want to know whether they like the taste, the texture, the affordability or value, the variety available, the health benefits, or something else. Never assume they know what you mean--always be specific and remove all doubt.

6) Thou shalt live by the rules of MECE
What does MECE mean? Write this down, because you're about to learn your fact of the day, and if you remember only one thing from this article, this should be it. MECE stands for "Mutually Exclusive, Collectively Exhaustive." Repeat that to yourself--I promise, it's actually important in areas outside of survey writing, too. This should be applied to both questions and answer choices. You can't ask "Do you think this event was affordable and informative?" Those are actually two different questions! It's also very important to be MECE in answer choices (and this is a common mistake I see all the time). Try to find the problem with the following question:
How many children do you have?
-0
-1 to 2
-2 to 3
-3 to 5
-5 or more

See it? The responses overlap! These are not mutually exclusive. If I have 2, 3, or 5 children, I'm not sure which answer to choose. This gives you very muddy data. Try to find the problem with the next question:
Please describe your area of specialization (choose one):
-Finance
-Marketing and Statistics
-Data analysis
-Graphic Design

Oh dear. These choices are not mutually exclusive. What if I specialize in Statistics and Data Analysis? What if I specialize in Marketing, but not Statistics? I don't know which to choose! Also, this list is not collectively exhaustive. It's a very short list for a very broad question. I could specialize in accounting, management, negotiations, sales, customer service, and any number of other things and still not have an answer that suits me. Aim to cover about 90% of the possible responses, and include an "Other" selection for things you might have missed. If you end up with a large amount of people who choose "Other," you probably didn't do a good job including an exhaustive list. If your exhaustive list is too long and burdensome, your question probably is not specific enough.

7) Thou shalt use balanced lists
The easy side of this is to say a question like, "How satisfied or dissatisfied were you with the service?" would have an equal number of responses ranging from strongly dissatisfied to strongly satisfied, with a neutral response in the middle. Easy, right? It gets trickier when you get to questions with weighted answers, like income. Consider the following question:
What was your total pre-tax household income in 2012?
-$0-$49,999
-$50,000-$99,999
-$100,000-$150,000
-$150,000-$200,000

First of all, I should point out that the question was worded specifically enough, and the answer choices were mutually exclusive. Good job. The problem is, if you have any awareness of household income in the United States, you know that most of the answers will come lumped in the first two options, and the last two options will not have many responses at all. While it makes sense for the answer ranges to be equal (in this case, about $50K), you can make reasonable exceptions to that, and you also should have some idea (when possible) what your ranges of responses might be in order to format accordingly. In this case, you can consult the latest census results. Let's try that again:
What was your total pre-tax household income in 2012?
-Less than $30,000
-$30,000-$49,999
-$50,000-$69,999
-$70,000-$89,999
-$90,000-$109,999
-$110,000 or more

Bear in mind that there is no one right way to ask for income. If my survey was going out to a very affluent group of people, my response choices would look very different. Also note that the response choices are not equal--the first bucket is $30K, the groups in the middle are only a $20K range, and the range that is above $110K in the last bucket is going to be even larger. But if I know that the majority of my responses are going to land somewhere in the middle, and I've spaced out the ranges in the middle enough that I can do the analysis that I need to do, then this format is appropriate and much better.

8) Thou shalt use caution with complex questions
Questions like open-ended response, multi-choice grids, ratings or rankings etc. (apologies if you don't know what those are--maybe that will be another blog post), are laborious in that the respondent needs to think harder about their answers, and they take longer to complete. You should be careful to only include a reasonable number of these in the survey. If you over-burden the respondent, they might either quit the survey, or just fill in bubbles without even reading the question! That's not good...

9) Thou shalt avoid "nice to know" questions
It is VERY tempting to get greedy when we're collecting data. If we have someone's captive attention, we want to keep questioning them! Ask them all sorts of things! Just keep going and who knows, maybe something brilliant will come out of it! Whoa there, tiger, don't get too excited. More is definitely not better when it comes to surveys. The longer the survey gets, the more tired or annoyed the respondents become at the end, and the more likely they are to become fatigued and either drop out, or give faulty answers. You should have in mind exactly what kind of analysis you are going to perform, and what questions you need for that analysis. Ask those things--no more, no less.

10) Thou shalt mind the overall length of the questionnaire
I won't go into detail on this because I've already said it (see the irony?). But it bears repeating.