The Ten Commandments of Survey Writing

Ah, surveys. They're used everywhere--election polls, censuses, major corporations, even "mom and pop" stores are doing them. It seems like everyone wants to ask other people some questions to get feedback. And everyone seems to think that anyone can write a survey. How hard could it be? Just write down some questions and see what kind of answers you get, right?

Sadly, that is not the case. Survey writing is actually a specific skill. The reason for that is, if you want the data you get in return to be accurate enough that you can actually do something with it, it is important to format the survey as a whole and all of its individual questions in the right way. Otherwise, you could run into problems like different folks interpreting questions/answers differently (giving you inconsistent results), becoming confused and quitting the survey (lowering your response rate), or becoming nervous and not answering truthfully (or completely misunderstanding the question and answering incorrectly). It's difficult to know when you get data back if they are correct or not (because you can't actually compare it to what's in people's heads), but you can reduce the potential for error by following some basic rules.

1) Thou shalt format the survey as a funnel.
The respondents are not as familiar with the subject matter at the beginning of the survey as they are at the end, and they are typically hesitant to answer difficult questions or share personal things immediately in the relationship they have with you, the researcher. So start out with more broad, easy questions to get them introduced to the topic. Court them. More difficult or lengthy questions like grids or multi-choice questions should be near the end. And demographics, being very personal in nature, should always be optional, and should (almost) always be at the end (exception--if a demographic is needed as a screener, but that's for another blog entry).

2) Thou shalt utilize skip patterns and page breaks correctly (for online surveys)
Having surveys online gives a lot more options that aren't possible otherwise, like using skip patterns and page breaks. Take advantage of these, and use them correctly! Spread the survey out over multiple pages because having the entire questionnaire on one page can be too daunting or overwhelming. If you want some people to answer a particular question but not other people, program it so that only those people see that question, rather than collecting data from everyone which may be confusing to some who find the question not applicable. Never let the respondent guess what they are supposed to do if you can avoid it!

3) Thou shalt use common or layman's language
It's tempting to assume everyone knows what you're talking about when you use certain words or phrases. But be aware that a) not everyone is as educated as you are, b) some of your words or terms may be specific to your industry, so even though you think everyone knows it, it may just seem that way from your perspective and c) some words or phrases may have different meanings to different people. When in doubt, always clarify what you mean.

4) Thou shalt avoid leading or loaded questions
Never give the slightest hint that you anticipate or guess how the respondent might answer. You must remain completely neutral! The way you phrase a question must indicate that any answer (and any opinion) is "correct." Bad idea: "So be honest, how drunk did you really get on Saturday night?" Good idea: "Which option best describes how many drinks you had on Saturday night?" The good question here not only asks in a non-leading way, but it also asks for a specific answer, and not a relative one--that means that one person will interpret it the same way as the next person. Which leads to the next commandment....

5) Thou shalt be as specific as possible
Without being specific, your respondents might all be answering different questions. If you ask, "do you like oatmeal?", your respondents might wonder if you want to know whether they like the taste, the texture, the affordability or value, the variety available, the health benefits, or something else. Never assume they know what you mean--always be specific and remove all doubt.

6) Thou shalt live by the rules of MECE
What does MECE mean? Write this down, because you're about to learn your fact of the day, and if you remember only one thing from this article, this should be it. MECE stands for "Mutually Exclusive, Collectively Exhaustive." Repeat that to yourself--I promise, it's actually important in areas outside of survey writing, too. This should be applied to both questions and answer choices. You can't ask "Do you think this event was affordable and informative?" Those are actually two different questions! It's also very important to be MECE in answer choices (and this is a common mistake I see all the time). Try to find the problem with the following question:
How many children do you have?
-0
-1 to 2
-2 to 3
-3 to 5
-5 or more

See it? The responses overlap! These are not mutually exclusive. If I have 2, 3, or 5 children, I'm not sure which answer to choose. This gives you very muddy data. Try to find the problem with the next question:
Please describe your area of specialization (choose one):
-Finance
-Marketing and Statistics
-Data analysis
-Graphic Design

Oh dear. These choices are not mutually exclusive. What if I specialize in Statistics and Data Analysis? What if I specialize in Marketing, but not Statistics? I don't know which to choose! Also, this list is not collectively exhaustive. It's a very short list for a very broad question. I could specialize in accounting, management, negotiations, sales, customer service, and any number of other things and still not have an answer that suits me. Aim to cover about 90% of the possible responses, and include an "Other" selection for things you might have missed. If you end up with a large amount of people who choose "Other," you probably didn't do a good job including an exhaustive list. If your exhaustive list is too long and burdensome, your question probably is not specific enough.

7) Thou shalt use balanced lists
The easy side of this is to say a question like, "How satisfied or dissatisfied were you with the service?" would have an equal number of responses ranging from strongly dissatisfied to strongly satisfied, with a neutral response in the middle. Easy, right? It gets trickier when you get to questions with weighted answers, like income. Consider the following question:
What was your total pre-tax household income in 2012?
-$0-$49,999
-$50,000-$99,999
-$100,000-$150,000
-$150,000-$200,000

First of all, I should point out that the question was worded specifically enough, and the answer choices were mutually exclusive. Good job. The problem is, if you have any awareness of household income in the United States, you know that most of the answers will come lumped in the first two options, and the last two options will not have many responses at all. While it makes sense for the answer ranges to be equal (in this case, about $50K), you can make reasonable exceptions to that, and you also should have some idea (when possible) what your ranges of responses might be in order to format accordingly. In this case, you can consult the latest census results. Let's try that again:
What was your total pre-tax household income in 2012?
-Less than $30,000
-$30,000-$49,999
-$50,000-$69,999
-$70,000-$89,999
-$90,000-$109,999
-$110,000 or more

Bear in mind that there is no one right way to ask for income. If my survey was going out to a very affluent group of people, my response choices would look very different. Also note that the response choices are not equal--the first bucket is $30K, the groups in the middle are only a $20K range, and the range that is above $110K in the last bucket is going to be even larger. But if I know that the majority of my responses are going to land somewhere in the middle, and I've spaced out the ranges in the middle enough that I can do the analysis that I need to do, then this format is appropriate and much better.

8) Thou shalt use caution with complex questions
Questions like open-ended response, multi-choice grids, ratings or rankings etc. (apologies if you don't know what those are--maybe that will be another blog post), are laborious in that the respondent needs to think harder about their answers, and they take longer to complete. You should be careful to only include a reasonable number of these in the survey. If you over-burden the respondent, they might either quit the survey, or just fill in bubbles without even reading the question! That's not good...

9) Thou shalt avoid "nice to know" questions
It is VERY tempting to get greedy when we're collecting data. If we have someone's captive attention, we want to keep questioning them! Ask them all sorts of things! Just keep going and who knows, maybe something brilliant will come out of it! Whoa there, tiger, don't get too excited. More is definitely not better when it comes to surveys. The longer the survey gets, the more tired or annoyed the respondents become at the end, and the more likely they are to become fatigued and either drop out, or give faulty answers. You should have in mind exactly what kind of analysis you are going to perform, and what questions you need for that analysis. Ask those things--no more, no less.

10) Thou shalt mind the overall length of the questionnaire
I won't go into detail on this because I've already said it (see the irony?). But it bears repeating.

Tardy to Tweet, but I'm No Twit

I know, I know, I'm a little late getting on this bandwagon. I have used Twitter before as a researcher, and it has been a great tool for tracking trends and searching for information. But I intentionally stopped short of joining it personally up until now because it always seemed a bit narcissistic. Facebook can be great to keep in touch with people you actually know, but to broadcast your thoughts to the entire world and try to get more "followers" out of people you don't actually know? I'm not sure it's possible to create a tool more driven by narcissism.

In the last couple of months that I have had to play around with Twitter, though, while it is true that narcissism is alive and well in some users (and there seems to be almost a game among many to see how many followers they can get), I have actually come to like it and appreciate it for reasons I did not quite realize before I hopped on the bandwagon.

1.  Sharing content. This reason I was already aware of through previous research work that I've done, but it's quite different to see it in action. It's overwhelming the amounts and variety of content that is available on Twitter, and it spans as far as your imagination can reach. I can see how it would be possible to obsess over watching your Twitter feed constantly over hours at a time, but I've avoided that so far and hope I never reach that point.
2.  Making new connections.  Like many people, I would imagine, I utilize Facebook mainly for personal connections, and LinkedIn mainly for professional connections. But in both cases, I don't often interact with people I don't know. In LinkedIn I may get contacted from time to time by headhunters and such, but when I share information there, I highly doubt many people pay attention--even those I am actually connected to. But with Twitter, the more valuable the content that I share and the more ways I find to share that content with appropriate audiences directly and engage with key people in my industry, the more exposure I get within my industry and the more valuable connections I am able to make with my peers. By sharing one article, I got myself featured in one industry newsletter's website for the day, and while that is a small accomplishment, it astounded me by the power and possibility that Twitter has.
3.  Building a personal/professional brand.  I have always believed in the importance of making a name for yourself and developing a digital brand. This blog is the result of that, and even though I don't find the time to update it as often as I'd like, it is an outlet I keep for sharing my thoughts and showing the digital world who I am. There are many outlets for digital branding, and Twitter is another one. The difference is, whereas it is difficult to find time to write a blog entry, it is quite easy to throw something out in 140 characters, and I find it a good way to show a little bit of both my personal and professional persona, as a good businesswoman can be her true self in both capacities, and someone is much more interesting to do business with if you get to know him or her a little bit on a more personal level.

These are just a few of my initial thoughts on Twitter. I am sure I have only scratched the surface on what it is capable of (especially since this only really covers the individual level and not the business/corporate brand level), so I look forward to exploring it a bit more.

P.S. If you haven't already found me on Twitter, you can check me out @beckybmw

What's your dream job?

The other day in an interview, someone asked me what would be my dream job. It's a common question, and one I know exactly how to answer in an interview.  But when the moment came that I realized that the job I was describing actually was the job I was interviewing for, I wondered if this job really was that great, or if it was time that I re-visited what my idea of a dream job could be, and think about more of the finer details.

This is a question that I've thought about often in my career, but I've gotten to a point where my dream job could be reachable soon, so it actually is time to start ironing out specifics.

I think we've all had a dream position and/or a dream company in mind.  But have we thought about all the details a dream job would entail? What would a dream job look like according to level of company influence, size of company, company culture, work style, work environment, location, industry, etc, etc?

It's a fun thing to dream about, but it's also a good practical exercise. Once you identify exactly what it is that you want, you can see the differences between what you want and where you currently are in your career. Then you can think about what steps you might need to take to get to the dream.

I'd recommend repeating this exercise every few years, both to see if the dream has changed at all, and to monitor your progress toward the goal. Not to mention, it's also just fun and inspiring to daydream a little bit now and then, to help you believe that the thing that you want is out there, and you can make it happen.

Happy dreams!

Super Bowl XLVII Commercials Review!

Here we go again, it's time for my annual post reviewing the Super Bowl commercials.  If you're joining me for the first time this year, this is something I do every year (except I missed last year--sorry about that).  I, like so many people, usually watch the Super Bowl just for the commercials (pro football? couldn't give a flying fart....), and while many people throw around their one or two favorites, I enjoy taking a bit more time to dissect the commercials of the year.  So I take notes on every last one, pick out my favorites, and add some commentary on my choices.  If you'd like to go back in time and catch up, you can find my posts about Super Bowl Ad History, Super Bowl XLIV, and Super Bowl XLV.

This year's commercials overall were much better quality than last year's.  Although I did not write up a review last year, I do remember that not very many of them stood out to me, and I certainly cannot remember any of them now (which says a lot).  A few overall interesting observations: one, this year's ads were not nearly as exploitative and overly-sexual in nature as the usual (exception: Go Daddy).  An interesting and welcome change of pace, in my opinion.  Two, while Doritos typically offers 3 or 4 spots during the game, all incredibly witty and entertaining, this year they only had one, which was kind of a dud in my book.

Alright, let's get right to it!  My top 5 of the year are:

5.  Taco Bell--"Viva Young"


Not everyone in my house got this one, but the track playing is fun.'s "We Are Young," but sung in Spanish--Sort of a "Taco Bell" joke, if you will.  The song goes well with the video of some folks sneaking out of the retirement home for a little fun out on the town.  The theme of living forever young goes well with the Taco Bell brand, and even though it wasn't my house's favorite, it made me smile.

4.  Best Buy--"Questioning Amy Poehler"


This holds a soft spot in my heart because the spot is for a retailer. But in addition, there is no denying how awesome Amy Poehler is. And, not only is this ad clever and funny, but it really sets up Best Buy as a service provider--one who can answer all those questions.  In an age when Best Buy is fighting off the "showroom" phenomenon with their business being threatened by the internet, it's an important leg to stand on.

3.  Coca Cola--"See the World Differently"

>

This commercial actually isn't new--it came out last June, although if you're like me and have been a little bit out of the commercial loop this year, this may have been your first time seeing it.  (Also because it's been around a while, the internet police have already snatched it away from a lot of places online, it was a lot harder to find a video that I could share.  If the above embed doesn't work, try this link for the source).  And yet, even though it was an older commerical, it still stood out to me.  I have always loved the sense of optimism that is a theme in Coca-Cola's ads, and it is always a bright spot in a string of Super Bowl commercials that feature beer, sex, slutty girls, lazy guys, and other content that just makes me roll my eyes.  Perhaps that is one of the reasons why Coca Cola's brand is the biggest brand of the world.  Perhaps that is why they always have spots in my own personal top 5.  Well, whatever.  I just liked it.


2.  Oreo--"Silent Fight"


The internet BLEW. UP. after this ad aired, and so did my house.  We loved this.  The debate between cream or cookie is pretty straight-forward when it comes to Oreos (BTW, I'm on team Cookie), but the delivery of a whisper is clever and makes you pay attention to it amidst all the other commercial clutter.  Loved the extremes that it develops into.

1.  Budweiser--"Brotherhood"


*sniff* I always find it amazing how Budweiser can manage to create an entire mini-drama that is touching, heartwarming, and a complete story, all in a short clip that's just over a minute long.  This one was a standout tonight in the way it did all those things, and still stuck to strong brand position that's been around for years (and yet has not gotten old or stale!).  Good job, Budweiser.

Honorable mentions:

• M&M's "Anything for Love" - They took the classic (?) Meatloaf song and spun it into the drama that the red M&M faces in his relationship with a human lady, and where he draws the line.  Cute.
• Samsung "The Next Big Thing" - Paul Rudd and Seth Rogan are invited to brainstorm ways for Samsung to advertise the next big thing.  Their ideas range from stereotypical Super Bowl commercial fare to quirky weird oddities to things that can be found in the very ad they are featured in.  I loved the clever plays, like "If you get Lebron James, you won't need anyone else..."
• Jeep "Coming Home" - Featuring Oprah's voiceover, this was a tribute to our troops.  I loved the idea of Jeep saluting the troops, and using Oprah to read the script was a bold statement.  It was just too long and somewhat repetitive to make it in my top 5.
• Dodge RAM "God Made a Farmer" - Beautiful nod to all the amazing qualities of our country's farmers (who of course, get it all done with their Dodge trucks). Again, a bit lengthy for me, but I loved the spot, and it was a good brand fit.

Duds:

•  Budweiser Black Crown - There were a couple of these spots, virtually identical, and very beer commercial cliche--sexy people drinking beer at a party. Seems like they are introducing a new product that is supposed to compete in a premium market, but any premium beer drinker would laugh at the idea of "Budweiser" being considered premium.  Maybe they should have thought more carefully about using the Budweiser name for that.
• Go Daddy - They nearly always make my dud list, and while I do give them credit for making an ad that everyone will be talking about the next day (and the "shock and awe" factor is pretty much what they always stand for), this one really pushed the envelope even more than usual.  Not in a good way, either.  I didn't mind the fact that the hot girl was making out with a geek on principle, but I didn't really need to hear the detailed saliva-swapping audio of such an encounter.
• Subway - What's that?  Jarred's anniversary, lots of famous people who can't pronounce February or FebruANY or whatever? Meh, forgettable.  Nice try, though, I guess.
• E*Trade - I know that the talking baby is party of their brand legacy now and maybe I'm just going to have to accept it.  I just hate that thing.
• Pistachios - Crackin' Gangnam Style?  Oh please.

 Until next year!

New Year, New Perspective

Happy New Year!

I'm not normally a person who does resolutions and such for New Years--my general belief is that I can be motivated to make a change in my life at any time of the year, and I take advantage of that when the motivation is pure and strong, not when it is dictated by a holiday.  However, perhaps because my life is currently at a bit of a crossroads, I have found myself recently reflecting on a few things and thought it would be good to write them down (if for no one else other than myself).

First, I've thought about what has happened in the past year and how I have grown.  Since I have worked in very fast-paced industries in the past, I've always been used to frequent and rapid change.  But 2012 has definitely shown me how to roll with the punches in my personal life, as well.  People always say, "don't sweat the small stuff," and we all believe that we live by that rule.  But usually what happens is we tell ourselves that small things don't bother us, and secretly they fester under the surface and make us unhappy.  This year I think I've made more progress on really living that mantra.  This year I was laid off along with 40 or so other colleagues in a company restructuring, and although I understand very well my colleagues' desire to lash out against said company with verbal slander, I just did not find it productive for myself since there was nothing I could have done to change the situation, and I needed to move on with my life.  That's a big event, too--I've found that the small things roll off my back fairly easily now as well, without even much thought.  I think in past years I have told myself that things should not bother me, but the reality was that they did.  In a sense, we can't control what bothers us and what doesn't, at least in the short-term.  But perhaps between all the times I told myself that things shouldn't bother me, and going through a period when a lot of big stuff happened, I have gained that deep-seeded perspective that if there isn't anything I can do about a situation, it isn't worth getting upset or stressed-out over.  I think it's important that I recognize this change in myself and be grateful, before I launch into all the things I want to change.

So, about that.  The coming year is going to present a lot of changes for me, of that I am sure.  I do firmly believe that I will be in another job soon (it just remains to be seen where it will be).  And regardless of where that job will be, I will also have a new place to live soon.  I have to recognize that all of these changes afford me the ability to "start fresh," set a new routine, and make my life into what I want it to be.  Now is the time for me to decide what that is.  A few things I want to change:

1. It's okay to be a little selfish.  I've thought about others my entire life.  Practically my entire existence has been about making others happy, trying to make others like me, and doing the right thing for everyone.  While this is admirable and I will always continue this to a degree, I've found that balance is necessary so that others don't take advantage of me.  Maybe this is a little bit of the East Coast rubbing off on me, where everyone only thinks about "number 1" and if I am always thinking about others, I always lose.  I'm not going to lose anymore.  I deserve to live a good life, too.
2. Get back into volunteering.  Some things will never change, though, and working for my community and for a cause is a big part of who I am.  I've fallen away from it a bit in the past few years as I've dealt with a lot of personal issues, but it's time to start back up again.
3. Assume nothing.  It is so easy to get into an argument or disagreement or misunderstanding by not saying something because I assume the other person already knows, or by assuming that the other person would act in the same way I would.  They are not mind-readers, just like I am not a mind-reader.  State the obvious--I'll avoid the problems, and if it really is too obvious, we'll both have a laugh over it.
4. Be grateful.  There will always be terrible things that happen to me in life.  But I get through all of them when I remember that no matter what happens, it can always be worse.  Focusing on the positive things in my life helps adjust my attitude away from dwelling on those terrible things.  I like the idea of waking up every morning and thinking of at least 3 things that make me happy, or that I'm grateful for.  Thinking of happy things is a good way to start every day, don't you think?  It makes a big difference in my attitude for the entire day.
5. No waiting around for life--life is short.  Since I think I've gotten down the "don't sweat the small stuff" mantra in practice, this is one I'd like to work on as a new long-term thought mantra.  I've lived my life carefully for the most part, and while that can be a good thing in many ways, it may have stifled my ability to take risks or take advantage of opportunities that could end up being great or even life-changing.  I don't want to look back at my life in 20 years and wish I had done more or tried more.  The older I've gotten, the more I've realized that it all moves very fast, and the years I have ahead of me are not infinite.  There's a saying that hit me like a ton of bricks the first time I read it, and I like to remind myself of it often, whenever I catch myself "waiting" for happiness to find me:

“For a long time it seemed to me that life was about to begin – real life. But there was always some obstacle in the way, something to be gotten through first, some unfinished business, time still to be served, a debt to be paid. At last it dawned on me that these obstacles were my life. This perspective has helped me to see there is no way to happiness. Happiness is the way. So treasure every moment you have and remember that time waits for no one. Happiness is a journey, not a destination.” ~Souza

Anyway, that's enough deep philosophy for now.  Good luck to all of you in your goals, whether they are weight loss, quitting smoking, or attitude adjustments (like mine).