Mistake #1: Starting your questionnaire with demographics
Clients tell me that members of their dissertation committee insist on demographics at the beginning of a questionnaire. This is so WRONG! Please convince them otherwise. It is best to leave the demographics for the end. Here are some reasons why:
- The introduction to the questionnaire, either on the first page or in your recruitment email, included a call to action—Please give your opinion, your perceptions or attitudes about XYZ. They click on the link to your questionnaire, ready to provide their opinion, and what do they see? What is your age? Such a turnoff!
- Some respondents can find these questions (i.e., age, sex, income) intrusive and are not always comfortable answering them at the beginning of a survey. They are more likely to volunteer this type of information after they are sure the survey is legitimate, and they have taken the time to fill it out completely.
- Even if they quit before answering all the demographic questions, you still have their responses to the items that are at the heart of your study. You may be able to use that data in analyses that are not connected to demographic variables.
Mistake #2: Inadequate or inappropriate response options
I find that when asking study participants to answer questions about age or experience, the best option is a text box. Why?
- It is easy to not give a full range of possible responses. For example, you are surveying teachers. You ask them how many years they have been in education with ranges from 1-2 years, 2-5 years, 5-10 years, and 10 or more years. In many studies, the differences in attitudes and perceptions of the issue at hand can be significantly different among those with 1 year, 5 years, and 25 years. Do you want to lose that granularity or specificity by asking the teacher participants to limit their responses to the four you chose?
- Instead, ask them to type in their years of experience. It is easy on the online survey platforms to validate a text box to ensure that the respondents simply type in a whole number between 0 and, say, 50. Now you have the entire range of your participants’ experience.
- With that range of experience, you can conduct regression and correlation analyses or find your own (more relevant) ranges to use in analyses of variance such as t test, ANOVA, and MANOVA. It is easy to recode experience into ranges that may make more sense to you and your committee.
Important note: Income is a threatening question and should be dealt with using ranges. However, I have found that age is not a threatening question and few people leave that question blank. Again, having the luxury of the entire range of participants’ ages opens up all possible analysis options.
Mistake #3: Too many open-ended questions
Please, PLEASE, do not ask questions that require study participants to type more than 2 or 3 digits! If you must ask an open-ended question, make it the last question before the demographics and DO NOT make it mandatory. If your study’s research questions require open-ended questionnaire items, you should rethink your methodology. Interviews or focus groups may serve you much better. (Note: an open-ended questionnaire is NOT an interview.)
Tip: When discussing your questionnaire in your methodology section, please refer to it as one questionnaire. You may use several instruments to collect your variables of interest; however, they are all put into ONE questionnaire.