Surveys are a data collection technique where data is collected via a non-integrative self-report from the participants. Surveys are useful for gathering information about a large group of people quickly by asking people about their behavior, attitudes, or characteristics. The format of the questions can vary as appropriate for the research questions and administering a survey is typically less time-consuming that conducting interviews and focus groups.
Surveys provide a snapshot of behavior, attitudes, or characteristics at a given point in time. Multiple administrations of a survey can be compared so show a change over time as in the Pre-Test/Post-Test study designs.
Data collected via a survey are an important complement to empirical research. They can provide additional details about participants behavior and attitudes that can be explanatory to the data collected via other means. While surveys can be the sole data collection mechanism to answer a research question, they can support and strengthen other empirical evidence.
Surveys are used to answer research questions related to:
Where possible, consider using existing and/or validated instruments. But if there isn’t an appropriate instrument available for the variable you’re interested in measuring, you may need to create your own survey questions.
The wording of the survey question can determine the quality and accuracy of the responses. Some survey question wording problems are:
To write good survey questions, consider the following guidelines:
Some poorly worded survey questions are given below along with their problems:
I did NOT like the new testing tool. 1 = Strongly Disagree and 5 = Strongly Agree
This question is leading and uses negative wording.
Do you test and document your code each day?
This question is leading and double-barreled.
Do you still check modules into the code repository without thorough testing?
This question is loaded in an accusatory manner.
Do you think that the statistical variation of the amount of time spent by developers in formalized code review is a problem?
This question is loaded and overloads working memory.
Responses to survey questions can take on many forms. The main categorization of survey responses is closed-ended, where all the possible responses are provided, and open-ended, where the participant can fill in their response. Open-ended responses will require qualitative analysis and coding to identify themes and patterns in the data, which can lead to a richer understanding of the phenomenon of interest.
Closed-ended responses will provide a limited set of response items as a single select (e.g., How many credit hours have you completed?), multiple select (e.g., Which of the following describes your prior computing experience (select all that apply)), or scale answer (e.g., I can debug my programs successfully. Strongly Disagree to Strongly Agree). A common type of response is the Likert-scale with provides the participant with a full or partially labels set of responses over a continuum. For example, a 5- or 7-point Likert scale from Strongly Disagree to Strongly Agree. An even number of Likert responses can be used if you want a participant to commit to an answer in one direction or the other rather than selecting a neutral middle response.
When utilizing scaled responses, keep the response scales consistent across the survey and across multiple surveys!
When creating a survey, focus on the research questions that can be addressed with the types of questions that surveys are good for: attitudes and beliefs, facts and demographics, and behaviors! When creating a survey, consider the following:
Survey fatigue is when participants don’t complete the survey because the survey is too long or there may be too many surveys they are requested to complete. Focus your surveys on the key items that are necessary to answer your research questions and recognize in pre-test/post-test designs there will be some attrition of participants.
Pilot your surveys to ensure that the questions are interpreted in way that you intended. The best way to pilot a survey is through a think-aloud session where the participant will describe their interpretation of the survey questions. From the think-aloud information, you can refine your survey to avoid poorly worded or confusing questions.
Surveys can be administered in several ways (paper, electronic, in-person, mail/email, etc.) and there are trade-offs in response rates for the survey administration techniques. In-person surveys on paper tend to have the highest response rates, but the challenge is transcribing the survey answers to an electronic format for analysis. In-person surveys administered electronically via a form software can also have high response rates without the challenge of transcribing the data. Sending surveys out via mail or email will typically lower the response rate, but may be the appropriate means for administration for a distributed group of people or a survey of a large, international community.