Taking time to validate your instruments or to pilot your research design and data collection can help you avoid problems during the actual study. The purpose of validation is to try out the study design and not to gather actual data to answer your research questions. Any data gathered during a validation step is not eligible for analysis or reporting. Additional information about reliability and validity are available on csedresearch.org.
One way to validate an instrument, like a survey, is to conduct a think-aloud session with several people similar to who you plan to study. In a think-aloud, the participant will verbalize their thoughts as they work through the instrument. They’ll explain any confusion with questions or clarify their understanding. If the participant’s understanding is different than the what the researcher intended, the opportunity is available to clarify the question to ensure the right thing is asked so that you’re measuring something appropriate to the variable.
When operationalizing variables, you want to ensure that the measurements are appropriate for the phenomenon on interest. Expert review can help with determining if the measure is appropriate for the variable. Comparison with similar instruments may also be involved. Expert review and comparison with other validated work increases content validity.
The entire study process can be piloted with a group of participants similar to the ones that you intend to study. A pilot study can help you determine if your research design and data collection processes are going to work in the actual context and will gather the information necessary to answer the research questions.
If you are creating or working with a content inventory to measure knowledge of a specific topic, then item response theory can help determine the validity and reliability of the questions in the concept inventory. Item Response theory is a technique for identifying if the possible answers to a multiple choice question demonstrate strong item discrimination between answers, appropriate difficulty, and a low guessing probably. [T10].