Before going into depth on what empirical methods are, it is helpful to understand what they are not and remove some misconceptions.
People often connect the term Empirical Study with the concept of a controlled experiment that collects and analyzes a large amount of quantitative data. This type of study is certainly an example of an empirical method, but it is by no means synonymous with the term empirical method. There are many types of empirical methods that employ other research approaches.
Sometime researchers believe that they can conduct one study about a topic, draw conclusions based upon the data gathered, and then move on to another topic. The assumption here is that the results from that one study provide the definitive answer on the subject. This view is a misconception. Empirical studies, especially those that involve human subjects (as CS Education studies likely will), must be replicated in different settings and with different participants to fully understand a phenomenon.
Here we argue that CS Education is a Laboratory Science. Therefore, understanding the CS Education discipline involves using the scientific method: Observation, reflection, model building, experimentation. This process involves iterations that refine the tool, method, intervention, or other study focus as the researcher learns from each iteration.
Following on the last misconception, the goal of an empirical study is generally not to say ‘yes’ or ‘no’ to an intervention (e.g. a pedagogical technique). Rather, the idea is for the study to yield insights that can help the researcher evolve their methods or better understand the context within which those methods are most appropriate.
For example, imagine we created new automated grading system that we believe will provide helpful feedback to the students. If we run a study to evaluate this new tool we are not looking for an answer like “The new tool doesn’t (or does) work.” Rather, we are looking for an answer like “The new tool is better (or worse) than another approach in our environment.” This second type of answer means that we must define important terms like “environment” and “better (or worse)”. By thinking more deeply about how we want to compare approaches and clearly defining these measures, we can provide better insight into the concept we are studying.