Most CER venues, especially those sponsored by SIGCSE, require a review of previous, related work to the study and research questions described in a manuscript. While the research question will drive your literature review, you may find that your literature review informs your research question, which then further informs your literature review and your study design.
A review of relevant literature provides an understanding of the common methods in CER, particularly around your research question. By exploring outside of computing, you may find studies from other disciplines that will help with or provide evidence to support your research direction. A literature review will provide an overview of the novelty, or the lack thereof, in your area of research. If your research question isn’t novel, don’t worry, consider a replication!
A key aspect of literature reviews is to situate your work in what’s already been done in the broader community. This can help you identify your contributions related to your research question.
Here are the key steps for a literature review that is appropriate to create a related works section in a CER paper. We’ll use the example of automated grading of Java programs in a CS1 class.
Systematic literature reviews are research reviews that attempt to synthesize the literature in a research area, and are not needed for CER studies.
From your research question identify key words that would be useful in your search. You may have several search strings. Most databases allow for logical operations.
For our example of automated grading, you may start with the search string:
automated grading
After looking at the results, you may find that you need to refine your search. For example, the top hit on the ACM’s Digital Library for automated grading is related to grading design diagrams, which isn’t relevant for a study involving introductory programming course. Further refining the search string may help:
automated grading AND CS1
Standard databases for CER are the ACM Digital Library and IEEE Xplore. The SIGCSE conferences and journal are all archived on the ACM Digital Library. Your librarian can help you identify other sources (Taylor and Francis, Elsevier, etc.).
Reviewing recent proceedings for relevant papers can provide the most recent work in the area.
Conferences
Journals
As you read papers, pay close attention to their related works section. That can help you identify key references that you should either prioritize or add to your list.
When conducting a literature review, not all literature will be relevant to your research question or area. Use the abstract to see if the paper is appropriate for further reading. The intro and discussion/conclusions sections can determine if a deeper read is needed. If the paper is very close to your research question or utilizes a study design you want to emulate, then a deep read is needed.
For a literature review, you should summarize the findings of the key papers, synthesize results, and relate to your research questions and work.
One of the hardest things with literature reviews is knowing when you’re done. When you start a literature review, you’ll be adding papers to your list to read quickly. As you map out a research area, you’ll add fewer and fewer papers. At that point, you’re most likely done… at least until the next conference proceedings are posted!
A lack of novelty in your research direction doesn’t mean that there is nothing else to contribute in that space. Instead, you should consider a replication of an earlier study in your institutional context to contribute additional evidence related to the research question. There are several different approaches to replication. We quote the definitions in the NSF’s and IES’s Companion Guidelines on Replication & Reproducibility in Education Research and from [Schmidt’s 2009 Replication paper], below:
Reproducibility: the ability to achieve the same findings as another investigator using extant data from a prior study
Replication: involved collecting and analyzing data to determine if the new studies (in whole or in part) yield the same findings as a previous study
Replication studies are further broken down into two categories:
Direct Replication: seek to replicate findings from a previous study using the same, or as similar as possible, research methods and procedures as a previous study
Conceptual Replication: seek to determine whether similar results are found when certain aspects of a previous study’s method and/or procedures are systematically varied
Other definitions and nuances to replication may be found in the following references: