How a Good Idea Becomes Bad Research

A good research project starts with a good research question. By this, we don’t necessarily mean something that is “good” or “important” or “meaningful,” although these concepts do affect the discussion. A project that is investigating the most impactful or meaningful concept can be immediately made impossible if the actual research qustion that is being proposed isn’t effective.

As an example, consider how many (but, of course, not all) research projects come to be in computer science education (CSEd). The pattern often goes like this:

1. Think of new [thing] you want to try in a class / program / project / assignment / etc.
2. Do the [thing] during the semester
3. Notice at the end of the semester: “Hey - that [thing] seemed to work well!”
4. Look at the previous semester and realize that you did not do that [thing] then and use that as a “baseline”
5. Quickly send a survey (or two!) to the students in the current class to get their reaction to [thing]
6. Write up the [thing] for the next SIGCSE Technical Symposium

While a bit cynical, it’s not hard to see this happening to many faculty who focus on teaching. Educators with large teaching loads are often spending the bulk of their time preparing and running their courses and don’t feel like they have time for extended research project planning.

It’s not to say that there isn’t value in the reflective “here’s how [thing] went” papers. In fact, there’s an entire track for that at the SIGCSE Technical Symposium called “Experience Reports and Tools!” But the goal of the DEERS project is to help researchers move from these more reactive and reflective research papers to more planned, empirical projects.

Consider some of the issues that can come from presenting these experience reports as research projects.

• Bias: When the educator created the [thing], they obviously wanted it to work well. If this person is also the person investigating whether the [thing] worked or not, there could be some bias introduced into the research, unintentional or not.
• Missing Data: Going back to examine the [thing] after the fact, you may find that you needed some piece of crucial data gathered along the way and that chance has now passed.
• Proper Study Setup: Or to be more direct, you may have needed to get approval from your Internal Review Board (IRB) before beginning a research project with human subjects.
• Hypothesis Hunting: Consider this comic, courtesy of XKCD:

Sometimes, you “find what you’re looking for,” even when that’s not what you were originally out to prove.

---