- Where do I find my topic and research question?
- What sort of study should I do? Quantitative or qualitative? Or mixed methods?
- How much should I address – the sun, the moon and the stars?
- Evaluation studies can be tricky
- No significant findings?
- “So what?”
In this unit we explore planning your study and related topics. A chunk of the unit is spent on the important issue of identifying the places to look for a good question, and then breaking the question down into its operational or measurable parts; operationally. By the end of the unit you should be able to formulate your research question, and have started identifying the evidence you will need to gather in order to answer it.
How do you come up with a thesis topic and research question?
Clearly it needs to be a topic that you are interested in, even passionate about exploring because it will need to sustain you through a long journey.
There are several ways to look for your topic:
- Through a grant-supported project that already has a research history. Check the research projects in your department and the profs who are in charge of them. This is frequently a good route to go, since there are defined topics that need to be addressed, and a well developed research team will give you support. The senior researcher will be on hand to assist you as well as allocate a piece of the data to you, and may well become your primary supervisor.
- One of my favorite approaches is to look for a recent review article in the field I want to study. Sources such as “The International Handbooks on…” or “Annual Review of” often contain thorough reviews of the history and current status of a topic, and they usually point out the research questions that need to be addressed. There are several collections of reviews, each a compendium of research in a field or topic. Check with your librarian for a review of a topic you want to investigate.
- During your coursework you will have been exposed to controversies in your field. There will be rival arguments to support opposing claims. This is a good source for a thesis study; presenting evidence that addresses the controversy and adds data to the claims. [example as a blog piece? For example, my own thesis was written at a time when there was a major political controversy about whether the IQ of black Americans was on average lower than that of white Americans. One possible ‘intervening variable’ (spanner in the works) was that economic status, an environmental or nurture variable which correlated with race in the U.S., was possibly responsible for any differences in IQ. My test was to take two groups of Canadian children, both Caucasian, one came from a low-economic area and the other from a high-economic area of a Canadian city. I replicated the black-white study and found the same results! Conclusion – IQ is not solely race-related if at all. Did I have a predisposition to support one side of the argument at the outset of the thesis? Indeed I did, but I could only express it at the end of the thesis when I had presented my evidence to that elusive reader (see unit 1).
- Look for an unfinished or outstanding argument. Most writers of academic scholarly papers in the literature point out the shortcomings of their studies. It’s one of the rules of academic publishing. Or you may read a paper and see a glaring omission that may affect how you interpret the findings. Those shortcomings are fertile ground for a thesis topic if you can figure out how to address them in your own study.
- Rival arguments in practice lead to important research studies. The practices in a field may be divided by different views about which treatment (intervention, instructional direction, etc.) is the best to address a practical problem. For example, “are children placed in care in a different jurisdiction less or more vulnerable than those left with oversight in their original community?” In the second part of this unit, we show how to break down a question like this to create a viable study: children – what age? Background?; in care – foster home or institution or hotel?; different jurisdiction – how far from first family? Linguistically and cultural different?; original community – type and resources?; oversight – type and who?; vulnerable – multiple possible measures? Etc.
Such studies are usually evaluation-based. That is, they compare groups, (children in care in their own or distant communities), sometimes over time and with or without a treatment (with or without oversight? Culturally and linguistically similar or different?) and derive their data from an outcome measure (less or more vulnerable as defined or measured by length of care needed, self report, police report, by school grades) or by something else. Evaluation studies are often selected by thesis candidates for their thesis because they are important both for societal, theoretical and policy reasons. There are however some requirements for evaluation studies that will be addressed later in this unit.
I’ve heard many times “I want to do a qualitative research study because it’s easier than a quantitative one with stats and math”
Not so! It’s not that one is easy and the other not – they are different! They address different questions. They cannot be compared as methods because their intent is to provide vastly different forms of data, but they both require that your thesis question yields evidence that the reader can judge to be true.
Quantitative studies rest on exploring a wide sample, representing a population, on one or a few variables (factors that may cause differences to emerge among the sample participants). The studies ask a question that is answered by being a wide and shallow sampling of a group of cases (people or instances). They are the basis of epidemiological studies of the effects of drug treatments and medical procedures, etc. They addresses questions like “Is group A different from group B on characteristics X,Y and Z?” “Did group A benefit more than Group B from the training programs that we gave them?” Hence quantitative studies are important for evaluating treatments (training programs, interventions etc.). They require that samples be of sufficient size – the rule of thumb is a minimum 30 per group for a single comparison, more with an increase in the number of comparisons to be made.
Qualitative studies examine fewer cases in greater depth and with acknowledgement of the complex characteristics that surround each case. They may be studies of a single case, documented over time or phases. They may be a group that is investigated in depth to identify its characteristics. As such, qualitative studies don’t usually lend themselves to evaluating the effects of a training procedure, intervention or treatment. So if you want to dig into whether a program, a treatment, or a type of intervention works (i.e. an evaluation of the effects of some treatment), qualitative study is likely to be a weak tool to address your question. It will yield anecdotal data, which may be what you want, but it won’t be generalizable beyond the people or cases you have studied; that is, applicable as a way to do business with other cases.
It will give you a deep and rich profile of the events, people or cases you are exploring. If this is the question you want to address, this is your research approach.
By the way, if length is a criterion for you, quantitative theses do require that you get to understand your data as statistics. But generally such theses are shorter and more concise than qualitative studies which often require considerable amounts of excerpted transcripts as data.
Try this technique with your own research question. Write down your question along a right hand column of a large sheet of paper, then begin to map out how each phrase can be operationalized; that is, how would you recognize each of the terms in your question if you were observing it? This is a strategy to move toward identifying the variable characteristics of each of the main elements of your research question as part of your method of enquiry. Note that it isn’t yet finished – you may still need a third column in which you define the participants/cases/events of your study and add the measures/criteria that will answer the question in ways that the reader can verify.
See Unit 5 for an example of how to do this in order to develop your question further.
Often a thesis candidate sees this work as their ‘magnum opus’ – their major production, and as a result they try to include every possible spin. They want to address multiple questions rather than limiting their subject pool, controlling their data set and eliminating some of the variables. In short they want to investigate the sun, the moon and the stars.
This is not a good idea because the study will already be sufficiently complex by limiting the investigation to a few cases, variables or sample characteristics. As we see in the next unit, research questions grow. But in any case, you may want to see yourself as a researcher with long term career prospects; one who builds his or her field of research in increments, addressing each new question as it emerges from the last. So make the thesis address the first question in a potential series; one that you will be comfortable investigating and defending. One star now, the universe later.
A word about evaluation studies. I often run into thesis proposals that are evaluations of a treatment. They take the form of :
“If I do A and B (the treatment), will the learners, recipients (subjects) benefit?
Or “What is the outcome of this intervention (e.g. instructional approach) or treatment (e.g. added resources) on how XYZ (the outcome or dependent measure) happens? In these types of studies there are two time points; before, when a baseline is established for the dependent measure, and after, to see if change has occurred. In between the two time points a treatment or intervention is administered . There may be a comparison or control group that is given the same measures of the dependent measure at both the before and after points, but is not given the intervention. Sometimes authors might not recognize an evaluation study for what it is. They may therefore select a qualitative approach or a case study as their method, in the expectation that any changes in their subject(s) can be attributed to the treatment. Without a comparison or control group though, no conclusions are dependable.
If you have a promising technique for doing something, and you believe it has an impact on a group of participants, you are inevitably going to need to design an evaluation study. In order to demonstrate that your approach (treatment) has a significant impact on its recipients you may require a comparison or control group to show that, without it, the same gains are not made, or at least are not significant.
This brings in the inevitable “Hawthorne effect”, in which individuals modify an aspect of their behaviour in response to their awareness of being observed. People will show changes in behaviour not directly as a result of the treatment but because they are getting attention, or because their behaviour is being shaped to please the investigator.
Do not be discouraged however if you want to undertake an evaluation study. There are several ways to counter the effects, but know that you will be moving toward a possibly quantitative study involving pre- and post-testing of a treatment, and that it may require a time study in which you compare the progress of two groups or more over periods of time when they are both receiving and not receiving the treatment.
This is one of the scariest prospects for a thesis candidate. After all the work, your data fails to show the differences you anticipated. What if the results don’t work out, as in the case of one student whose subject for the thesis case study was a teacher designated as exemplary. The teacher proved to be less than exemplary however when the student began in-depth observation and recording.
This can indeed be disheartening. The best strategy is to revisit the premise of the thesis and ask what came out of the study that is surprising – and how can that become the core of the question.
This is a big problem at the final defense! It has to do with findings that are trivial and that don’t add to the overall knowledge base in the field of study. After a long and detailed exposition of a candidate’s thesis at the oral defense, a favorite question from an examiner is “So what?” The question asks the candidate to locate their most significant findings in the literature and in the wider field of application or practice so that the findings are not deemed to be self-evident. Your objective is to be able to tell the examiner that this is not a trivial finding but that it adds an increment of knowledge to the field.
Here are two example that I have encountered of the ‘so what?’ question:
Example 1: The candidate has selected a well-developed, standardized test to explore the characteristics of a large sample of subjects. Let’s say it’s an aptitude test given to children. Before the test is administered a great deal of organization, ethical reviews and permissions are collected. The test itself is then administered under rigorous conditions. The results show that the children’s scores vary on the test, representing a normal distribution of scores, with the largest number scoring the median mark. The researcher then draws conclusions about the variation in the students, and how they are comparable in performance to the students on which the test was normed.
So what? The findings really say little about the students who took the test, because they are reflecting how well the test was originally constructed. The candidate’s test administration, despite its cost in time and effort, provides little more than a test-retest reliability statistic of the original test’s construction. The children are performing as they would be expected to perform. What more might the research need from this study in order to add an increment of knowledge to the findings?
Example 2: The candidate administered parallel questionnaires to three groups; teachers, parents and students. The thesis examined the main themes in each group’s responses and pointed out the similarities among the three groups. Even though there were ostensible differences in the patterns of responses of each group, the candidate did not address the differences between groups in the results. After hearing the separately reported results of each group, the examiner asked “So what? You have clear differences between your groups, but to what do you attribute them?” The similarities in the positions taken by each group were predictable in the context of the study. They were the centre of the results but the examiner did not think they were contributing anything that wasn’t self-evident – hence the “so what?” question. The examiner was pressing for the candidate to recognize the need for another step that the candidate had not taken; to examine why the prevailing differences occurred between the groups, perhaps by interviewing or otherwise following up with individuals in each group who took opposing positions. This is where a mixed methods study shows the benefits of combining the two approaches; quantitatively-based broad description of a population followed by in depth qualitative case studies or interviews to shed light on the population trends.
Step back from your findings, and ask what they mean in the larger context. It may lead to a second analysis or a sub study, and could be crucial in supporting the conclusions and inferences you draw in your discussion of the findings.
Keep the “So what” question in mind when you formulate your research question. It’s better to have a question that is going to yield interesting results no matter which way the evidence falls than to anticipate the outcome and fail to find it.