1. How do I write a good question? – a strategy
a. Operationalizing the question
b. Topic creation
2. Convincing your reader : Validity, veracity, reliability and confidence
3. What if the results don’t work out?
4. The “so-what?” question
You have your topic, and a good idea of what you want to investigate. You’ve become conversant with the literature in your field of interest (Unit 4), and you’ve been able to situate your study in this field. You are able to make an argument that your study will add an increment of knowledge to this field. You’ve figured out that your question will require a quantitative or a qualitative method of investigation .
How do you generate a viable thesis question from the material you have?
There are a couple of ways to go about formulating a researchable question.
A much loved mentor and professor of mine during my student days used the following technique:
Write the possible research question down the left-hand side of a page. You can leave articles with nouns, but every noun, verb and adjective needs its own row:
Figure 5-1 Example: Breaking down the question
placed in care
in a different jurisdiction
less or more vulnerable
than those left
in their original community?
Using the question we asked above, we generated this format in Figure 5-1
Now write beside each row what is important about each word or phrase, from the perspective of your reader who is seeking to be convinced of your findings:
Figure 5-2 Defining the elements
Are children those in care of the Province; aged between birth and 18 years
placed in care already designated as being a ward of a Children’s Aid Society or similar. What different mechanisms are there for placing a child in care and will these result in different outcomes?
in a different jurisdiction how far from the first family; Largely inaccessible by first family members? Or First Family not available to visit? Or a measure – more vs. less available for visits from birth parents?
less or more vulnerable a central measure or documentation of the outcome.
This is a major part of convincing the reader that factors leading up to this outcome is the result of differences in the variables to be investigated – school completion, grades, record of having been abused, criminal record? A long-term measure of financial and social independence in society after age 21 …or? A collection of measures to yield a rating scale of vulnerability/resilience?
Or a case study approach with the stories of one or more teens or adults who went through the child care system as children? Selection criteria? Purpose of selection?
than those indicates a comparison group or cases.
left with oversight ‘left’ – as in ‘contact with first family’? Or within distance for visits? What is oversight? Case study documentation? Seek differences in more than a single story?
in their original community as Southern or Northern Canada? Culturally and linguistically similar to or different from the ‘different jurisdiction’?
Notes – what are the critical variables? – contact opportunities for first families? Cultural contrasts? Linguistic contrasts?
Are there complicating factors – medical, age, siblings? Do these need to be added?
As you work through the example in Fig. 5-2 you see how complicated a study can become. But this tells you exactly what you need to do to trim your study to fit your resources. For example, if you have access to specific records or if you are wanting to conduct an in-depth profile of these children, you will want to tailor the research question accordingly:
Figure 5-3 Defining the participants
“The study addresses children between the ages of 5 and 16 who were living in Northern Ontario communities, who have been placed in the care of a Provincially-recognized child care service in Ontario, and are now living in a different jurisdiction and cannot communicate with their first family. The research question asks whether these children are less or more vulnerable (to xyz outcomes) than those left with oversight in their original community.”
It’s getting better – but now there are clearly some emerging variables yet to be defined (operationalized):
Figure 5-4 Operationalizing the variables
children between the ages of 5 and 16,
who were living in Northern Ontario communities,
and who have been placed in the care of a Provincially-recognized child care service in Ontario
and are now living in a different jurisdiction
and/or? cannot communicate with their first family,
less or more vulnerable,
compared to those left with oversight
in their original community as indicated by their secondary school graduation status, and post secondary financial independence?
This is how you develop your proposal for a thesis – breaking down the question until you can see what the elements are that you will need to define, measure and present to answer your question. You are ‘operationalizing the variables’ because you are moving toward an operational definition of each element or variable in your study.
Of critical importance is choosing your outcome variable; the measure that you will offer to your reader as a gauge of the impact of your study.
The question raised in the example is a complex one. It would lend itself to either a quantitative study or a qualitative study. In the case of a quantitative or measures-based study, you would gather data from a group of children in care who had been sent away from their communities, and compare these data with those of a group who had stayed. There would be large differences in the types of care (the dependent variable) that each group received that would need to be addressed in the thesis. The data you would be collecting are the independent or outcome measures that make explicit in measurement terms what you mean by ‘vulnerable’.
The outcome measures would need to be appropriate for the larger group from which these children are drawn.
Long-term, is graduation from school a realistic and valid measure of success for these children? How long do you have to collect long-term outcome data?
Or do you need an immediate outcome measure that will convince your reader of the source of any differences found? Would the children’s opinions or self-reports of their adjustment to their current home life fill this bill? Or would the opinions of their care workers, foster parents etc., or both, meet this objective?
Alternatively using a qualitative, case-based approach, you could select a few cases from each group and document in depth what had happened to these children after being placed in care. This would be a very different thesis from a cross-sectional quantitative study, but it could handle the complexities of the profiles and histories of the children’s cases much better than a quantitative approach with a broad set of measures. You would need to consider to what extent, however, the conclusions would be generalizable to other children. Depending how you select the cases, the outcomes will likely not be representative of other children in similar situations. The stories told by the children and their care workers and foster parents would be rich and possibly unique to each case. Indeed does generalizability even matter in this case-based study, or is your intention to document the complexity of the factors that interplay in each child’s story?
The question will need further development. In this example, it will become better defined and more accurately measured as you uncover the circumstances of the Child Care system and their policies for taking children into care (parent- or court-referral, loss of a parent?) and other factors, such as what the community, the schools and the child care agency itself would regard as a positive outcome for its children.
The second way is attributed to John McGarvey at http://johnmcgarvey.com/apworld/student/thesiscreator.html
The author breaks down the thesis statement into elements:
- The topic of your study
- Your position – what you believe to be true
- Qualifications to your position; exceptions, or extenuating circumstances, Is yours an absolute or qualified position?
- Your reasons for taking your position
Keep in mind that, while this is exactly what you will need to defend your thesis at the end, when you start developing your thesis topic you are aiming to operationalize your question in a way that you hope will produce the evidence that will lead the reader to agree with your position about it. If it doesn’t work out, you may need to change or at the least qualify your position.
So here is THE MAIN challenge in writing a thesis:
Will your research question, the methods and measures you select to answer it, and the conclusions you draw, lead to evidence that will be convincing to your reader?
In other words, will your reader be able to follow your arguments and agree with you in interpreting what you found?
What will it take to convince your reader that your findings –
a) answer the research question and answer it in a way that the reader can believe to be true (that is, the evidence presented to answer the question is valid and reliable)?
b) take account of the limitations that your study has, so that nonetheless the results can be believed.
Note that every study has its limitations in trying to address the research question. Being aware of and acknowledging these is one way you will defend what you find, by limiting your conclusions to what they represent. More about limitations in Unit 6.
These questions concern the validity, veracity and reliability of your evidence or data for claiming that your study findings can be trusted
In quantitative research, there is a statistical mechanism for convincing your reader that the results can be trusted. It is called confidence estimates or limits, and it tells the reader what is the likelihood that the findings being presented simply occurred at random, or by chance? When I hear the weather forecast and are told that ‘today there is a 30% chance of rain” I smile. The chance of rain is a probability estimate. It does not mean that for 30% of your day you will need your umbrella, but that there is a 30% or a 3 in 10 chance that it is going to rain at all. That is, the odds are 7 (out of 10 ) to 3 (out of 10) in favour of no rain. But 30% is enough of a chance to take my umbrella when I head out.
Similarly, statistical analysis gives the reader the odds of the finding being trustworthy or believable. By convention, the odds are usually set at p<.01, (the probability is 1 in 100 that the findings occurred by chance and therefore are wrong), or p<.05 (the probability is 5 in 100 that the findings occurred by chance). Those are the kinds of confidence estimates you are willing to accept since the probability of them being wrong is exceedingly small. If the probability is 5 in 100 chances that it will rain today, I am leaving that umbrella behind.
In quantitative theses, you will try to convince your reader that the evidence you present as part of your results could only have occurred by some quirk of fate or chance or be wrong either once or no more than 5 times in every hundred times. Therefore they can be confident that the finding reflects reality!
In a quantitative thesis you will want to demonstrate that your measures are up to the job of measuring what you say they are supposed to be measuring. To convince the reader, again, you present the statistical confidence estimates that the measures are up to that job:
Validity is an estimate of the extent to which your findings and /or measures are truly reflecting what you intended them to measure. They reflect confidence that the reader can place in the findings or the measures that they truly reflecting a phenomenon out there in the world.
Reliability is similar except that it tells the reader what the probability is of the composition of the measure or findings being random or the result of chance. That is, they reflect the confidence that the reader can place in the stability and composition of the measure. They reflect the estimate of confidence that the internal characteristics of the measure are stable and can be believed.
Figure 5 -5
Validity and reliability (TBA)
In qualitative research a similar requirement is demanded by the reader. The reader wants to be sure that the findings you report about your sample of people or events, while not necessarily statistical and generalizable to a broad population, are valid and can be believed. We talk about the veracity of the findings.
Qualitative research offers a different look at a complex phenomenon from a quantitative approach. It is primarily exploratory, in that it explores in depth the multiple, interacting factors or variables that contribute to a phenomenon, usually one of human behavior. Qualitative research is also effective in investigating intangible factors, such as social norms, socioeconomic status, gender and ethnicity, that contribute characteristics that may not be readily apparent at first glance.
Qualitative research techniques include in-depth interviewing, participant observations, third-part analyses and transcript verification, to answer questions about the rich and complex contributors to human behavior and events. To convince the reader of the veracity of the findings, researchers use analytical techniques that might include verification of transcripts by the interviewee, 3rd party verification of thematic analysis, demonstrating stability of findings over time and over other populations.
However, although the results of qualitative research can often be extended to people who share characteristics with the research participants, the usual aim of the researcher is to gain a rich and deep understanding of the specific social context and the phenomena rather than to generalize the data to other contexts, people or places.
Mixed methods. Sometimes, a mix of both quantitative and qualitative research techniques give an even more convincing presentation of the phenomena under study. A mixed methods study would be carried out in two stages; the first stage charting a cross section of an event or behaviour, and the second stage using in depth interviews or observations of some of the cases to fill out the details that contribute to the broader trends.
As an example, you analyse the results of a questionnaire that was completed by a number of people. You find that two distinct perspectives are emerging from the data. You select two or more of the respondents who seem to represent the differing perspectives and you interview them about their responses to the questionnaire.
This gives you both a broad picture of the sample of people who responded and some insight into the depth and complexity of why they responded in apparently disparate ways. Presenting both to your reader clearly improves the veracity and believability of your findings.
Every candidate is concerned that the results may not work out as planned. But don’t worry! Findings that change, qualify or even contradict the ‘thesis’ that you held going into the process can be worthy of explaining and defending. Indeed, they may add to the knowledge base.
You always have the option of rethinking your original research questions, and adapting the front end of your thesis to lead to your actual findings.
You also have the option of re-examining the theoretical or empirical base that led you into the study, and then supplying a critical or enhanced new version in your discussion.
Here is an example:
Certain authors hold a strong and definite perspective on certain hypothesized societal injustices. These include views about the hegemonic repression by the social majority of minorities including minority races, the LBGTQ communities, indigenous people, and women, etc.. Although thoroughly documented, persuasive and convincing, these writings are theories or opinions, not facts. So thesis writers who take them at face value as a starting point for their research may find that the evidence they expected did not emerge in their particular study. This is not to say that the theories are wrong, but that the design of the study failed to find the expected trends. The thesis writer made the mistake of presenting the literature as factual rather than hypothesis or opinion. So when the evidence they collect tells a different story, they are not prepared to accommodate it.
They have a few options. They can decide that their research method was inadequate, in which case they have no thesis. Or they can delve into their data perhaps by collecting more evidence that allows them to qualify the claims made in the theories. Or they can return to the theories, looking for alternative explanations, perhaps arguing that the original source was overly encompassing and overstated and that the real world is more nuanced and complex than the theories imply.
Trivial findings: a common thesis downfall
In my experience this is a big problem! It has to do with findings that are trivial and that don’t add to the overall knowledge base in the field of study. After a long and detailed exposition of a candidate’s thesis at the oral defense, a favorite question from an examiner is “So what?” The question asks the candidate to locate their most significant findings in the literature and in the wider field of application or practice so that the findings are not deemed to be self-evident. Your objective is to be able to tell the examiner that this is not a trivial finding but that it adds an increment of knowledge to the field.
Here are two example that I have encountered of the ‘so what?’ question:
Example 1: The candidate has selected a well-developed, standardized test to explore the characteristics of a large sample of subjects. Let’s say it’s an aptitude test given to children. Before the test is administered a great deal of organization, ethical reviews and permissions are collected. The test itself is then administered under rigorous conditions. The results show that the children’s scores vary on the test, representing a normal distribution of scores, with the largest number scoring the median mark. The researcher then draws conclusions about the variation in the students, and how they are comparable in performance to the students on which the test was normed.
So what? The findings really say little about the students who took the test, because they are reflecting how well the test was originally constructed. This test administration, despite its cost in time and effort, provides little more than a test-retest reliability statistic of the original test’s construction. The children are performing as they would be expected to perform. What more might the research need from this study in order to add an increment of knowledge to the findings?
Example 2: The candidate administered parallel questionnaires to three groups; teachers, parents and students. The thesis examined the main themes in each group’s responses. Even though there were ostensible differences in the patterns of responses of each group, the candidate did not address the differences between groups in the results. After hearing the separately reported results of each group, the examiner asked “So what? You have clear differences between your groups, but to what do you attribute them?” The different positions taken by each group were predictable in the context of the study. They were the centre of the results but the examiner did not think they were contributing anything that wasn’t self-evident – hence the “so what?” question. The examiner was pressing for the candidate to recognize the need for another step that the candidate had not taken; to examine why the prevailing differences occurred between the groups, perhaps by interviewing or otherwise following up with individuals in each group who took opposing positions. This is where a mixed methods study shows the benefits of combining the two approaches; quantitatively-based broad description of a population followed by in depth qualitative case studies or interviews to shed light on the population trends.
Step back from your findings, and ask what they mean in the larger context. It may lead to a second analysis or a sub study, and could be crucial in supporting the conclusions and inferences you draw in your discussion of the findings.