Category Archives: Methodology

Can teachers really be researchers?

As the debate over evidence-based teaching continues there appears to be two separate strands emerging:

Strand 1: Teaching should be evidence-based (or evidence-informed).

I certainly have no issue with this, although the view of ‘what works’ is perhaps a secondary debate.

Strand 2: Teachers should also be researchers.

At a superficial level this appears like a pretty good idea – imagine the amount of evidence teachers could gather if they were all carrying out their own research studies within their own schools?

To be honest, it’s more than likely that strand 2 would lead to complete chaos. Let’s face it; there is enough bad educational research out there already, without research naïve teachers adding to it.

I view educational research through the lens of a psychologist and hold a very similar view to other psychologist-teachers (e.g. @turnfordblog) that (in terms of science) if psychology is in the Dark Ages then Education is in the Stone Age when it comes to research. Thus, I tend to make references to psychology when discussing education and, as a result, I take a positivist view of the research process.

So what’s the problem?

This list is by no means exhaustive, but it does represent at least some of the issues that need to be discussed before strand 2 can be fully recognised.

1. Sample Size:

Small sample sizes are common is school-based educational research. The majority of schools in their entirety don’t have enough participants to ensure an acceptable sample size, so carrying out a study using a few classes can only give us some small indication of the effect of any independent variable. The other issue is that we can’t force our pupils to take part in a study and even when they do volunteer, they must still be given the right to withdraw themselves (or the data) from the study.

2. Replication:

The replication debate is huge in psychological research at the moment. It has been found that many of our long-standing assumptions about human behaviour are based on studies that simply cannot be replicated. Replication studies are rarely published, null hypothesis studies end up in a dusty cupboard somewhere or in a folder marked ‘failed studies’. One study doesn’t make a theory so implementing interventions based on the results of single teacher conducting a study in a single school with a small sample tells us very little about anything.

3. Generalizabilty:

Does a study conducted in a middle-class school with a low number of students receiving free school meals, below average number of ethnic minority pupils and low number of special need pupils tell us anything about those pupils in deprived, inner-city or ethically-mixed schools? If research is to be useful then it needs to inform us about learning, not just about learning is a particular school (although this data can be useful at a more local level).

4. Bias:

Accept it or not, we all want to be ‘right’. Bias is a major problem in psychology and there is no reason to believe it won’t also be an issue with the teacher who is trying to support a hypothesis (or perhaps prove a point). Bias is usually unconscious but can often also be deliberate.

5. Lack of research training:

I’m a teacher and a Chartered Psychologist. I’m also conducting research as part of a part-time PhD at the University of York. Even though I have a psychology degree, a Masters in Education, and have attended more research methods workshops and seminars than I care to recall, the process of research still often baffles me. Those teachers whose degrees have not included a substantial amount of science or social science research training are far from equipped to carry out serious research.

6. Analysing results – the use and abuse of statistics:

How would a teacher know if the results they have obtained are significant? Most undergraduates are baffled by statistics, as are many post-graduates and post-docs. A non-British academic recently told me that us Brits don’t do stats very well, mainly because the support isn’t there at undergraduate and post-graduate level. Statistical analysis is confusing and often very time consuming and even the best statistical software packages can seem like a long-forgotten language. What about p values? Should we trust effect sizes? Is it alright for us to shave off some of our outliers to get an acceptable level of significance (it’s not by the way – it’s called p-hacking and it very much a no no)? What about Type 1 and Type 2 errors?

What can be done?

I can only offer a few suggestions:

1. Don’t go it alone.

Partner up with other schools to get bigger sample sizes. Partner up with universities or research centres that can give advise on how to carry out research or involve you in some of their own research.

2. Replicate replicate, replicate.

Get other teachers in other schools to carry out the same study.

3. Publish/Blog results (even null hypotheses) – and accept advice/criticism.

Make others aware of what you are doing and take on board the advice offered. Let’s face it; sometime it’s hard to get past the egos that dominate the Internet. We all have something to say and we don’t like being criticised for it but if we’re serious about using research to inform our teaching we really need to get over it.

4. Get some training.

CPD is a major issue and most isn’t worth the time and effort (or money) involved. Introductory workshops in research methods are often cheap (and sometimes free) and there are plenty of resources available online (try OpenLearn from the Open University). Linking such CPD to a recognised research qualification would be a great incentive.

It’s perhaps time to move away from the debate about the acceptability of teacher-researchers and try to work out how it can be practically done. Even though there is a great opportunity here for educational research, there is an equally realistic possibility that it could end in disaster and confusion.

In which I ponder my life as part-time researcher.

This year I am celebrating ten years as a teacher. To be honest, I never thought I would make it this far and I have lost count of the amount of times I have considered packing it all in. I recall the exasperated looks on the faces of family andBusinessman juggling friends when I informed them that I was going to do a PGCE, “Really!” they exclaimed, “Are you sure?”

I was met with somewhat the same response when, this time last year, I decided to embark on a part-time, self-funded PhD. Some, I believe, thought me insane or at the very least suffering from some sort of psychotic break – but this was something I felt compelled to do and I’m glad that I took the deep and dangerous plunge.

So, here I am, almost at the end of my first year of what could turn out to be a six year obsession. Juggling teaching with research is a tricky task and I often feel that I would be better off doing one or the other rather than both simultaneously. Nevertheless, exposure to the research side of education is beginning to inform my teaching more and more and I am more than ever aware of the role such research could play in teaching generally. I also know a little bit more about the lives of my students because of my chosen area of research. Through tentative exploratory investigations I have gleaned information about how they see themselves as learners, how they feel about their own resilience and what they believe about intelligence (and the possible relationships between these factors). What I have certainly realised is that I want to understand more about their daily struggles; about why some of my sixth formers engage deeply with their studies while others don’t appear to be bothered about anything related to school. I’m also starting to understand a little bit more about the interaction between school and student, the complex connections between school culture and student attitudes.

I’ve also started to look at research methods in a new light and am currently investigating some exiting methods for studying students’ daily lives such experience sampling and regular ‘snippets’ of study-related conversation. I intend to blog about some of these soon.

On a more practical level, every day is hard. As a part-time student I have to organise my time carefully. I have supervision meetings, research group seminars and other events to carefully slot into my limited time. I have managed to attend only two events this year – the Institute for Effective Education conference at the University of York (for which I was granted time off from school) and the NTEN ResearchED event which was held on a Saturday. Both were excellent events – far better than any CPD course I’ve ever attended (and whole lot cheaper – an added bonus seeing as I have to foot the bill myself). I will, however, miss the British Psychological Society conference this year as well as the ResearchED conference in September but this all part of the process whereby I need to select what is necessary and what is feasible. More stressing, perhaps, is the ever-present possibility that I will run out of money before the completion of my research. There are few funding opportunities available for part-time postgraduate students and those that are available remain highly competitive – this is partly why the drop-out rate for part-timers is so high. There is also a kind of self-imposed pressure to publish during the course of the PhD and many would argue that publication is more important than the final thesis. I’m desperate to publish but there is a need to prioritise and my teaching always takes precedence.

Would I advise other teachers to embark on a PhD?
It’s a difficult question one that depends upon individual circumstances. A good supervisor is perhaps the key and I’m lucky in that respect but there are more pressing concerns related to time and finances. At some point I will need to approach other schools in order to obtain a more representative sample and this will eat further into my limited time. Family life is also an important factor to bear in mind and dividing time between study, work and family can become quite a juggling act. If you think you can do it then it can be very rewarding but remember that you don’t need to do a PhD to do research.

Teachers are ideally placed to engage in all kinds of research and there are plenty of ways to learn the basics of research design and implementation. The rewards are unlikely to be financial – a PhD just makes you an over-qualified teacher so it’s worth thinking about why you want to do it in the first place. You might be looking for a career change but for most, I suspect, it’s more to do with personal development. The financial commitment is perhaps the biggest concern (most PhD’s will set you back around £2000 a year and this could be for up to six years) and there is still that nagging feeling that self-funded PhD’s are somehow less worthy.

A PhD is challenging at the best of times but for teachers (and anyone who has to balance work with study) I suspect it’s more so. The benefits can only outweigh the costs however, both in terms of personal fulfillment and the opportunity to add to the research literature.

True and false might not be as simple as you think.

[Note: This post was partly inspired by a talk given at the NTEN ResearchED event held at Huntington School, York on 3rd May, 2014. Also note that I am not a statistician – which might appear obvious!]

Many of us in the educational community are at last coming around to the realisation that research does have something to offer. We read it all the time on social media and have all witnessed discussions that seem to go on for days about whichweb-UCScalendar-2012-ScientificTruthPauls2011 research is best or how to be critical about things we’ve read. There is an assumption that if research supports our particular view then we have permission to take the high ground and shoot down all those who refuse to accept the ‘evidence’. Of course, we need evidence (that’s the whole point) but we also need to be critical of it.

A great deal of educational research is positivist; like psychology, educational research often assumes that outcomes can be measured using scientific principles and anyone who is familiar with academic papers in psychology will have noticed that there is an awful lot of numbers and bizarre equations involved. The scientific method is one of hypothesis testing – to paraphrase Richard Feynman, the first thing you do is make a guess and then you test the guess by conducting an experiment. If your experiment doesn’t support your guess then your guess is wrong.

In psychology (and many social sciences) what we are looking for is a statistical significance, the nuts and bolts of which are dependent upon statistical tests. The main criteria we use in order to establish significance is something called a p value (or probability value). Psychologists often set the p value as 5% and represent this using the statement p≤0.05.

What does p≤0.05 actually mean?

This is actually quite straightforward, despite the looks on by students’ faces when presented with it: “Maths! In psychology… nooooo.”

All it means is that there is a 5% (or less) chance that the results were due to something other than the manipulation of the independent variable (i.e. something the researcher was unable to control for). The p value is fairly arbitrary but there is a general consensus that 0.05 is a good place to start. We could set it higher but this might mean that we accept our hypothesis based on a false positive (a Type 1 error), or we could set it lower – but then we face the possibility that we reject our hypothesis and accept our null hypothesis when, in fact, the difference was significant (a Type 2 error). So, sometimes that which is true is actually false and that which is false is actually true (cue Robin Thicke).

P values are a hot topic at the moment with many suggesting that effect size might be a better measure to use (there are problems here as well). Nevertheless, while p values remain so influential, we need to be mindful that errors do occur. More worrying, perhaps (if less common) is the phenomenon of p-hacking. P-hacking involves removing the data which prevents the p value from being reached, thus manipulating the data in order to create a positive result. So a researcher might remove all the outliers, sometimes under the ruse that there was something ‘wrong’ with these results. P-hacking (and other such dubious practices) are often uncovered due to the inability to replicate the results – so be wary of single studies (especially if they are a few years old) with no recent studies to support them.

So, to claim that true and false (or right and wrong) are absolute in research is perhaps to misunderstand the workings of the scientific method as it applies to real people. Other factors such as bias, demand characterises and individual differences can blur the lines even further. This is perhaps the reason for the oft-used line ‘research suggests’ because there is always the probability (however small) that the results aren’t as statistically significant as we thought.

Boredom – now that’s interesting.

boredDid you know that there are at least 5 types of boredom? To be honest, neither did I until quite recently.

Well, there are (they are outlined in the table below), but it’s the fifth type (apathetic boredom), which seems to be the most interesting and the one that could have implications for teaching and learning. In a paper published last year (in the journal Motivation and Emotion) Thomas Goetz from the University of Konstanz, Germany and colleagues from Germany, Canada and the US, reported on series of studies utilising experience sampling techniques in order to establish the reasons why people (in this case university and high school students) get bored and, more importantly, if previous research that identified four types of boredom stand up to further experimental scrutiny (Goetz et al., 2013)

BoredomTypes

Experience sampling, as a research methodology, has quite a long history but has really come into its own since the advent of the mobile communications device (that’s mobile phones to you and me). The method is mainly associated with Reed Larson and Mihaly Csikszentmihalyi and originally involved participants providing self-reports by means of a pager or alarm watch. This allowed researchers to gather specific information at any moment in time. Times have moved on and now bespoke software and PDA’s or phones have replaced pagers and alarms even though the principle behind the method remains the same. The experience sampling method (or ESM) allows the researcher to obtain a number of snapshots pertaining to emotions, obstacles or other factors that impact on our day-to-day lives.

Goetz and his team supplied each participant with Personal Digital Assistant (PDA) loaded with specially designed software. The PDA’s would then emit a number of audible sounds throughout the day and participants would complete a questionnaire that appeared on the screen (the procedure was slightly different between the two group – university or high school students). The questionnaires required likert-responses to identify levels of boredom, wellbeing, satisfaction, enjoyment, anger and anxiety. If they identified themselves as being bored, they were asked a second set of questions on arousal and valence (the extent to which they were attracted or repelled by the task).

Results suggested the existence of a fifth type of boredom – apathetic boredom, which appeared widely prevalent amongst both groups of students. The interesting point here is that the team identified apathetic boredom as possessing characterises related to learned-helplessness (a condition associated with depression), making apathetic boredom a very unpleasant experience indeed. The implications for teaching and learning are as yet unknown but might suggest there is a learning process involved in certain types of boredom. On the other hand, there might also be some speculation involved in the findings and the ‘types of boredom’ might simply be the result of the statistical analysis rather than anything ‘real’ (so-called ‘reification’). Nevertheless, the mere suggestion should ignite further research, certainly in terms of pupil wellbeing and factors such as day-to-day resilience (academic buoyancy).

On the methodological side, the use of ESM in educational settings could provide a very rich source of data for understanding the complex daily lives of learners as well as an antidote to the current, often misplaced, attraction of Randomised Controlled Trials (RCT’s).

References:
Goetz, T., Frenzel, A. C., Hall, N. C., Nett, U. E., Pekrun, R., & Lipnevich, A. a. (2013). Types of boredom: An experience sampling approach. Motivation and Emotion. doi:10.1007/s11031-013-9385-y

I want to tell you a story…

jackanoryI’m rapidly realising that research with young people is more fraught with problems than I originally surmised. Ironically, the very area of my research should have highlighted this possibility, in that young people will go to often extraordinary lengths to safe-guard their self esteem and provide an answer that they believe the researcher will judge at ‘right’. Unfortunately, as researchers, we are rarely interested in the right answer – usually because there isn’t one.

As a result, it can become difficult for us to confirm that any answer provided by young people is honestly given as opposed to an attempt to ‘get it right’. Furthermore, understanding the reasoning behind their answer could actually provide a more accurate indication of factors such as self-esteem and resilience than the very measure itself. This calls into question the reliability of measuring, say, academic resilience using psychometric methods such as scales.

What if we could make these questions more objective and still use the answers as a means to providing subjective measures?

One area I have decided to explore is the use of scenarios (or vignettes). The basic premise is fairly simple: Rather than presenting participants with questionnaires that prompt a subjective response  using a likert-type scale like this:

abs1

 

 

…the questions are presented once the participant has read a fictional vignette.

For example:

Frank is studying A-level maths and his teacher has just given him the results of his latest test. Frank’s target grade is B but the grade he has achieved for the test is a D.

On a scale of 1 to 10, decide how this bad result would have affected Frank’s confidence in maths (where 1 is ‘a great deal’ and 10 is ‘not much at all’).

Vignette methodology is widely used in the social sciences, including sociology and education. Vignettes provide a way for the participant to distance himself or herself from the question by ensuring that on the surface the questions aren’t necessarily about them. The assumption from the researchers point of view, however, is that the participant will provide an answer that is more in keeping with their views but limits the desire to safe-guard self-esteem.