Category Archives: Psychology

Psychology and Education: It’s Good To Be Critical.

Psychology is still a very young discipline and one fraught with competing theories, multiple paradigms and methodological problems. It’s also become very popular with teachers, especially with those who are eager to understand how psychology can be used in their own practice.

This interest has led to a flurry of Edu-bloggers and Tweeters taking up the mantle of advising others on how they might go about applying psychology to their classrooms. Often, these amateur psychologists use the knowledge they have gained through reading or attending conferences to engage others and further the knowledge of others.

What I have noticed, however, is that many are so taken by what they have learned that they become uncritical of the information they have acquired. They may be approaching psychology from a cognitive perspective or from any one of the other paradigms available, without applying any kind of critical gaze. When others read these blog posts or engage in online discussion, they often assimilate the information in an equally uncritical way.

If you study psychology formally (as an undergraduate of even an A level student) criticality becomes vital. All paradigms (from Freudian Psychodynamics through to Behavioural, Cognitive, Evolutionary and everything in between) are assessed for their validity and relevance. Indeed, this criticality and understanding of competing paradigms are necessary for a Psychology degree course to be accredited by the British Psychological Society – the first step on a long road to becoming a Chartered Psychologist.

Take, for example, cognitive psychology. We can learn a great deal from studies within this paradigm, especially as it applies to education (cognitive psychology is also the current dominant paradigm). Nevertheless, studies into memory and other cognitive processes are often conducted in highly controlled artificial environments with many of the participants drawn from the psychology undergraduate population. This means that such studies suffer from both low ecological validity (results can’t necessarily be applied to the real world) and sample bias. A 2010 study found that 96 percent of participants in psychology studies represented only 12 percent of the world’s population, while others have found that a large percentage of studies rely solely on student samples (a detail that is often omitted from the published paper). This also raises concerns over the use of non-naïve participants. It’s easier for more knowledgeable participants to second guess the purpose of the study and alter their behaviour according (generally referred to as demand characterises).

Many studies don’t travel well from the laboratory to the classroom. For example, Dweck’s work on Growth Mindsets fails to support her earlier findings when they are introduced into a real world environment. However, with such a widely known theory, it remains difficult to find any genuinely naïve participants, certainly within the teaching profession.

What about genes? Surely this is proper science that can’t be disputed? The more recent discipline of behavioural genetics certainly shows promise but also prompts controversy. Much of the controversy concerns the equally controversial construct of general intelligence, or G, (measured as IQ). Statistically, there is little doubt that IQ correlates highly with a large number of positive outcomes. Children of high IQ parents are more likely to have similar IQ’s to their parents (even when raised outside the family) and studies using identical twins support the genetic link, yet the concordance rates are never 100 percent, indeed, they range from about 40 percent up to about 80 percent in some studies. It’s difficult to argue with the statistics, however, those who wholeheartedly support the genetic basis of intelligence are often also those who are highly critical of correlational studies – it’s often their first point of criticism – correlation doesn’t imply causation. Indeed, G itself is a correlation, prompting some to argue that IQ is merely a reification, something that has been made concrete even though it should only exist in abstraction.

Whether we support the existence of a genetic basis of intelligence or not (and I must admit that I don’t doubt the findings), without applying a critical gaze we are in danger of omitting important aspects.

Genes do interact with the environment through what is known as genotype-environment correlation, the view that both the environment and our individual genotype interact in different ways. These correlations can be passive (for example, low achieving/low aspiring parents will pass both their genes and their attitude to education on to their children – providing them with a home life which is educationally uninspiring). Correlations can also be evocative whereby certain behaviours are promoted through the child’s genetic predispositions. Finally, such a correlation can be active in that the child actively seeks out opportunities and challenges based on their genetic predisposition (see Plomin and Asbury for a much deeper discussion on this). Such considerations are often conspicuously absent from blog posts, perhaps as a way of supporting a personal ideology, but most likely through a lack of criticality, understanding or simply not reading widely enough.

On a superficial level, even the amateur psychologist understands that there are multiple, often competing paradigms in psychology. While learning tends to be cognitive based, behaviour management is often behaviourist in nature, differing little from the early studies of Pavlov, Watson and Skinner, where teachers are urged to keep order through simple methods of reward and punishment. The potential problem here is that we ignore the ever present social processes taking place, the kids who will endure the sanction because of the belief that they will win points from within their peer group, or emotional aspects where students employ defensive tactics as a way of coping with anxiety or other psychological problems.

Curiously, in a bid to become more critical, many teachers appear to be becoming less so, accepting what they are told by self-styled educational gurus or accepting only the evidence that supports their own view. Unfortunately, Twitter is a poor platform for wider debate and few minds have been changed in 140 characters.

Advertisements

The Introvert in the Classroom.

In a world full of noise the quiet ones often get left behind, their voices drowned out by the cacophony of braver, bolder and more confident children. In a culture where extroversion appears so highly valued, it is all too easy to neglect the introverts, viewing them as merely shy and in need of encouragement in order to be coaxed out of their shell. Those teachers who see introversion as something to be fixed are in danger of stifling the talents of the less vocal pupils in their class. Introverts are often more creative and think more deeply and emotively than their extrovert classmates, teachers who are able to understand the nature of introversion are more equipped to help the quiet ones to achieve and to release their full potential.

Introversion is what in known as a personality trait. Generally speaking, a trait is a habitual pattern of thought, behaviour and emotion that remains relatively stable over time. This suggests that the behaviour you see in a toddler will be equally visible in later life, although research is beginning to find that certain traits can change with growing age and maturity. Brent Roberts and Wendy Delvecchio of the University of Tulsa found that the consistency of personality increases up to the age of about 30 then stabilises between 50 and 70 (Roberts & DelVecchio, 2000). Research conducted by Avshalom Caspi found similar results (Caspi, 2000).

Caspi followed a group of children from age three up to the age of twenty-one and found that three-year-olds who were viewed as uncontrollable grew up to be impulsive, unreliable and antisocial and that those children viewed as inhibited displayed higher levels of depression, were more likely to be unassertive and had fewer sources of social support later on in life.

However, a more recent longitudinal study found no correlation between personality measures taken at 14 and again at 77 (but see Christian Jarrett’s discussion of these findings here).

Nevertheless, psychologists who study personality are in general agreement that there are five specific personality traits, imaginatively known as the Big 5 (or the Five Factor Model). Introversion doesn’t appear as a specific trait but, rather, as the undesirable opposite of extroversion. See the table below.

Brain studies have identified small but significant differences between extroverts and introverts, particularly in those regions of the brain concerned with attention. Generally speaking, while extrovert’s brains are designed to focus outwards, introverts have a tendency to focus inwards, so the child in the classroom who appears to be daydreaming is more likely to be running things through in his or her mind; weighing up possible solutions and reaching creative conclusions. It’s not all positive, however; results from eye-tracking studies have found that shyness has a negative impact on word learning in very young children.

Schools can be stressful places for many children but introverted children appear to be more affected by busy, noisy classrooms than those children displaying more extroverted personality traits. For an introverted child, the classroom provokes anxiety and social encounters can prove exhausting. Introverts are not anti-social, despite often appearing as such; rather, they are over-vigilant, resulting in a tendency to approach new people, places and events with extreme caution. Meeting new children for the first time or being placed in a group with unfamiliar faces can cause anxiety and feelings of being overwhelmed, ultimately resulting in further withdrawal. Novel situations are often approached cautiously, sometimes leading to the child appearing as if disengaged. Even though introversion doesn’t necessarily reflect lack of confidence, teachers and other education professions can help those introverted children to achieve without trying to turn them into extroverts – a strategy that will ultimately fail.

The sensitive nature of introverts often means that their self-esteem is more fragile; unfortunately they are also better skilled at picking up cues related to disapproval, such as a frustrated sigh, an angry facial expression or subtle criticism, resulting in further withdrawal. The introverted child might also react badly to being labelled as ‘shy’, with all its negative connotations – the introvert is fully aware that introversion is in no way synonymous with shyness.

Certain classroom techniques that involve children working in groups can be a nightmare for introverted children; furthermore, they gain less from such strategies than more extroverted pupils. Introverts prefer to be one of the early guests at any party and become anxious when forced to join a pre-existing group.

Pair work is therefore preferable in the early stages; pairs can be joined later to form larger groups, so that less dominant voices can get a chance to be heard and grow in confidence.

Helping introverted children to cope with novel situations will eventually lead to fewer anxious reactions to new environments. Children new to your class or school will want to understand the rules as well as the practical elements of their surroundings, such where the toilets are; they will try and work these things out for themselves (because that’s what introverts do) but by making them explicitly clear there’s less for them to worry about.

The existence of introversion (like any personality trait) is somewhat of a contested issue, with some denying the very existence of personality traits outright. The discussion perhaps really lies with the consistency and stability of these traits, for example, people display certain traits in specific situations (children might be introverts in the classroom but livewires in the playground when they are with their friends). Additionally, teenagers are often reluctant to speak out in lessons, which can create the impression of introversion. However, teenagers might be risk-takers in some environments and not in others (they don’t tend to take risks in the classroom and speaking out or attempting to ask and answer questions is viewed as taking a risk to many teens).

Nevertheless, those students who do display behaviour related to introversion, can benefit from a slightly different approach than those who appear more lively.

Could Lessons In Personality Help Teens Cope With Social Stressors?

personalityA new study from David Yeager suggests that teaching teenagers about social and personality traits could help them cope with certain social challenges such as bullying, which in turn could help with stress and lead to higher academic achievement.

Peer groups are vitally important to adolescents, much more so than for adults. While adults aren’t always worried about not fitting in with their peer group, teenagers possess a heightened desire to be accepted by and into the group. Social exclusion causes them anxiety which can in turn impact young people’s wellbeing and academic achievement. Teenagers cause no end of frustration for teachers and parents due to their change in behaviour when they have been excluded from a friendship group, and while we might comfort them there is little that can be done to calm the anxiety exclusion can cause. This anxiety (or the attempt to prevent it) coupled with a brain unable to inhibit risk taking, means that teenagers are highly influenced by the group they are worried about being excluded from. The transition from primary school to high school can be particularly difficult.

According to Yeager:

Adolescents are very focused on peer social hierarchy and status, and when they transition into high school, they are put into a situation where they have to figure out where they stand. Often, teenagers think if it’s going to be hard now, it’s going to be hard forever. That’s stressful for them.

Yeager suggest that teaching students that socially relevant traits are malleable, rather than fixed, can make them feel better prepared to face social challengers as opposed to viewing them as threats and thinking of them as lasting realities. Yeager’s research (to be published in Psychological Science) used two double-blind studies to monitor teenagers physiological responses to stress and how the lessons in personality could improve cognitive, physiological and behavioural responses to stressful situations as well as academic performance.

In the first study, Yeager and his team monitored cardiovascular responses as sixty teenagers between the ages of 14 and 17 prepared and delivered a short speech on what makes people popular before completing a series of maths equations. Prior to undertaking the task, half of the teenagers were told that people and their socially relevant traits were changeable. Those teenagers who were exposed to this idea reported feeling less threatened by the task, exhibited higher cardiac efficiency and lower levels of the stress hormone cortisol. They also performed better on speeches and the maths problems.

In the second study, 205 ninth grade students were tracked throughout the school year. Half of the sample received lessons on the idea that people could change (the intervention group). The students were asked to complete daily diaries where they reported all the stressful things that had happened to them, were asked how much they could deal with the stressors noted in the diaries and provided samples of saliva to measure the levels of stress hormones. Those students in the intervention group coped better on the days where they reported more stressors and were also exhibiting higher Grade Point Averages than their peers seven months later.

The study builds on work by Carol Dweck and other self-theorists whose research indicates that by exposing students to messages surrounding change and adaptation, it is possible to help them cope with stress, raise levels of resilience and obtain higher academic success. Although Yeager stresses that such psychological interventions don’t represent ‘magic bullets’ but can be seen as a ‘progressive step forward in the research process of addressing the wider public health issue of teenage stress’.

Mental Health, Concept Creep and Moral Panic.

mental-healthI don’t tend to read ‘columns’; I find them opinionated and often lacking in substance (a bit like a discussion on Twitter or some of my blogs). However, I was directed to a recent piece by Tom Bennett in the TES on mental health.

Now, I think I’ve made it clear in the past that I don’t feel comfortable commenting on the current debate surrounding the existence or otherwise of a child mental health crisis. This isn’t to say that I’m uncomfortable commenting on mental health, it’s just that the changing landscape of diagnosis and classification along with the controversies surrounding the latest diagnostic criteria mean that I just don’t know what to think. Nevertheless, here I am commenting on that very topic.

Bennett, behaviour advisor to the government and director of IdeologyED ResearchED, makes a number of pertinent points, some of which I alluded to in a blog post back in May. The most important point is that mental health is something that we need to get right – it’s far too important to fuck up.

But a moral panic helps no-one, least of all those vulnerable young people who need help most.

The point, I think, that we both agree on is that child mental health and it’s relationship to education policy is creating a moral panic rather than a need to establish the facts. Debra Kidd, in her recent blog ‘No Mental Health Crisis?’ makes a brave and valiant attempt to support the crisis hypothesis, but Kidd (like Bennett and like me) isn’t a mental health professional – we are amateur observers who interpret the findings and read the reports from altered perspectives.

In a recent paper Psychologist Nick Haslam makes some important observations related to this  (Haslam, 2016). Haslam suggests that society is taking psychological concepts and applying them erroneously to normative human behaviours; everyday ‘worry’ becomes pathologised as ‘anxiety disorder’, sadness as ‘depression’. So-called ‘concept creep’ works on a number of levels, reshaping society and creating more sensitive populations who must, in some way, be protected from everyday horrors; the need for resilience has replaced ordinary courage.

One worrying observation is that mental health appears to be populated by non-professionals, especially when it comes to child mental health and mental health in schools. The appointment of the recently deposed ‘Mental Health Champion for Schools’ Natasha Devon is perhaps one such example. Devon’s intentions are certainly laudable and the bravery with which she has taken on such a dispassionate and emotionally barren government is, in my mind, worthy of praise. Whether her knowledge of the complexity of mental illness, it’s diagnosis, classification and treatment is strong enough to warrant her influence, however, remains questionable.

So who do we listen to?

This is a major problem. There are a number of individuals on the speaker circuit who advise on mental health issues and have published books on how to respond to mental health problems in schools. Many aren’t mental health professionals and it is obvious that some have only a passing understanding of the psychology behind mental health and the complexities of diagnosis and classification. I recently received feedback from a publisher on a manuscript I’m currently circulating about the role of emotions on learning. To be clear, the manuscript details normative functions, the way they impact learning and how teachers can work with them. There is a chapter on anxiety but, again, in terms of everyday processes rather than extreme circumstances. The publisher liked the idea but insisted that I include more extreme behaviours (they actually singled out suicidal tendencies). I could have complied and perhaps even been looking at publication, but such areas are far beyond my knowledge and expertise, so I declined and submitted elsewhere. There is certainly a desire for such publications – I would argue against a need for them however, especially when written by amateur observers.

As I (and Bennett) have said, we need to get mental health right and we need to receive the right information from the right people. There will always be controversies and I suspect that it will be some time before we manage to reconcile the medical model of mental health (with its emphasis on neurochemical and genetic explanations) with more nurture related and environmental theories. Teachers aren’t mental health professionals and the danger is that those most in need of help get lost in the moral panic.

References and Further Reading:

Haslam, N. (2016). Concept creep: psychology’s expanding concepts of harm and pathology. Psychological Inquiry. 27 (1). p.pp. 1–17.

How Americans Became So Sensitive to Harm, The Atlantic.

Self-esteem and Academic Achievement

If sources are to be trusted, there is a mental health crisis in our schools at present. I’m not sure of how accurate these claims are or if we are indeed witnessing the ‘emotionalisation’ of education. Nick Haslam, a psychologist at the University of Melbourne does suggest that psychological concepts are being expanded so that they encompass broader areas; certain behaviours are becoming ‘pathologised’: Fear of maths becomes ‘math anxiety’ and fear of taking tests becomes ‘test anxiety’, while the NUS has banned clapping in an attempt to prevent trauma (clapping anxiety?). Of course, such anxieties are real to the individual and can be reduced through support, but whether such ‘concept creep’ is desirable is yet to be seen.

There also exists the misguided notion that raising self-esteem in young people can somehow protect them from further psychological distress and raise academic attainment. But high self-esteem can have a dark side. Roy Baumeister found that the most aggressive people tend towards high levels of self-esteem, suggesting that violence becomes more likely when other people and situations contradict a persons highly favourable view of themselves. Very high levels of self-esteem might also lead to arrogance and fragile self-concept. This results in such individuals being easily threatened and more likely to use violence in order to protect their fragile and inflated sense of self-worth. People high in self-esteem often have a mistaken impression of themselves and are more likely to claim to be more likeable and attractive, to have better relationships and to make better impressions on people than those with low self-esteem. Objective measures however, appear to cast doubt on these beliefs, leading Baumeister to conclude that ‘narcissists can be charming as first, but tend to alienate others eventually’.

There are certainly benefits to having high levels of self-esteem. Self-esteem is strongly correlated with happiness, although without a clearly established causation. What this means is that there is no way to confirm that the high levels of self-esteem cause people to be happier. It could be that happy people develop higher levels of self-esteem or that other factors indirectly lead to higher levels of happiness. It is clear, however, that low self-esteem is related to depression and stress, but the direction remains unclear. High levels of self-esteem have also been shown to foster experimentation among children, leading to an increase in risky behaviours such as smoking, drinking, drug taking and underage sex. However, high self-esteem does reduce the risk of bulimia in females.

Self-esteem doesn’t impact academic performance.

While some studies have identified a link between high self-esteem and academic attainment, correlations have been modest at best. High self-esteem, therefore, appears to have little impact on academic outcomes but findings do suggest that high levels of self-esteem emerge from good school performance. Furthermore, efforts to boost the self-esteem of school pupils have not resulted in improvements in academic performance and, in some cases, have been found to be counterproductive. Similar results have been found in adults, in that job performance is sometimes related to self-esteem but such correlations vary widely and, again, direction of causality hasn’t been reliably established. Certainly, high levels of occupational success may boost self-esteem.

It would appear, therefore, that while self-esteem is related to some positive outcomes, there is little to suggest that students with high self-esteem do any better academically than those with low self-esteem. This is not to say that schools should abandon such notions, but that interventions should be directed towards well-being rather than that formal academic outcomes. Harter suggests that scholastic competence can be seen as a specific type of self-esteem that might be better placed as a means of investigating the role of self-theories on academic outcomes. A more developed concept similar to that proposed by Harter is that of ‘academic self-concept’, of which I have written previously.

Psychology: It gets more complicated after twenty-odd years.

A few weeks before I left my last teaching job I was observed by my line manager (no, this isn’t a rant about observations). He asked me why I insisted on students repeating what they had learned and why I then had them explain it to each other.

“Because recalling information strengthens the memory trace of that information,” I explained.

He didn’t seem convinced, but then again, a week earlier he had distributed a revision pack to sixth form students that included information on learning styles. At the time of our discussion we were also seated next to my wonderful display on ‘Evidenced-based Revision Techniques’ (taken from the Dunlosky paper).

Towards the end of the meeting he asked politely what I was intending to do when I left. I explained briefly that I was interested in the application of psychology to the classroom and that I was hoping to do some work in that area.

“Well”, he said, “It’s very popular. My wife is doing the same thing – she’s training to be a mindfulness teacher”.

I nodded politely. To be fair, I wouldn’t necessarily expect a Spanish teacher (or any non-psychology teacher) to fully comprehend the deeper meaning of what I said. Although of course, he was charged with observing my lessons, so a modicum of knowledge could have been useful. Furthermore, this is simply an observation, not a criticism (Okay, a little bit of a criticism). Despite a growing interest in the use of psychology in teaching, the majority of teachers I would guess are not always aware that they are using psychological techniques in their classrooms already.

I’ve been studying or teaching psychology for more than twenty years. In that time I have studied at undergraduate and post-graduate level, designed and delivered introductory psychology courses to adult learners and spent 12 years teaching A-level Psychology; I was awarded ‘Chartered’ status by the British Psychological Society and three years after that Associate Fellowship (a fairly rare achievement for an A-level Psychology teacher). Over the years I’ve published in academic journals, popular magazines and everything in-between – I’ll stop now, but I often feel that I have to justify myself.

Some time ago I briefly engaged in a discussion around the legitimacy of non-psychology specialists to offer advice, consult and blog on psychological theories and criticise the research underpinning its methodology. The argument was that if you don’t have formal qualifications in psychology then you should keep out of it. On the one hand there was a kind of logic attached to this point of view – I suspect there are fewer English teachers offering advice on Maths than there are English teachers offering advice on Psychology, so why should Psychology be any different? On the other hand (and this was my stance during the exchange), if an English teacher (or a History teacher, or even a Spanish teacher!) wants to share what they have learned then why not? We are all the richer for it – or are we?

My concern is that our understanding of what psychology represents and how it can influence teaching and learning becomes very narrowly defined. With psychology seemingly in ‘crisis’ I’ve even noticed that many are now using the term ‘cognitive science’ and even ‘behavioural economics’ to distance themselves from what they see as a discipline in meltdown. It’s my former line manager viewing psychology as a self-help guide rather than a fledgling scientific discipline, still trying to find its place in the grand scheme of things.

Certainly, my own interests have shifted. Ten years ago it was all about memory for me, then came the ‘resilience’ years (the shift from cognitive to so-called non-cognitive). Recently I’ve somewhat shifted back with a growing (near obsessional – 60,000 words and counting type of obsessional) interest in the interaction between cognition and emotion and its impact on learning. Many teacher blogs I have read describe learning as a cognitive process, when it has been clear for some time that it’s an emotional (and social) one as well. We pick the definitions we need to justify our own position and develop a narrow framework based on limited reading – it gets more complicated than that after twenty-odd years.

Of course, like other disciplines, psychology is fragmented – biological, cognitive, social, developmental, behavioural and, yes, even the Freudians still walk the ghostly halls, unaware that they died some time ago (*waits for backlash and the accusations of inappropriate feelings towards mothers*).

Nevertheless, there is less conflict in psychology than there once was and while the replication crisis is multifaceted, it does, I believe, provide psychology with the opportunity to re-think its priorities. Unfortunately, while psychology attempts to reconcile its differences, many teachers appear hell-bent on exacerbating them.    

Is Guessing the Answer?

In which year were the series of peace treaties known as the Peace of Westphalia signed?

Of course you could go and Google it, but it’s a year that is branded into my brain. Before studying for a degree in Psychology I was a student of International Relations and Politics and, seeing as the date was crucial in the development of international cooperation, it’s become one of those dates I will always remember.

Don’t know? Have a guess, you might get it right.

We’ve all said the same thing to our students, right? When their frightened faces look up in response to the sound of their name being called and stare at us like a rabbit caught in the headlights of a speeding car.

But is guessing helpful?

Elizabeth Loftus, cognitive psychologist extraordinaire and world-renowned expert on eyewitness testimony thinks not. In fact, she thinks guessing can be downright dangerous. In 1978 Loftus, along with Reid Hastie and Robert Landsman, found that when individuals are encouraged to guess on a test, their incorrect answer often crops up on a later test (Hastie et al., 1978).

Elizabeth Marsh and Henry Roediger (along with Robert and Elizabeth Bjork) also reached similar conclusions in their 2007 study, concluding that when people make errors on multiple choice tests the errors can persist on later cued-recall tests (when participants are given ‘cues’ to help them recall previously seen material) (Marsh et al., 2007)

These and other research studies have led leading cognitive psychologists and experts on eyewitness testimony to suggest that guessing can be dangerous because, when people guess, they might later recall their incorrect guesses as being correct. The problem, then, is one of memory; when people are forced to guess the answer on a test they often remember their guesses as being part of the original to be learned list, which perhaps explains why teachers continue to receive incorrect answers from students even when told that the answer they have given is wrong.

The problem with this, however, is that results can often be inconsistent. Other studies have identified the benefits of unsuccessful retrieval to learning. As long as the correct answer is, in the end, generated by the student or provided by the teacher then the error shouldn’t carry over to subsequent tests. Bridgid Finn found that when unsuccessful retrieval attempts were followed by feedback, long-term retention was better than when the correct answer was just given (Finn et al., 2012).

This shows that not only is the testing effect replicated, but also that feedback is vital in order to correct any errors or misconceptions (it also highlight the fallibility of memory, something for next time perhaps).

And the answer to the question?

The treaties brought to a close the series of related conflicts known as the Thirty Years’ War, which lasted from 1618 to the signing of the treaties in 1648.

References.

Finn, B., Roediger, H. & Rosenzweig, E. (2012). Reconsolidation from negative emotional pictures: Is successful retrieval required? Memory & Cognition. 40 (7). p.pp. 1031–1045.

Hastie, R., Landsman, R. & Loftus, E.L. (1978). Eyewitness Testimony: The Danger of Guessing. Jurimetrics Journal. (Fall). p.pp. 1–8.

Marsh, E.J., Roediger, H.L., Bjork, R.A. & Bjork, E.L. (2007). The memorial consequences of multiple-choice testing. Psychonomic Bulletin & Review. 14 (2). p.pp. 194–199.