A recipe for (attempting to) replicate existing findings in psychology

Regular readers of this blog will know that social psychology has gone through a traumatic time of late. Some of its most high profile proponents have been found guilty of research fraud. And some of the field's landmark findings have turned out to be less robust than hoped. This has led to soul searching and one proposal for strengthening the discipline is to encourage more replication attempts of existing research findings.

To this end, some journals have introduced dedicated replication article formats and pressure is building on others to follow suit. As the momentum for reform builds, an international team of psychologists has now published an open-access article outlining their "Replication Recipe" - key steps for conducting and publishing a convincing replication.

This is an important development because when high-profile findings have failed to replicate there's been a tendency in recent times for ill-feeling and controversy to ensue. In particular, on more than one occasion the authors of the original findings have complained that the failed replication attempt was of poor quality or not suitably similar to the original methods. In fact it's notable that one of the co-authors of this new paper is Ap Dijksterhuis who published his own tetchy response to a failed replication of his work earlier this year.

Dijksterhuis and the others, led by Mark Brandt at Tilburg University (coincidentally the institution of the disgraced psychologist Diederik Stapel), outline five main ingredients for a successful replication recipe:

1. Carefully defining the effects and methods that the researcher intends to replicate;
2. Following as exactly as possible the methods of the original study (including
participant recruitment, instructions, stimuli, measures, procedures, and analyses);
3. Having high statistical power [this usually means having a large enough sample];
4. Making complete details about the replication available, so that interested experts can
fully evaluate the replication attempt (or attempt another replication themselves);
5. Evaluating replication results, and comparing them critically to the results of the original study.

To help the would-be replicator, the authors have also compiled a checklist of 36 decisions that should be made, and items of information that should be collated, before the replication attempt begins. It appears as table 1 in their open-access article and they've made it freely available for completion and storage as a template on the Open Science Framework.

Here are some more highlights from their paper:

In deciding which past findings are worth attempting to replicate, Brandt and his team urge researchers to choose based on an effect's theoretical importance, its value to society, and the existing confidence in the effect.

They remind readers that a perfect replication is of course impossible - replications inevitably will happen at a different time, probably in a different place, and almost certainly with different participants. While they recommend contact with the authors of the original study in order to replicate the original methodology as closely as possible, they also point out that a replication in a different time or place may actually require a change in methodology for the purpose of mimicking the context of the original. For instance, a study originally conducted in the US involving baseballs might do well to switch to cricket balls if replicated in the UK. Also bear in mind that something as seemingly innocuous as the brand of the computer monitor used to display stimuli could have a bearing on the results.

Another important point they make is that replicators should set out to measure any factors that they anticipate may cause the new findings to deviate from the original. Not only will this help achieve a successful replication but it furthers scientific understanding by establishing boundary conditions for the original effect.

On their point about having enough statistical power, Brandt and his colleagues urge replicators to err on the side of caution, towards having a larger sample size. Where calculating the necessary statistical power is tricky, they suggest a simple rule of thumb: aim for 2.5 times the sample size of the original study.

It's not enough to say simply that a replication has failed or succeeded, Brandt and co also advise. Instead replicators should conduct two tests: first establish whether the effect of interest was statistically significant in the new study; and second, establish whether the findings from the attempted replication differ statistically from the findings of the original. A meta-analysis that combines results from the original study and the replication attempt is also recommended.

"By conducting high-powered replication studies of important findings we can build a cumulative science," the authors conclude. "With our Replication Recipe, we hope to encourage more researchers to conduct convincing replications that contribute to theoretical development, confirmation and disconfirmation."

_________________________________

Mark J. Brandt, Hans IJzerman, Ap Dijksterhuis, Frank J. Farach, Jason Gellerd, Roger Giner-Sorollae, James A. Grange, Marco Perugini, Jeffrey R. Spies, and Anna van 't Veer (2013). The Replication Recipe: What Makes for a Convincing Replication? Journal of Experimental Social Psychology DOI: 10.1016/j.jesp.2013.10.005

Further reading
Psychologist magazine special issue on replications.

Post written by Christian Jarrett (@psych_writer) for the BPS Research Digest.

Students assume psychology is less scientific/important than the natural sciences, says study with scientific limitations

Students see test tubes as more scientific than questionnaires

Despite over 130 years passing since the opening of its first laboratory, psychology still struggles to be taken seriously as a science. A new paper by psychologists in the USA suggests this is due in part to superficial assumptions made about the subject matter and methods of behavioural science.

Douglas Krull and David Silvera asked 73 college students (49 women) to rate various topics and pieces of equipment on a 9-point scale in terms of how scientific they thought they were. On average, the students consistently rated topics from the natural sciences (e.g. brain, solar flares), and natural science equipment (e.g. microscope, magnetic resonance imaging) as more scientific than behavioural science topics and equipment (e.g. attitudes and questionnaires) - the average ratings were 7.86, 5.06, 7 and 4.34, respectively.

A follow-up study involving 71 more college students was similar but this time students rated the scientific status of 20 brief scenarios. These varied according to whether the topic was natural or behavioural science and whether the equipment used was natural or behavioural (e.g. "Dr Thompson studies cancer. To do this research, Dr Thompson uses interviews" is an example of a natural science topic using behavioural science methods.) Natural science topics and equipment were again rated as more scientific than their behavioural science counterparts. And this was additive, so that natural science topics studied with natural science methods were assumed to be the most scientific of all.

A third and final study was almost identical but this time the 94 college students revealed their belief that the natural sciences are more important than the behavioural sciences. "Even though the scientific enterprise is defined by its method, people seem to be influenced by the content of the research," Krull and Silvera concluded. They added that this could have serious adverse consequences including students interested in science not going into psychology; psychology findings not being taken seriously; and funding being diverted from psychology to other sciences. "Misperceptions of science have the potential to hinder research and applications of research that could otherwise produce positive changes in society," they said.

Unfortunately for a paper on the reputation of psychological science, the paper contains a series of serious scientific limitations. For instance, not only are all three samples restricted to college students, we're also told nothing about the background of these students; not even whether they were humanities or science students. There is also no detail on how the students construed the meaning of "scientific". If students assume the meaning of scientific has more to do with subject matter than with method then the findings from the first two studies are simply tautological.

Apart from a couple of exceptions, we are also given no information on how the researchers categorised their list of topics and equipment as belonging either to natural or behavioural science. Sometimes it's obvious, but not always. For instance, how was "computer programmes" categorised? Where the categorisation is revealed it doesn't always seem justified. Is "the brain" exclusively a natural science topic and not a behavioural science topic? In truth psychologists often make inferences about the brain based on behavioural data. Obviously carving up scientific disciplines is a tricky business, but the issue is not really addressed by Krull and Silvera. In terms of terminology, their paper starts off distinguishing between natural and behavioural science, with psychology given as an example of a behavioural science. Their discussion then focuses largely on psychology.

Lastly, it's unfortunate that Krull and Silvera more than once refer to the seductive allure of brain scans as an example of the way that people are swayed by the superficial merit of natural science. Presumably they wrote their paper before the seductive allure of brain scans was thoroughly debunked earlier this year. They can't be blamed for not seeing into the future, but it was perhaps scientifically naive to place so much faith in a single study.

_________________________________

Douglas S. Krull and David H. Silvera (2013). The stereotyping of science: superficial details influence perceptions of what is scientific. Journal of Applied Social Psychology DOI: 10.1111/jasp.12118

--Further reading--
Child's play! The developmental roots of the misconception that psychology is easy
From The Psychologist magazine news archive: A US psychologist has urged the psychological community to do more to challenge the public's scepticism of our science.

Post written by Christian Jarrett (@psych_writer) for the BPS Research Digest.

Scanning a brain that believes it is dead

What is going on in the brain of someone who has the deluded belief that they are brain dead? A team of researchers led by neuropsychologist Vanessa Charland-Varville at CHU Sart-Tilman Hospital and the University of Liege has attempted to find out by scanning the brain of a depressed patient who held this very belief.

The researchers used a Positron Emission Tomography (PET) scanner, which is the first time this scanning technology has been used on a patient with this kind of delusion - known as Cotard's syndrome after the French neurologist Jules Cotard. The 48-year-old patient had developed Cotard's after attempting to take his own life by electrocution. Eight months later he arrived at his general practitioner complaining that his brain was dead, and that he therefore no longer needed to eat or sleep. He acknowledged that he still had a mind, but (in the words of the researchers) he said he was "condemned to a kind of half-life, with a dead brain in a living body."

The researchers used the PET scanner to monitor levels of metabolic activity across the patient's brain as he rested. Compared with 39 healthy, age-matched controls, he showed substantially reduced activity across a swathe of frontal and temporal brain regions incorporating many key parts of what's known as the "default mode network". This is a hub of brain regions that shows increased activity when people's brains are at rest, disengaged from the outside world. It's been proposed that activity in this network is crucial for our sense of self.

"Our data suggest that the profound disturbance of thought and experience, revealed by Cotard's delusion, reflects a profound disturbance in the brain regions responsible for 'core consciousness' and our abiding sense of self," the researchers concluded.

Unfortunately the study has a number of serious limitations beyond the fact that it is of course a single case study. As well as having a diagnosis of Cotard's Delusion, the patient was also depressed and on an intense drug regimen, including sedative, antidepressant and antipsychotic medication. It's unclear therefore whether his distinctive brain activity was due to Cotard's, depression or his drugs, although the researchers counter that such an extreme reduction in brain metabolism is not normally seen in patients with depression or on those drugs.

Another issue is with the lack of detail on the scanning procedure. Perhaps this is due to the short article format (a "Letter to the Editor"), but it's not clear for how long the patient and controls were scanned, nor what they were instructed to do in the scanner. For example, did they have their eyes open or closed? What did they think about?

But perhaps most problematic is the issue of how to interpret the findings. Does the patient have Cotard's Delusion because of his abnormal brain activity, or does he have that unusual pattern of brain activity because of his deluded beliefs? Relevant here, but not mentioned by the researchers, are studies showing that trained meditators also show reduced activity in the default mode network. This provides a graphic illustration of the limits to a purely biological approach to mental disorder. It seems diminished activity in the default mode network can be associated both with feelings of being brain dead or feelings of tranquil oneness with the world, it depends on who is doing the feeling. Understanding how this can be will likely require researchers to think outside of the brain.

_________________________________

Charland-Verville, V., Bruno, M., Bahri, M., Demertzi, A., Desseilles, M., Chatelle, C., Vanhaudenhuyse, A., Hustinx, R., Bernard, C., Tshibanda, L., Laureys, S., and Zeman, A. (2013). Brain dead yet mind alive: A positron emission tomography case study of brain metabolism in Cotard's syndrome. Cortex DOI: 10.1016/j.cortex.2013.03.003

Post written by Christian Jarrett (@psych_writer) for the BPS Research Digest.

Serious power failure threatens the entire field of neuroscience

Psychology has had a torrid time of late, with fraud scandals and question marks about the replicability of many of the discipline's key findings. Today it is joined in the dock by its more biologically oriented sibling: Neuroscience. A team led by Katherine Button at the School of Experimental Psychology in Bristol, and including psychologist Brian Nosek, founder of the new Center for Open Science, make the case in a new paper that the majority of neuroscience studies involve woefully small sample sizes, rendering their results highly unreliable. "Low statistical power is an endemic problem in neuroscience," they write.

At the heart of their case is a comprehensive analysis of 49 neuroscience meta-analyses published in 2011 (that's all the meta-analyses published that year that contained the information required for their purposes). This took in 730 individual papers, including genetic studies, drug research and papers on brain abnormalities.

Meta-analyses collate all the findings in a given field as a way to provide the most accurate estimate possible about the size of any relevant effects. Button's team compared these effect size estimates for neuroscience's subfields against the average sample sizes used in those same areas of research. If the meta-analyses for a particular subfield suggested an effect - such as a brain abnormality associated with a mental illness - is real, but subtle, then this would indicate that suitable investigations in that field ought to involve large samples in order to be adequately powered. A larger effect size would require more modest samples.

Based on this, the researchers' estimate is that the median statistical power of a neuroscience study is 21 per cent. This means that the vast majority (around 79 per cent) of real effects in brain science are likely being missed. More worrying still, when underpowered studies do uncover a significant result, the lack of power means the chances are increased that the finding is spurious. Thirdly, significant effect sizes uncovered by underpowered studies tend to be overestimates of the true effect size, even when the reported effect is in fact real. This is because, by their very nature, underpowered studies are only likely to turn up significant results in data where the effect size happens to be large.

It gets more worrying. The aforementioned issues are what you get when all else in the methodology is sound, bar the inadequate sample size. Trouble is, Button and her colleagues say underpowered studies often have other problems too. For instance, small studies are more vulnerable to the "file-drawer effect", in which negative results tend to get swept under the carpet (simply because it's easier to ignore a quick and easy study than a massive, expensive one). Underpowered studies are also more vulnerable to an issue known as "vibration of effects" whereby the results vary considerably with the particular choice of analysis. And yes, there is often a huge choice of analysis methods in neuroscience. A recent paper documented how 241 fMRI studies involved 223 unique analysis strategies.

Because of the relative paucity of brain imaging papers in their main analysis, Button's team also turned their attention specifically to the brain imaging field. Based on findings from 461 studies published between 2006 and 2009, they estimate that the median statistical power in the sub-discipline of brain volume abnormality research is just 8 per cent.

Switching targets to the field of animal research (focusing on studies involving rats and mazes), they estimate most studies had a "severely" inadequate statistical power in the range of 18 to 31 per cent. This raises important ethical issues, Button's team said, because it makes it highly likely that animals are being sacrificed with minimal chance of discovering true effects. It's clearly a sensitive area, but one logical implication is that it would be more justifiable to conduct studies with larger samples of animals, because at least then there would be a more realistic chance of discovering the effects under investigation (a similar logic can also be applied to human studies).

The prevalence of inadequately powered studies in neuroscience is all the more disconcerting, Button and her colleagues conclude, because most of the low-lying fruit in brain science has already been picked. Today, the discipline is largely on the search for more subtle effects, and for this mission, suitable studies need to be as highly powered as possible. Yet sample sizes have stood still, while at the same time it has become easier than ever to run repeated, varied analyses on the same data, until a seemingly positive result crops up. This leads to a "disquieting conclusion", the researchers said - "a dramatic increase in the likelihood that statistically significant findings are spurious." They end their paper with a number of suggestions for how to rehabilitate the field, including performing routine power calculations prior to conducting studies (to ensure they are suitably powered), disclosing methods and findings transparently, and working collaboratively to increase study power.
_________________________________

KS Button, JPA Ioannidis, C Mokrysz, BA Nosek, J Flint, ESJ Robinson, and; MR Munafo (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience : 10.1038/nrn3475

Post written by Christian Jarrett (@psych_writer) for the BPS Research Digest.

Working memory training does not live up to the hype

According to CogMed, one of the larger providers of computerised working memory training, the benefits of such training is "comprehensive" and includes "being able to stay focused, resist distractions, plan activities, complete tasks, and follow and contribute to complex discussions." Similar claims are made by other providers such as Jungle Memory and Cognifit, which is endorsed by neuroscientist Susan Greenfield.

Working memory describes our ability to hold relevant information in mind for use in mental tasks, while ignoring irrelevant information. If it were possible to improve our working memory capacity and discipline through training, it makes sense that this would have widespread benefits. But that's a big if.

A new meta-analysis by Monica Melby-Lervåg and Charles Hulme has just been published in the February issue of the respected APA journal Developmental Psychology, which combined the results from 23 studies of working memory training completed up to 2011 (PDF is freely available). To be included, studies had to compare outcomes for a working memory training treatment group against outcomes in a control group. Most of the studies available are on healthy adults or children, with just a few involving children with developmental conditions such as ADHD.

The results were absolutely clear. Working memory training leads to short-term gains on working memory performance on tests that are the same as, or similar to, those used in the training. "However," Melby-Lervåg and Hulme write, "there is no evidence that working memory training produces generalisable gains to the other skills that have been investigated (verbal ability, word decoding, arithmetic), even when assessments take place immediately after training."

There was a modest, short-term benefit of the training on non-verbal intelligence but this disappeared when only considering the studies with a robust design (i.e. those that randomised participants across conditions and which enrolled control participants in some kind of activity). Similarly, there was a modest benefit of the training on a test of attentional control, but this disappeared at follow-up.

All of this suggests that working memory training isn't increasing people's working memory capacity in such a way that they benefit whenever they engage in any kind of task that leans on working memory. Rather, people who complete the training simply seem to have improved at the specific kinds of exercises used in the training, or possibly even just at computer tasks - effects which, anyway, wear off over time.

Overall, Melby-Lervåg and Hulme note that the studies that have looked at the benefits of working memory training have been poor in design. In particular, they tend not to bother enrolling the control group in any kind of intervention, which means any observed benefits of the working memory training could be related simply to the fun and expectations of being in a training programme, never mind the specifics of what that entails. Related to that, some dubious studies reported far-reaching benefits of the working memory training, without finding any improvements in working memory, thus supporting the notion that these benefits had to do with participant expectations and motivation.

A problem with all meta-analyses, this one included, is that they tend to rely on published studies, which means any unpublished results stuck in a filing cabinet get neglected. But of course, it's usually negative results that get left in the drawer, so if anything, the current meta-analysis presents an overly rosy view of the benefits of working memory training.

Melby-Lervåg and Hulme's ultimate conclusion was stark: "there is no evidence that these programmes are suitable as methods of treatment for children with developmental cognitive disorders or as ways of effecting general improvements in adults' or children's cognitive skills or scholastic achievements."

_________________________________

Melby-Lervåg M, and Hulme C (2013). Is working memory training effective? A meta-analytic review. Developmental psychology, 49 (2), 270-91 PMID: 22612437 Free, full PDF of the study.

--NB.--
This meta-analysis only took in reviews published up to 2011. If you know of any quality studies into the effects of working memory training published since that time, please do share the relevant links via comments. 

--Further reading--
Brain training games don't work.
Brain training for babies actually works (short term, at least)

Post written by Christian Jarrett (@psych_writer) for the BPS Research Digest.

Emotion research gets real - Was this person just told a joke or told they have great hair?

How accurately could you tell from a person's display of behaviour and emotions what just happened to them?  Dhanya Pillai and her colleagues call this "retrodictive mindreading" and they say it's a more realistic example of how we perceive emotions in everyday life, as compared with the approach taken by traditional psychological research, in which volunteers name the emotions displayed in static photos of people's faces.

In Pillai's study, the task of a group of 35 male and female participants wasn't to look at pictures and name the facial expression. Instead, the participants watched clips of people reacting to a real-life social scenario and they had to deduce what scenario had led to that emotional display.

Half the challenge Pillai and her colleagues faced was to create the stimuli for this research. They recruited 40 men and women who thought they were going to be doing the usual thing and categorising emotional facial expressions. In fact, it was their own responses that were to become the stimuli for the study proper.

While these volunteers were sitting down ready for the "study" to start, one of four scenarios unfolded. The female researcher either told them a joke ("why did the woman wear a helmet at the dinner table? She was on a crash diet"); told them a story about a series of misfortunes she'd encountered on the way to work; paid them a compliment (e.g. "you've got really great hair, what shampoo do you use?"); or made them wait 5 minutes while she had a drink and did some texting. In each case the volunteers' emotional responses were recorded on film and formed the stimuli for the real experiment.

The researchers ended up with 40 silent clips, lasting 3 to 9 seconds each, comprising ten clips for each of the four scenarios. The real participants for the study proper were first shown footage of the researcher in the four scenarios and how these were categorised as joke, story, compliment or waiting. Then these observer participants watched the 40 clips of the earlier volunteers, and their task in each case was to say which scenario the person in the video was responding to.

The observing participants' performance was far from perfect - they averaged 60 per cent accuracy - but it was far better than the 25 per cent level you'd expect if they were merely guessing. By far, they were most skilled at recognising when a person was responding to the waiting scenario (90 per cent accuracy). Their accuracy was even for the other scenarios at around 50 per cent. They achieved this success level despite the huge amount of variety in the way the different volunteers responded to the different scenarios. "From observing just a few seconds of a person's reaction, it appears we can gauge what kind of event might have happened to that individual with considerable success," the researchers said.

A surprise detail came from the recordings of the observing participants' eye movements. They focused more on the mouth region rather than the eyes. Based on past research (much of it using static facial displays), Pillai and her colleagues thought that better accuracy would go hand-in-hand with more attention paid to the eye region of the targets' faces. In fact, for three of the scenarios (all except the joke), the opposite was true. This may be because focusing on the eye region is more beneficial when naming specific mental states, as opposed to the "retrodictive mindreading"challenge involved in the current study.

In contrast to much of the existing psychology literature, Pillia and her team concluded that theirs was an important step towards devising tasks "that closely approximate how we understand other people's behaviour in real life situations."

_________________________________

Pillai, D., Sheppard, E., and Mitchell, P. (2012). Can People Guess What Happened to Others from Their Reactions? PLoS ONE, 7 (11) DOI: 10.1371/journal.pone.0049859

Note: the picture above is for illustrative purposes only and was not used in the study.

Post written by Christian Jarrett (@psych_writer) for the BPS Research Digest.

A new test for finding out what people really think of their personality

A problem with your standard personality questionnaire is that most people like to make a good impression. This is especially the case when questionnaires are used for job candidates. One way around this is to use so-called implicit measures of personality, designed to probe subconscious beliefs. The famous Rorschach ink-blot test is one example, but many psychologists criticise it for its unreliability. A more modern example is a version of the implicit association test, in which people are timed using the same response key for self-referential words and various personality traits. If they associate the trait with themselves, they should be quicker to answer. Now a team led by Florin Sava have proposed a brand-new test based on what's called the "semantic misattribution procedure".

Nearly a hundred participants watched as personality traits were flashed one at a time for a fifth of a second on a computer screen. After each trait (e.g. "anxious"), a neutral-looking Chinese pictograph was flashed on-screen. The participants didn't know what these Chinese symbols meant. Their task was to ignore the flashed personality traits and to say whether they'd like each Chinese symbol to be printed on a personalised t-shirt for them or not, to reflect their personality.

This method is based on past research showing that we tend to automatically misattribute the meaning of briefly presented words to subsequent neutral stimuli. So, in the example above, participants would be expected to attribute, at a subconscious level, the meaning of "anxious" to the Chinese symbol. When assessing the suitability of the symbol for their t-shirt, it feels subjectively as if they are merely guessing, or making their judgment based on its visual properties. But in fact their choice of whether the symbol is suitable will be influenced by the anxious meaning they've attributed to it, and, crucially, whether or not they have an implicit belief that they are anxious.

In this initial study, and two more involving nearly 300 participants, Sava and his colleagues showed that participants' scores on this test for conscientiousness, neuroticism and extraversion correlated with explicit measures of the same traits. The new implicit test also did a better job than explicit measures alone of predicting relevant behaviours, such as church attendance, perseverance on a lab task, and punctuality. The implicit scores for extraversion showed good consistency over 6 months. Finally, the new implicit test showed fewer signs of being influenced by social desirability concerns, as compared with traditional explicit measures. Next, the researchers plan to test whether their new implicit measure is immune to attempts at deliberate fakery.

"The present study suggests that the Semantic Misattribution Procedure is an effective alternative for measuring implicit personality self- concept," the researchers said.

_________________________________

Sava, F., MaricuΤoiu, L., Rusu, S., Macsinga, I., Vîrgă, D., Cheng, C., and Payne, B. (2012). An Inkblot for the Implicit Assessment of Personality: The Semantic Misattribution Procedure. European Journal of Personality, 26 (6), 613-628 DOI: 10.1002/per.1861

--Further reading--
A personality test that can't be faked

Post written by Christian Jarrett (@psych_writer) for the BPS Research Digest.

Labs worldwide report converging evidence that undermines the low-sugar theory of depleted willpower

One of the main findings in willpower research is that it's a limited resource. Use self-control up in one situation and you have less left over afterwards - an effect known as "ego-depletion". This discovery led to a search for the underlying physiological mechanism. In 2007, Roy Baumeister, a pioneer in the field, and his colleagues reported that the physiological correlate of ego-depletion is low glucose. Self-control leads the brain to metabolise more glucose, so the theory goes, and when glucose gets too low, we're left with less willpower.

The breakthrough 2007 study showed that ego-depleted participants had low blood glucose levels, but those who subsequently consumed a glucose drink were able to sustain their self-control on a second task. In the intervening years the finding has been replicated and the glucose-willpower link has come to be stated as fact.

"No glucose, no willpower," wrote Baumeister and his journalist co-author John Tierney in their best-selling popular psychology book Willpower: Rediscovering Our Greatest Strength (Allen Lane, 2012). The claim was also endorsed in a guide to willpower published by the American Psychological Association earlier this year. "Maintaining steady blood-glucose levels, such as by eating regular healthy meals and snacks, may help prevent the effects of willpower depletion," the report claims.

But now two studies have come along at once (following another published earlier in the year) that together cast doubt on the idea that depleted willpower is caused by a lack of glucose availability in the brain. In the first, Matthew Sanders and his colleagues in the US report what they call the "Gargle effect". They had dozens of students look through a stats book and cross out just the Es, a tiresome task designed to tax their self-control levels. Next, they completed the famous Stroop task - naming the ink colour of words while ignoring their meaning. Crucially, half the participants completed the Stroop challenge while gargling sugary lemonade, the others while gargling lemonade sweetened artificially with Splenda. The participants who gargled, but did not swallow, the sugary (i.e. glucose-containing) lemonade performed much better on the Stroop task.

The participants in the glucose condition didn't consume the glucose and even if they had, there was no time for it to be metabolised. So this effect can't be about restoring low glucose levels. Rather, Sanders' team think glucose binds to receptors in the mouth, which has the effect of activating brain regions involved in reward and self-control - the anterior cingulate cortex and striatum.

The other study that's just come out was conducted by Martin Hagger and Nikos Chatzisarantis based in Australia and the UK. Their approach was similar to Sanders' except that participants gargled and spat out a glucose or artificially sweetened solution prior to performing a second taxing task, rather than during. Also, this research involved a series of 5 experiments involving many different ways of testing people's self-control, including: resisting delicious cookies; reading boring text in an expressive style; unsolvable puzzles; and squeezing hand-grips. But the take-home finding was the same - participants who gargled, but did not swallow, a glucose drink performed better on a subsequent test of their willpower; participants who gargled an artificially sweetened drink did not. So again, willpower was restored without topping up glucose levels. Moreover, the benefit of gargling glucose was displayed only by participants who'd had their self-control taxed in an initial task. It made no difference to participants who were already in an untaxed state.

Hagger and Chatzisarantis agree with the interpretation of the Sanders' group, except they make a distinction. The effect of glucose binding to receptors in the mouth could either stimulate activity in brain regions like the anterior cingulate that tend to show fatigue after a taxing task. Or they say that glucose in the mouth could trigger reward-related activity that prompts participants to interpret a task as more rewarding, thus boosting their motivation. The explanations are complementary and need not be mutually exclusive.

The key point is the new results suggest depleted willpower is about motivation and the allocation of glucose resources, not about a lack of glucose. These findings don't prove that consuming glucose has no benefit for restoring willpower, but they suggest strongly that it's not the principle mechanism. It's notable that the new findings complement previous research in the sports science literature showing that gargling (without ingesting) glucose can boost cycling performance.

"While our findings are consistent with the predictions of the resource-depletion account, they also contribute to an increasing literature that glucose may not be a candidate physiological analog for self-control resources," write Hagger and Chatzisarantis. "Instead ego-depletion may be due to problems of self-control resource allocation rather than availability." An important next step is to conduct brain-imaging and related studies to observe the physiological effects of gargling glucose on the brain, and on motivational beliefs. There are also tantalising applications from the new research - for example, could the gargle effect (perhaps in the form of glucose-infused chewing gum) be used as a willpower aid for dieters and people trying to give up smoking?

_________________________________

Hagger, M., and Chatzisarantis, N. (2012). The Sweet Taste of Success: The Presence of Glucose in the Oral Cavity Moderates the Depletion of Self-Control Resources. Personality and Social Psychology Bulletin DOI: 10.1177/0146167212459912

Sanders, M., Shirk, S., Burgin, C., and Martin, L. (2012). The Gargle Effect: Rinsing the Mouth With Glucose Enhances Self-Control. Psychological Science DOI: 10.1177/0956797612450034

--Further reading--
From The Psychologist: Roy F. Baumeister outlines intriguing and important research into willpower and ego depletion.

Post written by Christian Jarrett for the BPS Research Digest.

Who gets aggressive at the late-night bar and why?

The exhaustive analysis in Steven Pinker's latest book shows that we are living in the most peaceable age for thousands of years. To anyone who spends time in late-night bars, this might come as a surprise. In these temples to hedonism, spilled drinks and unwelcome gropes all too often provoke violent brawls.

Kathryn Graham and her colleagues trained 148 observers and sent them out to 118 bars in early-hours Toronto where they recorded 1,057 instances of aggression from 1,334 visits. Where the majority of psychology research on aggression is based on laboratory simulations (often involving participants zapping each other with loud noise or spiking each other's food with chilli sauce), Graham's team collected real-life observational data to find out who gets aggressive and why.

The researchers followed the Theory of Coercive Actions, according to which aggressive acts have one or more motives: compliance (getting someone to do something, or stop doing something); grievance; social identity (to prove one's status and power); and thrill-seeking.

Unsurprisingly, the vast majority (77.5 per cent) of aggressive acts were instigated by men. Men more than women were driven to aggression by identity and thrill-seeking motives; by contrast female aggression was more often motivated by compliance and grievance. This often had a defensive intent, as a reaction against unwanted sexual advances.

As well as being particularly severe, aggression that was ignited by patrons who felt threats to their identity was also particularly likely to escalate, "because," the researchers said, "their strong identity motivation reflects a situation where the person is already invested in winning or besting the other person." Aggressive acts motivated by grievance were also likely to escalate, because of people feeling their actions were justified.

The researchers found that greater intoxication led to more serious aggression in women, but not men - perhaps because the latter were emboldened enough already. Younger men and bigger men also tended to engage in more serious aggressive acts, replicating past research showing that larger, intoxicated men are more likely to get aggressive than their smaller counterparts.

Graham and her colleagues said their findings could help contribute to preventative policies in late-night bars. For example, given the incendiary role of identity motives in aggressive incidents, efforts could be made to challenge traditional cultural norms that say masculine identity is about power and strength. Because of the escalating effect of grievance motives, security staff could be trained to diffuse situations early - for example, by replacing spilled drinks free of charge. And because so much female aggression was provoked by sexual harassment, the researchers advised establishments to create an atmosphere that discourages "invasive and aggressive sexual overtures whilst still maintaining an exciting venue where young people can explore their sexuality and meet potential partners."

These recommendations sound well-intentioned and supported by the new evidence, but are they really achievable? What do you think?

_________________________________

Kathryn Graham, Sharon Bernards, D. Wayne Osgood, Michael Parks, Antonia Abbey, Richard B. Felson, Robert F. Saltz, and Samantha Wells (2012). Apparent motives for aggression in the social context of the bar. Psychology of Violence DOI: 10.1037/a0029677

Post written by Christian Jarrett for the BPS Research Digest.

Anonymity may spoil the accuracy of data collected through questionnaires

Thousands of psychology papers are based on data derived from questionnaires that were filled out anonymously. That's because most psychologists have reasoned that the way to get people to be honest about their practice of undesirable behaviours is to promise them anonymity. But in a new analysis, Yphtach Lelkes and his colleagues point out that anonymity comes with a price. Participants will feel less accountable and may be less motivated to answer questions accurately.

To test this, Lelkes' team devised a cunning methodology in which dozens of undergrads conducted internet research for what they thought was a study into the way that people search for information on the web.

After each student had spent 45 minutes researching the mountain pygmy-possum, a researcher made a show of deleting the student's search history before their eyes, ostensibly to prevent the next participant from accessing the browser's archives. In fact, a spyware programme was installed on the computer and kept track of all the sites visited. After the research session, each student answered a questionnaire about their use of the internet in general and their experience of the internet research task, including which sites they'd searched. Crucially, half the students were instructed to fill out their name and other personal details at the top of the questionnaire; the others were told to leave it blank to ensure anonymity.

Students who answered the questionnaire anonymously admitted to more embarrassing internet behaviours in general, such as looking at porn, but regards their searches specifically during the research task, they answered with less accuracy. There was also evidence of a lack of variety in many of the anonymous students' later answers, consistent with the idea that they were putting less thought and effort into the questionnaires as they grew tired.

Two follow-up studies involved dozens more students having the opportunity to eat M&M sweets and jelly beans while they completed questionnaires. A question at the end asked them to report how many they'd eaten and once again, students who answered anonymously were less accurate about how much they'd indulged. This was the case whether anonymity was promised before or after the opportunity to eat the snacks.

Lelkes and his colleagues were cautious about how far these findings can be generalised. For example, the same problems might not apply when people are interviewed face-to-face but promised confidentiality. However, they warned researchers against assuming that promising participants anonymity means that they will provide better quality answers. "Particularly among college students who often complete questionnaires to fulfil course requirements, such a guarantee may serve to sanction half-hearted survey completion rather than freeing students up to respond with greater honesty."

_________________________________

Yphtach Lelkes, Jon A. Krosnick, David M. Marx, Charles M. Judd, and Bernadette Park (2012). Complete anonymity compromises the accuracy of self-reports. Journal of Experimental Social Psychology DOI: 10.1016/j.jesp.2012.07.002

--Further reading-- Another recent study found that anonymous web participants provided quality data for psychological experiments.
Our bias for the left-hand side of space could be distorting large-scale surveys.
Is a taste for extreme answers distorting cross-cultural comparisons of personality?

Post written by Christian Jarrett for the BPS Research Digest.

Parents underestimate their children's worry levels and overestimate their optimism

It's well-established that parents frequently overestimate their children's intelligence and the amount of exercise they get. Now a team led by Kristin Lagattuta has uncovered evidence suggesting that parents have an unrealistically rosy impression of their kiddies' emotional lives too. It's a finding with important implications for clinicians and child researchers who often rely on parental reports of young children's psychological wellbeing.

It's previously been assumed that children younger than seven will struggle to answer questions about their emotions. Undeterred, Lagattuta and her colleagues simplified the language used in a popular measure of older children's anxiety and they developed a pictorial scoring system that involved the children pointing to rectangles filled with different amounts of colour. Time was taken to ensure the child participants understood how to use the scale.

An initial study with 228 psychologically healthy children aged 4 to 11 from relatively affluent backgrounds found that the children's answers to oral questions about their experience of worry (including general anxiety, panic, social phobia and separation anxiety) failed to correlate with their parents' (usually the mother's) written responses to questions about the children's experience of worry. Specifically, the parents tended to underestimate how much anxiety their children experienced.

A second study was similar, but this time the researchers ensured the parents and children answered items that were worded in exactly the same way; the parents were reassured that it was normal for children to experience some negative emotion; and the parents were able to place their completed questionnaires in envelopes for confidentiality. Still the children's answers about their own emotions failed to correlate with parents' answers, with the parents again underestimating the amount of worry experienced by their children.

A revealing detail in this study was that parents also answered questions about their own emotions. Their scores for their own emotions correlated with the answers they gave for their children's experiences. "These data suggest that even parents from a low-risk, non-clinical sample may have difficulty separating their emotional perspective from that of their child," the researchers said.

Finally, 90 more children aged 5 to 10 answered questions about their optimism, whilst their parents also answered questions about their own and their children's optimism. Again, parents' and children's verdicts on the children's emotions failed to correlate, with the parents now overestimating their children's experience of optimism. And once more, parents' own optimism was related to how they interpreted their children's optimism.

Lagattuta and her colleagues admitted that it's theoretically possible that the children were the ones showing a distorted view of their own emotions, and it's the parents who were painting the true picture. However, they think this is highly unlikely. For starters it's revealing that parents underestimated their children's negative emotion and yet over-estimated their positive emotion, which argues against the idea that the children were simply answering more conservatively, or giving systematically extreme answers in one direction. Moreover, the new findings fit with the wider literature showing how parents tend to have an unrealistically rosy impression of their children's wellbeing. An obvious study limitation is the focus on middle class US participants, so there is of course a need to replicate with people from other backgrounds and cultures.

"From the standpoint of research and clinical practice, this mismatch between parent and child perceptions raises a red flag," the researchers concluded. "Internally consistent self-report data can be acquired from young children regarding their emotional experiences. Obtaining reports from multiple informants - including the child - needs to be the standard."
_________________________________

  Lagattuta KH, Sayfan L, and Bamford C (2012). Do you know how I feel? Parents underestimate worry and overestimate optimism compared to child self-report. Journal of experimental child psychology, 113 (2), 211-32 PMID: 22727673

Post written by Christian Jarrett for the BPS Research Digest.

Most brain imaging papers fail to provide enough methodological detail to allow replication

Amidst recent fraud scandals in social psychology and other sciences, leading academics are calling for a greater emphasis to be placed on the replicability of research. "Replication is our best friend because it keeps us honest," wrote the psychologists Chris Chambers and Petroc Sumner recently.

For replication to be possible, scientists need to provide sufficient methodological detail in their papers for other labs to copy their procedures. Focusing specifically on fMRI-based brain imaging research (a field that's no stranger to controversy), University of Michigan psychology grad student Joshua Carp has reported a worrying observation - the vast majority of papers he sampled failed to provide enough methodological detail to allow other labs to replicate their work.

Carp searched the literature from 2007 to 2011 looking for open-access human studies that mentioned "fMRI" and "brain" in their abstracts. Of the 1392 papers he identified, Carp analysed a random sample of 241 brain imaging articles from 68 journals, including PLoS One, NeuroImage, PNAS, Cerebral Cortex and the Journal of Neuroscience. Where an article featured supplementary information published elsewhere, Carp considered this too.

There was huge variability in the methodological detail reported in different studies, and often the amount of detail was woeful, as Carp explains:

"Over one third of studies did not describe the number of trials, trial duration, and the range and distribution of inter-trial intervals. Fewer than half reported the number of subjects rejected from analysis; the reasons for rejection; how or whether subjects were compensated for participation; and the resolution, coverage, and slice order of functional brain images."

Other crucial detail that was often omitted included information on correcting for slice acquisition timing, co-registering to high-resolution scans, and the modelling of temporal auto-correlations. In all, Carp looked at 179 methodological decisions. To non-specialists, some of these will sound like highly technical detail, but brain imagers know that varying these parameters can make a major difference to the results that are obtained.

One factor that non-specialists will appreciate relates to corrections made for problematic head-movements in the scanner. Only 21.6 per cent of analysed studies described the criteria for rejecting data based on head movements. Another factor that non-specialists can easily relate to is the need to correct for multiple comparisons. Of the 59 per cent of studies that reported using a formal correction technique, nearly one third failed to reveal what that technique was.

"The widespread omission of these parameters from research reports, documented here, poses a serious challenge to researchers who seek to replicate and build on published studies," Carp said.

As well as looking at the amount of methodological detail shared by brain imagers, Carp was also interested in the variety of techniques used. This is important because the more analytical techniques and parameters available for tweaking, the more risk there is of researchers trying different approaches until they hit on a significant result.

Carp found 207 combinations of analytical techniques (including 16 unique data analysis software packages) - that's nearly as many different methodological approaches as studies. Although there's no evidence that brain imagers are indulging in selective reporting, the abundance of analytical techniques and parameters is worrying. "If some methods yield more favourable results than others," Carp said, "investigators may choose to report only the pipelines that yield favourable results, a practice known as selective analysis reporting."

The field of medical research has adopted standardised guidelines for reporting randomised clinical trials. Carp advocates the adoption of similar standardised reporting rules for fMRI-based brain imaging research. Relevant guidelines were proposed by Russell Poldrack and colleagues in 2008, although these may now need updating.

Carp said the reporting practices he uncovered were unlikely to reflect malice or dishonesty. He thinks researchers are merely following the norms in the field. "Unfortunately," he said, "these norms do not encourage researchers to provide enough methodological detail for the independent replication of their findings."

_________________________________

Carp J (2012). The secret lives of experiments: Methods reporting in the fMRI literature. NeuroImage, 63 (1), 289-300 PMID: 22796459

--Further reading-- Psychologist magazine opinion special on replication.
An uncanny number of psychology findings manage to scrape into statistical significance.
Questionable research practices are rife in psychology, survey finds.

Post written by Christian Jarrett for the BPS Research Digest.