Friday, 29 December 2006

Satisfaction guaranteed!

Our bias for the left-hand side of space could be distorting large-scale surveys. Past research has shown that when people are asked to bisect a horizontal line down the centre, most will cross the line too far to the left. This leftward bias is thought to stem from the right hemisphere – it plays a dominant role in allocating our attention and is also responsible for processing the left-hand side of space. It may also be related to a cultural tendency to read from left to right. Now Andrea Loftus and colleagues have reported this spatial bias could be distorting survey results.

The researchers presented two groups of students with the same questionnaire statements about their experience at university (e.g. “My course has been enjoyable”), except that one group answered using a 5-item Likert scale that ranged left-to-right, from ‘definitely disagree’ to ‘definitely agree’, whereas the other group answered using a scale that ranged left-to-right across the page, from ‘definitely agree’ to ‘definitely disagree’. The positive questionnaire statements were the same as those used by the Higher Education Funding Council for England (HEFCE) in its survey of 250,000 students.

In the current study, the students’ natural bias for the left meant those answering using the Likert scale that started on the left with ‘definitely agree’, responded with that answer to 27 per cent more statements than did the other group of students – that is, their views came out as more positive. By contrast, those students who answered using the scale that began on the left with ‘definitely disagree’ responded more often with ‘mostly disagree’, meaning their views came out overall as more negative.

The observation has profound implications for surveys, such as that conducted by the HEFCE, that seek respondents’ agreement, or not, with consistently positive or negative statements, and which use the same Likert scale for answers throughout. The researchers said one solution in the future is for the Likert scale direction to be reversed for half of the survey sample.
__________________________________

Nicholls, M.E.R., Orr, C.A., Okubo, M., and Loftus, A. (2006). Satisfaction Guaranteed. The Effect of Spatial Biases on Responses to Likert Scales. Psychological Science, 17, 1027-1028.

Post written by Christian Jarrett (@psych_writer) for the BPS Research Digest.

6 comments:

  1. Wow.. how much of a bias was there? Can we find out without buying the article?

    ReplyDelete
  2. hi Andy, I've added in one quantifying comparison I found in the paper - 'definitely agree' responses were 27 per cent more common for the left-oriented answer scale vs. the right-oriented.

    ReplyDelete
  3. Anonymous12:30 am

    Did the students read the directions? I know I have started answering questionnaires one way because the scale is always the same way. Sometimes, when I notice the scale is reversed, I forget halfway through and answer how I am used to answering. I have heard it said that misinterpretted data is worse than no data. This should be studied further.

    ReplyDelete
  4. Anonymous3:54 pm

    In the 1970s until the 1990s I ran hundreds of surveys - we measured what people felt about the tv programmes they had seen; these programme appreciation index scales were always posed in the same direction (demands of bureaucracy) and I do not know if there was a bias, as according to the new findings. Incidentally, there might be different patterns of response from among left handers or those who had a first literate language which was semitic ...(reading/writing from right to left ...); I do not know; we didnt investigate ...
    There were also additional questionnaires about everything under the sun. again the response scales were printed all the same way
    but
    I often ran questions which presnted the same idea, expressed in an opposite direction thus (- I'll dig out exact quotes if anyone insists, but to illustrate the point, saying) "the monarchy is valuable and should continue to exist" then later on "the monarchy is outmoded and should be discontinued ..."
    the point is that the distribution of responses most usually was "harmonious" in that the proportion who agreed to one proposition tended to disagree with its opposite, and the correlation between the two items was markedly negative ...
    therefore, I reckoned we might trust the validity of the findings ....though there might indeed have been an element of shift in some cases ...
    J.M. Wober

    ReplyDelete
  5. Anonymous5:26 am

    A very interesting article, however do we know if the samples were matched (ie equal gender split, equal age ranges, equal split by course/lecturer, etc)? I think more testing may be required to be more sure. Most surveys I am involved with these days involve some form of rotation/randomisation in order to minimise bias and fatigue - I guess this article highlights one more issue to be aware of when designing formats for large scale surveys.

    ReplyDelete
  6. Anonymous9:10 am

    Having not read the article, I don't know if the authors recommended anything more interesting than reversing the scale for half the sample - this is not a practical option for anyone using a survey in a non-research context! For instance, many people in a business environment use '360' feedback questionnaires with rating scales. I wonder if up/down lists are treated differently from left/right ones.

    ReplyDelete