Can you tell a snake from a pretzel? Some can’t – and their experiences are revealing how the brain builds up a coherent picture of the world

AFTER her minor stroke, BP started to feel as if her eyes were playing tricks on her. TV shows became confusing: in one film, she was surprised to see a character reel as if punched by an invisible man. Sometimes BP would miss seeing things that were right before her eyes, causing her to bump into furniture or people.

BP’s stroke had damaged a key part of her visual system, giving rise to a rare disorder called simultanagnosia. This meant that she often saw just one object at a time. When looking at her place setting on the dinner table, for example, BP might see just a spoon, with everything else a blur (Brain, vol 114, p 1523).

BP’s problems are just one example of a group of disorders known collectively as visual agnosias, usually caused by some kind of brain damage. Another form results in people having trouble recognising and naming objects, as experienced by the agnosic immortalised in the title of Oliver Sacks’s 1985 best-seller The Man Who Mistook His Wife for a Hat.

Agnosias have become particularly interesting to neuroscientists in the past decade or so, as advances in brain scanning techniques have allowed them to close in on what’s going on in the brain. This gives researchers a unique opportunity to work out how the brain normally makes sense of the world. “Humans are naturally so good at this, it’s difficult to see our inner workings,” says Marlene Behrmann, a psychologist who studies vision at Carnegie Mellon University in Pittsburgh, Pennsylvania. Cases like BP’s are even shedding light on how our unconscious informs our conscious mind. “Agnosias allow us to adopt a reverse-engineering approach and infer how [the brain] would normally work,” says Behrmann.

Although we may not give it much thought, our ability to perceive our world visually is no mean feat; the most sophisticated robots in the world cannot yet match it. From a splash of photons falling on the retina – a 3-centimetre-wide patch of light-sensitive cells – we can discern complex scenes comprising multiple items, some near, some far, some well lit, some shaded, and with many objects partly obscured by others.

The information from the photons hitting a particular spot on the retina is restricted to their wavelength (which we perceive as colour), and their number (which determines brightness). Turning that data into meaningful mental images is a tough challenge, because so many variables are involved. For example, the number of photons bouncing off an object depends both on the brightness of the light source and on how pale or dark the object is. “The information that the visual system receives is very impoverished,” says Behrmann.

It is in the visual cortex, located at the back of the brain, where much of the processing goes on. When items obscure each other, the brain must work out where one thing ends and another begins, and take a stab at their underlying shapes. It must recognise things from different perspectives: consider the image of a chair viewed from the side compared with from above. Then there’s the challenge of recognising novel objects – a futuristic new chair, for example. “Somehow, almost magically, we derive a meaningful interpretation of complex scenes very rapidly,” says Behrmann. “How we do this is the million-dollar question in vision research.”

So how does the brain work its magic? In the early 20th century, European psychologists used simple experiments on people with normal vision to glean some basic rules that they called the “gestalt principles”. For example, the brain groups two elements in an image together if they look similar, having the same colour, shape or size, for example. And if not all of an object is visible, we mentally fill in the gaps – that’s the “closure principle” (see “Constructing reality”).

The gestalt principles can only go part of the way to describing visual perception, though. They cover how we separate the different objects in a scene, but they cannot explain how we know what those objects are. How, for example, do we know that a teacup is a teacup whether we see it from above or from the side, in light or in shadow?

It’s here that people with visual agnosias come in handy. Behrmann had previously studied people with integrative agnosia, who have difficulty recognising and naming complex objects as a whole, and instead seem to pay unusual attention to their individual features. One person, for example, mistook a picture of a harmonica for a computer keyboard, presumably thinking the row of air-holes in the mouthpiece were computer keys (Journal of Experimental Psychology: Human Perception and Performance, vol 29, p 19). Others have mistaken a picture of an octopus for a spider, and a pretzel for a snake.