Why do some of us believe in ghosts, and others don't?

It's Halloween, which means the dead are walking the earth and we're celebrating by stuffing our faces with sweets.

But while this minor celebration of all things undead comes just once a year, it turns out we believe in ghouls and ghosts all year round. Recent studies by YouGov in the UK and the USA show that between 30% and 50% of the population says they believe in ghosts.

But why? The idea of a bodyless soul or spirit walking around defies all the laws of physics we currently spend most of our lives living by. So what's different? Well, our belief in ghosts comes from two places: our heads, and our hearts.

A head for hallucinations

Firstly let's look at what's happening in our heads. The catch-all term for spotting something out of this world is a "hallucination", which is broadly defined by the NHS as an event where someone sees, hears, smells, tastes or feels things that don't exist outside their mind. And they are surprisingly common.

As far back as 1892, the Society for Psychical Research began collecting thousands of first-hand reports of people seeing, hearing or feeling the presence of a recently deceased person.

More recently, research suggests that the majority of elderly bereaved people experience visual or auditory hallucinations of their departed loved ones that persist for a few months. There are several reasons why this could be happening, according to neuroscientists.

To best of our knowledge, the way our brains work is by building a model of the world based upon information it receives from our bodies, and information it may have had on an experience that feels similar.

Sleep states 💤

Because our brains can't go out there and touch things for themselves, it has to build a rough idea of what that might be like to experience. Most of the time it does a pretty good job, which gets confirmed to us by other people reporting on feeling and sensing the same thing.

But sometimes, it gets it wrong. That could be because we might be ill - literature is full of people having hallucinations while sick - taking drugs - be it medicinal or recreational - damage from a stroke or injury, or even just feeling tired.

When the brain is deprived of a stream of information or activity, it fills in the gaps to ensure the story remains consistent. This is what happens when people lose limbs in a condition known as Phantom Limb Syndrome, for example. Patients can still sense their limb when it's no longer there.

Almost all amputees feel that their lost limb is still part of their body, and can even experience pain in the space where their limb once was. The brain is quite literally filling in the gaps.

One of the most common areas where people report seeing ghosts is when they're in that period between being awake and being asleep - see Freddie Kruger for more details. When we're in those fuzzy moments, the brain is doing a lot of guesswork, which opens the door for hallucinations.

For some of us, we'd dismiss what we're sensing as being half asleep. For others however, they take those messages and interpret them as more supernatural. This is where our hearts come in.

A heart for horror 💗

The next part of why we believe in ghosts, and what makes a bump in the night feel like we're living in a haunted house is our willingness to believe.

Consider what might happen if you were in a reputedly haunted house at night and you saw something moving in the corner of your eye.

If you believe in ghosts, you might interpret what you saw as a ghost. This is what psychologists call an example of top-down perception in which what we see is influenced by what we expect to see.

When it's dark, it might be difficult to see properly, so our brain fills in the gaps and leaves it up to us to decipher whether what we saw is what we think is likely – a ghost or a gust of wind.

When we believe in a phenomenon, we are more likely to experience it. This is why history is stuffed with supernatural visions. Before scientists began unpacking the brain and how it works, we had little to no understanding over how it might be filling in the gaps.

The interpretation then falls to how our societies interpret those gaps. In the days of Shakespeare, if you were poor life was hard, and often cruel and unfair.

A desire for justice and the belief in karma, or supernatural protection (which exists in almost all major religions) helps people make sense of their crappy experiences. Shakespeare used this idea to great effect

In Hamlet, the protagonist is visited by the ghost of his murdered father seeking revenge on his murderer. In Macbeth, meanwhile, the murdered Banquo points an accusing finger at the man responsible for his death.

What a ghoul believes

Those ideas continue to endure today. In Kenya, a murdered person may become an ngoma, a spirit who pursues their murderer, sometimes causing them to give themself up to the police.

In Russia, the rusalka is the spirit of a dead woman who died by drowning and now lures men to their death. She may be released when her death is avenged.

Not all spirits are justice warriors, but the belief in the supernatural, paired with a desire to make sense out of what can be quite often seemingly random or unfair events, can push us to believe in almost anything.

It might give us comfort when trying to come to terms with the death of a loved one. It might give us a little dash of excitement to know we can hang out with the dead. Or it might be nothing more than a brain fart, a quirk in the infinitely complex grey matter in our heads. It all depends on what you choose to believe.

According to the Dutch philosopher Baruch Spinoza, belief comes quickly and naturally, whereas scepticism is slow and unnatural.

And at this time of the year, entertaining the afterlife is a welcome distraction to some of the more challenging issues we're facing together.

Happy Halloween everyone.

Ok, where can I learn more?

• What do hallucinations tell us about the brain? Over on nature, they take a fascinating look at what happens when the brain's wiring goes wrong, and how it's helped us learn about the mysteries of perception.
• Lessons from a terrified horror researcher - Head on over to the TED stage to hear horror researcher Mathias Clausen talking about the science of why we love to be scared half to death.
• The Strange Order of Things - Professor Antonio Damasio takes you on a brilliant journey into how the way you see and exist in the world is influenced by everyone that has come before you.