Why lie? I don’t mean the small, social lies that keep things running smoothly, but what appear to be enormous ones – invented academic posts, fabricated childhood, feats of endurance that never happened.

And having decided to lie, why choose stories so extravagant that they invite disbelief, and so specific that anybody with a spare afternoon and an internet connection can take them apart?

These are questions I’ve been asked about Jason Arday, who resigned last week from his position as professor of sociology education at Cambridge, after the university announced an investigation into his qualifications.

He denies being a liar, he has explanations for his claims about endurance running and fundraising, and when Liverpool John Moores University examined plagiarism allegations against his 2015 doctorate they were not upheld. Nothing has been proved against him and the investigations remain open.

I should say that I’ve never met Jason Arday, and so have no idea what has been going on in his mind, or how much truth there may or may not be in his life story. But having spent 20 years as a psychiatrist listening to people who tell me things that turn out not to be true, I can speak generally about why people find themselves in these sorts of situations, because the pattern is well established.

There is an old descriptive psychiatric term, pseudologia fantastica, coined in the 1890s for people whose lies are elaborate, sustained and built around a fantasised version of themselves rather than for obvious material gain.

You may think that hardly fits here, given this story apparently brought Jason Arday a professorial chair, several honorary doctorates and, he claimed, a 1.4million book deal.

But the gain is usually a by-product of this pattern of behaviour rather than the object, and you can usually tell this from the type of lies.

Jason Arday resigned as a professor at Cambridge University after an investigation was launched into his qualifications

Someone who, for instance, wants a certain job often embroiders their qualifications. But they stop there, as anything beyond that is pure risk, and tall tales about marathons they’ve run, funds they’ve raised and the desperate childhood they’ve endured do little for a selection panel except draw questions that up the chance of being caught out.

The people I’ve met who fit the description of pseudologia fantastica tend not to stop lying. They are seldom cold or calculating. They tend to be charming and, perhaps most importantly, fragile underneath. What strikes me most is how often they appear to half-believe the story they’ve invented. The lie is not really being told to the listener at all, it is being told to the ego of the person telling it. The listener is only there to confirm it.

Lying, I think, starts with the memory, which is not really a straightforward recording of life’s experiences, more a reconstruction of events. Every time we remember or retrieve something from our memory, we rebuild it and return it slightly altered.

All of us do this, because in our minds we tend to shift to the version of ourselves we want to be.

Someone comfortable in their own skin has little to gain from the process, so the edits stay small. Someone who is not comfortable will find that each retelling shaves away a little more of the person they cannot stand being, and by the time a story has been told two hundred times it has not only been changed but rehearsed so much it eventually begins to feel like a real memory.

The stories are hardly ever straightforward boasts of success either, because someone who simply claims to be clever or rich invites envy – and envy is not what these people are looking for.

Instead the tales are about suffering that has been overcome, odds that were insurmountable, the diagnosis nobody believed, the childhood that should not have been survived, and then the triumph over all this. It is a far more effective structure for any fantasy, because it makes doubting it feel like an act of cruelty.

It is also the structure our whole culture of aspiration is built on, the one that fills memoirs and Ted talks and the back-stories on TV talent shows. This is why anyone constructing an identity out of borrowed material will reach for the template. It’s the stuff of fairytales and legends, the struggle in the face of adversity, a struggle that demonstrates positive things about an individual’s personality and inner character.

Leonardo DiCaprio seen starring in the hit film Catch Me If You Can, about a con man who poses as a pilot, doctor and lawyer

I once saw a man, and I have changed key details that might identify him, who had claimed to the woman he married that he was a barrister. He left the house in a suit every morning and came home with stories about court.

What struck me was that he could not tell me exactly when he had decided to lie. There had been a conversation at a party where somebody misheard him and he did not correct them.

It’s easy to see how it can happen. It was a series of slow, incremental lies that built and built.

Each time he got positive feedback from those around him and he embellished a little more. Each time the feedback provided a sort of affirmation until the story had grown into something unrecognisable from where it started.

It was after this that he met his future wife and had to double down on the deception.

When I saw him he had spent almost a decade maintaining a fictional life that had begun with an innocuous misunderstanding at a party.

There is a neurological basis to this. Researchers at University College London put volunteers in a scanner and let them lie for their own benefit. They found the amygdala, an area of the brain that drives emotional response to wrongdoing, lit up strongly at the first lie, then less after each one.

The size of that fading response predicted the size of the next lie. Our brain starts to extinguish that awkward, guilty feeling. The guard-rails that prevent us living out a fantasy are easily moved by incremental falsehoods.

Why do the stories become so extravagant? Part of it is this notion that an audience gets used to them, just as the exponent of the lie does. So what produced gasps last year produces only nods this time, and applause that has stopped coming leaves an empty feeling.

So the story has to grow, and it grows in whatever direction was rewarded most loudly last time. You can’t backtrack once a story is out there, you can only embellish and elaborate.

And why make up stories that are so easily disproved? A conman would never behave like this, as the object is to get away with it. The fantasist is running no calculation at all, as there is no auditor or moderator in their head.

The only person they picture on the other side of their story is somebody looking impressed. A claim is not weighed for whether it will survive scrutiny, it is weighed for whether it will trigger the required impression and response. The claims that do best in that regard are precisely the ones nobody sensible would invent.

Every fantasist I’ve known kept going with their stories long past the point where anybody was still convinced, because the alternative was to confess it was a sham and allow people to destroy the false identity they had built for themselves, so become a nobody.

Which is why I don’t feel triumphant about Jason Arday’s downfall. When a world collapses it is a terrible thing to live through, and no less terrible for having been constructed of tissue paper.

He built it and must answer for it. But he did not build it on his own, and the institutions – not least Cambridge University and the diversity, equality and inclusion industry – that made him their poster boy and handed him the bricks have gone very quiet.

For that they bear considerable responsibility in this sorry saga and should reflect on it in shame.