If Anthony Albanese has a fatal flaw – and every politician has one – it is his stubbornness.

That flaw has been on full display this week as the prime minister has been engulfed by a scandal entirely of his own making: melon-gate has embarrassed and damaged Albanese’s standing, to Australia’s detriment, and hurt Australia-Japan relations.

It has also hurt Albanese’s standing with his colleagues. During private conversations this week with Labor people, across factions and states, from the backbench to the cabinet, it’s clear they remain unconvinced by the prime minister’s explanations and excuses.

It was nearly six weeks ago that Albanese appeared on a bawdy podcast, Bush Deep (the title is a big hint), with comedian Nikki Osborne. He took the bait when she asked him about Japanese Prime Minister Sanae Takaichi’s gift of a pair of crown melons during her visit in May. Japan’s former ambassador to Australia Shingo Yamagami reignited the matter when he wrote a blistering opinion piece in The Australian, labelling Albanese’s gesture “insulting” and ignorant.

Yamagami, a political conservative who remains close to Coalition figures including Tony Abbott and Peta Credlin, told me this week it was up to Albanese whether to apologise to Takaichi.

Initially reluctant to address it, Yamagami said he changed his mind after reading a piece in the English-language The Japan Times by journalist Waka Ikeda. The article carried the headline “Sanae Takaichi: The prime minister nobody defended” and the kicker: “Anthony Albanese’s melon joke revealed which women leaders the world feels safe mocking.” It took feminists to task for not defending the conservative Japanese prime minister.

“Mutual respect should be the basis of our ties,” Yamagami told me. “This was at the expense of the first female prime minister of Japan. So this aspect of the issue needed to be made clear to the Australian audience.”

He added: “I was rather hesitant to write this article because, as far as I’m concerned, this joke by Albanese was, candidly, at the undergraduate, locker-room level. But after I saw that column in The Japan Times, I thought … perhaps as a former ambassador I could contribute.”

Albanese might have played a straight bat to Osborne’s question comparing the melons to two of actor Pamela Anderson’s better-known assets. Instead, he tried to out-funny the comedian.

It was a mistake, and one that politicians often make in so-called “relatable” interviews on podcasts or FM radio. They’re supposed to showcase the “real” person behind the politician. Recall Tony Abbott’s infamous comment in 2012 that then-prime minister Julia Gillard should not be “swanning around in New York talking to Africans”. Gillard was lobbying, successfully, for Australia to win a two-year spot on the United Nations Security Council.

But politicians, whether the prime minister or an obscure backbencher, are held to a higher standard because they are our representatives. In interviews like this, the key is to play the straight man, even if you end up as the butt of a few jokes. You must safeguard the dignity of elected office. For a moment, Albanese forgot that.

That was Albanese’s first mistake.

His second mistake was digging in over the need to apologise or walk back the “couple of melons” comment, as he had already done when he suggested on the same podcast that Kylie Minogue was his pick in a game of “shag, marry, date”.

The prime minister’s stubborn refusal to apologise when the issue blew up again this week has only elevated it – to the point that it has dominated political debate for days.

And there’s a pattern now of denying the blatantly obvious: think of Albanese’s insistence that he didn’t stumble on stage during the election campaign despite footage showing otherwise; remember his implausible explanation after he drew fire for describing Grace Tame as “difficult”; consider his refusal to delay, if not cancel, the Voice referendum when it was clear it would lose.

This practice of refusing to take a backward step damages Albanese more than the missteps. It entrenches cynicism about politicians when they deny what we can all plainly see. “She brought two – as you do,” he said, gesturing in front of his chest, in response to Osborne’s Pamela Anderson gag.

And it has been excruciating watching a conga line of senior Labor women, including Minister for the Status of Women Katy Gallagher, Foreign Minister Penny Wong, Housing Minister Clare O’Neil and Social Services Minister Tanya Plibersek, having to defend Albanese. It’s worth noting, at this point, that Albanese’s personal approval in the Resolve Political Monitor trails the Labor Party’s standing in the same poll. He is, arguably, a drag on Labor’s vote.

As one member of cabinet told me: “It hasn’t been ideal. He should have just apologised. And it’s super awkward for people like Tanya and Katy, who have been defending him. The way it has blown up now, I worry about the long-term damage it could do to the [Japan-Australia] relationship.”

Clearly, Japan would have preferred he apologised and moved on, given its nothing-to-say-and-nothing-to-see-here statement on Wednesday afternoon.

But after nearly 40 years in the political trenches, working as a staffer and then as an MP, the prime minister has developed a thick skin. You don’t get to the top of politics, from the minority Left faction in a state branch that for generations has been run by the Right, without it. It’s not as thick as some of our former leaders – and if there were an award for prime minister most likely to tear up at a moment’s notice, Albanese would win it.

But he is not used to taking a backward step. He rarely loses out in policy or political fights these days, and he avoids giving ground to his opponents at all costs. That stubborn streak and refusal to back down have been to his detriment on this and several other notable occasions.

Rather than leaning on his talented female ministers (and some men) to defend him, Albanese should have fronted up and owned his mistake. It’s probably too late for that now, given Tokyo’s stance, and given how many of his colleagues have been sent out to defend the indefensible.

But he needs to learn the art of the strategic retreat.

James Massola is chief political commentator at The Sydney Morning Herald and The Age.

The Opinion newsletter is a weekly wrap of views that will challenge, champion and inform. Sign up here.