When I was growing up there in the 1960s, Petersham was not the neighbourhood of well-to-do Greens and Bollinger socialists it has since become. It was then an unfashionable suburb, relatively poor, much favoured by migrants who had fled war-ravaged Europe to seek a new life in Australia. Many of the families on my street were from Greece and Italy. Our next-door neighbours were Chinese.

At the local convent school, St Thomas’s Lewisham (where one of my classmates was Greg Sheridan, who would become the prominent journalist), the kids came from the same diverse mix of migrant families. The nuns taught us that all souls are equal in the sight of God.

The word “multiculturalism” hadn’t been invented then, but for me, multicultural Australia is the only Australia I have ever known. Anthony Albanese grew up in much the same milieu in neighbouring Marrickville. Tony Abbott, raised in the then monocultural suburbs north of the harbour, no doubt had a different childhood experience.

The “White Australia policy”, which had been a bipartisan consensus since Federation, was formally rescinded at that time, when the Holt government passed the Migration Act of 1966. This was the culmination of efforts by several coalition ministers from the late 1950s onward. The immigration minister, Sir Hubert Opperman, a leading critic of the policy, played an important role in unwinding it.

Sir Robert Menzies himself had always been uncomfortable with it. As well as his moral objections, Menzies was concerned about the message the policy sent to the Asian nations with which Australia was increasingly engaging. It would be the Colombo Plan – the visionary scheme which brought thousands of Asian students to study at Australian universities – which opened the door to non-European migration, since Opperman’s reforms allowed them to become permanent residents, then citizens, after they had finished their studies.

When Pauline Hanson recently regretted the ending of the policy “by Gough Whitlam”, she was falling victim to one of the most commonly practised frauds on Australian history – Labor claiming credit for the liberalising reforms of Liberal governments. While Whitlam’s government removed the last administrative vestiges of the policy, by the time he was elected in 1972 it was already a dead letter. As the leading historian of the Menzies government, David Kemp, has written, Menzies and his ministers “did most of the heavy lifting” in its abolition.

Their approach was incremental. Menzies – who always avoided using the term “White Australia” – was mindful that the policy enjoyed strong public support. That support was reinforced by the trenchant opposition of Labor and the trade unions to any change. Arthur Calwell, Labor’s leader until 1967, was a strident advocate of White Australia, often in explicitly racist terms. He famously once told parliament: “Two Wongs don’t make a white.” (Calwell later claimed he had been making a pun on the surname of Menzies’ first immigration minister. Sir Thomas White. Why he thought that made the remark any less racist was never explained.) Calwell’s successor Whitlam drove the change of attitude within Labor from the mid-’60s.

Menzies detested racism. In April 1942, at a time when Australians lived in daily fear of Japanese invasion, he made a remarkable radio broadcast (part of the “Forgotten People” series) condemning the Curtin government for inciting racial hatred of the Japanese in its wartime propaganda. “It is, in my opinion, poor policy to try to persuade people to despise the Japanese,” said Menzies. “It is an offence to an honest citizen to imagine that the cold, evil and repulsive spirit of racial hatred must be substituted for … a noble and decent cause.”

Menzies’ attitude echoed that of his father, James, a state MP, who in August 1916, at the height of wartime anti-German frenzy, rose in the Victorian parliament to defend German settlers, many of whom were his constituents, and vouch for their patriotism. The occasion was a debate on legislation to close Lutheran schools, which the government justified as a wartime security measure partly on the grounds that some classes were taught in the German language. Menzies’ speech was defiant in its disdain for fear-mongering, racial populism. Young Robert Menzies evidently learnt the virtues of liberal tolerance at his father’s knee.

Menzies, who was immensely proud of his own Scottish heritage, welcomed cultural diversity. He said in 1963, when laying the foundation stone of the Kew Jewish Centre: “There are a lot of people you will encounter who think that we all ought to be the same, that there ought to be a species of drab uniformity ... This passion for uniformity is not to be encouraged. What we need in Australia is that every person coming from some particular race or faith who comes into the Australian community should retain his quality in that sense, but should add it to the qualities of all the other people in the community …”

Twelve years later, Malcolm Fraser, then-opposition leader, in a speech written by his 27-year-old adviser Petro Georgiou, formally adopted the policy of multiculturalism, the architecture of which he went on to construct as prime minister through many initiatives, notably the creation of SBS.

Recently, Abbott upset a lot of Liberals – and repudiated the policy of more than half a century of Liberal governments (including his own) – when he pronounced multiculturalism a failure. In the same speech, he also said “diversity is not our strength”. His words bespeak an outlook very distant from Menzies’ liberalism, with its commitment to a culturally tolerant Australia, all the richer for its diversity.

If only Tony had grown up in Petersham.

George Brandis is a former high commissioner to the UK, and a former Liberal senator and federal attorney-general. He is now a professor at the ANU’s National Security College.

More: