NSW Premier Barry O'Farrell has criticised comments by federal Attorney-General George Brandis that people have the right to be bigoted, declaring that vilification on the grounds of race or religion is ''always wrong''.

Opening a Museum of Sydney exhibition documenting the history of the Chinese community in Sydney on Thursday, Mr O'Farrell said the work was important because of ''issues that are happening at the federal level''.

''In commendably seeking to protect freedom of speech, we must not lower our defences against the evil of racial and religious intolerance,'' he said.

''Bigotry should never be sanctioned, whether intentionally or unintentionally. Vilification on the grounds of race or religion is always wrong. There's no place for inciting hatred within our Australia society.''

Senator Brandis made his comments this week in response to criticism by Labor of the federal government moves to change section 18C of the Racial Discrimination Act.

It was used to prosecute News Corp columnist Andrew Bolt over an article he wrote attacking ''fair-skinned'' Aborigines.

Under questioning from Labor Senator Nova Peris, Senator Brandis said: ''People do have a right to be bigots, you know. In a free country, people do have rights to say things that other people find insulting or offensive or bigoted.''

But Mr O'Farrell said Australians ''enjoy a history as a state and a nation of which we can be overwhelmingly proud''.

''But we must never forget that includes appalling examples of the consequences of intolerance and hatred,'' he said.

''No government,  no organisation, no citizen can afford to be less than vigilant in combating bigotry, intolerance and hatred. And frankly, our way of life depends on that vigilance.''

Senator Brandis released a draft exposure bill on Tuesday after a lengthy cabinet meeting on Monday night in which he was forced to water down his original proposal for changes to sections of the Act.

The draft allows the government's position to remain fluid and community groups to react. The changes proposed to the act in an exposure draft on release to the government party room on Tuesday contained a weakening of Senator Brandis' original proposals.

The outcome represented what one minister described as a compromise between the conservative and moderate factions. One minister said: ''George has really drunk the right-wing Kool-Aid.''

Another minister said Mr Brandis' original proposal was ''much worse'' than the agreed text and he had been forced to back down. A third minister present at the meeting said the original bill had been ''terrible''.

Asked if the cabinet had forced the change from a bill to an exposure draft, that minister said ''things are evolving all the time'' and that the exposure draft still ''needs to be changed quite substantially''.

The exposure draft released has proposed section 18C, which makes it unlawful for someone to act in a manner likely to ''offend, insult, humiliate or intimidate'' someone because of their race or ethnicity, would be repealed while section 18D, which provides protections for freedom of speech, will be removed and replaced by a new section.

The changes remove the words ''offend, insult and humiliate'', leave in ''intimidate'' and adds the word ''vilify'' for the first time.

But a  passage  in the exposure draft that exempts words and images "in public discussion of any political, social, cultural, religious, artistic, academic or scientific matter", has attracted a storm of criticism for being too broad and weakening current protections.

Small Business Minister Bruce Billson declined to give his view on the proposed legislation, citing cabinet confidentiality, but admitted building consensus around the changes would be a ''real test''.

''I'm a cabinet minister and my view is cabinet's view,'' he said.

''It will be a real test for the Australian community and political stakeholders whether this important issue can be discussed in sober, measured terms.''

''There is a fair degree of shrillness circulating at the moment, this is too important to have a battle of screaming banshees. There needs to be thoughtfulness and engagement and that’s what I would encourage.''

Labor's shadow attorney-general, Mark Dreyfus, said the ''extreme'' changes proposed by Senator Brandis would ''give the green light to racism and hate speech''.

''Having three cabinet ministers briefing against the Attorney-General in today's papers is proof the Abbott cabinet has no confidence in him,'' he said.

''They know it shows this government’s twisted priorities – not even Tony Abbott’s closest supporters want to be associated with this.

''This shows how completely untrue the Attorney-General's claim that his proposal is strengthening protections against racial vilification is.''

Mr Dreyfus said the five-week time frame for consultation on the proposed changes was also too short.

Environment Minister Greg Hunt, however, defended Senator Brandis on Thursday saying the Attorney-General had been ''badly misrepresented''.

''It's the distinction between two different things here. When somebody says something that is vilifying or utterly offensive and deeply inappropriate, what is the boundary between what should be criticised and what should be criminalised,'' Mr Hunt said.

''I said earlier on in the week there is no place for bigotry in Australia, the question is always what should be criticised and what should be criminalised.''

Victoria's Multicultural Affairs Minister, Matthew Guy, has also questioned the proposed changes, telling the State Parliament the Coalition government would formally raise its concerns with the Commonwealth.

"I am concerned there may be some harmful and unintended impacts upon our community should the exposure draft as it stands be enacted," Mr Guy said.

with Peter Hartcher and James Massola