Drunks and drug users have been accused of spoiling the picturesque Walled Garden within Gadebridge Park.
Months of hard work and preparation have gone into ensuring the enclosed area with links to Henry VIII is blooming this spring.
But amongst the sweet smelling hyacinths and colourful pansies are foul-mouthed drunks, according to visitors.
Grandfather Jeffrey Brain, who lives in the town, said: “All is ruined by a bunch of drunks and drug addicts swearing, fighting and being abusive. The smell of urine is quite disgusting.
“My grandson used to like wandering up the steps at the back of the gardens to sit on the wall and overlook the gardens.
“The police do move them on occasionally but they just move to St Mary’s churchyard.
“They openly carry and drink from beer cans, which I thought was against the law.”
Dacorum Borough Council’s gardeners start planning the annual impressive spring display during the previous summer and spend hours tending to the flower beds.
Among the beds on the northern wall of the garden is a plaque commemorating the men and women of Dacorum who served in the South East Asia Command and to the Burma Star Association, in memory of those who died in the Far East in World War Two.
The garden is the location of Hemel Hempstead’s original Bury House and it is claimed that King Henry VIII stayed there with Anne Boleyn. All that remains of the house today is the Charter Tower.
Another Gazette reader said: “The glorious walled secret garden in Gadebridge Park is now occupied by drunks.
“It ruins the experience and discourages visitors.”
The borough council’s antisocial behaviour team is now stepping up patrols of the garden.