The neighbour of a new Wednesday market says its traders arrive as early as 3am and wake him up every week.
Stuart Wells, 38, of Berkhamsted High Street, has two daughters and says the night-time noise is making them sleepy at school.
He said if it does not get quieter, he will consider legal action against the market’s organisers.
He said: “I have been a resident here for seven years and think we have got a right to peace and quiet on working days. We have never once complained about the weekend markets.”
He says the market had made High Street traffic a lot worse, too.
Organisers Emy and Michael Smith said they have ordered quieter equipment to unload their stock and employed another worker so that they can set up later in the morning.
But Mr Smith claimed Mr Wells had been aggressive towards him and used bad language.
Mr Wells said: “At three o’clock in the morning, if you are crashing around outside someone’s property, you are not going to get an: ‘Excuse me, do you mind keeping the noise down?’
“You are going to get a: ‘Get the hell out of here.’”
Berkhamsted Town Council previously complained that the first it heard about the new market was through the Gazette.
Mr Wells said: “The traffic noise, the lorries and the set-up of market noise is horrendous.
“They have gone and taken two hours of my sleep pattern away from me without even consulting me.
“And the other night, they took five hours off me without so much as an apology from the organisers.”
The Smiths said they did not contact the council as the market is ‘not really anything concerning them’, but said councillors could talk to them at their fruit and veg stall on any Wednesday.
Mr Smith said: “We have not come along to try and upset people. We have come along to try and give the people of Berkhamsted what they want.”