Demolition workers at Martindale School have been rapped for starting an unauthorised bonfire with 15ft-high flames last week.
Staff from DSR Demolition started the fire which residents said led to fumes blowing into children’s bedrooms on Thursday evening.
Fire crews had to be called to the scene to put out the blaze.
Dacorum Borough Council has confirmed that the fire was not authorised, but added that no chemicals or harmful materials were burnt.
Resident Paul Mortimore told the Gazette: “My three-year-old twin daughters’ bedroom backs onto the site.
“I didn’t know what they were burning or if it contained asbestos, but the fumes were going into the girls’ window.
“They shouldn’t have been burning anything – and if that’s the case, who knew what was on that fire?”
Other residents claim that fires had been started on the site throughout the week but only Thursday’s blaze caused the emergency services to be summoned.
Mr Mortimore added: “The fumes were 15ft high and there was an awful stink.
“There were embers flying around all the trees too - any one of them could have gone up in flames.
“The fire service was called and got the fire under control, but I went there just after 11pm and it was still burning.”
Martindale School closed in 2008, although demolition work at the Warners End site only started a few months ago.
Dacorum Borough Council now owns the site, and plans to build 43 new homes there.
Julia Hedger, the council’s group manager for development said: “We have strongly addressed the issue with our contractor as we did not agree to this activity and have been reassured that it will not happen again.
“We apologise for any concern the residents had but want to reassure them that the air quality around the site is safe.
“All notifiable asbestos had previously been removed under the required controlled conditions.
“As well as the contractor monitoring the air quality, we also employed as an extra precaution an independent company to separately monitor the air quality.
“During the asbestos removal process there were no issues with our air testing or the air quality around the site.”
DSR Demolition refused to comment.
Anyone who has concerns regarding the site, should contact the council’s housing development manager David Barrett on 01296 228211 or Julia Hedger on 01442 228363.