Residents are fighting to hold on to the last remains of Georgian history in Hemel Hempstead.
Nash Mills House avoided demolition in 2008 thanks to protests by the local History And Museum Society to preserve the site.
A deal was struck for it to be used for housing, office space and community uses, through co-operation with the society, the developers and Dacorum Borough Council.
A new planning proposal wants to remove the community use for the building, making it completely residential.
Michael Stanyon, vice-president of the society, said: “I believe that members of our society may have opinions about the retention of this building which is one of the last survivors locally of the Georgian age.
“I hope people will join with me to request that our council provides a clear requirement that the building should retain as much of its historic character as possible.”
Residents were initially told that the ground floor would be used as for a coffee shop and a community meeting place.
Mr Stanyon said: “Local people may also complain about a lack of parking but my issue is about the history of the house.
“Ideally I would like to preserve the meeting room and I would like to see some kind of signage on the outside detailing the history. Everybody has been expecting a meeting room and coffee bar and now this new application has taken that away.”
One objector, who lives in Butterfly Crescent, said: “These plans do not suggest a sympathetic restoration project but merely a cramming exercise to maximise profit.”
A spokesman for Dacorum Borough Council said: “The application is still under consideration and the consultation is open for comments.”