Following the death of Miss Florence Mathews, the City of Norwalk purchased the 20-acre Elm Park estate for "for park purposes," funded by a $200,000 bond issued by the CT Assembly. In 1942 the City sold the Mathews furnishings and the Lockwood carpets at a local auction so that it could use the Mansion to house some of its war offices, then its planning department, and to store voting machines and records. The City soon demolished the extensive greenhouses and gave away the iron fence surrounding the estate. During this time the south end of the property was condemned for interstate I-95, and later more land was taken for its interchange with Route 7. Also in the 1950's a brick building to house a Police Station, Court House and Jail rose up behind the Mansion and a Department of Public Works garage was constructed on the site of the greenhouses.
Norwalk Planning Commission in the Billiards Room, 1952
Finally in 1959 when the City announced that they planned to demolish the Mansion there was a public outcry. Angry citizens incorporated to form the Common Interest Group to preserve for public use what was left of the park and the abandoned Mansion. A public referendum in November 1962 "to renovate and preserve" the Mansion passed by a three to two vote but was ignored by the Norwalk Common Council, even with the personal appearance and testimony of Dr. William J. Murtagh, the Director of Education for the National Trust for Historic Preservation, who declared: "...the magnificent interior of the building has the best frescoed walls I have ever seen in this country, and the lavishness of the marble and wood inlay work almost defies description in the museum quality of its workmanship..."
The Common Interest Group then took the case to court to prevent the City from carrying out its demolition plans and won. But the struggle was still not over because the City appealed. Ultimately the Connecticut Supreme Court of Errors in its unanimous decision upheld the lower court's decision against the City in what has become a landmark decision affecting other historic properties in America. This all transpired prior to Congressional passage of the National Historic Preservation Act of 1966.
With a seriously deteriorated Mansion now on its hands in 1965, and the Common Interest Group without sufficient funds, the City and its new Mayor responded gladly to a proposal from the Junior League of Stamford-Norwalk, with members from the suburbs of Westport, Wilton, Ridgefield, New Canaan, and Darien as well as from the cities of Norwalk and Stamford, to lease the building from the City. The agreement stated that the Junior League would establish a Museum Corporation board which would include its own members, members from the board of the Common Interest Group, and members of the general public; it would restore and operate the Mansion as a public museum, a national center for Victorian studies, and a community center for exhibits and concerts.