Robert Muzzy was one of the first settlers in Ipswich and was admitted a freeman on September 3, 1634. Benjamin Dutch bought property from his estate on the west side of the intersection of Summer Street with County Street, and built his house at 9 Summer Street. In 1718 he sold Samuel Harris the lot at 11 Summer Street, who sold it five years later to Joseph Bennett.

Joseph Bennett built this early Second Period house in 1725. In that same year his 25 year old wife Elizabeth died. Bennett then married Mary Jewett, but he died 6 years later at the age of 31, leaving Mary a widow. Early Massachusetts law allowed women to keep only one third of the estate, and Joseph Lakeman Ross purchased the remaining interest. Ross, his wife Mary, and the widow’s (apparent) daughter, a spinster also named Mary Bennett sold the entire house to Daniel Holland in 1796.

In 1818 the house was sold to Capt. Sylvanus Caldwell, who engaged in maritime trade along the coast from Massachusetts to Maine for a half century and was noted for his honesty and integrity. Sylvanius Caldwell died in 1864. A correspondent of the Kennebec Journal gave the following tribute to his memory, recorded in the Caldwell family records:

windowview
Looking through the Indian Shutter windows in the Bennett-Caldwell house toward the Benjamin Dutch house on the corner of County and Summer Streets.

Died in Ipswich, Mass., Aug. 26 1864, Capt. Sylvanus Caldwell, in the 78th year of his age. In the community where he was born, where he reared his family, and where he has ended his well-spent life, the tribute to the memory of Capt. Caldwell is one of universal and profound respect. In this State, and especially in the valley of the Kennebec, he has been well known for nearly half a century from continued business connections in which his engagements were always performed with the utmost punctuality and honor. Capt. Caldwell was a fine specimen of the best type of New England character. He was honest, brave, self-reliant, modest and generous ; and to these traits and gifts of nature was super-added the grace of an earnest religious faith, manifesting itself in a life of personal purity and in daily deeds of charity and good will. He was a man of great industry, having no time to idle, and yet so fond of society and of friends that he was never too busy for their enjoyment. His house was the abode of hospitality, and a wide circle of relatives and acquaintances will recall many a pleasant hour passed beneath his roof. He was a cheerful giver ; and ‘ though a man of moderate means his habits of punctuality and economy enabled him to contribute liberally to all worthy objects. And what he gave he had a right to give. Every dollar was his own, honestly earned without wrong to any man. His death, though sudden, was not unprepared for, and probably not unexpected. He was a sheaf fully ripe. His work in this world was done. And she who had shared his joys and sorrows for more than fifty years preceded him but a few months. Together they had reared a large family — all of whom are comfortably and honorably located in life — and together they have gone to their reward. 

Caldwell’s daughter Sarah Perkins Caldwell lived her life in the “comfortable old mansion” and died in 1919 at the age of 86. Interior features include original Indian shutters throughout, pine floors in the bedrooms, exposed beams in the dining Room, raised panel doors, and a feather-painted door in the master bedroom.

Sources: 

Photo of the Bennet-Caldwell house in the late 20th Century, from the MACRIS site
Photo of the Bennet-Caldwell house in the late 20th Century, from the MACRIS site