HARDWICK AND ITS NEIGHBOUR HENCOTE WERE MANORS HELD BY THE ABBEY OF BURY ST EDMUNDS from 945 until the Dissolution of the Monasteries, when Hardwick was granted to Sir Thomas Darcy, Lord Darcy of Chiche, and Hencote to the Drury family of nearby Hawstead. Hardwick was owned by a number of families in the latter part of the sixteenth century, until in 1610 it was purchased by Sir Robert Drury, thus bringing into common ownership the Hardwick, Hencote and Hawstead estates. In 1656 Sir Robert's wife's nephew, Sir Christopher Wray of Ashby, Lincolnshire, sold the property to Sir Thomas Cullum, first baronet (who had been Sheriff of London in 1646) for £17,787. The Cullum family owned Hardwick House for the remainder of its existence.

The Drury and Cullum families were associated with a number of other Suffolk houses which have been lost. Sir Robert Drury's wife was a daughter of Sir Nicholas Bacon of Redgrave. The second Cullum baronet, also Sir Thomas, married Dudley, daughter of Sir Henry North of the Manor House, Mildenhall. Her sister, Peregrine, married William Hanmer whose son Thomas's marriage to Elizabeth Folkes brought Barton Hall into his family. Hawstead Place was abandoned as the Cullum residence in the 1730s, and was gradually reduced in size over the next ninety years until it was finally demolished. Redgrave, Mildenhall and Barton have also disappeared.

Hardwick House, which stood some one and a half miles south of Bury, was a Jacobean house of about 1612 thought to have incorporated the medieval Abbey Lodge. The portico surmounted with the Drury arms in stonework survived until the demolition of the house. Sir Robert Drury was, it is said, responsible for the installation in 1612 of panelling with sixty-eight painted panels which were removed from Hawstead. These were included in the sale of the contents of the house in 1924, when they were described as having been in the Painted Closet, which was ‘probably the Oratory of the last Lady Drury’.

The Cullum family was distinguished for its historical and scientific interests: plants and gardening particularly were a continuing passion over many generations. Sir Thomas Cullum, the second baronet, was the noted gardener who built a greenhouse fifty-eight feet by fourteen feet (very large for its time) at Hawstead.