BROME HALL, FOR FOUR CENTURIES ONE OF THE SEATS OF THE CORNWALLIS FAMILY, STOOD NEAR THE SUFFOLK/NORFOLK BOUNDARY BETWEEN EYE AND DISS. The family inherited the manor of Ling in the parish of Brome early in the fifteenth century following the marriage of Philippa, heiress of Robert Buckton, to John, son of Thomas Cornwallis, a London merchant. His descendant Sir Thomas was Comptroller of the Household in the reign of Queen Mary. After her death, he was suspected of being a Catholic recusant and is said to have retired to Brome, where he rebuilt the hall, which remained in his family until 1823.

The Cornwallis family was prominent in public life through the seventeenth and eighteenth centuries, providing an ambassador to Spain in the reign of King James I, a Treasurer of the Royal Household and Surveyor of Customs in the reign of King Charles II and a Paymaster General in that of King George I. Frederick Cornwallis was raised to the peerage by King Charles II, and the fifth Baron was created Viscount Brome and Earl Cornwallis in 1753. His son Charles served in the American War of Independence where his involvement in the surrender of Yorktown in 1781 does not appear to have affected his prospects. Five years later he became Governor General of Bengal where he served until 1793 and again in 1805. He died in India soon after his return there in 1805 and was buried at Ghazipore near Varanasi (Benares), having been promoted to the rank of General, made a Knight of the Garter and created a Marquess.

The Cornwallis family inherited the Culford estate near Bury St Edmunds in the seventeenth century, and in the early 1790s the first Marquess commissioned the remodelling by James Wyatt of his Tudor house there. Brome became the family's secondary seat and rebuilding there did not take place for another twenty years until after the second Marquess had inherited the estate. He, however, died without children in 1823 and the Cornwallis family's ownership of Brome came to an end with its sale.

The purchaser was Matthias Kerrison, the son of a Norfolk clergyman, who had made a substantial fortune in the grain trade in Bungay, where he also had other business interests including the development of the navigations on the Ouse and Waveney rivers.