FORNHAM HALL STOOD IN THE PARISH OF FORNHAM ST GENEVIEVE SOME THREE MILES NORTH-WEST OF BURY ST EDMUNDS, and the manor belonged, prior to the Dissolution of the Monasteries, to the Abbey there. In 1539/40 Fornham passed with nearby Hengrave to Sir Thomas Kytson and later, following the death of his son Thomas, to Sir William Gage. It then belonged to the Gipps family before the estate was purchased in 1731 by Samuel Kent, a London distiller, who left it to his son-in-law, Sir Charles Egleton, a London goldsmith, who had already acquired land at neighbouring Fornham St Martin. On Egleton's death in 1769 the estate passed to his son Charles, who changed his name to Kent and was knighted in 1782. It was he who was responsible for the building of the house, but he did not retain the estate for many years after its completion, selling it in 1789 to Bernard Edward Howard who became the twelfth Duke of Norfolk in 1815.

During the Duke's time substantial alterations to the house were undertaken but, after his death in 1842, the estate was sold to Lord Manners, a kinsman of the Duke of Rutland, for a reputed figure of £75,500. The next owner, from 1862 until his death in 1896, was Sir William Gilstrap, a Nottinghamshire brewer, who paid £85,000 for the property and enlarged the estate by purchasing land in the adjoining parish of Fornham All Saints. From him the estate passed to George Espec John Manners, whose wife was a Gilstrap and who himself was a descendant of both the thirteenth Duke of Norfolk and the fifth Duke of Rutland. When George Manners, who was knighted in 1920, died in 1939 Captain Duncan MacRae of Dunoon became life tenant under Sir William Gilstrap's will. It was during his tenure of the estate that the house was sold, with demolition following shortly thereafter.

WHEN CHARLES KENT decided to improve the existing house at the beginning of the 1780s he initially consulted Lancelot ‘Capability’ Brown who visited Fornham in February 1782 and again in September that year. It seems that it was originally intended that Brown would undertake work to the existing house, to the church (which had been damaged by fire in 1780 reputedly owing to the negligence of a man shooting magpies) and to the surrounding park.