DOWNHAM HALL STOOD ON LOW GROUND ON THE SOUTH SIDE OF THE LITTLE OUSE RIVER, three miles from Brandon in the village of Santon Downham, a parish created by the merger of Santon across the river in Norfolk and Downham in Suffolk. In 1668 the village was subject to what was described as a ‘sand-flood’, when a wave of sand (believed to have originated near Lakenheath) overwhelmed the village where ‘many houses were overthrown and buried, and their pastures and meadows which for so small a town were considerable were over-run and destroyed’.

In the late Middle Ages the village was held partly by the Abbey of Bury St Edmunds and partly by the Priory of Ixworth. After the Dissolution of the Monasteries the manor was granted to Sir Thomas Kytson. From the middle of the seventeenth century Downham Hall was owned by the Wright family. Thomas Wright's house had twelve hearths in 1674. The last member of his family to own it was Ann Wright from whom the estate was acquired in 1778 by Charles Sloane, first Earl Cadogan (of the second creation). He died in 1807 and the estate subsequently, in 1830, was bought by Lord William John Frederick Powlett, later Duke of Cleveland. On his death in 1864 the estate was left for the use of his widow, Caroline, a daughter of the first Earl of Lonsdale, but it was put on the market in 1870.

The purchaser was Edward Mackenzie of Fawley Court near Henley on Thames, Oxfordshire. Edward Mackenzie's father was a Scottish-born engineer and the family had been heavily involved in canal and railway ventures. It was, no doubt, from these that their wealth derived. Downham became the residence of Edward Mackenzie's second son Edward Phillipe who also had property in Kirkcudbright. Edward Mackenzie bought the Downham estate (which included the Home and three other farms) of 5,921 acres for £81,500. His elder son William inherited the property with Edward Phillipe having the right to live there as long as he wished.

THEWRIGHT family's original house was replaced by, or remodelled as, a Georgian building in the late eighteenth century either by the last Wright owner or more probably after Earl Cadogan's acquisition of what was then the 3,134-acre estate.