Ludstone was one of the Norman manors that were held by Roger de Montgomery, Earl of Shrewsbury, but later passed to the College of St Mary Magdalen in Bridgnorth which farmed it as a monastic grange. In circa 1403, the then Dean of Bridgnorth, Thomas of Tutbury, began to rebuild the house although his works were apparently never completed. This was one of the two manor houses which are said to have stood on the site before the present house was built for John Whitmore in circa 1607. That which existed prior to that was possibly built for the Brooke family, who bought the property in circa 1548 but then sold it in 1557 to concentrate on Madeley Court (q.v.) and their other estates. The Whitmore building, which now exists, has a pair of seemingly contemporary lodges to the east, although they stand on masonry which may be associated with the earlier house.

The rectangular moat of the earlier house also remains, creating a magical setting for the brick building dressed in red sandstone, with its handsome curvilinear shaped gables. Formed from an H-plan, with strong balanced elevations and a broad central recessed, south-facing gable, set with a two-storeyed, curved mullioned and transomed bay, the house is of a highly sophisticated design for Shropshire in the first decade of the seventeenth century.

Arthur Oswald, writing in the mid-twentieth century, surmised that the house might have been by a master mason associated with a house such as Aston Hall, Warwickshire. Meanwhile, recent research by Mark Girouard has highlighted the fact that Whitmore’s elder son had matriculated from Wadham College, Oxford, in 1617 which had recently been completed to designs by William Arnold. Most of his other documented work is centred around the West Country, at Montacute, Somerset, and at Lulworth Castle, Dorset. At both of these houses he used pairs of unusual arched, scallop-headed niches. Although Ludstone has pairs of niches to the sides of the segmental bay, they are of a conventional type. Their placing in this low location, though, is unusual suggesting that the design, rather than detailed execution, may have involved Arnold or someone in his circle.

The entrance into the house is asymmetrically placed in the gabled projection to the left of the curved bay window.