In later life, the 3rd Duke of Buccleuch would look back on his early childhood as a profoundly unhappy one, characterised above all by a sense of neglect. Lady Dalkeith, herself the product of an unorthodox upbringing, seems to have been a distant and at times vindictive mother whose parenting style was described by one observer as little short of ‘domestic tyranny’. She had no qualms in sending Henry off at ‘a very young age’ to a private boarding school, Dr Fountaine's of Marylebone, where, he would later recall, he languished ‘almost neglected by my mother, neglected in every respect as to my learning by the masters of the school unknown to my family and connections’. Although presenting itself as an exclusive seminary for gentlemen of rank, Fountaine's was in reality, as one of the Duke's fellow-pupils there would later remark, ‘more of a nursery than a school’. Boys were admitted as soon as they had ‘breeched’ (usually between the ages of three and seven) and left around the age of twelve or thirteen. Despite him being taught very little there – he later recalled spending most of his time ‘living with, and indulged by’ the school's servants – the one thing the teachers did instil in the young Duke was an acute awareness of his aristocratic standing. Indeed, as the highest-ranking boy in the school he was often paraded in front of visitors to bolster the school's credentials; as he later recalled,

The marriage of Charles Townshend to Lady Dalkeith in September 1755, shortly after the Duke's ninth birthday, did indeed mark a turning point in young Henry's life. Although it seems to have been largely a strategic marriage Townshend did take a genuine and active interest in the upbringing of all four of his stepchildren, and, most significantly for the Duke, in the direction of their education.