James Fitzjames Stephen was born on 3 March 1829 into the upper-middle classes of an England of which George IV was king, the Duke of Wellington Prime Minister with the Great Reform Act and the death of Jeremy Bentham still three years distant. The marriage, in 1818, of his father, James Stephen, to his mother, Jane Venn, represented a union of the Stephens, a recently established line of lawyers originating in Scotland, with the Venns, a clerical and scholastic dynasty stretching back to Tudor times. Both families brought to the partnership a pronounced evangelical work ethic coupled with a high regard for solid intellectual accomplishment. It was a marriage which also constituteda further connecting strand in a genealogical web of distinguished families that included (or would soon include) the Wilberforces, Macaulays, Diceys, Elliotts, Trevelyans, Stracheys, Thackerays and Arnolds.

Unsurprisingly, the Stephen children, particularly Fitzjames and his younger brother Leslie, were much influenced by Sir James Stephen's complex character and temperament. Though he was principally known for his career in government service, it was not until 1825, after initially practising at the Chancery Bar, that Sir James Stephen took up a full-time post in the Colonial Office, rising to Under Secretary in 1836. Even before reaching this level of seniority Sir James, through a combination of ability, appetite for work and unwillingness to delegate, ‘Virtually ruled the Colonial Empire’.