Benedetto Accolti was born in 1415, the third child of Michele di Santi Accolti and Margherita di Rosello Roselli. His elder brother Antonio was born in 1410 and his sister Agnesa in 1413. He had three younger brothers: Francesco, born in 1416; Donato, in 1421; and Giovanni, in 1423. His two younger sisters, Tommasa and Nanna, were born between 1427 and 1441. Two of Accolti's brothers died in their youth: Antonio between 1430 and 1445, and Giovanni between 1443 and 1456.

Benedetto Accolti was depicted as a man of preeminent intelligence by Vespasiano da Bisticci, who particularly praised his memory, declaring that it was a marvel, unequalled by any of his contemporaries. This prodigious intellect was an endowment shared by his equally brilliant brother, Francesco. Their father, Michele, must have been a man of considerable ablity to restore his family from the oblivion of the Casentine hills to social prominence in Arezzo and forge for himself a prosperous legal career in Florence without the benefit of a previous legal tradition in his family. On their mother's side, Benedetto and Francesco benefited from the intellectual versatility of the Roselli family. Their grandfather, Rosello, the son of a notary, had become one of the leading lawyers of Florence by the turn of the fifteenth century; he was the founder of a veritable legal dynasty, numbering many lawyers in succeeding generations. The Roselli family was distinguished too by a remarkable series of Italian poets in the fifteenth century, and members of the family showed an interest in Latin letters as well.