The future Baldwin IV was born in the early summer of 1161 to Amalric, count of Jaffa, and his wife, Agnes of Courtenay. Amalric's elder brother, King Baldwin III, stood godfather to his nephew who was named after him, and the story was later told that when a member of the court asked what christening present he intended to give the child he laughingly replied, ‘The Kingdom of Jerusalem.’ It must have seemed a frivolous remark at the time, for Baldwin was only thirty-one and had recently married a young and beautiful wife, so that the likelihood of his nephew's inheriting the throne appeared remote. Yet less than two years later Baldwin died childless. His death had important consequences for his little nephew, for not only did his father become king, but his parents' marriage was dissolved.

Amalric was Baldwin III's only brother and therefore the undisputed heir apparent, yet when the members of the High Court met with the senior clergy to consider the succession, they refused to recognise him as king unless he repudiated his wife. Their spokesman was the patriarch of Jerusalem, Amalric of Nesle, who objected that the couple were related within the prohibited degrees. In fact they were related in the fourth degree, having a common great-great-grandfather, Burchard of Monthléry. In the twelfth century impediments of consanguinity were reckoned to extend to the sixth or even the seventh degrees, but it was extremely unusual for an objection of this kind to be made to a well-established marriage and this suggests that the canonical objection masked some more deep-seated animosity towards Agnes on the part of the baronage.

By 1163 Amalric and Agnes had been married for six years and had two children, Baldwin and his elder sister Sibyl. There certainly could have been no objection to Agnes becoming queen on grounds of her birth. Her father, Joscelin II of Courtenay, count of Edessa, was the second cousin of Queen Melisende of Jerusalem and of her sisters, Alice, princess of Antioch and Hodierna, countess of Tripoli, so that Agnes was related to all the ruling families in the Frankish East.