On returning to Whitehall as Secretary of State for Social Services Joseph was reunited with the chauffeur, Molly, who had driven him when he was at Housing. Coincidentally Richard Crossman, who had inherited Molly from Sir Keith in 1964, was the departing minister at the DHSS. On his last day in office Crossman's current driver was away, so Molly picked him up from Paddington Station. Crossman reflected in his diary that Molly used to enjoy working for Sir Keith, because she could “mother” him. He was glad that Joseph had been picked as his replacement; “He is a civilised, cultivated man, who is certainly intelligent enough”. But Crossman wondered whether he could stand up to the strain. His misgivings were confirmed when he encountered Molly again some time later. Asked if she was glad to be working once more for her favourite minister, she shook her head sadly and replied, “At least you were never sick, sir”. This time round Molly would have to be more nurse than mother. Joseph had at least one minor operation during this period, but the nature of the problem was not disclosed to the press, then or later.

Apparently Joseph's health had been reasonably good in the run-up to the election, but Crossman must have been watching him more closely than the commentators who saw him as someone who could withstand even the pressures of the premiership. It was a stressful job that Joseph had undertaken, comparable to his first Cabinet post.