Samuel Osborne Barber II—the full name the composer later gave up—was born on March 9, 1910, at 35 South High Street, West Chester, near Philadelphia. His paternal grandfather, the first Samuel O. Barber, was a manufacturer of shipping tags who had moved to West Chester in 1888 with the Denny Tag Company. He set up his own highly successful firm, the Keystone Tag Company, in 1901, which the family ran until 1941. The composer's maternal grandfather was a Presbyterian minister, Dr. William Trimble Beatty, who founded Shady Side Presbyterian Church in Pittsburgh. He was also instrumental in founding the Pennsylvania College for Women. He died at only age forty-eight and left his widow with eight children; they moved to West Chester in 1885.

Helen John was in high school with Barber and recalled, “West Chester was plainer than most towns, quite strict with a strong Quaker influence.” One of the Beatty children became the famous opera singer Louise Homer. Her husband, Sidney Homer, recalled that West Chester “was the most retired retiring town one could imagine. Even the houses seemed to retire behind the massive maple trees. So quiet were the streets that when you walked you walked softly, if you had to speak you spoke softly.” Louise Homer's younger sister was Barber's mother, Marguerite McLeod Beatty, known as Daisy. Ruth Dean, a neighbor, recalled that all the Beattys had fine singing voices. They were descendants of the inventor Robert Fulton (1765–1815), who developed steamboats and an early submarine. There was less music on the Barber side of the family, although Ada Strode, Samuel's aunt, was a piano teacher.

Samuel Le Roy Barber, the composer's father, married Daisy on October 17, 1905, in the First Presbyterian Church, West Chester. The couple had two children. Samuel Barber grew up close to his sister, Sara Fulton Barber (1913–61). She sang beautifully, and it was a great loss when she died prematurely of asthma. Roy Barber was a doctor, widely respected in the small-town community where everybody knew each other's religion and politics. As Henry James explained: “This profession in America has constantly been held in honor, and more successfully than elsewhere has put forward a claim to the epithet of ‘liberal.’