Met his wife Anne-Marie Raffo when he joined the Greenwich House (a performing arts and social service house) where her father, pottery maker
Victor Raffo conducted some works. James and Anne were married for 69 years, until his death. They have four children.