Ilse was one of four children born to a Dutch-Jewish father and a German-Jewish mother. In 1924 she married Franz Ledermann, a successful lawyer who was 15 years her senior. The couple made their home in Berlin, where they raised two daughters, Barbara and Susanne.