Yae had six brothers and sisters, although two died when they were young.
Her older brother, Kakuma Yamamoto, was seventeen years older than she was. He studied Western science and artillery in Edo (now Tokyo) and later became a teacher at the Aizu domain school, Nisshinkan.

Her younger brother Saburo was wounded during the Battle of Toba-Fushimi in Kyoto and died at the age of nineteen.