Prenatal depressive symptoms are linked to negative child behavioral and cognitive outcomes and predict later psychopathology in adolescent children. Prior work links prenatal depressive symptoms to child brain structure in regions like the amygdala; however, the relationship between symptoms and the development of brain structure over time remains unclear. We measured maternal depressive symptoms during pregnancy and acquired longitudinal T1-weighted and diffusion imaging data in children (n = 111; 60 females) between 2.6 and 8 years of age. Controlling for postnatal symptoms, we used linear mixed effects models to test relationships between prenatal depressive symptoms and age-related changes in (i) amygdala and hippocampal volume and (ii) structural properties of the limbic and default-mode networks using graph theory. Higher prenatal depressive symptoms in the second trimester were associated with more curvilinear trajectories of left amygdala volume changes. Higher prenatal depressive symptoms in the third trimester were associated with slower age-related changes in limbic global efficiency and average node degree across childhood. Our work provides evidence that moderate symptoms of prenatal depression in a low sociodemographic risk sample are associated with structural brain development in regions and networks implicated in emotion processing.