Phthalates are ubiquitous environmental endocrine-disrupting chemicals suspected to interfere with developmental androgen action leading to adverse effects on male reproductive function. Prenatal exposure studies in rodents show cryptorchidism, hypospadias and reduced testicular volume (TV), testosterone and anogenital distance in males. It is postulated that there is a developmental window in utero when phthalate exposure has the most potent adverse effects. Some human studies show associations between prenatal phthalate exposure and reduced calculated “free” serum testosterone in infant boys and shorter anogenital distance. However, there are no data available yet which link antenatal exposure to long-term effects in men. We aimed to correlate antenatal phthalate exposure with adult TV, semen parameters and serum reproductive hormone concentrations. 913 men from the Western Australian (Raine) Pregnancy Cohort were contacted aged 20–22 years. 423 (56%) agreed to participate; 404 underwent testicular ultrasound examination; 365 provided semen samples, and reproductive hormones were measured in 384. Maternal antenatal serum phthalate metabolite measurements were available for 185 and 111 men, who provided serum and semen, respectively. Maternal serum collected at 18 and 34 weeks gestation, stored at −80°C, was pooled and analyzed for 32 phthalate metabolites by liquid chromatography-tandem mass spectrometry. TV was calculated, semen analysis performed by WHO approved methods, and serum concentrations of gonadotrophins, inhibin B, and testosterone measured. Eleven phthalate metabolites were detected. Primary and secondary metabolites of di-(2-ethyl-hexyl) phthalate (DEHP) and di-iso-nonyl phthalate (DiNP) were positively correlated. After correction for adult height, BMI, presence of a varicocele and exposure to maternal smoking mono-iso-nonyl phthalate (MiNP) (r = −0.22) and sums of DEHP and DiNP metabolites (r = −0.24) and the sum of the metabolites of the high molecular weight phthalates (r = −0.21) were negatively correlated with TV (all p < 0.05). After adjustment for BMI adult serum total testosterone was positively associated with exposure to the following antenatal serum phthalate metabolites: mono-(2-ethylhexyl) phthalate (r = 0.26), MiNP (r = 0.18), the sum of metabolites for DEHP (r = 0.21) and DiNP (r = 0.18), and the sum of high molecular phthalates (r = 0.20) (p = 0.0005 to p = 0.02). Given sample size, storage duration and confounding through postnatal exposures, further studies are required.