Populus cathayana was employed as a model species to study sexual differences in plant growth, physiological, biochemical, and ultrastructural responses to manganese (Mn) stress with two different nitrogen sources (NH4 + and NO3 −). Results showed that NH4 +-N (3.75 mM) and NO3 −-N (3.75 mM) significantly increased plant growth, gas exchange capacity, as well as superoxide dismutase (SOD) and peroxidase (POD) activities in the leaves of both male and female P. cathayana. In addition, NH4 +-N supply could improve tolerance to Mn stress in both male and female P. cathayana with higher photosynthetic capabilities, biomass accumulation, antioxidant enzyme activities, superoxide radicals (O2 −) and thiobarbituric acid-reactive substances (TBARS) in leaves. Under the same condition, female plants grown under NO3 − supply exhibited significantly lower biomass accumulation, photosynthetic capabilities, and antioxidant enzyme activities than male plants because of higher Mn accumulation in both above- and belowground organs and higher ROS releasing rate in leaves. Moreover, damage to cellular ultrastructure was also observed in leaves of males and females exposed to Mn stress, but more strongly in females, with more visible degradation of mitochondrion in plant leave cells. The difference in the preferences for different N sources between male and female P. cathayana under Mn stress provides important guidance in fertilizing strategy to minimise intraspecific competition between female and male plants.