Objectives: To investigate the association between lean mass (LM) and fat mass (FM) with fatigue and vitality before and after exercise in patients with prostate cancer already undergoing androgen-deprivation therapy (ADT). Subjects and Methods: Cross-sectional associations between LM and FM with fatigue and/or vitality measures were examined in 229 patients (aged 43–90 years). Prospective analysis was undertaken in 129 patients who underwent a supervised 3–6 months exercise programme (predominantly resistance + aerobic). Whole body and appendicular LM, and total and trunk FM were assessed by dual X-ray absorptiometry. Fatigue was assessed using the European Organisation for Research and Treatment of Cancer Quality of Life Questionnaire-Core 30 (EORTC QLQ-30) and vitality using the Short Form-36. Results: Based on the EORTC QLQ-30, 19% of patients had clinically relevant fatigue. There was no association between LM and fatigue; however, total (P = 0.013), trunk (P = 0.015) and percentage (P = 0.008) FM were higher in fatigued than not fatigued patients, with total and trunk FM 5.0 and 2.6 kg higher, respectively. For quartiles of vitality, a similar pattern emerged for FM with those in the lowest quartile of vitality having the highest FM values (P = 0.014–0.034). In contrast, following supervised exercise, change in fatigue and vitality were associated with change in total LM (r = −0.182, P = 0.042 and r = 0.309, P = 0.001, respectively) but not FM. Patients fatigued at baseline but not fatigued following the exercise programme gained a median (interquartile range) of 2.1 (0.7–3.2) kg LM. Conclusion: In patients with prostate cancer treated with ADT, body composition is associated with fatigue, with higher total and trunk FM in those with clinically relevant fatigue. However, following exercise those no longer fatigued had an accompanying substantial increase in LM. Modifying body composition, both LM and FM, in patients with prostate cancer may favourably alter cancer-related fatigue levels and should be a target of exercise medicine in this population.