Purpose: Research on the association between physical activity and the risk of prostate cancer is inconsistent. The aim of this study was to investigate whether the timing, intensity, and type of recreational physical activity influence prostate cancer risk. Methods: A population-based case–control study was conducted in Western Australia in 2001–2002. Data were collected on lifetime recreational physical activity from a self-reported questionnaire. The estimated effects of recreational physical activity on prostate cancer risk were analyzed using logistic regression, adjusting for demographic and lifestyle factors. This analysis included 569 incident cases and 443 controls. Results: There was a significant, inverse dose–response relationship between vigorous-intensity recreational physical activity between the ages 19 and 34 years and the risk of prostate cancer (p Trend = 0.013). Participants in the most active quartile of vigorous-intensity physical activity in this age period had a 33% lower risk of prostate cancer than participants in the least active quartile (Adjusted Odds Ratio = 0.67, 95% confidence interval = 0.45–1.01). Moderate-intensity recreational physical activity was not associated with the risk of prostate cancer. Recreational physical activity performed over the lifetime showed no association with prostate cancer risk. Weight training performed from early adulthood onwards showed a non-significant but consistent inverse association with prostate cancer risk. There was no strong evidence that physical activity was differentially associated with the risks of low-grade and medium-to-high grade prostate cancers. Conclusions: A high level of vigorous recreational physical activity in early adulthood may be required to reduce the risk of prostate cancer.