Androgens, notably testosterone (T), have been implicated in development of several common cancers and prostate cancer; however, precise mechanisms remain unclear. This study assessed prospective associations of serum T, dihydrotestosterone (DHT) and estradiol (E2) with overall cancer (excluding skin cancer), prostate, colorectal and lung cancer risk in 1574 community-dwelling men aged 25–84 years. Sex hormones were assayed using mass spectrometry and men were followed for 20 years with outcomes ascertained using data linkage. Over 20 years, there were 289, 116, 48 and 22 men who developed any cancer, prostate cancer, colorectal cancer and lung cancer, respectively. Androgens in the lowest quartile were associated with an increased overall cancer risk (HR = 1.36, 95% CI 1.05–1.76, p = 0.020 for T; and HR = 1.30, 95% CI 1.00–1.69, p = 0.049 for DHT comparing the lowest vs other quartiles). T in the lowest quartile was associated with an increased risk of prostate cancer (HR = 1.53, 95% CI 1.02–2.29, p = 0.038 comparing the lowest vs other quartiles). The association between androgens and overall cancer risk remained similar after excluding prostate cancer outcomes; however, results were not significant. There were no associations of T, DHT or E2 with colorectal or lung cancer risk; however, LH in the highest quartile was associated with an increased risk of lung cancer (HR = 4.55, 95% CI 1.70–12.19, p = 0.003 for the highest vs other quartiles). Whether T is a biomarker of poor health in men with any cancer or prostate cancer requires further confirmation as does the nature and mechanism of the association of a high LH with future lung cancer.