Objective: Testosterone and sex hormone-binding globulin (SHBG) concentrations are reported to decline during male ageing, but whether these changes reflect physiological ageing or age-related comorbidities remains uncertain. We examined longitudinal changes in serum testosterone and SHBG concentrations in middle-aged to older men, concordance between baseline and follow-up values and relationships with concomitant changes in lifestyle and medical factors. Design: Population-based longitudinal cohort study. Participants: Community-dwelling men aged 40–69 years. Measurements: Immunoassay serum total testosterone (n = 7812) and SHBG (n = 6491) at baseline (2006–2010) and follow-up (2012–2013). Free testosterone (cFT) was calculated. Bland–Altman analyses and concordance correlation of repeated measurements were conducted. Associations of changes in hormone concentrations with lifestyle and medical factors were explored using Spearman's rank correlation. Results: Over 4.3 years follow-up, there was a negligible mean change (±SE) in serum total testosterone concentration (+0.06 ± 0.03 nmol/L), whereas mean SHBG concentration increased (+3.69 ± 0.12 nmol/L) and cFT decreased (−10.7 ± 0.7 pmol/L). Concordance estimates were 0.67 (95% confidence interval [CI]: 0.66–0.69) for total testosterone, 0.83 (CI = 0.82–0.84) for SHBG and 0.56 (CI = 0.54–0.58) for cFT. Changes in serum total testosterone correlated with changes in SHBG (Spearman's rank ρ = 0.33, CI = 0.30–0.35), and inversely with changes in body mass index (BMI) (ρ = −0.18, CI = −0.20 to −0.16) and waist circumference (ρ = −0.13, CI = −0.15 to −0.11) and in SHBG with changes in BMI (ρ = −0.34, CI = −0.36 to −0.32) and waist circumference (ρ = −0.21, CI = −0.24 to −0.19). Conclusion: In relatively healthy middle-aged to older men, mean serum total testosterone concentration is stable with ageing, while mean SHBG concentration increases. Both total testosterone and SHBG concentrations were highly concordant over time.