Age-related decrease in testosterone levels is a potential risk factor for cognitive decline in older men. However, observational studies and clinical trials have reported inconsistent results on the effects of testosterone on individual cognitive domains. Null findings may be attributed to factors that studies have yet to consider. In particular, individual variations in polyglutamine (CAG) length in the androgen receptor (AR) gene could alter androgenic activity in brain regions associated with cognitive processes including memory and executive functions. However, the role of AR CAG repeat length as a moderator of the relationship between testosterone levels and cognition has not been investigated. Therefore, we aimed to examine the relationship between baseline calculated free testosterone (cFT) levels, change in cFT levels over 18 months and CAG repeat length on cognitive performance in memory, executive function, language, attention and processing speed domains. These relationships were examined in 304 cognitively normal older male participants of the Australian Imaging, Biomarkers and Lifestyle (AIBL) Study of Ageing. In the attention and processing speed domain, a short CAG repeat length appears to exacerbate the effects of low baseline cFT levels that are also lower than expected at follow-up. These results highlight that individual variations in AR CAG repeat length should be considered in future studies and clinical trials that examine the complex relationship between testosterone and cognition.