The aim of this paper was to investigate the association of three well-recognised dietary patterns with cognitive change over a 3-year period. Five hundred and twenty-seven healthy participants from the Australian Imaging, Biomarkers and Lifestyle study of ageing completed the Cancer Council of Victoria food frequency questionnaire at baseline and underwent a comprehensive neuropsychological assessment at baseline, 18 and 36 months follow-up. Individual neuropsychological test scores were used to construct composite scores for six cognitive domains and a global cognitive score. Based on self-reported consumption, scores for three dietary patterns, (1) Australian-style Mediterranean diet (AusMeDi), (2) western diet and (3) prudent diet were generated for each individual. Linear mixed model analyses were conducted to examine the relationship between diet scores and cognitive change in each cognitive domain and for the global score. Higher baseline adherence to the AusMeDi was associated with better performance in the executive function cognitive domain after 36 months in apolipoprotein E (APOE) ε4 allele carriers (P<0.01). Higher baseline western diet adherence was associated with greater cognitive decline after 36 months in the visuospatial cognitive domain in APOE ε4 allele non-carriers (P<0.01). All other results were not significant. Our findings in this well-characterised Australian cohort indicate that adherence to a healthy diet is important to reduce risk for cognitive decline, with the converse being true for the western diet. Executive function and visuospatial functioning appear to be particularly susceptible to the influence of diet.