White matter (WM) microstructure is a sensitive marker to distinguish individuals at risk of Alzheimer’s disease. The association of objective physical fitness (PF) measures and WM microstructure has not been explored and mixed results reported with physical activity (PA). Longitudinal studies of WM with PA and PF measures have had limited investigation. This study explored the relationship between objective PF measures over 24-months with “normal-appearing” WM microstructure. Data acquired on magnetic resonance imaging was used to measure “normal-appearing” WM microstructure at baseline and 24-months. Clinical variables such as cognitive and blood-based measures were collected longitudinally. Also, as part of the randomized controlled trial of a PA, extensive measures of PA and fitness were obtained over the 24 months. Bilateral corticospinal tracts (CST) and the corpus callosum showed a significant association between PF performance over 24-months and baseline WM microstructural measures. There was no significant longitudinal effect of the intervention or PF performance over 24-months. Baseline WM microstructural measures were significantly associated with PF performance over 24-months in this cohort of participants with vascular risk factors and at risk of Alzheimer’s disease with distinctive patterns for each PF test.