Introduction: For decades, researchers have tried to understand the moderating effect of APOE ε4 carriage on the relationship between physical activity (PA), brain health and dementia risk. However, this field has produced inconsistent findings. Method: We conducted a systematic review of the literature, searching for observational and interventional studies examining the effect of APOE ε4 carriage on the relationships between PA, dementia risk and different markers of brain health. Results: Observational studies using dementia risk as a primary outcome measure generally found that in shorter follow-up periods (up to 10 years) both APOE ε4 carriers and non-carriers benefit from PA, although longer follow-ups showed mixed results. In neuroimaging studies, mainly carriers or both groups showed benefits. Additionally, the association between PA and amyloid burden was more evident among carriers. Overall, studies with greater samples of active APOE ε4 carriers are more likely to report benefits within this group in terms of lower dementia risk and reduced brain pathology. Discussion: Although we have identified some patterns for the modulating effect of APOE ε4 on PA and dementia or brain pathology, the available data is, overall, inconclusive. Heterogeneity in study design, methodology, and outcomes blur the ability to detect clear associations.