It is well-established that task-irrelevant sounds deviating from an otherwise predictable auditory sequence capture attention and disrupt ongoing performance by delaying responses in the ongoing task. In visual tasks, larger distraction by unexpected sounds (deviance distraction) has been reported in older than in young adults. However, past studies based this conclusion on the comparisons of absolute response times (RT) and did not control for the general slowing typically observed in older adults. Hence, it remains unclear whether this difference in deviance distraction between the two age groups reflects a genuine effect of aging or a proportional effect of similar size in both groups. We addressed this issue by using a proportional measure of distraction (PMD) to reanalyze the data from four past studies and used Bayesian estimation to generate credible estimates of the age-related difference in deviance distraction and its effect size. The results were unambiguous: older adults exhibited greater deviance distraction than young adults when controlling for baseline response speed (in each individual study and in the combined data set). Bayesian estimation revealed a proportional lengthening of RT by unexpected sounds that was about twice as large in older than in young adults (corresponding to a large statistical effect size). A similar analysis was carried out on the proportion of correct responses (PC) and produced converging results. Finally, an additional Bayesian analysis comparing data from cross-modal and uni-modal studies confirmed the selective effect of aging on distraction in the first and not the second. Overall, our study shows that older adults performing a visual categorization task do exhibit greater distraction by unexpected sounds than young adults and that this effect is not explicable by age-related general slowing.