Background: Survival following colorectal cancer (CRC) survival may be influenced by a number of factors including family history, individual medical history, and comorbidities. The impact of these factors may vary based on the patient’s age. Methods: The study cohort consisted of individuals born in Western Australia between 1945 and 1996, who had been diagnosed with CRC prior to 2015 (n = 3220). Hospital, cancer, and mortality data were extracted for each patient from state health records and were used to identify potential risk factors associated with CRC survival. Family linkage data, in combination with cancer registry data, were used to identify first-degree family members with a history of CRC. The association between survival following CRC diagnosis and identified risk factors was examined using Cox proportional hazard models. Results: Age and sex were not significantly associated with survival in young patients. However, in middle-aged patients increasing age (HR 1.03, 95% CI 1.01–1.05, p = 0.003) and being male (HR 0.72, 95% CI 0.60–0.87, p < 0.001) were associated with reduced survival. Being diagnosed with polyps and having a colonoscopy prior to CRC diagnosis were associated with improved survival in both young and middle-aged patients, while a history of non-CRC and liver disease was associated with reduced survival. In middle-aged patients, having diabetes-related hospital admissions (HR 1.53, 95% CI 1.15–2.03, p = 0.004) was associated with reduced survival. Conclusions: In both young and middle-aged patients with CRC, factors associated with early screening and detection were associated with increased CRC survival while a history of liver disease and non-CRC was associated with decreased CRC survival.