Purpose: The purposes of this study are to investigate factors influencing non-participation in a structured exercise program for cancer survivors and to explore survivors’ experiences and attitudes in relation to physical activity participation. Methods: Face-to-face or telephone interviews were conducted with individuals who had registered for, or engaged in, the ‘Life Now’ exercise program run by Cancer Council Western Australia. Participants were 20 cancer survivors (mean age 63.90 years, SD 15.29) who had either cancelled their registration or withdrawn from the exercise program during the preceding 2 years. Interview transcripts were analysed using thematic analysis. Results: Seven main themes emerged: availability of the program; access, time and cost; lack of motivation or confidence; unwell or fatigued; physical activity preferences; knowledge of physical activity guidelines; and lack of referral or advice. The main barriers were contextual and included availability of, and access to, the program. Participants expressed a preference for home-based physical activity. Conclusions: Interventions aimed at promoting physical activity in cancer survivors should offer home-based programs and include referral and advice from oncologists. Implications for cancer survivors: Increasing cancer survivors’ participation in, and compliance with, exercise programs may require home-based strategies and referrals from oncologists to allied health professionals to individualise care.