Misalignment of gas and stellar rotation in galaxies can give clues to the origin and processing of accreted gas. Integral field spectroscopic observations of 1213 galaxies from the Sydney-AAO Multi-object Integral field spectrograph (SAMI) Galaxy Survey show that 11 per cent of galaxies with fitted gas and stellar rotation are misaligned by more than 30◦ in both field/group and cluster environments. Using SAMI morphological classifications and Sérsic indices, the misalignment fraction is 45 ± 6 per cent in early-type galaxies (ETGs), but only 5 ± 1 per cent in late-type galaxies (LTGs). The distribution of position angle offsets is used to test the physical drivers of this difference. Slower dynamical settling time of the gas in elliptical stellar mass distributions accounts for a small increase in misalignment in early-type galaxies. However, gravitational dynamical settling time is insufficient to fully explain the observed differences between ETGs and LTGs in the distributions of the gas/stellar position angle offsets. LTGs have primarily accreted gas close to aligned rather than settled from misaligned based on analysis of the skewed distribution of PA offsets compared to a dynamical settling model. Local environment density is less important in setting the misalignment fractions than morphology, suggesting that mergers are not the main source of accreted gas in these discs. Cluster environments are found to have gas misalignment driven primarily by cluster processes not by gas accretion.