We investigate the Tully-Fisher relation (TFR) for a morphologically and kinematically diverse sample of galaxies from the Sydney-AAO Multi-object Integral field spectrograph (SAMI) Galaxy Survey using two-dimensional spatially resolved Ha velocity maps and find a well-defined relation across the stellar mass range of 8.0 <log (M*/M-circle dot) <11.5. We use an adaptation of kinemetry to parametrize the kinematic Ha asymmetry of all galaxies in the sample, and find a correlation between scatter (i.e. residuals off the TFR) and asymmetry. This effect is pronounced at low stellar mass, corresponding to the inverse relationship between stellar mass and kinematic asymmetry found in previous work. For galaxies with log (M*/M-circle dot) <9.5, 25 +/- 3 per cent are scattered below the root mean square (RMS) of the TFR, whereas for galaxies with log (M*/M-circle dot) > 9.5 the fraction is 10 +/- 1 per cent. We use 'simulated slits' to directly compare our results with those from long slit spectroscopy and find that aligning slits with the photometric, rather than the kinematic, position angle, increases global scatter below the TFR. Further, kinematic asymmetry is correlated with misalignment between the photometric and kinematic position angles. This work demonstrates the value of 2D spatially resolved kinematics for accurate TFR studies; integral field spectroscopy reduces the underestimation of rotation velocity that can occur from slit positioning off the kinematic axis.