We present an analysis of asymmetries in global HI spectra from the extended GALEX Arecibo SDSS Survey (xGASS), a stellar mass-selected and gas fraction-limited survey which is representative of the HI properties of galaxies in the local Universe. We demonstrate that the asymmetry in a HI spectrum is strongly linked to its signal-to-noise meaning that, contrary to what was done in previous works, asymmetry distributions for different samples cannot be compared at face value. We develop a method to account for noise-induced asymmetry and find that the typical galaxy detected by xGASS exhibits higher asymmetry than what can be attributed to noise alone, with 37 per cent of the sample showing asymmetry greater than 10 per cent at an 80 per cent confidence level. We find that asymmetric galaxies contain, on average, 29 per cent less HI mass compared to their symmetric counterparts matched in both stellar mass and signal-to-noise. We also present clear evidence that satellite galaxies, as a population, exhibit more asymmetric HI spectra than centrals and that group central galaxies show a slightly higher rate of HI asymmetries compared to isolated centrals. All these results support a scenario in which environmental processes, in particular those responsible for gas removal, are the dominant driver of asymmetry in xGASS.