Recent cosmological hydrodynamical simulations suggest that integral field spectroscopy can connect the high-order stellar kinematic moments h3 (∼skewness) and h4 (∼kurtosis) in galaxies to their cosmological assembly history. Here, we assess these results by measuring the stellar kinematics on a sample of 315 galaxies, without a morphological selection, using two-dimensional integral field data from the SAMI Galaxy Survey. Proxies for the spin parameter (λRe) and ellipticity (∈e) are used to separate fast and slow rotators; there exists a good correspondence to regular and non-regular rotators, respectively, as also seen in earlier studies. We confirm that regular rotators show a strong h3 versus V/σ anti-correlation, whereas quasi-regular and non-regular rotators show a more vertical relation in h3 and V/σ. Motivated by recent cosmological simulations, we develop an alternative approach to kinematically classify galaxies from their individual h3 versus V/σ signatures. Within the SAMI Galaxy Survey, we identify five classes of high-order stellar kinematic signatures using Gaussian mixture models. Class 1 corresponds to slow rotators, whereas Classes 2-5 correspond to fast rotators. We find that galaxies with similar λRe - ∈e values can show distinctly different h3 - V/σ signatures. Class 5 objects are previously unidentified fast rotators that show a weak h3 versus V/σ anti-correlation. From simulations, these objects are predicted to be disk-less galaxies formed by gas-poor mergers. From morphological examination, however, there is evidence for large stellar disks. Instead, Class 5 objects are more likely disturbed galaxies, have counter-rotating bulges, or bars in edge-on galaxies. Finally, we interpret the strong anti-correlation in h3 versus V/σ as evidence for disks in most fast rotators, suggesting a dearth of gas-poor mergers among fast rotators.