The confinement of high frequency phonons approaching 1 GHz is demonstrated in phonon-trapping acoustic cavities at cryogenic temperatures using a low-coupled network approach. The frequency range is extended by nearly an order of magnitude, with excitation at greater than the 200th overtone achieved for the first time. Such a high frequency operation reveals Rayleigh-type phonon scattering losses due to highly diluted lattice impurities and corresponding glasslike behavior, with a maximum QL×f product of 8.6×1017 at 3.8 K and 4×1017 at 15 mK. This suggests a limit on the Q×f product due to unavoidable crystal disorder. Operation at 15 mK is high enough in frequency that the average phonon occupation number is less than unity, with a loaded quality factor above half a billion. This work represents significant progress towards the utilization of such acoustic cavities for hybrid quantum systems. © 2013 American Physical Society.