We observe magnetic effects in ultrahigh-quality factor crystalline quartz bulk acoustic wave resonators at milli-Kelvin temperature. The study reveals the existence of hysteresis loops, jumps, and memory effects of acoustical resonance frequencies. These loops arise as a response to the external magnetic field, and they span a few Hertz for modes with linewidths of about 25mHz, which constitute a frequency shift of order 60 linewidths. The effects are broadband but get stronger toward higher frequencies where both nonlinear effects and losses are limited by two-level systems. This suggests that the observed effects are due to the ferromagnet-like phase of a spin ensemble coupled to mechanical modes. The observed coupling between mechanical and spin degrees of freedom in the ultralow-loss regime brings about new possibilities for the emerging class of quantum hybrid systems.