Using the deepest 1.2 mm continuum map to date in the Hubble Ultra Deep Field, which was obtained as part of the ALMA Spectroscopic Survey (ASPECS) large program, we measure the cosmic density of dust and implied gas (H2+H i) mass in galaxies as a function of look-back time. We do so by stacking the contribution from all H-band selected galaxies above a given stellar mass in distinct redshift bins, and . At all redshifts, and grow rapidly as M decreases down to 1010 M o, but this growth slows down toward lower stellar masses. This flattening implies that at our stellar mass-completeness limits (108 M o and 108.9 M o at z 0.4 and z 3), both quantities converge toward the total cosmic dust and gas mass densities in galaxies. The cosmic dust and gas mass densities increase at early cosmic time, peak around z 2, and decrease by a factor 4 and 7, when compared to the density of dust and molecular gas in the local universe, respectively. The contribution of quiescent galaxies (i.e., with little on-going star formation) to the cosmic dust and gas mass densities is minor (10%). The redshift evolution of the cosmic gas mass density resembles that of the SFR density, as previously found by CO-based measurements. This confirms that galaxies have relatively constant star formation efficiencies (within a factor 2) across cosmic time. Our results also imply that by z 0, a large fraction (90%) of dust formed in galaxies across cosmic time has either been destroyed or ejected to the intergalactic medium.