We present an analysis of the global stellar populations of galaxies in the SAMI (Sydney-AAO Multi-object Integral field spectrograph) Galaxy Survey. Our sample consists of 1319 galaxies spanning four orders of magnitude in stellar mass and includes all morphologies and environments. We derive luminosity-weighted, single stellar population equivalent stellar ages, metallicities and alpha enhancements from spectra integrated within one effective radius apertures. Variations in galaxy size explain the majority of the scatter in the age-mass and metallicity-mass relations. Stellar populations vary systematically in the plane of galaxy size and stellar mass, such that galaxies with high stellar surface mass density are older, more metal rich and alpha enhanced than less dense galaxies. Galaxies with high surface mass densities have a very narrow range of metallicities; however, at fixed mass, the spread in metallicity increases substantially with increasing galaxy size (decreasing density). We identify residual correlations with morphology and environment. At fixed mass and size, galaxies with late-type morphologies, small bulges and low Sersic n are younger than early type, high n, high bulge-to-total galaxies. Both age and metallicity show small residual correlations with environment; at fixed mass and size, galaxies in denser environments or more massive haloes are older and somewhat more metal rich than those in less dense environments. We connect these trends to evolutionary tracks within the size-mass plane.