Primacy and recency effects are common benchmarks for models of free recall and episodic memory. In this work, we show that RT distributions carry diagnostic information about how items enter into competition for recall, and how that competition impacts on the dynamics of recall and leads to novel conclusions about the forms of primacy and recency effects. We jointly fit RT distributions and serial position functions for free recall initiation with both a racing diffusion model and the linear ballistic accumulator (LBA: Brown & Heathcote, 2008). The models were fit in a hierarchical Bayesian framework, factorially varying different assumptions of how primacy and recency are generated. Recency functions were either exponential or power law in shape. Primacy was treated either as a strength boost to the early list items so that both primacy and recency items jointly compete to be retrieved; a mixture of primacy and recency gradients reflecting the usage of different retrieval cues; or a primacy-as-recency account in which primacy items are functionally recent due to the contribution of rehearsal. Although serial position curves do not distinguish between these accounts, they make distinct predictions about how RT distributions vary across serial positions. Results from a number of data sets strongly favor an exponential recency function along with a mixture model of primacy and recency gradients. These results suggest that complete RT distributions can provide informative constraints on models of free recall.