Fruit is seed-bearing structures specific to angiosperm that form from the gynoecium after flowering. Fruit size is an important fitness character for plant evolution and an agronomical trait for crop domestication/improvement. Despite the functional and economic importance of fruit size, the underlying genes and mechanisms are poorly understood, especially for dry fruit types. Improving our understanding of the genomic basis for fruit size opens the potential to apply gene-editing technology such as CRISPR/Cas to modulate fruit size in a range of species. This review examines the genes involved in the regulation of fruit size and identifies their genetic/signalling pathways, including the phytohormones, transcription and elongation factors, ubiquitin-proteasome and microRNA pathways, G-protein and receptor kinases signalling, arabinogalactan and RNA-binding proteins. Interestingly, different plant taxa have conserved functions for various fruit size regulators, suggesting that common genome edits across species may have similar outcomes. Many fruit size regulators identified to date are pleiotropic and affect other organs such as seeds, flowers and leaves, indicating a coordinated regulation. The relationships between fruit size and fruit number/seed number per fruit/seed size, as well as future research questions, are also discussed.