Large distribution of roots in topsoil layers allow more uptake of soil water and nutrients during the vegetative growth, but it may be disadvantageous if soil water deficit develops during the reproductive stage. The relationship between the distribution of roots in topsoil (0–0.4 m) and soil water use, dry matter and nitrogen (N) accumulations, and grain yield was examined in winter wheat (Triticum aestivum L.) with contrasting root size in the topsoil. Two old landraces (CW134 and JM47, larger root length and biomass in the topsoil) and two modern wheat cultivars (CH58 and LH7, smaller root system size in the topsoil), were grown in the field during two seasons (2016–2017 and 2017–2018) under rainfed and irrigation conditions in the semi-arid farmland on the Loess Plateau. Root biomass and root length density (RLD) in topsoil (0–0.4 m) was significantly higher in the old landraces than in the modern cultivars (P < 0.05) under rainfed and irrigation in both seasons (no such difference in subsoil, 0.4–1.0 m). The modern cultivars had significantly higher grain yield, grain N concentration, water-use efficiency (WUE), and 1000-grain weight (P < 0.05). Seasonal water use was similar among all cultivars, but post-anthesis water use was higher in the modern cultivars, particularly under rainfed conditions in both seasons. Root biomass and RLD in the topsoil was positively correlated with pre-anthesis water use, but negatively correlated with after anthesis water use. Post-anthesis water use was closely related to post-anthesis dry matter, N accumulation, yield and WUE under rainfed conditions. To conclude, large distribution of roots in the topsoil had non-advantage for wheat grown under rainfed conditions. The small distribution of roots in the topsoil (characteristics of modern wheat cultivars) enhanced post-anthesis water use, increased post-anthesis dry matter and N accumulation, and hence attained higher grain yield and grain N when grown in the semi-arid environment.