The lack of water availability led scientists to breed short duration rice varieties despite their lower yield potential than the long duration rice varieties to ensure sustainability in lowland rice fields. It is not yet clear whether this reduction in yield is due to the reduced crop duration and/or is associated with the reduction in biomass and nutrient accumulation rates. Field experiments were conducted at low-fertile and fertile sites using 22 rice varieties differing in their duration (short, medium, and long), and yield potential (low and high). Straw and grain dry weights (DW), nitrogen (N), phosphorus (P), and potassium (K) concentrations were measured. For rice varieties grown in the low-fertility site, straw and grain DWs, biomass accumulation rate (g plant−1day−1), and N, P, and K accumulation rates (mg plant−1day−1) were reduced by 63, 61, 56, 44, 79, and 43%, respectively, than those observed in the fertile site. At the low-fertile site, short duration rice varieties had only 64% straw and 76% grain DWs, 67% biomass accumulation rate, and 87% N, 82% P, and 64% K accumulation rates than those observed in medium and long duration rice varieties. The differences were less pronounced at the fertile site. Reduced biomass, N, P, and K accumulation rates, apart from the reduced crop duration, hindered grain yield and thereby the sustainability of short duration rice varieties, particularly at low-fertile sites. Thus, rice breeding programs for low-fertile soils and short duration varieties should specifically be designed.