The Bayan Obo area in the North China Craton (NCC) contains the largest rare earth element (REE) resources in the world. The key to this gigantic and efficient REE accumulation has been the focal point of century-long research efforts. Intense differentiation of mantle-derived alkaline magmas is envisaged as a highly viable process for REE enrichment based on the first discovery of mafic-alkaline-carbonatitic suite in the Bayan Obo region. Geochronological and Sr-Nd-Hf-O isotopic analyses revealed that these rocks derived from a common mantle source and evolved with magmatic differentiation at 1.31 Ga to 1.32 Ga. Elemental geochemistry and Mg isotopes in this magma series (alkaline diabase sill, syenite dyke and carbonatite) unambiguously support the differentiation model. The earliest ferrocarbonatite, with nearly undifferentiated REE concentrations, has mantle-like δ26Mg (−0.27 ± 0.03‰), and therefore can be considered as the original product of the mantle derived alkaline magma at Bayan Obo. The subsequent magnesiocarbonatites have a LREE enrichment feature, which is consistent with the alkaline diabase sills. The magnesiocarbonatites possess light Mg isotopic composition (from − 0.42‰ to − 0.50‰), whereas the alkaline diabase sills have heavy Mg isotopic composition (from − 0.03‰ to − 0.05‰). This rock assemblage should present two end members, which differentiated likely by immiscibility in the chamber from a common alkaline magma. The subsequent syenite dyke has no obvious Mg isotopic variation (from − 0.44‰ to − 0.39‰) during differentiation from the above alkaline magma. Notably, as the most important REE hosts, fine-grained ferrocarbonatites show large Mg isotopic variations (from − 0.85‰ to + 0.13‰) and present positive correlations with total MgO and CaO contents, but negative with total Fe2O3, P2O5 and REE contents. In fine-grained ferrocarbonatite, closely associated apatite, monazite, hematite and magnetite minerals occur as a special intergrowth texture. This unique paragenetic assemblage is likely the mineralization record of a Fe-P-REE-rich melt, which shows particular affinity of hydrothermal fluid and light Mg isotope in carbonatitic magma. By contrast, magnesian siderite phenocrysts as the early cumulates in carbonatitic magma are enriched in heavy Mg isotope (from − 0.07‰ to + 0.21‰). Immiscibility of Fe-P-REE-rich melt and fractional crystallization of carbonate minerals are responsible for large Mg isotope fractionation and giant REE enrichment in the Bayan Obo deposit.