Increasing food demand has led to more intensive farming, which threatens our ecosystem and human health due to toxic elements accumulation. This study aimed to estimate the vulnerability of different agricultural systems with unequal high fertilizer input practices regarding toxic element pollution in the greenhouse, kiwifruit orchard, cereal field, and forest/grassland. Soil samples were collected from 181 sites across Shaanxi Province, China, and analyzed for selected characteristics and toxic elements (As, Cd, Cr, Cu, Hg, Pb, and Zn). The contamination factor (CFx) represents the ratio of the measured value of the toxic element in the soil over the soil background values. The CFx values of all the toxic elements were above background values, while Cd and Hg contamination levels were more severe than those of Zn, Cu, As, Cr, and Pb. Kiwifruit orchards and greenhouse soils were contaminated with Cd, Hg, Cu, and Zn, but cereal fields and forest/grassland soils were contaminated with As, Cd, Hg, and Hg. Overall, the cumulative pollution load (PLI) of toxic elements indicated moderate contamination. The cumulative ecological risk (RI) results indicated that greenhouse (178.81) and forest/grassland (156.25) soils were at moderate ecological risks, whereas kiwifruit orchards (120.97) and cereal field (139.72) soils were at low ecological risks. According to a Pearson correlation analysis, Cd, Hg, Cu, and Zn were substantially linked with soil organic matter (SOM), total nitrogen (TN), total phosphorous (TP), and total potassium (TK). The primary sources of toxic elements were phosphate and potash fertilizers, manure, composts, and pesticides in a greenhouse, kiwifruit orchards, and cereal fields, whereas, in forest/grassland soils parent material and atmospheric deposition were the sources identified by positive matrix factorization (PMF). Furthermore, the partial least square structural equation model (PLS-SEM) demonstrated that agriculture inputs largely influenced toxic elements accumulation. We conclude that high fertilizer inputs in greenhouse soils should be considered carefully so that toxic element pollution may be minimized.