Terraces are an important cultivated land resource. Terrace abandonment affects the soil quality, soil and water conservation benefits, and biodiversity of terraces. Therefore, it is important to quantify the number and spatial distribution of abandoned terraces to protect cultivated land and food security. However, the traditional remote sensing method cannot identify small plots and make accurate assessment of abandoned farmland quickly in mountainous areas. To accurately identifying abandoned terraces, this study used semantic segmentation based on deep learning and change detection to identify abandoned terraces and their spatial distribution in a small watershed on the Loess Plateau in 2021. A comparative analysis of the accuracy of three deep learning models revealed that RefineNet is superior to DeepLabv3+ and DeepLabv3 for identifying abandoned terraces. The user's accuracy, producer's accuracy, overall accuracy, and appa values for RefineNet were 0.817, 0.894, 0.800, and 0.539, respectively. For change detection, the corresponding values were 0.821, 0.753, 0.731, and 0.426, respectively. In addition, semantic segmentation produced better recognition results than change detection in complex terrain and geomorphological areas. The abandoned terraces in the study area were mainly distributed in mountainous areas far from residential areas and more likely at high elevations with large slopes. This study provides a new method for recognizing abandoned terraces and spatial distribution information for managing and utilizing abandoned terraces.