The focus of this paper is to analyze earth pressure against a rigid retaining wall under various wall movement modes with a contact model considering inter-particle rolling resistance implemented into the distinct element method (DEM). Firstly, a contact model considering rolling resistance in particles was generally explained and implemented into the DEM. The parameters of the contact model are determined from DEM simulation of biaxial tests on a sandy specimen. Then, the influence of inter-particle rolling resistance in the backfill is discussed by comparing the active and passive earth pressure against a rigid wall subjected to a translational displacement with and without inter-particle rolling resistance in the material. Third, the DEM model considering the rolling resistance is used to investigate active and passive earth pressures while the rigid wall moves in a more general manner such as rotation or translation. The influence of rolling resistance on the earth pressures is examined from the microscopic particle scale (e.g., shear strain field) as well as the macroscopic scale (e.g., the magnitude and action point of resultant earth pressures). Finally, the effect of the initial density and the particle size of the backfill are discussed. The results show that when rolling resistance in the particles is taken into account in the DEM simulation, the simulation results are more appropriate and are in line with practical situation. Hence, particles rolling resistance should be taken into account to get more realistic results in DEM analyses.