In this paper, a reweighted smoothed l0-norm algorithm is proposed for direction-of-arrival (DOA) estimation in monostatic multiple-input multiple-output (MIMO) radar. The proposed method firstly performs the vectorization operation on the covariance matrix, which is calculated from the latest received data matrix obtained by a reduced dimensional transformation. Then a weighted matrix is introduced to transform the covariance estimation errors into a Gaussian white vector, and the proposed method further constructs the other reweighted vector to enhance sparse solution. Finally, a reweighted smoothed l0-norm minimization framework with a reweighted continuous function is designed, based on which the sparse solution is obtained by using a decreasing parameter sequence and the steepest ascent algorithm. Consequently, DOA estimation is accomplished by searching the spectrum of the solution. Compared with the conventional l1-norm minimization based methods, the proposed reweighted smoothed l0-norm algorithm significantly reduces the computation time of DOA estimation. The proposed method is about two orders of magnitude faster than the l1-SVD, reweighted l1-SVD and RV l1-SRACV algorithms. Meanwhile, it provides higher spatial angular resolution and better angle estimation performance. Simulation results are used to verify the effectiveness and advantages of the proposed method.