Permeability is the most crucial property of coal in relation to coalbed methane production and CO2 sequestration. Due to coal’s anisotropic structure and mechanical properties, its permeability exhibits strong anisotropy. The main factors controlling coal permeability evolution are effective stress, anisotropic swelling/shrinkage near fracture surfaces (internal swelling/shrinkage), and gas rarefaction effects. Combined impacts of the above mechanisms make coal permeability evolution complex and difficult to predict. In this study, we establish a fully anisotropic coal permeability model incorporating stress sensitivity, anisotropic internal swelling/shrinkage, and gas rarefaction effects. Specifically, a mechanical-property-based internal swelling model is established to link up anisotropic internal swelling/shrinkage with mechanical anisotropy using the energy balance theory. A Knudsen-number-based model is utilized to describe gas rarefactions effects. The comparison with coal anisotropic swelling data and anisotropic permeability evolution data demonstrates the permeability model’s reliability. Results show that anisotropic internal swelling/shrinkage mainly determines the overall shape of permeability curves, the evolution trend, the range of permeability change in all directions, and the anisotropy level during evolution. It partially or totally offsets the permeability change caused by effective stress variation under certain stress conditions. Effective stress variation starts to dominate permeability evolution when the variation exceeds a certain value. Permeability increment/reduction caused by the gas rarefaction phenomenon enhancement/weakening is dependent on fracture (pore) pressure and aperture, but its influence on permeability is not as strong as that of anisotropic internal swelling/shrinkage. Anisotropic internal swelling/shrinkage and the gas rarefaction phenomenon show a synergistic influence on anisotropic permeability evolution with fracture (pore) pressure changing. The permeability model is applicable for different permeability measurement conditions.