Background: Social genetic effects (SGE) are the effects of the genotype of one animal on the phenotypes of other animals within a social group. Because SGE contribute to variation in economically important traits for pigs, the inclusion of SGE in statistical models could increase responses to selection (RS) in breeding programs. In such models, increasing the relatedness of members within groups further increases RS when using pedigree-based relationships; however, this has not been demonstrated with genomic-based relationships or with a constraint on inbreeding. In this study, we compared the use of statistical models with and without SGE and compared groups composed at random versus groups composed of families in genomic selection breeding programs with a constraint on the rate of inbreeding. Results: When SGE were of a moderate magnitude, inclusion of SGE in the statistical model substantially increased RS when SGE were considered for selection. However, when SGE were included in the model but not considered for selection, the increase in RS and in accuracy of predicted direct genetic effects (DGE) depended on the correlation between SGE and DGE. When SGE were of a low magnitude, inclusion of SGE in the model did not increase RS, probably because of the poor separation of effects and convergence issues of the algorithms. Compared to a random group composition design, groups composed of families led to higher RS. The difference in RS between the two group compositions was slightly reduced when using genomic-based compared to pedigree-based relationships. Conclusions: The use of a statistical model that includes SGE can substantially improve response to selection at a fixed rate of inbreeding, because it allows the heritable variation from SGE to be accounted for and capitalized on. Compared to having random groups, family groups result in greater response to selection in the presence of SGE but the advantage of using family groups decreases when genomic-based relationships are used.