© 2015 IOP Publishing Ltd and Deutsche Physikalische Gesellschaft. The speed meter concept has been identified as a technique that can potentially provide laser-interferometric measurements at a sensitivity level which surpasses the standard quantum limit (SQL) over a broad frequency range. As with other sub-SQL measurement techniques, losses play a central role in speed meter interferometers and they ultimately determine the quantum noise limited sensitivity that can be achieved. So far in the literature, the quantum noise limited sensitivity has only been derived for lossless or lossy cases using certain approximations (for instance that the arm cavity round trip loss is small compared to the arm cavity mirror transmission). In this article we present a generalized, analytical treatment of losses in speed meters that allows accurate calculation of the quantum noise limited sensitivity of Sagnac speed meters with arm cavities. In addition, our analysis allows us to take into account potential imperfections in the interferometer such as an asymmetric beam splitter or differences of the reflectivities of the two arm cavity input mirrors. Finally, we use the examples of the proof-of-concept Sagnac speed meter currently under construction in Glasgow and a potential implementation of a Sagnac speed meter in the Einstein Telescope to illustrate how our findings affect Sagnac speed meters with metre- and kilometre-long baselines.