Quantum key distribution (QKD) is moving from research laboratories towards applications. As computing becomes more mobile, cashless as well as cardless payment solutions are introduced. A possible route to increase the security of wireless communications is to incorporate QKD in a mobile device. Handheld devices present a particular challenge as the orientation and the phase of a qubit will depend on device motion. This problem is addressed by the reference frame independent (RFI) QKD scheme. The scheme tolerates an unknown phase between logical states that vary slowly compared to the rate of particle repetition. Here we experimentally demonstrate the feasibility of RFI QKD over a free-space link in a prepare and measure scheme using polarization encoding. We extend the security analysis of the RFI QKD scheme to be able to deal with uncalibrated devices and a finite number of measurements. Together these advances are an important step towards mass production of handheld QKD devices.