Facial self similarity for sketch to photo matching

Automatic recognition of suspects from forensic sketches is of considerable interest to the law enforcement agencies. However, this task is complex due to the heterogenous nature of face sketches and photographs. To address this challenge, previous approaches generally learn a transformation of a sketch to photo or a photo to sketch at the image or feature level in order to reduce the modality gap. Such a transformation may be indeterministic and if learned from training data, is likely to over-fit the sketch artist's drawing technique. Instead, we formulate the problem in the context of matching local self similarities which are independently computed from a face sketch and a photo. The proposed Facial Self Similarity (FSS) descriptor is obtained by correlation of a small face patch with its local neighborhood. Thus, our approach avoids the need of a modality transformation, while implicitly reducing the inter-modality gap. The proposed FSS descriptor is evaluated on the CUHK Face Sketch database using sketch-photo pairs of 311 subjects. The FSS descriptor demonstrates high recognition accuracy of 99.53% and outperforms current techniques. We also evaluate the robustness of the descriptor to anomalies such as matching sketches to blurred photographs.