A Multi-modal, Discriminative and Spatially Invariant CNN for RGB-D Object Labeling

Research output: Contribution to journalArticlepeer-review

33 Citations (Scopus)


While deep convolutional neural networks have shown a remarkable success in image classification, the problems of inter-class similarities, intra-class variances, the effective combination of multimodal data, and the spatial variability in images of objects remain to be major challenges. To address these problems, this paper proposes a novel framework to learn a discriminative and spatially invariant classification model for object and indoor scene recognition using multimodal RGB-D imagery. This is achieved through three postulates: 1) spatial invariance - this is achieved by combining a spatial transformer network with a deep convolutional neural network to learn features which are invariant to spatial translations, rotations, and scale changes, 2) high discriminative capability - this is achieved by introducing Fisher encoding within the CNN architecture to learn features which have small inter-class similarities and large intra-class compactness, and 3) multimodal hierarchical fusion - this is achieved through the regularization of semantic segmentation to a multi-modal CNN architecture, where class probabilities are estimated at different hierarchical levels (i.e., imageand pixel-levels), and fused into a Conditional Random Field (CRF)- based inference hypothesis, the optimization of which produces consistent class labels in RGB-D images. Extensive experimental evaluations on RGB-D object and scene datasets, and live video streams (acquired from Kinect) show that our framework produces superior object and scene classification results compared to the state-of-the-art methods.

Original languageEnglish
Article number8022892
Pages (from-to)2051-2065
Number of pages15
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Issue number9
Publication statusPublished - 1 Sep 2018


Dive into the research topics of 'A Multi-modal, Discriminative and Spatially Invariant CNN for RGB-D Object Labeling'. Together they form a unique fingerprint.

Cite this