Hallucinating IDT Descriptors and I3D Optical Flow Features for Action Recognition with CNNs

Lei Wang, Piotr Koniusz, Du Huynh

Research output: Chapter in Book/Conference paperConference paperpeer-review

53 Citations (Scopus)
21 Downloads (Pure)


In this paper, we revive the use of old-fashioned handcrafted video representations for action recognition and put new life into these techniques via a CNN-based hallucination step. Despite of the use of RGB and optical flow frames, the I3D model (amongst others) thrives on combining its output with the Improved Dense Trajectory (IDT) and extracted with its low-level video descriptors encoded via Bag-of-Words (BoW) and Fisher Vectors (FV). Such a fusion of CNNs and handcrafted representations is time-consuming due to pre-processing, descriptor extraction, encoding and tuning parameters. Thus, we propose an end-to-end trainable network with streams which learn the IDT-based BoW/FV representations at the training stage and are simple to integrate with the I3D model. Specifically, each stream takes I3D feature maps ahead of the last 1D conv. layer and learns to `translate' these maps to BoW/FV representations. Thus, our model can hallucinate and use such synthesized BoW/FV representations at the testing stage. We show that even features of the entire I3D optical flow stream can be hallucinated thus simplifying the pipeline. Our model saves 20-55h of computations and yields state-of-the-art results on four publicly available datasets.
Original languageEnglish
Title of host publicationProceedings of the 2019 International Conference on Computer Vision
Place of PublicationUSA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Number of pages11
ISBN (Electronic)9781728148038
Publication statusPublished - Oct 2019
EventIEEE International Conference on Computer Vision 2019 - Seoul, Korea, Democratic People's Republic of
Duration: 27 Oct 20192 Nov 2019

Publication series

NameProceedings of the IEEE International Conference on Computer Vision
ISSN (Print)1550-5499


ConferenceIEEE International Conference on Computer Vision 2019
Abbreviated titleICCV2019
Country/TerritoryKorea, Democratic People's Republic of
OtherICCV is the premier international computer vision event comprising the main conference and several co-located workshops and tutorials. With its high quality and low cost, it provides an exceptional value for students, academics and industry researchers.
Internet address


Dive into the research topics of 'Hallucinating IDT Descriptors and I3D Optical Flow Features for Action Recognition with CNNs'. Together they form a unique fingerprint.

Cite this