Naturalizing neuromorphic vision event streams using generative adversarial networks

Dennis E. Robey, Wesley Thio, Herbert H.C. Iu, Jason K. Eshraghian

Research output: Chapter in Book/Conference paperConference paperpeer-review

2 Citations (Scopus)

Abstract

Dynamic vision sensors are able to operate at high temporal resolutions within resource constrained environments, though at the expense of capturing static content. The sparse nature of event streams enables efficient downstream processing tasks as they are suited for power-efficient spiking neural networks. One of the challenges associated with neuromorphic vision is the lack of interpretability of event streams. While most application use-cases do not intend for the event stream to be visually interpreted by anything other than a classification network, there is a lost opportunity to integrating these sensors in spaces that conventional high-speed CMOS sensors cannot go. For example, biologically invasive sensors such as endoscopes must fit within stringent power budgets, which do not allow MHz-speeds of image integration. While dynamic vision sensing can fill this void, the interpretation challenge remains and will degrade confidence in clinical diagnostics. The use of generative adversarial networks presents a possible solution to overcoming and compensating for a vision chip's poor spatial resolution and lack of interpretability. In this paper, we methodically apply the Pix2Pix network to naturalize the event stream from spike-converted CIFAR-10 and Linnaeus 5 datasets. The quality of the network is benchmarked by performing image classification of naturalized event streams, which converges to within 2.81% of equivalent raw images, and an associated improvement over unprocessed event streams by 13.19% for the CIFAR-10 and Linnaeus 5 datasets.

Original languageEnglish
Title of host publication2021 IEEE International Symposium on Circuits and Systems, ISCAS 2021 - Proceedings
PublisherIEEE, Institute of Electrical and Electronics Engineers
ISBN (Electronic)9781728192017
DOIs
Publication statusPublished - 2021
Event53rd IEEE International Symposium on Circuits and Systems, ISCAS 2021 - Daegu, Korea, Republic of
Duration: 22 May 202128 May 2021

Publication series

NameProceedings - IEEE International Symposium on Circuits and Systems
Volume2021-May
ISSN (Print)0271-4310

Conference

Conference53rd IEEE International Symposium on Circuits and Systems, ISCAS 2021
Abbreviated titleISCAS 2021
Country/TerritoryKorea, Republic of
CityDaegu
Period22/05/2128/05/21

Fingerprint

Dive into the research topics of 'Naturalizing neuromorphic vision event streams using generative adversarial networks'. Together they form a unique fingerprint.

Cite this