Multi-class Token Transformer for Weakly Supervised Semantic Segmentation

Lian Xu, Wanli Ouyang, Mohammed Bennamoun, Farid Boussaid, Dan Xu

Research output: Chapter in Book/Conference paperConference paperpeer-review

62 Citations (Scopus)
72 Downloads (Pure)


This paper proposes a new transformer-based framework to learn class-specific object localization maps as pseudo labels for weakly supervised semantic segmentation (WSSS). Inspired by the fact that the attended regions of the one-class token in the standard vision transformer can be leveraged to form a class-agnostic localization map, we investigate if the transformer model can also effectively capture class-specific attention for more discriminative object localization by learning multiple class tokens within the transformer. To this end, we propose a Multi-class Token Transformer, termed as MCTformer, which uses multiple class tokens to learn interactions between the class tokens and the patch tokens. The proposed MCTformer can successfully produce class-discriminative object localization maps from the class-to-patch attentions corresponding to different class tokens. We also propose to use a patchlevel pairwise affinity, which is extracted from the patchto-patch transformer attention, to further refine the localization maps. Moreover, the proposed framework is shown to fully complement the Class Activation Mapping (CAM) method, leading to remarkably superior WSSS results on the PASCAL VOC and MS COCO datasets. These results underline the importance of the class token for WSSS.(-1)
Original languageEnglish
Title of host publication2022 IEEE/CVF Conference on computer vision and pattern recognition (CVPR 2022)
Place of PublicationUSA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Number of pages10
ISBN (Electronic)9781665469463
ISBN (Print)9781665469463
Publication statusPublished - 2022
EventIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) - New Orleans, Lao People's Democratic Republic
Duration: 18 Jun 202224 Jun 2022


ConferenceIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Country/TerritoryLao People's Democratic Republic
CityNew Orleans


Dive into the research topics of 'Multi-class Token Transformer for Weakly Supervised Semantic Segmentation'. Together they form a unique fingerprint.

Cite this