Auxiliary Tasks Enhanced Dual-Affinity Learning for Weakly Supervised Semantic Segmentation

Research output: Contribution to journalArticlepeer-review

4 Downloads (Pure)


Most existing weakly supervised semantic segmentation (WSSS) methods rely on class activation mapping (CAM) to extract coarse class-specific localization maps using image-level labels. Prior works have commonly used an off-line heuristic thresholding process that combines the CAM maps with off-the-shelf saliency maps produced by a general pretrained saliency model to produce more accurate pseudo-segmentation labels. We propose AuxSegNet<inline-formula> <tex-math notation="LaTeX">$+$</tex-math> </inline-formula>, a weakly supervised auxiliary learning framework to explore the rich information from these saliency maps and the significant intertask correlation between saliency detection and semantic segmentation. In the proposed AuxSegNet<inline-formula> <tex-math notation="LaTeX">$+$</tex-math> </inline-formula>, saliency detection and multilabel image classification are used as auxiliary tasks to improve the primary task of semantic segmentation with only image-level ground-truth labels. We also propose a cross-task affinity learning mechanism to learn pixel-level affinities from the saliency and segmentation feature maps. In particular, we propose a cross-task dual-affinity learning module to learn both pairwise and unary affinities, which are used to enhance the task-specific features and predictions by aggregating both query-dependent and query-independent global context for both saliency detection and semantic segmentation. The learned cross-task pairwise affinity can also be used to refine and propagate CAM maps to provide better pseudo labels for both tasks. Iterative improvement of segmentation performance is enabled by cross-task affinity learning and pseudo-label updating. Extensive experiments demonstrate the effectiveness of the proposed approach with new state-of-the-art WSSS results on the challenging PASCAL VOC and MS COCO benchmarks.

Original languageEnglish
Pages (from-to)1-15
Number of pages15
JournalIEEE Transactions on Neural Networks and Learning Systems
Publication statusE-pub ahead of print - 13 Mar 2024


Dive into the research topics of 'Auxiliary Tasks Enhanced Dual-Affinity Learning for Weakly Supervised Semantic Segmentation'. Together they form a unique fingerprint.

Cite this