UniAfford: Token-Routed Multitask Learning for Generalizable 2D-3D Affordance Perception

Paper Overview

UniAfford-Data unifies pixel-level 2D affordance masks and point-level 3D affordance labels, while UniAfford routes MLLM hidden states into dense 2D and 3D prediction branches.

UniAfford-Data

🤗 Hugging Face Dataset

UniAfford-Data is a hybrid affordance dataset built by aligning heterogeneous 2D and 3D affordance resources under a shared object-affordance taxonomy. It preserves modality-specific supervision while exposing a unified instruction-driven interface for joint 2D-3D training.

2D Source

Pixel masks

RGB images with pixel-level affordance masks provide dense visual grounding for interaction regions such as graspable, supportable, or containable parts.

3D Source

Point labels

Point-cloud annotations describe affordance regions directly on object geometry, enabling point-wise evaluation and 3D interaction reasoning.

Language

Instructions

Natural-language prompts connect object categories, affordance labels, and modality-specific targets so the model can learn a common task interface.

UniAfford-Data 2D mask visualization examples — 2D examples: RGB objects paired with pixel-level affordance masks.

UniAfford-Data 3D point-cloud label visualization examples — 3D examples: point clouds with affordance-specific point labels.

Abstract

Affordance perception aims to highlight the actionable regions that support embodied interaction, yet 2D and 3D affordance grounding have evolved as separate problems, with different task definitions, supervision formats, datasets, and evaluation protocols. We propose Token Router for Tasks, a general multitask training paradigm for MLLM-based systems that routes contextual hidden states to task-specific branches instead of forcing downstream tasks to be represented by predefined textual markers. We instantiate this paradigm as UniAfford, a unified framework for generalizable 2D-3D affordance perception, together with UniAfford-Data, a unified dataset organized by a shared object-affordance taxonomy that integrates pixel-level 2D annotations, point-level 3D annotations, and language instructions. Experiments show strong zero-shot generalization on both 2D and 3D affordance benchmarks without target-specific fine-tuning.

Method Overview

Token Router

Soft branch assignment

The router predicts text / image / point-cloud probabilities for each MLLM hidden state. Supervision from <img_aff> and <pc_aff> anchors downstream affordance queries.

Similarity Alignment

Similarity-based 2D-3D grounding

Routed tokens are similarity-matched to pixel-level (2D) and point-level (3D) features in a shared normalized space with a learned logit scale.

Results

Metrics

Quantitative comparison

This section shows comparison results for some of the models in our paper.

2D Method	Benchmark	gIoU	cIoU
Affordance-R1	zero-shot on AGD20k	31.78	27.85
UniAfford (Ours)	zero-shot on AGD20k	27.52	25.22
AffordanceNet	zero-shot on AGD20k	14.10	20.04

3D Method	Benchmark	AUC	mIoU
UniAfford (Ours)	zero-shot on GEAL*	77.33	14.25
GREAT	zero-shot on GEAL*	73.61	8.82
IAGNet	zero-shot on GEAL*	71.84	7.95

2D Prediction

Image affordance comparison

The red part in the image is the model’s prediction result, processed according to the fixed binary threshold.

3D Prediction

Point-cloud affordance comparison

In the image, the red dots predicted by the model are the binarized result of the model’s output, while the GT shows the annotated heatmap.

Resources

Code

Repository

Installation, training, validation, and demo usage are documented in the repository README.

Open README

Data

Dataset

The dataset format and split convention are documented separately for release packaging.

Open DATASET.md

Rendering

Figures and Demos

Use the rendering utilities to export 2D overlays, 3D point-cloud figures, grids, and thumbnails.

Open rendering guide

Citation

@article{uniafford2026,
  title   = {UniAfford: Token-Routed Multitask Learning for Generalizable 2D-3D Affordance Perception},
  author  = {Yuhao Liu, Yiming Zhong, Hanqing Wang, Shaocheng Yan, Yuhang Zhang, Wenzhou Lyu, Ziyang Ding, Wei Zhang, Xue Zhao, Jin Pan, Yuexin Ma, Xinge Zhu },
  journal = {Coming soon},
  year    = {2026}
}