Fewer Tokens, More Self-Teaching: On-Policy Self-Distillation for Extreme Visual Token Reduction

Sep 26, 2026· Junxian Li*, Ruixuan Yang*, Tianao Zhang, Tiange Xu, Weisheng Dong, Yulun Zhang · 1 min read

* Equal contribution

Type
Publication
arXiv
publications

Overview

Recovering multimodal capabilities under extreme visual-token reduction through on-policy self-distillation and a progressive token-budget curriculum.

LT-OPD framework with a low-token student, a frozen full-token teacher, and a token-budget curriculum

arXiv 2026 · Preprint.

Read the paper · Download PDF · Code

Multimodal LLMs Visual Token Reduction On-Policy Distillation
Ruixuan Yang
Authors
Ruixuan Yang
Undergraduate Student

I am Ruixuan Yang, an undergraduate in the School of Mathematics and Statistics at Xi’an Jiaotong University (XJTU). I’m interested in Model Efficiency, Multimodal LLMs (MLLMs), Post-training & Reasoning, etc.

I am currently a visiting student in Prof. Yulun Zhang’s group at Shanghai Jiao Tong University and Prof. Huan Wang’s ENCODE Lab at Westlake University.