What multimodal training changed
MMDiff compares sparse autoencoders (SAEs) from a base language model and its multimodal adaptation to identify internal features changed by visual training. The authors study LLaVA‑MORE, PaliGemma 2 and InternVL3.5 on visual-spatial understanding, OCR and multimodal safety.
They report that removing discovered features selectively reduced target behavior by an average 12% on spatial tasks and 17% on OCR, while reducing multimodal safety-attack success by 24%, with no measured VQA impact. Steering the features improved spatial accuracy by 3.6 percentage points and OCR by 1.8 points on average over a standard single-layer steering baseline.
Why it matters
The work suggests a way to audit and adjust specific visual behaviors without retraining an entire model. Potential uses include reducing harmful instructions hidden in images or strengthening OCR and spatial functions. “Interpretable feature,” however, does not prove a perfect match to a human concept.
Research boundary
This is a preprint covering three model families and selected benchmarks; all numbers are author-reported. Independent reproduction is needed to establish whether the same selectivity and lack of side effects hold across other architectures, production traffic and compound attacks.