Canonical Color as a Lens into Concept Decodability in Vision Encoders and VLMs
A study constructs a dataset of objects with canonical colors and probes vision encoders for color and object identity using color and grayscale images. It finds that canonical color remains decodable from grayscale images and is tied to predicted object identity, indicating a conceptual link. Extending to full VLMs, VLM post-training can have a surprisingly large effect on color decodability in the vision encoder.
Researchers use canonical color as a controlled test case to investigate whether vision encoders make conceptual information linearly accessible even when color is removed from the input. They find that canonical color remains decodable from grayscale images and is tied to predicted object identity, indicating a conceptual link. Extending the analysis to full VLMs, they find that VLM post-training can have a surprisingly large effect on color decodability in the vision encoder. Overall, canonical color provides a usefully controllable lens for tracing object-level conceptual semantic information in vision encoders and VLMs.
The study demonstrates that vision encoders retain linearly accessible canonical color information even when color is absent from the input, suggesting that object identity and canonical color are entangled in the representation. The finding that VLM post-training significantly alters color decodability indicates that alignment or instruction tuning can reshape the internal feature geometry of the vision encoder, potentially overwriting or amplifying conceptual attributes.
This work highlights a measurable property of vision encoders that could be used to audit or debug conceptual leakage in deployed VLMs. If post-training changes decodability, model providers may need to monitor how fine-tuning affects internal representations, especially for applications where color or other conceptual attributes must be controlled or removed.
For companies building or fine-tuning VLMs, this research suggests a method to measure how much conceptual information is embedded in vision encoders and how post-training alters it. This could inform model selection, fine-tuning strategies, and safety audits where certain visual concepts must be suppressed or preserved.
Next signals include follow-up studies probing other conceptual attributes (e.g., texture, material, affordance) in vision encoders, and work examining whether decodability changes correlate with downstream task performance or bias. Researchers may also investigate whether interventions during post-training can deliberately control which concepts remain accessible.