Repository navigation
Load the tokenizer for models that take no images - #7774
qgallouedec wants to merge 1 commit into
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1e9e5821a1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # Processing class | ||
| if processing_class is None: | ||
| processing_class = AutoProcessor.from_pretrained( | ||
| processor_class = AutoProcessor if model_accepts_images(model) else AutoTokenizer |
There was a problem hiding this comment.
When processing_class is omitted for a multimodal checkpoint intentionally instantiated through AutoModelForCausalLM, the same failure remains in trl/experimental/tpo/tpo_trainer.py:340-348, sdft/sdft_trainer.py:301-308, and sdpo/sdpo_trainer.py:418-425: they still load AutoProcessor, so TPO rejects the resulting processor and the other trainers incorrectly enter multimodal preprocessing for the text-only model. Propagate this model-aware tokenizer/processor selection to those duplicated initialization blocks as well.
AGENTS.md reference: AGENTS.md:L35-L35
Useful? React with 👍 / 👎.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
| # Before transformers 5.21, several VLMs (e.g. Qwen3.5) report `input_modalities == "text"`, see | ||
| # huggingface/transformers#49677 | ||
| if Version(transformers.__version__) < Version("5.21.0"): | ||
| return "vision_config" in model.config.sub_configs |
There was a problem hiding this comment.
Checked how reliable the vision_config fallback is: over every class AutoModelForCausalLM and AutoModelForImageTextToText can load, it matches whether the model takes images for
- 251/258 on transformers main and
- 162/167 on 4.56.2.
The misses are
- Chameleon and Emu3 (image tokenizer under
vq_config), - Emu3 (image tokenizer under
vq_config), - Fuyu (no vision encoder),
- T5Gemma2 (encoder-decoders)
- Udop (encoder-decoders)
- VisionEncoderDecoder (encoder-decoders)
- PI0 (robotics).
None of them is tested in TRL or shows up in our telemetry, and passing a processor still works for them. From transformers 5.21 the util reads input_modalities instead.
This branch has not been deployed
1 participant
When no
processing_classis passed, the trainers loadAutoProcessor. A multimodal checkpoint loaded as a language model alone, like Qwen3.5 withAutoModelForCausalLM, then gets the VLM processor and goes through the VLM path, so users have to pass the tokenizer themselves. This addsmodel_accepts_images, and SFT, DPO, KTO, GRPO, RLOO and Distillation loadAutoTokenizerwhen the model takes no images.input_modalities. Before that, several VLMs (Qwen3.5 among them) report"text"there (Declare the input modalities of the VLMs that report text only transformers#49677), so it checks for avision_configinstead.Note
Medium Risk
Touches default initialization across six trainers and version-dependent modality detection; wrong classification could still route models through the wrong tokenizer/processor path.
Overview
Fixes default preprocessing when trainers auto-load a processing class: text-only models no longer always get
AutoProcessor.Adds
model_accepts_imagesin trainer utils. It returns true only when the loaded model actually handles images (usesinput_modalitieson transformers ≥5.21, otherwisevision_configinsub_configsfor VLMs that misreport modalities). A multimodal checkpoint loaded asAutoModelForCausalLM(language model only) correctly returns false.SFT, DPO, KTO, GRPO, RLOO, and Distillation now pick
AutoProcessorvsAutoTokenizerfrom that helper whenprocessing_classis omitted, so Qwen3.5-style LM-only loads use the tokenizer path instead of the VLM processor.TestModelAcceptsImagesparametrizes causal LM, full VLM, and Qwen3.5 LM-only cases.Reviewed by Cursor Bugbot for commit 1e9e582. Bugbot is set up for automated code reviews on this repo. Configure here.