Prevent VLM policies from generating the image placeholder token with vLLM

[albertvillanova]

This PR prevents VLM policies from generating their image placeholder token when completions are generated with vLLM, in both colocate and server mode.

Follow-up to #7695, which covers generation with transformers. See the root cause in #7692.

Motivation

When a VLM policy samples its image placeholder token in a completion, the forward pass over prompt + completion has more image tokens than image features. Training then crashes with ValueError: Image features and image tokens do not match. With vLLM generation, nothing prevents the policy from sampling this token.

Solution

When the processing class is a processor, VLLMGeneration sets logit_bias for the image placeholder token to -100, the lowest bias vLLM accepts. The token ID is taken from the model config, as in #7695. If users set logit_bias in generation_kwargs, their value overrides this.

Since GRPOTrainer, RLOOTrainer and DistillationTrainer all generate through VLLMGeneration, they are all covered, along with the experimental trainers built on it.

Verified with Qwen/Qwen2.5-VL-3B-Instruct, restricting generation to the image token and one other token through allowed_token_ids: on main, GRPO (colocate and server) and Distillation (colocate and server) crash with the error above, and with this PR they all train.

Changes

• Bias the image placeholder token out of vLLM sampling in VLLMGeneration for VLMs
• Add a GRPO test checking that vLLM generation suppresses the image placeholder token

---

Note

Low Risk
Targeted sampling guard for VLM vLLM paths with user override preserved; covered by a new GRPO test.

Overview
vLLM generation for VLMs now applies a default logit_bias of -100 on the model’s image_token_id when the processing class is a ProcessorMixin, so completions cannot sample the image placeholder token (which would break prompt+completion forward passes). User-supplied generation_kwargs still win because they are merged after this default.

A new GRPO integration test (test_vllm_generation_suppresses_image_token) runs colocated vLLM with allowed_token_ids limited to the image token and a low-ranked token, and asserts training completes without the image-token mismatch failure.

^{Reviewed by Cursor Bugbot for commit 98fc5b0. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a97314e. Configure here.}

[qgallouedec]

Thanks! So I checked myself from a gpu run: config is indeed the correct source: on the tiny VLMs it matches the processor everywhere (except Gemma3, where the processor gives <start_of_image> instead of the placeholder).

• The test only checks the kwargs. Restricting generation to the image token and one the model ranks far below it makes it behavioral: with allowed_token_ids=[image_token_id, 43476] on SmolVLM-256M and a reward function, one training step crashes on main and passes here 👍
• _place_features reads self.processing_class.image_token_id, so on Gemma3 in server mode it finds one run of length 1 at <start_of_image> instead of the 256 placeholder tokens. Storing model.config.image_token_id once in __init__ and using it in both places fixes it (checked just on the token runs, not end to end).

[albertvillanova]

Thanks for checking! The test now trains one step with generation restricted to the image token and one other token: it crashes on main (masked_scatter_size_check) and passes here.

The _place_features issue is a separate bug that predates this PR, so I opened #7772 and fixed it in #7773: with Gemma3 in server mode, vLLM was generating as if there were no image.