Place vLLM image features at the model config's image token

[albertvillanova]

This PR fixes VLM training with vLLM in server mode for Gemma3, where vLLM generated as if the prompts had no image.

Fix #7772.

Motivation

In server mode, _place_features located the image tokens with the processor's image_token_id, which for Gemma3 on transformers 5.x is <start_of_image> instead of the image placeholder token. The features were then sent to vLLM at the wrong position, and generations ignored the images (see the issue for the reproduction).

Solution

Read the image token from the model config, which holds the placeholder token for every supported VLM, as VLLMGeneration already does for the image token suppression in #7698.

With the reproduction from the issue, the completion becomes "Red".

Changes

• Locate the image tokens with model.config.image_token_id instead of the processor's image_token_id in VLLMGeneration._place_features

There is no test: the server-mode VLM test is skipped until there is a mock for the vLLM server, and the bug needs a real Gemma3 behind a vLLM server.

---

Note

Low Risk
Single-token-ID source change in server-mode feature placement; low scope but fixes incorrect multimodal alignment for affected VLMs.

Overview
Fixes vLLM server-mode VLM training (e.g. Gemma3) where generations ignored images because multimodal features were aligned to the wrong token positions.

In VLLMGeneration._place_features, image placeholder runs in the trainer’s prompt_ids are now found using model.config.image_token_id instead of the processor’s image_token_id. For some models (Gemma3 on transformers 5.x) those IDs differ, so features were sent to vLLM at incorrect offsets. This matches how TRL already picks the image token for suppression elsewhere in the vLLM path.

^{Reviewed by Cursor Bugbot for commit 2df7030. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.