Add Beam: tiny model, tests and SFT example

[qgallouedec]

Adds Beam to TRL: the tiny-model script, the tiny model in the SFT/DPO/GRPO/RLOO, chat-template, data and fused LM head tests, the GRPO tool-calling docs, and a LoRA SFT example on one 8xB300 node (DistributedConfig(fsdp_size=8, ep_size=8), FA4, packing, assistant-only loss on HuggingFaceH4/Multilingual-Thinking: 36 steps of 1.75 s, 223 GiB per GPU).

• The tests skip below transformers 5.20.1 and need trl-internal-testing/tiny-BeamForCausalLM, pushed by the new script.

Loss over 500 steps on a longer dataset:

---

Note

Low Risk
Docs, an isolated example, a tiny-model generator, and test parametrization only; no changes to core training logic.

Overview
Adds Beam (reflection/Beam-1.0-501B-A23B) as a documented and tested model family in TRL, without changing trainer implementation code.

A new examples/sft_beam LoRA SFT recipe runs on one 8×B300 node with load-time sharding via DistributedConfig(fsdp_size=8, ep_size=8), Flash Attention 4, sequence packing, and assistant-only loss on HuggingFaceH4/Multilingual-Thinking (assistant reasoning in a thinking field). README, docs index, and the examples index call this out; GRPO docs list Beam among tool-calling–tested models.

scripts/generate_tiny_models/for_causal_lm/beam_for_causal_lm.py builds and pushes trl-internal-testing/tiny-BeamForCausalLM (transformers ≥ 5.20.1). Tests wire that checkpoint into SFT/DPO/GRPO/RLOO smoke runs, chat-template utilities (including thinking like gpt-oss), data utils, fused LM head paths, and MoE aux-loss expectations (0 coef, no aux loss).

^{Reviewed by Cursor Bugbot for commit d8883d1. Bugbot is set up for automated code reviews on this repo. Configure here.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-11T13:56:11.908354Z	dd287d4	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 139892d. Configure here.}