Molt

Molt is agentic-first and PyTorch-native. The agent is the program; the trainer is a single actor; reward is any Python you write inside an Env or ChatAgent: graders, multi-turn tools, VLM environments, LLM-as-judge. Three components carry the rest: Ray for placement and async queues, vLLM for rollout, and NVIDIA AutoModel + FSDP2 for training in pure PyTorch.

That is the whole stack: about 9.2K lines of RL code that scale to 1T-class MoE on vLLM with TP / EP / CP, think DeepSeek-V3 at --fsdp.ep_size 256, with Adam CPU offload for the largest actors. One agent API, one trainable actor, clean enough to read end to end.

Read the Tech Report Code

Ray owns placement and the async queue between the three boxes; that is the entire runtime. The contract is token-first: token ids, logprobs, action ranges, rewards, and multimodal tensors stay aligned from rollout to training. Anything you can compute in Python is a valid reward, including LLM-as-judge calls back through the same vLLM engines that drive rollout.

Install from the repo (or pull the prebuilt container, recommended for the full CUDA stack):

git clone https://github.com/NVIDIA-NeMo/labs-molt.git
cd labs-molt
pip install -e ".[vllm]"      # or: docker pull hijkzzz/molt:latest
# checkout-free: pip install "molt-rl[vllm]"

Launch RL with one trainable actor and your agent as the reward:

python3 -m molt.cli.train_rl_ray \
  --actor.model_name_or_path /path/to/automodel \
  --data.prompt_dataset /path/to/prompts.jsonl \
  --data.input_key input \
  --train.agent_path examples/python/agents/math.py \
  --vllm.num_engines 2 \
  --vllm.tensor_parallel_size 2 \
  --rollout.batch_size 128 \
  --train.batch_size 128 \
  --train.micro_batch_size 1 \
  --algo.advantage.estimator reinforce \
  --algo.kl.init_coef 0 \
  --fsdp.attn_implementation te \
  --ckpt.output_dir ./ckpt/rl

Reference recipes under examples/scripts/ span Qwen3-4B dense text RL, Qwen3.6-35B-A3B VLM RL with a multi-turn Python tool, Nemotron-Omni-30B-A3B hybrid SSM MoE at CP8+EP8, on-policy distillation, and GLM-5.2 ~750B RL at EP256.

If you use Molt in your research, please cite:

@article{hu2026molt,
  title         = {Molt: A Scalable PyTorch-Native Training Framework for Agentic Reinforcement Learning},
  author        = {Hu, Jian and Li, Huiying and Zhang, Hao and Xu, Binfeng and Zhang, Yifan and Zhang, Shaokun and Desai, Hemil and Demoret, Michael and Molchanov, Pavlo and Kautz, Jan and Dong, Yi},
  year          = {2026},
  eprint        = {2607.21653},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2607.21653}
}

Copy BibTeX