Agentic RL · Agent Skills · Executable environments
Reinforcing agents with collective skills.
Public Agent Skills compiled into executable terminal environments, each with programmatic tests and a behavioral rubric.
We introduce Skill2Env, a human-aligned and scalable data recipe for modern agentic RL. The path to safe superintelligence requires behavioral alignment with collective human values and expertise. Public Agent Skills provide a promising source of this supervision: they represent a task distribution most relevant to human society, linking to real-world artifacts, packaging domain expertise and reusable workflows, and often defining success metrics.
We develop a pipeline to turn 3.4k web-crawled and filtered Agent Skills into 8k executable terminal environments with both programmatic tests and behavioral rubrics. Through extensive RL experiments, we observe promising improvements in general agentic abilities. With merely 300 steps of RL training using a DPPO variant, Qwen3.8-27B gains 4.7 percentage points on Terminal-Bench 2.1, and 4.3 percentage points on S2EBench, our hand-verified private benchmark featuring real-world agent use cases. Further analysis indicates increased behavioral alignment with original Agent Skills on similar problems.
An Agent Skill is a folder rooted at a SKILL.md file, with optional scripts, references and assets, that teaches an agent how to do one job. Read as data, a public Skill corpus is a sample of the tasks people want agents to handle, a set of pointers to real artifacts and tools, and a body of quality criteria that outcome tests alone cannot express. Skill2Env compiles one Skill into several tasks in the Harbor format. The planner and creator are Codex CLI agents running in Docker containers with network access, so they can read documentation, clone repositories and download public data while authoring.
The planner reads the whole Skill bundle and lists the distinct workflows it teaches, ranked by whether they are verifiable from the final state, well specified and solvable, and difficult for a reason inherent to the work. Each workflow gets a meta plan and a list of public assets to ground it.
The host draws a task archetype, verifier pattern and persona from a pool conditioned on the workflow, together with a complexity level, an instruction tone and a requester expertise level, so a research workflow is paired with a researcher, not an arbitrary draw.
A fresh creator agent builds the initial world, the instruction, the tests and the rubric, and only then the reference solution. The grading contract is frozen before the solution exists; a failing solution is a solution bug, not a reason to weaken the verifier.
A host gate with no model in the loop runs static checks, rejects leakage of tests or solutions into the environment, and pins every base image by digest. The reference solution must earn full reward and a no-op agent must earn zero.
Programmatic tests. tests/test.sh runs after the episode against the final container state and writes up to six named metrics in [0, 1], each covering a distinct facet of the workflow. Training averages them into one scalar reward, so most tasks carry partial credit rather than a binary outcome.
Behavioral rubric. tests/rubric.md lists Must-do, Must-avoid and Best-practice items distilled from the Skill's own methodology. The rubric may not introduce hidden hard requirements; it separates verifiable correctness from procedural quality.
task_<skill>_<id>/
instruction.md # prompt / request
task.toml # config and axis metadata
environment/Dockerfile # build context = initial world
tests/test.sh # writes /logs/verifier/reward.json
tests/rubric.md # Must-do / Must-avoid / Best-practice
solution/solve.sh # reference solution (Oracle)SkillHub. Over 5k Skills were crawled from public GitHub repositories and filtered for safety and licensing. A Skill is dropped if its workflow depends on an external account, API key or paid service, on physical hardware, or if it is a persona or copywriting prompt with no executable workflow; near-duplicates are removed by N-gram coverage. The result is SkillHub, about 3.4k Skills under MIT or Apache 2.0 licenses.
Coverage. The corpus used in the paper holds 7,971 tasks from about 3.4k Skills. Among the datasets compared in the paper, Skill2Env is the only one that covers all 13 domains, and the only one in which non-technical knowledge work (business, marketing, research, documents, product, design, productivity, health) forms a large share; software engineering accounts for under a quarter of it.
Fidelity to the source Skill. Using a task's instruction and rubric as the query, TF-IDF retrieval over the 3.4k SKILL.md files ranks the true source Skill first for 73.2% of tasks and within the top 10 for 94.6%, against a chance rate of 0.03%. The rubric alone retrieves its Skill at 68.5% top-1, so it carries Skill-specific methodology rather than generic advice.
S2EBench. A separate held-out set generated from SkillHub Skills and reviewed by hand: the instruction must be unambiguous, the task faithful to its Skill and meaningfully difficult, and the tests well-rounded and unbiased. 79 tasks remain.
The released dataset on Harbor Hub is generated with Kimi-K3-max under the same pipeline.
Setup. Qwen3.8-27B trained with group advantages and the binary-KL variant of DPPO: 8 rollouts per group, one optimizer update per 64 rollouts, with no KL penalty, entropy bonus or importance-sampling correction. Molt runs fully asynchronous training; Polar runs each rollout as an unmodified Pi agent in a sandbox and scores the final container state with the task's test.sh. Evaluation reports pass@1 over five trials per task with the Pi harness.
| Qwen3.8-27B | Terminal-Bench 2.1 | S2EBench pass@1 | S2EBench score |
|---|---|---|---|
| Base | 49.4 | 33.4 | 56.6 |
| Outcome RL | 54.1 | 37.7 | 75.1 |
| Outcome + Rubric RL | 50.1 | 34.7 | 63.6 |
Outcome-only RL raises S2EBench pass@1 by 4.3 points and its mean score by 18.5 points. The gain transfers to Terminal-Bench 2.1 by 4.7 points, and no Terminal-Bench task shares a 13-gram Jaccard similarity above 0.8 with the training corpus: a general gain in how the model plans, uses tools and finishes, not memorization of a task family.
Benchmarks score whether a task was finished; the Skills say how it should be done. For 200 tasks, a judge receives the source SKILL.md and two anonymized trajectories in random order and picks the one that follows the Skill more closely.
| Comparison | Prefers base | Prefers RL | Equally aligned |
|---|---|---|---|
| Base vs. Outcome RL | 33.5 | 54.5 | 12.0 |
| Base vs. Outcome + Rubric RL | 24.0 | 73.0 | 3.0 |
Outcome-only RL already moves the model toward the Skills, and the rubric-calibrated checkpoint moves it furthest. The benchmarks score only the outcome half, which is where the rubric run trails. The paper reads the preference result as support for a rubric-based reward dimension, especially for less verifiable work such as web development, report synthesis and open research.
If you find this work useful, please cite:
@techreport{xu2026skill2env,
title = {Reinforcing Agents with Collective Skills},
author = {Xu, Binfeng and Hu, Jian and Zhang, Hao and Zhang, Shaokun and Zhang, Yifan and Demoret, Michael and Kautz, Jan and Dong, Yi},
institution = {NVIDIA},
year = {2026},
url = {https://github.com/NVlabs/Skill2Env}
}