create-environments

Create Tasksets

Goal

Create native v1 tasksets that are installable and runnable with verifiers.

To start, ALWAYS use the CLI to create a package with the correct files:

uv run init my-task-v1

Add only the components the contract needs:

uv run init my-task-v1 -T      # task toolset
uv run init my-agent-v1 -H     # custom reusable harness

Often, the user does not want nor need a custom reusable harness, as verifiers offer a lot of built-in ones.

Re-use existing abstractions first

For some common tasks, there are existing, pre-built tasksets in the verifiers.v1.tasksets folder. These come with batteries included and should always be preferred. The most notable inclusion is the HarborTaskset, which allows the creation of Harbor-based tasksets within a few LoC (also see docs/v1/harbor.md).

Custom task images

When a task needs a custom container image (e.g. a Harbor task whose task.toml does not have docker_image), you can build and publish it with prime images push from the Prime CLI (Documentation). This builds in the cloud — no local Docker needed — and prints the full image reference to use as the task's image field.

Use the naming convention <env>.x86.<task>:latest for the image name (e.g. abc.x86.xyz:latest), where <env> is the taskset name and <task> is the individual task.

Define the needed values first

Before starting with the implementation, think about the following things:

• What is the dataset about, which fields does it have?
• Does it come with custom tools that are strictly necessary and not added by common harnesses? For example, a lot of harnesses come with bash or web search tools, which makes custom tools obsolete. Always prefer harnesses over custom tools
• Is the conversation driven by a user (scripted turns, a game engine, a modeled user)? That is env control flow (an interaction loop in run()), not a server.
• Does one rollout involve more than one agent run (attempts, a judge, game players)? Then the package also exports an Env subclass — or an existing bundled env (--env.id best-of-n|agentic-judge|user-sim) already covers it.
• Which rewards are needed for scoring? What additional metrics might be nice to have, either for debugging, training or potentially in the future?
• How should the tasks be scored, is a judge needed?

For a port, map source behavior one-to-one: rows, the exact prompts verbatim, harness restrictions, score extraction and exceptions.

Ask the user about unresolved semantic choices instead of inventing them. Present your evidence (both in code and in your questions) by commenting and linking to the exact source in the paper, the GitHub repo etc.

Native package contract

A package exports one vf.Taskset subclass — and optionally one vf.Env subclass (multi-agent control flow) and/or one vf.Harness subclass — through __all__. The taskset export happens automatically when you bootstrap a new taskset using uv run init.

Do not add load_environment(), load_taskset(), or load_harness() functions. The v1 loader resolves classes and their config types from __all__ and generic bases.

Use:

import verifiers.v1 as vf

Never mix v0 Environment, Rubric, Parser, SingleTurnEnv, MultiTurnEnv, or ToolEnv objects into a v1 taskset. Exclusively use functions, classes and objects from verifiers.v1.

Minimal implementation

import verifiers.v1 as vf


# One row's serializable data. Add references or other task-specific fields here.
class AdditionData(vf.TaskData):
    answer: int


# The behavior for that row. Decorated methods may request only the values they need;
# `trace` contains the full message graph and `self.data` is this task's row.
class AdditionTask(vf.Task[AdditionData]):
    @vf.reward
    async def exact_match(self, trace: vf.Trace) -> float:
        return float(trace.last_reply == str(self.data.answer))


# The taskset is the loader. Its config can be the empty base config.
class AdditionTaskset(vf.Taskset[AdditionTask, vf.TasksetConfig]):
    def load(self) -> list[AdditionTask]:
        # Construct one behavior object around each data row and the shared task config.
        return [
            AdditionTask(
                AdditionData(idx=i, prompt=f"What is {i} + {i}?", answer=2 * i),
                self.config.task,
            )
            for i in range(100)
        ]


# Export the taskset class so the v1 loader can discover it.
__all__ = ["AdditionTaskset"]

Do not override Taskset.__init__. Implement load() on the taskset and put hooks and scoring on the task.

Ownership rules

TaskData owns the immutable, serializable values for one row:

• prompts and optional system prompts;
• reference answers or other ground truth;
• container image, workdir, resources, and timeout requests;
• any additional typed fields scoring or a user/tool server needs.

Only TaskData is stored on the trace. Do not put live clients, runtime handles etc. here.

Task owns the behavior applied to that row:

• setup, finalize, and model-free validate hooks;
• stop conditions, metrics, and rewards;
• task-scoped tool declarations;
• task-facing configuration read from self.config.

Taskset owns loading and selection-time concerns. Its load() constructs the tasks, its direct config fields hold dataset/split/seed/sample-count knobs, and Taskset.toolsets may construct task-agnostic servers shared by one environment worker's rollouts.

The harness owns:

• the reusable agent or chat program;
• how that program is provisioned and launched;
• wiring model requests to the supplied interception endpoint and secret;
• harness-generic execution metrics.

Runtime config chooses where code executes. Task hooks should use the vf.Runtime interface they receive instead of assuming Docker-, Prime-, Modal-, or host-specific implementation details.

Scoring rules

• Prefer deterministic verification grounded in the task's actual artifact or answer.
• Use an LLM judge only when semantic judgment is unavoidable.
• Metrics are for observability and do not contribute to reward, but are useful. Use them deliberately and appropriately!
• Judgement that compares the sibling traces of one episode (best-of-n selection, zero-sum payoffs) lives on Env.finalize(task, episode) — attach via trace.record_reward/record_metric, in program order; no live runtime there.
• Raise ordinary Python exceptions from rollout hooks and scoring. The rollout records them as TaskError.

Validation and lifecycle

Implement Task.validate(self, runtime) whenever ground truth can be checked without a model. Keep rollout work on the task:

• setup(self, trace, runtime) — prepare files or services.
• harness execution — let the agent act.
• finalize(self, trace, runtime) — capture artifacts needed for scoring.
• @vf.reward / @vf.metric — evaluate while the runtime is still live.

Persist inspectable artifacts in JSON-serializable trace.info. Put counters and live coordination in a typed vf.State subclass.

Tools

Some tasksets require custom tools. These should be the exception as they don’t work with every harness and are registered as MCP servers.

class SearchToolset(vf.Toolset[vf.ToolsetConfig]):
    TOOL_PREFIX = "search"

    @vf.tool
    async def query(self, text: str) -> list[str]:
        # Tool docstrings are exposed to the model as the MCP tool description.
        """Search the task corpus."""
        return []


class SearchTaskConfig(vf.TaskConfig):
    tools: vf.ToolsetConfig = vf.ToolsetConfig()


class SearchTask(vf.Task[vf.TaskData, vf.State, SearchTaskConfig]):
    # Constructing on Task.toolsets gives it one-server-per-rollout scope.
    @classmethod
    def toolsets(cls, config: SearchTaskConfig) -> list[vf.Toolset]:
        return [SearchToolset(config.tools)]


if __name__ == "__main__":
    SearchToolset.run()

Choose placement from the tool's lifetime and filesystem needs:

• Task-scoped, own runtime: construct the server in Task.toolsets with a vf.ToolsetConfig field. One server is launched per rollout. The default subprocess runtime is inexpensive and host-side.
• Task-scoped, colocated: set colocated = true on its ToolsetConfig when the tool must see the harness's filesystem or processes. It still launches once per rollout.
• Taskset-scoped, shared: parameterize the toolset with vf.SharedToolsetConfig, put its config field directly on TasksetConfig, and construct the server in Taskset.toolsets.
• Existing remote service: set url on the toolset's config. Verifiers connects to the streamable-HTTP MCP endpoint instead of launching the class locally.

User simulation

There is one mechanism: the interaction — agents.<name>.interaction(task) in the env's run(); whoever calls turn() is the run's user, one harness segment per turn (the program yields, the caller answers, the next segment resumes the exchange with the answer). A prompt-less task is opened by the first turn(message); a prompted task speaks first (bare turn()); to hide a scenario prompt from the wire, hand the interaction a task copy with prompt=None and keep scoring on non-prompt fields (the user-sim shape). There is no user server to declare or place; who computes the turns is env control flow:

• Scripted user (replay pre-generated turns, step a game engine): a plain loop inside an Env.run() override — see environments/alphabet_sort or the bundled textarena taskset.
• Modeled user (an LLM playing the user): another agent role, driven live via agents.user.interaction(...) and relayed into the assistant's run — or just use the bundled user-sim env (--env.id user-sim), which does exactly this from the task's prompt-as-scenario.

The harness running the assistant must be able to resume an exchange: transcript-backed resume (`

---

Content truncated.