planning-visual-tasks

planning-visual-tasks

v1.2 — the brain of the agent loop. Inputs: a user instruction + optional task_state + the discovered tool / model / template inventory. Output: a Plan envelope (matching contracts/agent/plan.schema.json) that the host Orchestrator consumes via Executor.run(sub_goal, task_state).

The v1 keyword router (scripts/decompose.py + scripts/emit_dag.py) is preserved as a deterministic, no-LLM fallback. The host POST /api/chat/invoke route picks v1.2 when an LLM provider is configured (host.llm.provider.resolve_credentials() returns a key) and falls through to the v1 router otherwise. See host/api/chat_invoke.py::_llm_available and _invoke_stream_v12.

When this Skill applies

• Any natural-language image / video generation, edit, or composition request that the host has routed through POST /api/chat/invoke.
• Single-goal instructions still go through this Skill — the LLM planner is responsible for the "do not over-decompose" decision and emits a 1-step Plan in that case.
• Multi-step "generate then adjust the light" / "and then make it dance" instructions become a multi-subgoal DAG with explicit state://, template://, and model:// refs across steps.
• Cross-turn "and then" continuations consume the prior turn's TaskState.named_outputs so an earlier image can be referenced by name in the next turn.

Do not invoke when the caller already has a fully-specified comfyui_execute sub-goal — instantiate the Executor and call its comfyui_execute handler directly.

Scripts

• scripts/propose_plan.py — v1.2 LLM-driven planner. Stdin: {instruction, task_state, available_tools}. Stdout: a Plan envelope. Wraps host.agent.planner.Planner.propose.
• scripts/discover_tools.py — stdin {}, stdout {tools, models, templates, mcp_tools}. Lists every executor kind plus stubs from any installed Stream M / W / X surfaces.
• scripts/validate_plan.py — stdin Plan, stdout {ok, errors}. Checks schema, DAG acyclicity, ref resolution.
• scripts/legacy/decompose.py — v1 keyword router (no-LLM fallback). Heuristic clause-to-skill mapping. Still callable from the legacy code path for users without an LLM key.
• scripts/legacy/emit_dag.py — v1 DAG emitter for the legacy router output.
• scripts/decompose.py and scripts/emit_dag.py — copies of the legacy scripts kept in place so v1.1 callers (the existing host.api.chat_invoke.plan_instruction) keep working without modification. Removing these breaks the v1.1 fallback path.

Output contract — v1.2

The Plan envelope shape is canonical (contracts/agent/plan.schema.json): {plan_id, task_id, user_instruction, execution_order, subgoals[]}. Sub-goal kind values are limited to: workflow_search, model_resolve, model_download, comfyui_execute, llm_transform, evaluate, mcp_tool, checkpoint, wait_user. Cross-step refs use scheme://sgN/<name> format only (state, template, model, artifact schemes); the legacy $sgN.field form is rejected by the executor's resolve_inputs unless allow_legacy_refs=True is set explicitly.

The legacy v1 router emits a different shape ({steps, execution_order, created_at} keyed by skill+intent) — that shape is consumed by host.runner.runner.Runner.run_dag and is ONLY used on the no-LLM fallback path.

When this Skill applies

• The user issues an instruction that is not a trivial pass-through to one capability Skill (e.g. "generate a cyberpunk portrait, then replace the background with a neon street, then preserve the face across a style transfer"). Any instruction with ≥2 visual goals must go through the planner.
• Single-goal instructions still go through the planner — it is responsible for the "do not over-decompose" decision and emits a 1-step DAG in that case.
• Unusual requests that aren't covered by any bundled capability Skill route through searching-comfyui-solutions first; the planner consumes the ranked search results as candidates.

Do not invoke when the caller already has a fully-specified Visual Intent Object for a single Skill — call the capability Skill directly.

Pipeline

1. Parse the instruction: extract goals[], optional constraints, optional inputs.image_refs[].
2. If any clause has no matching keyword and the caller supplies search_candidates on stdin, the planner emits a search-grounded step for it. The planner itself does not shell out to search_local.py or search_web.py — the caller (host runner) is responsible for invoking the searching-comfyui-solutions Skill scripts and passing the ranked hits in. Each candidate must carry skill and goal fields declaring the target capability Skill; candidates without them are rejected (the planner fails loudly instead of guessing). See scripts/decompose.py VALID_SKILLS / VALID_GOALS.
3. Run scripts/decompose.py — heuristics map (goals + constraints + search candidates) → ordered list of {skill, intent, depends_on, evaluators, provenance}. Any clause that cannot be resolved (no keyword match and no usable candidate) causes the script to exit 2 with a no_match error; partial plans are rejected.
4. Run scripts/emit_dag.py — validates the decomposition, attaches provenance, returns the DAG JSON on stdout.
5. Caller (host) renders the DAG, gets user approval, then executes step by step through each capability Skill's compile.py and then comfyui-execution/submit_prompt.py.

Every step carries:

• provenance.source — "bundled" | "search:<url>" | "user-supplied"
• provenance.retrieved_at — ISO 8601 UTC timestamp (or null for bundled)
• provenance.trust_tier — one of bundled / web_trusted / web_unverified per ../searching-comfyui-solutions/reference/trust-tiers.md

One-step vs multi-step decision

See reference/strategies.md. Hard rule: if the instruction contains a single atomic visual action (generate, inpaint, replace bg, transfer style, preserve face) and no compositional connective ("then", "and after", "while keeping"), emit a 1-step DAG.

Reference files

• reference/intent-schema.md — Visual Intent Object fields
• reference/strategies.md — decomposition heuristics
• reference/dag-examples.md — four worked examples including the v1 acceptance test

Scripts

• scripts/decompose.py — pure Python; stdlib only; keyword + pattern match; no LLM. Input: {instruction, project_state?, search_candidates?}.
• scripts/emit_dag.py — validates and attaches provenance.

Error handling

Both scripts emit {"error": "...", "code": "..."} on bad input with exit code 2. Never raise to caller.

References

• knowledge/plans/v1/plan_v1.md §1, §7, §8.4
• knowledge/plans/v1/todo.md §6 (Stage 3)