sue-normalize-structure

Sue Normalize Structure

DeepResearch Root Contract (HIGHEST PRIORITY)

Before any SUE action in this skill, verify the workspace output-root contract. If any check fails, STOP and abort; do not proceed with discovery, planning, submission, monitoring, cleanup, reporting, or lessons.

1. The workspace path must be $<SANDBOX>_DEEPRESEARCH_ROOT/workspace/<workspace>/.
2. All scale-up outputs must be written under $<SANDBOX>_DEEPRESEARCH_ROOT/workspace/<workspace>/scale_up_outputs/${EXP_DIR}/ ($SUE_EXP_DIR), where EXP_DIR is declared in the bootstrap scale_up_outputs/runtime.yaml under paths.exp_dir. The only exceptions are the minimal bootstrap scale_up_outputs/runtime.yaml itself and sue-exp-history/ archives.
3. $<SANDBOX>_DEEPRESEARCH_ROOT must be resolvable via sue-context (or the selected sandbox's private config for remote backends). Do not fall back to $HOME, arbitrary scratch, or the source tree.

This contract takes precedence over sandbox selection, GPU accounting, manifest validation, and all downstream steps.

Hard Sandbox Repo Root Gate

Before any remote query, sync, cleanup, dryrun/fullrun submission, monitor, quota check, or output-path mutation, verify the selected backend exports a non-empty <SANDBOX>_DEEPRESEARCH_ROOT — one of LUMI_DEEPRESEARCH_ROOT, NM5_DEEPRESEARCH_ROOT, SNELLIUS_DEEPRESEARCH_ROOT, BREV_DEEPRESEARCH_ROOT, AUTODL_DEEPRESEARCH_ROOT, or RUNPOD_DEEPRESEARCH_ROOT — and that the directory exists. If the root is unset, empty, missing, or only inferable from cwd, scratch/project roots, PROJECT_ROOT, NM5_PROJECT_ROOT, workspace name, or marker walking, terminate immediately with a blocker naming the missing key. Do not derive a replacement root, do not continue with a guessed checkout, and do not run destructive or quota-consuming commands.

Per-Experiment Directory Contract

The canonical filesystem layout for SUE workspaces is defined in $<SANDBOX>_DEEPRESEARCH_ROOT/config/path_rule.yaml. Use that file as the source of truth for the per-experiment directory contract.

• runtime_key: runtime.yaml:paths.exp_dir
• env_var: SUE_EXP_DIR
• resolved: ${SCALE_UP_OUTPUTS_ROOT}/${EXP_DIR}/
• example: $<SANDBOX>_DEEPRESEARCH_ROOT/workspace/<workspace>/scale_up_outputs/<sandbox>_<experiment_family>/
• default_pattern: <sandbox>_<experiment_family>

Per-run artifacts live under ${SUE_EXP_DIR}/<run_id>/, and final result bundles live under ${SUE_EXP_DIR}/final_result/<run_id>/. Launchers must export SUE_EXP_DIR derived from runtime.yaml:paths.exp_dir.

Sandbox Communication

When this workflow communicates with a remote sandbox, keep SSH/API calls as control-plane actions: launch, inspect, fetch logs, or stop work. If a remote command is likely to run long, stream substantial output, or require repeated polling, start it inside a detached tmux session on the sandbox and return after verifying the session name and durable log path. Do not keep a local SSH connection open as the job supervisor.

Prefer tmux new-session -d -s <name> 'bash <script> 2>&1 | tee -a <log>'; avoid tmux send-keys. On direct-run sandboxes (Brev, AutoDL, RunPod), use tmux for long-running remote commands. If tmux is unavailable on the sandbox, stop and report the exact missing tool instead of silently keeping the connection open.

Purpose

Migrate misplaced files into the canonical DeepResearch layout:

• Workspace root keeps only the declared project surfaces plus scale_up_outputs/ (SUE bootstrap + active exp_dir) and sue-exp-history/ (archived epochs).
• scale_up_outputs/ root stays inode-light: only the minimal bootstrap runtime.yaml, the active ${EXP_DIR}/ directory, and sue-exp-history/ are allowed. All SUE-managed content buckets (config/, scripts/, logs/, state/, results/, artifacts/, datasets/, envs/, ckpt/, downloaded/, readiness/) live under ${SUE_EXP_DIR}/; everything else must be packed or relocated.
• Repo root stays free of workspace data, experiment outputs, datasets, or checkpoints.
• Nothing the SUE framework reads or writes lives outside $<SANDBOX>_DEEPRESEARCH_ROOT unless explicitly declared in runtime.yaml.environment.allowed_external_roots.

This skill is the fixer counterpart to sue-sandbox-boundary-check (which is read-only). It does not replace sue-cleanup (inode conservation via tar+remove of bulk image/benchmark trees) or sue-reset (archive-and-reset the whole scale_up_outputs/). It moves misplaced structural files into their canonical home; it does not pack or delete high-fanout trees.

Scope

In scope:

• Workspace-root leaks: data files, datasets, checkpoints, outputs, W&B local data, caches, venvs, or large generated artifacts at workspace/<p>/ root that belong under scale_up_outputs/${EXP_DIR}/ (i.e., $SUE_EXP_DIR) (or under the project's own code surfaces when they are source — src/, scripts/, exp/, docs/, artifacts/, preliminary_latex/, preliminary_code/).
• Workspace-root config-file duplication. A full runtime.yaml, scale-up.yaml, or runtime.<backend>.yaml living as a real file at workspace/<p>/ root is a violation — the canonical homes are ${SUE_EXP_DIR}/config/runtime.yaml and ${SUE_EXP_DIR}/config/scale-up.yaml (and runtime.json/scale-up.yaml at workspace root only when the workspace uses the Tao/preflight Pattern-A surfaces; see Contract step 2). The only runtime.yaml permitted at scale_up_outputs/ root is the minimal bootstrap pointer containing only paths.exp_dir. Treat any richer root-level runtime.yaml/scale-up.yaml as misplaced config and migrate it into ${SUE_EXP_DIR}/config/. A workspace-root runtime.yaml or scale-up.yaml symlink (or any workspace-root copy) is a structural violation: it duplicates the bootstrap pointer and creates ambiguity about which config SUE helpers consume. Remove it; tools resolve config from scale_up_outputs/runtime.yaml and ${SUE_EXP_DIR}/config/{runtime,scale-up}.yaml directly.
• Legacy archive-naming drift. Canonical SUE reset archives live at workspace/<p>/sue-exp-history/. The legacy hyphenated form scale-up-exp-history/ is a structural-naming violation: rename it to sue-exp-history/ (do not move it under ${SUE_EXP_DIR}/ — it stays a workspace-root archive, just under the canonical name).
• Per-workspace skill entries. workspace/<p>/.codex/skills/sue-*, workspace/<p>/.kimi/skills/sue-*, and workspace/<p>/.claude/skills/sue-*.md may exist only as symlink pointers to $<SANDBOX>_DEEPRESEARCH_ROOT/.codex/skills/sue-* or the matching SKILL.md. Canonical skill definitions live only at $<SANDBOX>_DEEPRESEARCH_ROOT/.codex/skills/. Project .kimi/ or .claude/ mirrors, if present, are noncanonical compatibility artifacts and not move targets. For each SUE entry found in a workspace skill tree:
  • If it is a symlink to the matching parent Codex source, leave it untouched.
  • If it is a broken symlink or points somewhere else, replace it with a symlink to the matching parent Codex source after reporting the exact target mismatch.
  • If the same skill name does not exist at the canonical project Codex level and the workspace entry is a real directory/file, move it into $<SANDBOX>_DEEPRESEARCH_ROOT/.codex/skills/ (e.g. workspace/<p>/.codex/skills/sue-foo/ -> $<SANDBOX>_DEEPRESEARCH_ROOT/.codex/skills/sue-foo/).
  • If the same skill name already exists at the project level and the workspace entry is a real directory/file, the workspace copy is a stale duplicate; remove it or replace it with a symlink pointer when local CLI discoverability needs that workspace path. Do not move workspace skill copies under scale_up_outputs/.
• scale_up_outputs/ structural violations: top-level files or directories other than the minimal bootstrap runtime.yaml, the per-experiment directory paths.exp_dir, and sue-exp-history/. Content-type buckets (config/, scripts/, logs/, state/, results/, artifacts/, datasets/, envs/, ckpt/, downloaded/, readiness/) must live under ${SUE_EXP_DIR}/, not directly under scale_up_outputs/. Run-specific directories must live under ${SUE_EXP_DIR}/<run_id>/, not directly under scale_up_outputs/.
• Repo-root leaks: workspace data, experiment outputs, datasets, checkpoints, W&B local data, or large artifacts at $<SANDBOX>_DEEPRESEARCH_ROOT/ root or under non-canonical top-level directories (not memory/, workspace/, deepresearch-sandbox/, docs/, config/, .codex/, .claude/, .kimi/, scripts/, tests/).
• Outside-root leaks: absolute paths under $HOME, /tmp, /scratch, or another project tree that the framework reads or writes without an allowed_external_roots declaration.

Out of scope (report only, never auto-fix):

• Writes inside preliminary_code/ or preliminary_latex/ — those are read-only vendored inputs per memory/project.md.
• Writes inside .tao/ orchestration state — owned by the Tao orchestrator.
• Writes inside sue-exp-history/ — archived epochs; only sue-reset manages them. (The legacy scale-up-exp-history/ rename above is in scope, but contents inside an already-canonical sue-exp-history/ are out of scope.)
• Read-mostly workspace cache. workspace/<p>/.cache/ is the canonical read-mostly cache root prescribed by config/path_rule.yaml (cache_root.base: ${WORKSPACE_ROOT}/.cache/, with .cache/huggingface/, .cache/torch/). Do not flag or move it. Non-canonical cache-named dirs at root (e.g. .cache_downloads/, .venv_download/) are violations.
• Gitignored local-dev state. workspace/<p>/.venv/, .pytest_cache/, .worktrees/, .mypy_cache/, .ruff_cache/ are local control-plane dev state, no

---

Content truncated.