omnidocbench-eval-helper

OmniDocBench evaluation helper

Help community users run OmniDocBench reproducibly. Prefer a practical, path-specific workflow over generic setup advice:

1. Validate GT/prediction paths and runtime access.
2. Generate an isolated config using stable container paths.
3. Run python pdf_validation.py --config ... in Docker when possible.
4. Parse *_metric_result.json, *_run_summary.json, *_stage_execution.json, and *_runtime_environment.json.
5. Report scores, output paths, and any warnings that affect score credibility.

Be concise for simple questions, but include copy-paste commands when the user gives paths.

What to ask for

Ask only for missing details that affect commands or config:

• Deployment mode: Docker, conda/source, or remote SSH. Recommend Docker when CDM is needed.
• Evaluation type: default to end2end for OmniDocBench JSON + page markdown predictions.
• Paths:
  • ground-truth JSON, commonly OmniDocBench.json with exact capitalization on Linux
  • prediction markdown directory, often a nested markdown/ folder such as .../mineru/markdown
  • output directory, or permission to create a timestamped output directory
• Whether CDM is required.
• CPU/RAM or node size if the user mentions slowness, OOM, or cluster execution.

If the user explicitly asks you to run a job on a remote host and provides an SSH command, treat that as authorization for that host/path scope. Avoid destructive operations: do not delete, overwrite, or chmod shared data/results.

Key facts

• Entrypoint: python pdf_validation.py --config <config_path>.
• Default config: configs/end2end.yaml.
• Package: omnidocbench-eval, Python >=3.10,<3.12.
• Recommended Docker image: ghcr.io/zeng-weijun/omnidocbench-eval:repro-ubuntu2204.
• CDM depends on TeX Live/CJK, ImageMagick PDF read/write, and Ghostscript. Docker avoids most CDM dependency failures.
• Worker keys:
  • dataset.match_workers: page matching
  • metrics.display_formula.cdm_workers: CDM rendering/comparison, memory-heavy
  • metrics.table.teds_workers: table TEDS
• Worker rule: on 4 CPU / 8 GB, use 2 for match/CDM/TEDS; if unstable, use 1. Do not keep README default 13 on small nodes.

Input validation before running

Always validate before launching a long Docker evaluation. For remote runs, execute this on the remote host via SSH.

GT=/path/to/OmniDocBench.json
PRED=/path/to/prediction/markdown

test -f "$GT" && echo "GT_OK=$GT" || echo "GT_MISSING=$GT"
test -d "$PRED" && echo "PRED_OK=$PRED" || echo "PRED_MISSING=$PRED"
[ -f "$GT" ] && wc -c "$GT"
[ -d "$PRED" ] && echo "md_count=$(find "$PRED" -maxdepth 1 -name '*.md' | wc -l)"
[ -d "$PRED" ] && echo "empty_md=$(find "$PRED" -maxdepth 1 -name '*.md' -empty | wc -l)"
[ -d "$PRED" ] && find "$PRED" -maxdepth 1 -name '*.md' | sort | head -5
nproc
free -h || true

Interpretation:

• Full OmniDocBench v1.6 has 1651 pages. md_count should usually be close to 1651 for a full run.
• md_count=0 usually means the user passed a parent directory instead of the actual markdown/ directory.
• empty_md > 0 is not fatal, but report it because empty predictions lower scores.
• Linux paths are case-sensitive: OmniDocBench.json and omnidocbench.json differ. If GT is missing, check nearby JSON filenames before concluding the data is absent.

Docker workflow

Use Docker when possible, especially for CDM.

Check Docker access

DOCKER=docker
if ! docker ps >/dev/null 2>&1; then
  if sudo -n docker ps >/dev/null 2>&1; then
    DOCKER="sudo -n docker"
  else
    echo "Docker daemon is not accessible by docker or sudo -n docker"
    exit 4
  fi
fi
$DOCKER --version

Use sudo -n docker only when the user is authorized on that host. Do not try interactive sudo in an automated workflow.

Local end2end command

Use stable container paths and write a custom config inside the container. This avoids editing the repository and keeps host paths out of YAML.

GT=/abs/path/to/OmniDocBench.json
PRED=/abs/path/to/prediction/markdown
OUT=/abs/path/to/output_dir
WORKERS=4   # use 2 on 4CPU/8G; use 1 if CDM OOMs
IMAGE=ghcr.io/zeng-weijun/omnidocbench-eval:repro-ubuntu2204

mkdir -p "$OUT"

DOCKER=docker
if ! docker ps >/dev/null 2>&1; then
  if sudo -n docker ps >/dev/null 2>&1; then DOCKER="sudo -n docker"; else exit 4; fi
fi

$DOCKER pull "$IMAGE"
$DOCKER run --rm --entrypoint bash \
  -v "$GT":/workspace/gt/OmniDocBench.json:ro \
  -v "$PRED":/workspace/data_md/predictions:ro \
  -v "$OUT":/workspace/result \
  "$IMAGE" \
  -lc "cat > configs/custom_end2end.yaml <<EOF
end2end_eval:
  metrics:
    text_block:
      metric: [Edit_dist]
    display_formula:
      metric: [Edit_dist, CDM]
      cdm_workers: ${WORKERS}
    table:
      metric: [TEDS, Edit_dist]
      teds_workers: ${WORKERS}
    reading_order:
      metric: [Edit_dist]
  dataset:
    dataset_name: end2end_dataset
    ground_truth:
      data_path: ./gt/OmniDocBench.json
    prediction:
      data_path: ./data_md/predictions
    match_method: quick_match
    match_workers: ${WORKERS}
    quick_match_truncated_timeout_sec: 300
    match_timeout_sec: 420
    timeout_fallback_max_chunk_span: 10
    timeout_fallback_order_penalty: 0.10
EOF
python pdf_validation.py --config configs/custom_end2end.yaml 2>&1 | tee /workspace/result/eval.log"

If CDM is not needed, remove CDM from display_formula.metric and remove cdm_workers. This avoids TeX/ImageMagick/Ghostscript dependency failures, but the final report will not include Formula CDM.

Remote SSH / H-cluster workflow

When the user provides an SSH command, run the same validation and Docker workflow on the remote host. Use a heredoc remote script to avoid nested quoting bugs.

ssh -CAXY user@host 'bash -s' <<'REMOTE'
set -euo pipefail

GT=/mnt/shared-storage-user/.../1.6/OmniDocBench.json
PRED=/mnt/shared-storage-user/.../mineru/markdown
OUT_ROOT=/mnt/shared-storage-user/.../omnidocbench_eval
OUT="$OUT_ROOT/mineru_$(date +%Y%m%d_%H%M%S)"
LATEST=/tmp/omnidocbench_latest_out.txt
IMAGE=ghcr.io/zeng-weijun/omnidocbench-eval:repro-ubuntu2204

mkdir -p "$OUT"
echo "$OUT" > "$LATEST"

test -f "$GT" || { echo "GT_MISSING=$GT"; exit 2; }
test -d "$PRED" || { echo "PRED_MISSING=$PRED"; exit 2; }
MD_COUNT=$(find "$PRED" -maxdepth 1 -name '*.md' | wc -l)
EMPTY_MD=$(find "$PRED" -maxdepth 1 -name '*.md' -empty | wc -l)
echo "GT_OK=$GT"
echo "PRED_OK=$PRED"
echo "md_count=$MD_COUNT"
echo "empty_md=$EMPTY_MD"
[ "$MD_COUNT" -gt 0 ] || { echo "No markdown files found; check nested markdown directory"; exit 3; }

DOCKER=docker
if ! docker ps >/dev/null 2>&1; then
  if sudo -n docker ps >/dev/null 2>&1; then
    DOCKER="sudo -n docker"
  else
    echo "Docker daemon is not accessible by docker or sudo -n docker"
    exit 4
  fi
fi

CPU=$(nproc)
MEM_KB=$(awk '/MemTotal/ {print $2}' /proc/meminfo 2>/dev/null || echo 0)
MEM_GB=$((MEM_KB / 1024 / 1024))
if [ -n "${FORCE_WORKERS:-}" ]; then
  WORKERS="$FORCE_WORKERS"
elif [ "$CPU" -le 4 ] || [ "$MEM_GB" -lt 10 ]; then
  WORKERS=2
else
  WORKERS=4
fi

echo "output_dir=$OUT"
echo "latest_pointer=$LATEST"
echo "cpu=$CPU mem_gb=$MEM_GB workers=$WORKERS"

$DOCKER pull "$IMAGE"
$DOCKER run --rm --entrypoint bash \
  -v "$GT":/workspace/gt/OmniDocBench.json:ro \
  -v "$PRED":/workspace/data_md/predictions:ro \
  -v "$OUT":/workspace/result \
  "$IMAGE" \
  -lc "cat > configs/custom_end2end.yaml <<EOF
end2end_eval:
  metrics:
    text_block:
      metric: [Edit_dist]
    display_formula:
      metric: [Edit_dist, CDM]
      cdm_workers: ${WORKERS}
    table:
      metric: [TEDS, Edit_dist]
      teds_workers: ${WORKERS}
    reading_order:
      metric: [Edit_dist]
  dataset:
    dataset_name: end2end_dataset
    ground_truth:
      data_path: ./gt/OmniDocBench.json
    prediction:
      data_path: ./data_md/predictions
    match_method: quick_match
    match_workers: ${WORKERS}
    quick_match_truncated_timeout_sec: 300
    match_timeout_sec: 420
    timeout_fallback_max_chunk_span: 10
    timeout_fallback_order_penalty: 0.10
EOF
python pdf_validation.py --config configs/custom_end2end.yaml 2>&1 | tee /workspace/result/eval.log"

echo "DONE_OUT=$OUT"
ls -lh "$OUT"
REMOTE

Practical H-cluster lessons from a real MinerU v1.6 run:

• ssh -CAXY may print X11 warnings such as No xauth data; they are harmless for CLI evaluation.
• Docker may be installed but inaccessible through the daemon socket. Check docker ps; if it fails and sudo -n docker ps works, use sudo -n docker for the run.
• Record the output directory in /tmp/omnidocbench_latest_out.txt or a user-specific variant so it can be recovered after disconnection.
• GT capitalization matters: /.../OmniDocBench.json worked where lowercase omnidocbench.json did not.
• MinerU predictions may live in a nested markdown/ directory. Passing the parent directory can produce zero or inflated markdown counts.
• For 4CPU/8G with CDM, start with workers 2; use 1 if unstable.

Bundled helper scripts

This skill includes scripts that can be copied into the repo or run from the skill directory:

• scripts/generate_end2end_config.py: generate an end2end YAML.
• scripts/parse_results.py: parse a result directory and print a compact score/validation report.

Example:

python scripts/generate_end2end_config.py \
  --gt ./gt/OmniDocBench.json \
  --pred ./data_md/predictions \
  --out configs/custom_end2end.yaml \
  --workers 2

python scripts/parse_results.py /path/to/result_dir

Use --no-cdm with generate_end2end_config.py to omit CDM and cdm_workers.

Conda/source workflow

Use only when Docker is unavailable or the user needs to modify OmniDocBench code.

conda create -n omnidocbench python=3.10 -y
conda activate omnidocbench
pip install -e .
python -c "from src.core.pipeline import run_config_file; print('OK')"

For CDM in source installs, verify:

pdflatex --version | head -2
kpsewhich CJK.sty && kp

---

*Content truncated.*