remote-compute

Remote Compute Skill

Run containerized jobs on KBase compute nodes via the CDM Task Service (CTS).

Overview

CTS runs containerized jobs on remote compute clusters. Two execution paths are available:

Always discover the live image registry first before deciding which path to take:

# JupyterHub
tscli = get_task_service_client()
tscli.get_images()        # returns list of registered images

# curl (any environment)
AUTH_TOKEN=$(grep "KBASE_AUTH_TOKEN" .env | cut -d'"' -f2)
curl -s -H "Authorization: Bearer $AUTH_TOKEN" \
  https://berdl.kbase.us/apis/cts/images | python3 -m json.tool

The registry grows over time — always check live rather than relying on any static list in this document.

When to use remote compute vs JupyterHub:

• JupyterHub: Interactive analysis, Spark SQL queries, exploratory work
• Remote compute: Batch processing, CPU/memory-intensive tools (CheckM2, Bakta, GTDB-Tk), long-running jobs, anything that would block your notebook

Preconditions

1. KBASE_AUTH_TOKEN set in environment or .env
2. MinIO client (mc) installed — see berdl-minio skill for installation
3. Proxy running (when off-cluster) — MinIO is not directly reachable from external networks. See the berdl-query skill's references/proxy-setup.md. The CTS API itself (https://berdl.kbase.us/apis/cts) does NOT require a proxy — only MinIO data staging does.
4. CTS access and your S3 paths — run whoami before staging any files. It confirms your role and tells you the exact S3 path prefixes you are allowed to use for input_files and output_dir. Every example in this skill uses cts/io/<username>/... as a placeholder — whoami tells you what to substitute.

# JupyterHub
tscli = get_task_service_client()
tscli.whoami()
# Returns, e.g.:
# {
#   "user": "mcashman",                       ← your username; use this in place of <username>
#   "roles": ["cts_user"],                    ← confirms CTS access; must include "cts_user" or "full_admin"
#   "allowed_paths": [
#     {"path": "cts/io/mcashman", "perm": "write"},   ← you can read and write here
#     {"path": "cts/io/mcashman", "perm": "read"}
#   ]
# }
# The "path" values are the S3 prefixes you use for input_files and output_dir.
# Example: input_files=["cts/io/mcashman/input/genome.fna.gz"]
#          output_dir="cts/io/mcashman/output/bakta_run"

# curl equivalent
curl -s -H "Authorization: Bearer $AUTH_TOKEN" \
  https://berdl.kbase.us/apis/cts/whoami | python3 -m json.tool

If roles is empty or does not include cts_user or full_admin, you do not have CTS access — ask a CTS admin to grant the cts_user role.

Two Interfaces

Python client — JupyterHub only

tscli = get_task_service_client()    # CTS client
minio = get_minio_client()           # MinIO client for file staging

REST API (curl) — any environment

All endpoints at https://berdl.kbase.us/apis/cts. Auth via Bearer token:

AUTH_TOKEN=$(grep "KBASE_AUTH_TOKEN" .env | cut -d'"' -f2)

Stage Input Files to MinIO

This step is required for both Path A and Path B. CTS does not read files from your local machine or from the BERDL Lakehouse directly — all input files must be pre-uploaded to MinIO before job submission. Every file must carry a CRC64NVME checksum or the submission will be rejected.

Find your MinIO path prefix

Run tscli.whoami() (see Preconditions) to find your allowed_paths. Your input and output directories should be nested under the writable path — typically cts/io/<username>/.

Upload with the MinIO CLI (mc)

# Off-cluster only: set proxy so mc can reach MinIO
export https_proxy=http://127.0.0.1:8123
export no_proxy=localhost,127.0.0.1

# Upload a single file with a CRC64NVME checksum
mc cp --checksum crc64nvme genome.fna.gz berdl-minio/cts/io/<username>/input/

# Upload a directory of files (e.g. all .fna.gz in a folder)
mc cp --checksum crc64nvme --recursive ./input_genomes/ berdl-minio/cts/io/<username>/input/

# Verify what was staged
mc ls berdl-minio/cts/io/<username>/input/

The berdl-minio alias must be configured. See the berdl-minio skill for setup instructions.

Upload from Python (JupyterHub)

minio = get_minio_client()
import os

local_file = "genome.fna.gz"
minio_path = f"io/<username>/input/{local_file}"  # path within the "cts" bucket

# Upload a file — the MinIO client handles checksums automatically on JupyterHub
minio.fput_object("cts", minio_path, local_file)
print(f"Staged: cts/{minio_path}")

# List staged files to confirm
for obj in minio.list_objects("cts", prefix="io/<username>/input/", recursive=True):
    print(obj.object_name, obj.size)

Build your input_files list

After staging, construct the input_files list for submit_job from the staged paths:

# Option 1: list them explicitly
input_files = [
    "cts/io/<username>/input/genome1.fna.gz",
    "cts/io/<username>/input/genome2.fna.gz",
]

# Option 2: discover from MinIO (useful for large batches)
minio = get_minio_client()
objs = list(minio.list_objects("cts", prefix="io/<username>/input/", recursive=True))
input_files = [f"cts/{o.object_name}" for o in objs]
print(f"{len(input_files)} files staged and ready")

Path A: Pre-built Tool Images

Pre-built images have the tool already installed and set as the container entrypoint. You do not write a bash script — you pass arguments directly to the tool. Some images include large reference databases (refdata) that CTS mounts automatically.

What is refdata?

Some tools require large reference databases (e.g., CheckM2's marker gene sets, GTDB-Tk's genome sketches, Bakta's annotation database). These are pre-staged by the CTS admins and mounted read-only into the container at a fixed path. The image registration records which refdata bundle it needs and where to mount it (default_refdata_mount_point).

You do not pass a refdata_id to submit_job — it is determined by the image. You only need to know the mount path so you can reference it in your args if needed (e.g., --db /ref_data). Most images bake the path into the tool config or environment variables so you do not need to reference it at all — see the per-tool notes below.

If you want to override the default mount point, pass refdata_mount_point="/your/path" inside the params via the Python client's submit_job call. This is rarely needed.

General pattern — pre-built image

tscli = get_task_service_client()

job = tscli.submit_job(
    "<image>:<tag>",          # full image name from the registry
    input_files,              # list of MinIO paths: ["cts/io/<user>/input/file1.fna.gz", ...]
    "cts/io/<user>/output/run_name",  # MinIO output directory (bucket/prefix)
    cluster="kbase",          # compute site — "kbase" for on-prem KBase hardware
    output_mount_point="/out", # path inside the container where the tool must write output;
                               # CTS copies everything under this path to S3 after the job finishes
    args=[...],               # arguments appended after the image entrypoint command
    num_containers=1,         # how many parallel containers to run (see per-tool guidance)
    cpus=4,                   # CPU cores per container
    memory="8GB",             # RAM per container
    runtime="PT30M",          # ISO 8601 duration: PT5M=5min, PT1H=1hour, PT2H30M=2.5hours
)

print(f"Job ID: {job.id}")    # save this — needed to check status or retrieve results later

Available pre-built images

Full per-tool examples, resource guidance, and special notes are in individual reference files under references/. Each tool has its own file named after the image pattern without the cdm_ prefix:

Claude should Read only the specific tool's reference file after identifying the user's tool from get_images().

Path B: Generic Ubuntu Script

Use this path wh

Content truncated.