docetl

DocETL Pipeline Development

DocETL is a system for creating LLM-powered data processing pipelines. This skill helps you build end-to-end pipelines: from data preparation to execution and optimization.

Workflow Overview: Iterative Data Analysis

Work like a data analyst: write → run → inspect → iterate. Never write all scripts at once and run them all at once. Each phase should be completed and validated before moving to the next.

Phase 1: Data Collection

1. Write data collection script
2. Run it immediately (with user permission)
3. Inspect the dataset - show the user:
   • Total document count
   • Keys/fields in each document
   • Sample documents (first 3-5)
   • Length distribution (avg chars, min/max)
   • Any other relevant statistics
4. Iterate if needed (e.g., collect more data, fix parsing issues)

Phase 2: Pipeline Development

1. Read sample documents to understand format
2. Write pipeline YAML with sample: 10-20 for testing
3. Run the test pipeline
4. Inspect intermediate results - show the user:
   • Extraction quality on samples
   • Domain/category distributions
   • Any validation failures
5. Iterate on prompts/schema based on results
6. Remove sample parameter and run full pipeline
7. Show final results - distributions, trends, key insights

Phase 3: Visualization & Presentation

1. Write visualization script based on actual output structure
2. Run and show the report to the user
3. Iterate on charts/tables if needed

Visualization Aesthetics:

• Clean and minimalist - no clutter, generous whitespace
• Warm and elegant color theme - 1-2 accent colors max
• Subtle borders - not too rounded (border-radius: 8-10px max)
• Sans-serif fonts - system fonts like -apple-system, Segoe UI, Roboto
• "Created by DocETL" - add subtitle after the main title
• Mix of charts and tables - charts for distributions, tables for detailed summaries
• Light background - off-white (#f5f5f5) with white cards for content

Report Structure:

1. Title + "Created by DocETL" subtitle
2. Key stats cards (document count, categories, etc.)
3. Distribution charts (bar charts, pie charts)
4. Summary table with detailed analysis
5. Minimal footer

Interactive Tables:

• All truncated content must be expandable - never use static "..." truncation
• Long text: Show first ~250 chars with "(show more)" toggle
• Long lists: Show first 4-6 items with "(+N more)" toggle
• Use JavaScript to toggle visibility, not page reloads

Source Document Links:

• Link aggregated results to source documents - users should be able to drill down
• Clickable links that open a modal/popup with source content
• Modal should show: extracted fields + original source text
• Original text can be collapsed by default with "Show original" toggle
• Embed source data as JSON in the page for JavaScript access

Key principle: The user should see results at every step. Don't proceed to the next phase until the current phase produces good results.

Step 1: Data Preparation

DocETL datasets must be JSON arrays or CSV files.

JSON Format

[
  {"id": 1, "text": "First document content...", "metadata": "value"},
  {"id": 2, "text": "Second document content...", "metadata": "value"}
]

CSV Format

id,text,metadata
1,"First document content...","value"
2,"Second document content...","value"

Data Collection Scripts

If user needs to collect data, write a Python script:

import json

# Collect/transform data
documents = []
for source in sources:
    documents.append({
        "id": source.id,
        "text": source.content,  # DO NOT truncate text
        # Add relevant fields
    })

# Save as DocETL dataset
with open("dataset.json", "w") as f:
    json.dump(documents, f, indent=2)

Important: Never truncate document text in collection scripts. DocETL operations like split handle long documents properly. Truncation loses information.

After Running Data Collection

Always run the collection script and inspect results before proceeding. Show the user:

import json
data = json.load(open("dataset.json"))

print(f"Total documents: {len(data)}")
print(f"Keys: {list(data[0].keys())}")
print(f"Avg length: {sum(len(str(d)) for d in data) // len(data)} chars")

# Show sample
print("\nSample document:")
print(json.dumps(data[0], indent=2)[:500])

Only proceed to pipeline development once the data looks correct.

Step 2: Read and Understand the Data

CRITICAL: Before writing any prompts, READ the actual input data to understand:

• The structure and format of documents
• The vocabulary and terminology used
• What information is present vs. absent
• Edge cases and variations

import json
with open("dataset.json") as f:
    data = json.load(f)
# Examine several examples
for doc in data[:5]:
    print(doc)

This understanding is essential for writing specific, effective prompts.

Step 3: Pipeline Structure

DocETL supports two equivalent approaches. Use whichever the user prefers:

Option A: YAML (low-code)

Create a YAML file with this structure:

default_model: gpt-5-nano

system_prompt:
  dataset_description: <describe the data based on what you observed>
  persona: <role for the LLM to adopt>

datasets:
  input_data:
    type: file
    path: "dataset.json"  # or dataset.csv

operations:
  - name: <operation_name>
    type: <operation_type>
    prompt: |
      <Detailed, specific prompt based on the actual data>
    output:
      schema:
        <field_name>: <type>

pipeline:
  steps:
    - name: process
      input: input_data
      operations:
        - <operation_name>
  output:
    type: file
    path: "output.json"
    intermediate_dir: "intermediates"  # ALWAYS set this for debugging

Option B: Python Frame API

The same pipeline as chainable Python calls:

import docetl

docetl.default_model = "gpt-5-nano"
docetl.intermediate_dir = "intermediates"

results = (
    docetl.read_json("dataset.json")          # or read_csv(), from_list()
    .map(
        prompt="...",
        output={"schema": {"field": "type"}},
    )
    .reduce(
        reduce_key="category",
        prompt="...",
        output={"schema": {"summary": "string"}},
    )
    .collect()                                 # returns list[dict]; .to_pandas() for a DataFrame
)

# Cost and token tracking
print(f"Cost: ${pipeline.total_cost:.4f}")

Key Frame API methods:

• Readers: docetl.read_json(), docetl.read_csv(), docetl.read_parquet(), docetl.from_list()
• Operations: .map(), .filter(), .reduce(), .resolve(), .equijoin(), .split(), .gather(), .unnest(), .code_map(), .code_filter(), .code_reduce()
• Terminal actions: .show() (run on sample, print results), .collect() (list of dicts), .to_pandas() (DataFrame), .write_json(), .write_csv(), .write_parquet()
• Inspection (no execution): .schema() (output schema), .count() (input doc count on bare datasets), .to_yaml() (export pipeline as YAML), .to_python()
• Config: docetl.default_model, docetl.max_threads, docetl.bypass_cache, docetl.rate_limits, docetl.intermediate_dir, docetl.agent_model, docetl.fallback_models

All operation parameters are the same between YAML and Python — just pass them as keyword arguments (e.g., validate=["len(output['items']) >= 1"], fold_prompt="...", fold_batch_size=100).

Tool-equipped agents (Python API only)

Use agent=docetl.Agent(...) on .map(), .filter(), or .reduce() when an operation needs tools before returning structured output:

@docetl.tool
def lookup_sla(customer_tier: str) -> dict[str, str | int]:
    """Return support entitlements for a customer tier."""
    return {
        "enterprise": {"response_hours": 1, "escalation": "page-on-call"},
        "growth": {"response_hours": 4, "escalation": "queue-lead"},
        "free": {"response_hours": 48, "escalation": "self-serve"},
    }.get(customer_tier.lower(), {"response_hours": 24, "escalation": "standard"})

agent = docetl.Agent(tools=[lookup_sla], max_turns=5, max_tool_calls=3)
rows = (
    docetl.from_list([{"ticket": "Production API latency is above SLO", "customer_tier": "enterprise"}])
    .map(
        prompt="Use lookup_sla to classify this ticket: {{ input.ticket }} / {{ input.customer_tier }}",
        output={"schema": {"priority": "str", "next_action": "str"}},
        model="azure/gpt-4o-mini",
        agent=agent,
    )
    .collect()
)

Key points:

• Agents are Python-only; do not put agent configs in YAML and do not export them with .to_yaml() / .to_python().
• The operation model= still selects the model. Python tools wrapped with @docetl.tool are the most provider-portable path through LiteLLM-compatible models.
• OpenAI-hosted tools (WebSearchTool, hosted ShellTool, docetl.tools.Sandbox.create(...)) require an OpenAI hosted-tool path. docetl.tools.Sandbox.create(...) creates a persistent OpenAI hosted container; sandbox.bash() returns a shell tool bound to that container.
• Specialist subagents can be exposed as manager-agent tools with specialist.as_tool(name=..., description=...).
• For richer examples and provider caveats, read:
  • docs/api-reference/python.md#tool-equipped-mapfilterreduce
  • docs/operators/map.md#tool-equipped-map-agents
  • docs/operators/filter.md#tool-equipped-filter-agents
  • docs/operators/reduce.md#tool-equipped-reduce-agents
  • docs/examples/tool-equipped-research-agents.md
  • OpenAI Agents docs: https://developers.openai.com/api/docs/guides/agents
  • OpenAI tools docs: https://openai.github.io/openai-agents-python/tools/
  • OpenAI sandbox docs: https://developers.openai.com/api/docs/guides/agents/sandboxes
  • OpenAI orchestration docs: https://developers.openai.com/api/docs/guides/agents/orchestration

Key Configuration

• default_model: U

---

Content truncated.