data-analysis

Data Analysis Skill

Comprehensive data analysis toolkit using Polars - a blazingly fast DataFrame library. This skill provides instructions, reference documentation, and ready-to-use scripts for common data analysis tasks.

Iteration Checkpoints

Quick Start

import polars as pl
from polars import col

# Load data
df = pl.read_csv("data.csv")

# Explore
print(df.shape, df.schema)
df.describe()

# Transform and analyze
result = (
    df.filter(col("value") > 0)
    .group_by("category")
    .agg(col("value").sum().alias("total"))
    .sort("total", descending=True)
)

# Export
result.write_csv("output.csv")

When to Use This Skill

• Loading datasets (CSV, JSON, Parquet, Excel, databases)
• Data cleaning, filtering, and transformation
• Aggregations, grouping, and pivot tables
• Statistical analysis and summary statistics
• Time series analysis and resampling
• Joining and merging multiple datasets
• Creating visualizations and charts
• Exporting results to various formats

Skill Contents

Reference Documentation

Detailed API reference and patterns for specific operations:

• reference/loading.md - Loading data from all supported formats
• reference/transformations.md - Column operations, filtering, sorting, type casting
• reference/aggregations.md - Group by, window functions, running totals
• reference/time_series.md - Date parsing, resampling, lag features
• reference/statistics.md - Correlations, distributions, hypothesis testing setup
• reference/visualization.md - Creating charts with matplotlib/plotly

Ready-to-Use Scripts

Executable Python scripts for common tasks:

• scripts/explore_data.py - Quick dataset exploration and profiling
• scripts/summary_stats.py - Generate comprehensive statistics report

Core Patterns

Loading Data

# CSV (most common)
df = pl.read_csv("data.csv")

# Lazy loading for large files
df = pl.scan_csv("large.csv").filter(col("x") > 0).collect()

# Parquet (recommended for large datasets)
df = pl.read_parquet("data.parquet")

# JSON
df = pl.read_json("data.json")
df = pl.read_ndjson("data.ndjson")  # Newline-delimited

Filtering and Selection

# Select columns
df.select("col1", "col2")
df.select(col("name"), col("value") * 2)

# Filter rows
df.filter(col("age") > 25)
df.filter((col("status") == "active") & (col("value") > 100))
df.filter(col("name").str.contains("Smith"))

Transformations

# Add/modify columns
df = df.with_columns(
    (col("price") * col("qty")).alias("total"),
    col("date_str").str.to_date("%Y-%m-%d").alias("date"),
)

# Conditional values
df = df.with_columns(
    pl.when(col("score") >= 90).then(pl.lit("A"))
    .when(col("score") >= 80).then(pl.lit("B"))
    .otherwise(pl.lit("C"))
    .alias("grade")
)

Aggregations

# Group by
df.group_by("category").agg(
    col("value").sum().alias("total"),
    col("value").mean().alias("avg"),
    pl.len().alias("count"),
)

# Window functions
df.with_columns(
    col("value").sum().over("group").alias("group_total"),
    col("value").rank().over("group").alias("rank_in_group"),
)

Exporting

df.write_csv("output.csv")
df.write_parquet("output.parquet")
df.write_json("output.json", row_oriented=True)

Best Practices

1. Use lazy evaluation for large datasets: pl.scan_csv() + .collect()
2. Filter early to reduce data volume before expensive operations
3. Select only needed columns to minimize memory usage
4. Prefer Parquet for storage - faster I/O, better compression
5. Use .explain() to understand and optimize query plans