Processes large-scale data with Apache Spark using DataFrames, RDDs, and Spark SQL. Use for big data ETL and analytics.
Install
mkdir -p .claude/skills/apache-spark && curl -L -o skill.zip "https://agentskills.codes/api/skills/download/16761" && unzip -o skill.zip -d .claude/skills/apache-spark && rm skill.zipInstalls to .claude/skills/apache-spark
Activation
This is the description your AI agent reads to decide when to run this skill — the better it matches your request, the more reliably it fires.
Processes large-scale data with Apache Spark using DataFrames, RDDs, and Spark SQL. Use for big data ETL and analytics.119 chars✓ has a “when” trigger
About this skill
Apache Spark
Unified analytics engine for large-scale data processing.
Quick Start
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("MyApp").getOrCreate()
df = spark.read.csv("data/*.csv", header=True, inferSchema=True)
df.groupBy("category").agg({"amount": "sum"}).show()
DataFrame API
from pyspark.sql.functions import col, avg, when
result = df.filter(col("age") > 18) \
.withColumn("adult", when(col("age") >= 21, "yes").otherwise("no")) \
.groupBy("department").agg(avg("salary").alias("avg_salary"))
Spark SQL
df.createOrReplaceTempView("users")
result = spark.sql("SELECT department, AVG(salary) as avg_salary FROM users GROUP BY department")
When to Use
- Terabyte-scale data processing
- ETL pipelines
- Interactive analytics
- ML feature engineering
Validation
- SparkContext initializes
- DataFrame operations execute
- Spark SQL queries return correct results