apache-spark
Enables large-scale data processing and analytics using Apache Spark frameworks.
Install
mkdir -p .claude/skills/apache-spark && curl -L -o skill.zip "https://agentskills.codes/api/skills/download/16761" && unzip -o skill.zip -d .claude/skills/apache-spark && rm skill.zipInstalls to .claude/skills/apache-spark
Activation
This is the description your AI agent reads to decide when to run this skill — the better it matches your request, the more reliably it fires.
Processes large-scale data with Apache Spark using DataFrames, RDDs, and Spark SQL. Use for big data ETL and analytics.Key capabilities
- →process terabyte-scale data
- →perform ETL operations
- →conduct interactive analytics
- →engineer features for machine learning
- →execute DataFrame operations
- →run Spark SQL queries
How it works
The skill initializes a SparkSession, reads CSV data into a DataFrame, and then performs operations like filtering, adding columns, and aggregating data.
Inputs & outputs
When to use apache-spark
- →ETL pipelines
- →Large-scale data processing
- →Interactive analytics
- →ML feature engineering
About this skill
Apache Spark
Unified analytics engine for large-scale data processing.
Quick Start
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("MyApp").getOrCreate()
df = spark.read.csv("data/*.csv", header=True, inferSchema=True)
df.groupBy("category").agg({"amount": "sum"}).show()
DataFrame API
from pyspark.sql.functions import col, avg, when
result = df.filter(col("age") > 18) \
.withColumn("adult", when(col("age") >= 21, "yes").otherwise("no")) \
.groupBy("department").agg(avg("salary").alias("avg_salary"))
Spark SQL
df.createOrReplaceTempView("users")
result = spark.sql("SELECT department, AVG(salary) as avg_salary FROM users GROUP BY department")
When to Use
- Terabyte-scale data processing
- ETL pipelines
- Interactive analytics
- ML feature engineering
Validation
- SparkContext initializes
- DataFrame operations execute
- Spark SQL queries return correct results
When not to use it
- →when data processing is not terabyte-scale
- →when ETL pipelines are not required
- →when interactive analytics are not needed
Limitations
- →limited to large-scale data processing
- →focused on ETL pipelines
- →designed for interactive analytics
How it compares
This skill provides a unified analytics engine for large-scale data processing, unlike manual data handling that would require separate tools for different data sizes and operations.
Compared to similar skills
apache-spark side by side with the closest alternatives in the catalog.
| Skill | Installs | Updated | Safety | Difficulty |
|---|---|---|---|---|
| apache-spark (this skill) | 0 | 2mo | No flags | Intermediate |
| hugging-face-datasets | 1 | 6mo | Review | Intermediate |
| quant-analyst | 103 | 3mo | No flags | Advanced |
| umap-learn | 6 | 2mo | Review | Intermediate |
Try saying
Example prompts that trigger this skill in your AI assistant.
More by ssrjkk
View all by ssrjkk →You might also like
hugging-face-datasets
patchy631
Create and manage datasets on Hugging Face Hub. Supports initializing repos, defining configs/system prompts, streaming row updates, and SQL-based dataset querying/transformation. Designed to work alongside HF MCP server for comprehensive dataset workflows.
quant-analyst
zenobi-us
Expert quantitative analyst specializing in financial modeling, algorithmic trading, and risk analytics. Masters statistical methods, derivatives pricing, and high-frequency trading with focus on mathematical rigor, performance optimization, and profitable strategy development.
umap-learn
K-Dense-AI
UMAP dimensionality reduction. Fast nonlinear manifold learning for 2D/3D visualization, clustering preprocessing (HDBSCAN), supervised/parametric UMAP, for high-dimensional data.
senior-data-engineer
davila7
World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, and modern data stack. Includes data modeling, pipeline orchestration, data quality, and DataOps. Use when designing data architectures, building data pipelines, optimizing data workflows, or implementing data governance.
embedding-strategies
wshobson
Select and optimize embedding models for semantic search and RAG applications. Use when choosing embedding models, implementing chunking strategies, or optimizing embedding quality for specific domains.
building-automl-pipelines
jeremylongshore
Build automated machine learning pipelines, including feature engineering, model selection, and performance evaluation.