gcp-spark
Assists with writing, optimizing, and debugging Spark jobs for Dataproc and BigQuery integration.
Install
mkdir -p .claude/skills/gcp-spark && curl -L -o skill.zip "https://agentskills.codes/api/skills/download/17924" && unzip -o skill.zip -d .claude/skills/gcp-spark && rm skill.zipInstalls to .claude/skills/gcp-spark
Activation
This is the description your AI agent reads to decide when to run this skill — the better it matches your request, the more reliably it fires.
Develops and executes Spark code on Dataproc Clusters and Serverless. Reads and writes data using BigLake Iceberg catalogs, BigQuery and Spanner. Debugs execution failures. Use when: - Writing Spark ETL pipelines on GCP. - Training or running inference with ML models with spark on GCP. - Managing Spark clusters, jobs, batches, and interactive sessions. Don't use when: - Writing generic Python scripts that don't use Spark. - Performing simple SQL queries that can be done directly in BigQuery.Key capabilities
- →Develop and execute Spark code on Dataproc Clusters and Serverless
- →Read and write data using BigLake Iceberg catalogs, BigQuery and Spanner
- →Debug execution failures
- →Train or run inference with ML models with Spark on GCP
How it works
The skill develops and executes Spark code on GCP Dataproc, integrating with BigLake, BigQuery, and Spanner, while enforcing best practices for schema understanding and code generation.
Inputs & outputs
When to use gcp-spark
- →Write Spark ETL pipelines
- →Manage Dataproc cluster jobs
- →Debug Spark execution failures
- →Train ML models on Spark
About gcp-spark
Generates Spark code in notebook format while enforcing GCP-specific best practices for data schema and cluster management. It integrates with BigLake, BigQuery, and Spanner to build efficient data pipelines.
|
When not to use it
- →When writing generic Python scripts that don't use Spark
- →When performing simple SQL queries that can be done directly in BigQuery
- →When converting a PySpark dataframe to Pandas by calling toPandas() for large datasets
Limitations
- →Converting a PySpark dataframe to Pandas by calling toPandas() can lead to out of memory errors
- →CSV files read without header and inferSchema will result in the header row becoming data and all columns as strings
- →IAM requirements for Dataproc service account include roles/dataproc.worker, roles/biglake.admin, roles/bigquery.jobUser, roles/storage.objectUser, and roles/spanner.databaseUser
How it compares
This skill provides a structured workflow for Spark development on GCP, emphasizing schema validation and specific optimization guidelines for Dataproc, BigLake, BigQuery, and Spanner.
Compared to similar skills
gcp-spark side by side with the closest alternatives in the catalog.
| Skill | Installs | Updated | Safety | Difficulty |
|---|---|---|---|---|
| gcp-spark (this skill) | 0 | 24d | No flags | Advanced |
| feast-user-guide | 0 | 20d | Review | Intermediate |
| cocoindex | 6 | 9mo | Review | Intermediate |
| similarity-search-patterns | 3 | 2mo | No flags | Advanced |
Try saying
Example prompts that trigger this skill in your AI assistant.
More by ironkid90
View all by ironkid90 →You might also like
feast-user-guide
feast-dev
Guide for working with Feast (Feature Store) — defining features, configuring feature_store.yaml, retrieving features online/offline, using the CLI, and building RAG retrieval pipelines. Use when the user asks about creating entities, feature views, on-demand feature views, stream feature views, fea
cocoindex
cocoindex-io
Comprehensive toolkit for developing with the CocoIndex library. Use when users need to create data transformation pipelines (flows), write custom functions, or operate flows via CLI or API. Covers building ETL workflows for AI data processing, including embedding documents into vector databases, building knowledge graphs, creating search indexes, or processing data streams with incremental updates.
similarity-search-patterns
wshobson
Implement efficient similarity search with vector databases. Use when building semantic search, implementing nearest neighbor queries, or optimizing retrieval performance.
senior-data-scientist
davila7
World-class data science skill for statistical modeling, experimentation, causal inference, and advanced analytics. Expertise in Python (NumPy, Pandas, Scikit-learn), R, SQL, statistical methods, A/B testing, time series, and business intelligence. Includes experiment design, feature engineering, model evaluation, and stakeholder communication. Use when designing experiments, building predictive models, performing causal analysis, or driving data-driven decisions.
spark-optimization
wshobson
Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.
hugging-face-datasets
patchy631
Create and manage datasets on Hugging Face Hub. Supports initializing repos, defining configs/system prompts, streaming row updates, and SQL-based dataset querying/transformation. Designed to work alongside HF MCP server for comprehensive dataset workflows.