deduplication
This skill cleans event streams by using reputation scoring and content grouping to identify canonical records.
Install
mkdir -p .claude/skills/deduplication && curl -L -o skill.zip "https://agentskills.codes/api/skills/download/6694" && unzip -o skill.zip -d .claude/skills/deduplication && rm skill.zipInstalls to .claude/skills/deduplication
Activation
This is the description your AI agent reads to decide when to run this skill — the better it matches your request, the more reliably it fires.
Event deduplication with canonical selection, reputation scoring, and hash-based grouping for multi-source data aggregation. Handles both ID-based and content-based deduplication.Key capabilities
- →Perform ID-based deduplication
- →Execute content-based semantic grouping
- →Calculate reputation scores for sources
- →Select canonical records from duplicate groups
- →Track deduplication reduction metrics
How it works
It groups items by normalized content keys or IDs, then applies a scoring function based on source reputation to select the canonical version.
Inputs & outputs
When to use deduplication
- →Deduplicating user records from multiple sources
- →Cleaning event logs for data analysis
- →Removing redundant data entries in pipelines
- →Selecting the best canonical data points
About deduplication
Performs data cleaning by deduplicating event streams. It utilizes reputation scoring and content-based grouping to select canonical records from aggregated datasets.
Event deduplication with canonical selection, reputation scoring, and hash-based grouping for multi-source data aggregation. Handles both ID-based and content-based deduplication.
When not to use it
- →When data requires complex fuzzy matching beyond simple normalization
- →When source reputation is not a relevant factor
Prerequisites
Limitations
- →Relies on predefined source reputation tiers
- →Normalization logic is limited to title and date
How it compares
It moves beyond simple URL-based filtering by using semantic normalization and reputation-weighted selection to preserve high-quality data.
Compared to similar skills
deduplication side by side with the closest alternatives in the catalog.
| Skill | Installs | Updated | Safety | Difficulty |
|---|---|---|---|---|
| deduplication (this skill) | 1 | 6mo | No flags | Intermediate |
| data-engineering | 13 | 7mo | Review | Advanced |
| crawl4ai | 21 | 8mo | Review | Intermediate |
| data-cleaning-pipeline | 13 | 5mo | Review | Intermediate |
Try saying
Example prompts that trigger this skill in your AI assistant.
More by dadbodgeoff
View all by dadbodgeoff →You might also like
data-engineering
pluginagentmarketplace
ETL pipelines, Apache Spark, data warehousing, and big data processing. Use for building data pipelines, processing large datasets, or data infrastructure.
crawl4ai
basher83
This skill should be used when users need to scrape websites, extract structured data, handle JavaScript-heavy pages, crawl multiple URLs, or build automated web data pipelines. Includes optimized extraction patterns with schema generation for efficient, LLM-free extraction.
data-cleaning-pipeline
aj-geddes
Build robust processes for data cleaning, missing value imputation, outlier handling, and data transformation for data preprocessing, data quality, and data pipeline automation
paddle-ocr-validation
jgtolentino
PaddleOCR-based receipt and BIR form extraction with validation
apify
vm0-ai
Web scraping and automation platform with pre-built Actors for common tasks
ocr
trpc-group
Extract text from images using Tesseract OCR