incremental-fetch
Ensures reliable data pipelines using a two-watermark system to prevent duplicates and data gaps.
Install
mkdir -p .claude/skills/incremental-fetch && curl -L -o skill.zip "https://agentskills.codes/api/skills/download/15524" && unzip -o skill.zip -d .claude/skills/incremental-fetch && rm skill.zipInstalls to .claude/skills/incremental-fetch
Activation
This is the description your AI agent reads to decide when to run this skill — the better it matches your request, the more reliably it fires.
Build resilient data ingestion pipelines from APIs. Use when creating scripts that fetch paginated data from external APIs (Twitter, exchanges, any REST API) and need to track progress, avoid duplicates, handle rate limits, and support both incremental updates and historical backfills. Triggers: 'ingest data from API', 'pull tweets', 'fetch historical data', 'sync from X', 'build a data pipeline', 'fetch without re-downloading', 'resume the download', 'backfill older data'. NOT for: simple one-shot API calls, websocket/streaming connections, file downloads, or APIs without pagination.Key capabilities
- →Track progress using two watermarks for forward and backward fetching
- →Save data records after each page fetch
- →Update watermarks only at the end of a successful run
- →Handle API rate limits gracefully
- →Adapt to different pagination types like cursor or timestamp
How it works
This skill builds resilient data ingestion pipelines from APIs by using a two-watermark pattern to track progress and manage data fetching.
Inputs & outputs
When to use incremental-fetch
- →Ingesting paginated data from REST APIs
- →Syncing tweets or feed data
- →Building a resilient backfill pipeline
About incremental-fetch
Implements persistent fetching pipelines that track progress through watermarks. This ensures the ability to resume downloads, avoid redundant calls, and perform historical backfills safely.
Build resilient data ingestion pipelines from APIs. Use when creating scripts that fetch paginated data from external APIs (Twitter, exchanges, any REST API) and need to track progress, avoid duplicates, handle rate limits, and support both incremental updates and historical backfills. Triggers: 'in
When not to use it
- →For simple one-shot API calls
- →For websocket/streaming connections
- →For file downloads
Limitations
- →The skill is not for APIs without pagination
- →The skill focuses on ID-based pagination
- →The skill requires a database for state management
How it compares
This skill implements a persistent fetching pipeline that prevents data loss and re-fetching, which is more reliable than simple API calls.
Compared to similar skills
incremental-fetch side by side with the closest alternatives in the catalog.
| Skill | Installs | Updated | Safety | Difficulty |
|---|---|---|---|---|
| incremental-fetch (this skill) | 0 | 7mo | No flags | Advanced |
| crawl4ai | 21 | 10mo | Review | Intermediate |
| apify | 9 | 5mo | Review | Intermediate |
| douyin-scraper-skill | 0 | 5mo | Review | Advanced |
Try saying
Example prompts that trigger this skill in your AI assistant.
More by xiangteng007
View all by xiangteng007 →You might also like
crawl4ai
basher83
This skill should be used when users need to scrape websites, extract structured data, handle JavaScript-heavy pages, crawl multiple URLs, or build automated web data pipelines. Includes optimized extraction patterns with schema generation for efficient, LLM-free extraction.
apify
vm0-ai
Web scraping and automation platform with pre-built Actors for common tasks
douyin-scraper-skill
orange-suli
douyin-scraper-skill — an agent skill by orange-suli.
web-scraper
KevanPatira
Web scraping inteligente multi-estrategia. Extrai dados estruturados de paginas web (tabelas, listas, precos). Paginacao, monitoramento e export CSV/JSON.
apify-ultimate-scraper
Anhvu1107
ALWAYS use this when the request matches Apify Ultimate Scraper: AI-driven data extraction from 55+ Actors across all major platforms.
ax
yusukebe
Use the ax CLI instead of curl + throwaway parsing scripts whenever you fetch a URL, explore an unknown web page, or extract structured data from HTML. Trigger whenever you are about to write an inline script (python3 heredoc, node -e, regex over HTML) or a bare curl for one-off web fetching, scrapi