web-scraping
Web scraping assistant using a command-line tool for single or recursive page extraction.
Install
mkdir -p .claude/skills/web-scraping-astrabert && curl -L -o skill.zip "https://agentskills.codes/api/skills/download/14718" && unzip -o skill.zip -d .claude/skills/web-scraping-astrabert && rm skill.zipInstalls to .claude/skills/web-scraping-astrabert
Activation
This is the description your AI agent reads to decide when to run this skill — the better it matches your request, the more reliably it fires.
Scrape web pages based on a provided URL using the scpr CLI app.Key capabilities
- →Scrape a single web page based on a provided URL
- →Perform recursive scraping of linked pages within the same domain
- →Control the maximum depth of pages to follow during recursive scraping
- →Execute parallel scraping with multiple concurrent threads
- →Specify allowed domains for recursive scraping
- →Save scraped content as markdown files to an output directory
How it works
The skill uses the `scpr` command-line interface to fetch web content from a given URL, supporting single-page or recursive domain-wide scraping with concurrency options.
Inputs & outputs
When to use web-scraping
- →Extracting content from websites
- →Parallel crawling for data analysis
- →Recursive site documentation gathering
About this skill
When asked to scrape a web page, use the scpr command line interface.
Basic usage (scrape a single page):
scpr --url https://example.com --output ./scraped
This will scrape the page and save it as a markdown file in the ./scraped folder.
Recursive scraping
To scrape a page and all linked pages within the same domain:
scpr --url https://example.com --output ./scraped --recursive --allowed example.com --max 3
Parallel scraping
Speed up recursive scraping with multiple threads:
scpr --url https://example.com --output ./scraped --recursive --allowed example.com --max 2 --parallel 5
Additional options
--log- Set logging level (info, debug, warn, error)--max- Maximum depth of pages to follow (default: 1)--parallel- Number of concurrent threads (default: 1)--allowed- Allowed domains for recursive scraping (can be specified multiple times)
For more details, run:
scpr --help
Once you are done with scraping, you should scan the output folder to find the content the user asked you for, here is an example flow:
scpr --url https://example.com --output ./scraped --recursive --allowed example.com --max 2
cd ./scraped
grep -r "pattern of interest"
When not to use it
- →When the user wants to interact with web pages (e.g., fill forms, click buttons)
- →When the goal is to extract data from non-HTML sources
- →When the `scpr` CLI tool is not available
Prerequisites
Limitations
- →Requires the `scpr` CLI app to be installed
- →Output is primarily markdown files
- →Does not support interactive web page operations
How it compares
This skill provides a CLI-based web scraping solution with built-in recursive and parallel capabilities, specifically designed for saving content as markdown, unlike general-purpose web crawlers or browser automation tools.
Compared to similar skills
web-scraping side by side with the closest alternatives in the catalog.
| Skill | Installs | Updated | Safety | Difficulty |
|---|---|---|---|---|
| web-scraping (this skill) | 0 | 6mo | Review | Beginner |
| reddit-fetch | 9 | 2mo | Review | Beginner |
| brightdata-web-mcp | 9 | 6mo | Review | Intermediate |
| firecrawl-scrape | 5 | 7mo | Review | Beginner |
Try saying
Example prompts that trigger this skill in your AI assistant.
You might also like
reddit-fetch
ykdojo
Fetch content from Reddit using Gemini CLI when WebFetch is blocked. Use when accessing Reddit URLs, researching topics on Reddit, or when Reddit returns 403/blocked errors.
brightdata-web-mcp
patchy631
Search the web, scrape websites, extract structured data from URLs, and automate browsers using Bright Data's Web MCP. Use when fetching live web content, bypassing blocks/CAPTCHAs, getting product data from Amazon/eBay, social media posts, or when standard requests fail.
firecrawl-scrape
parcadei
Scrape web pages and extract content via Firecrawl MCP
tavily-web
davila7
Web search, content extraction, crawling, and research capabilities using Tavily API
webclaw
0xmassi
Web extraction engine with antibot bypass. Scrape, crawl, extract, summarize, search, map, diff, monitor, research, and analyze any URL — including Cloudflare-protected sites. Use when you need reliable web content, the built-in web_fetch fails, or you need structured data extraction from web pages.
batch-research
miantiao-me
批量数据采集技能，负责分批并发调度 researcher agent 抓取所有数据源。