web-scraper
Extract and crawl data from websites using structured prompts and batch processing.
Install
mkdir -p .claude/skills/web-scraper-shobcoder && curl -L -o skill.zip "https://agentskills.codes/api/skills/download/10989" && unzip -o skill.zip -d .claude/skills/web-scraper-shobcoder && rm skill.zipInstalls to .claude/skills/web-scraper-shobcoder
Activation
This is the description your AI agent reads to decide when to run this skill — the better it matches your request, the more reliably it fires.
Scrape, crawl, and extract data from websites. Use when users ask to scrape web pages, extract content, crawl websites, or collect data from the internet.Key capabilities
- →Crawls websites
- →Extracts structured data
- →Supports batch URL processing
- →Handles dynamic content
How it works
It uses various scraping techniques, including browser rendering, to extract structured content from websites based on user prompts.
Inputs & outputs
When to use web-scraper
- →Scraping articles for analysis
- →Gathering product price lists
- →Extracting structured data from web pages
About web-scraper
Automates website crawling and content extraction into structured JSON formats, supporting research and data collection tasks.
Scrape, crawl, and extract data from websites. Use when users ask to scrape web pages, extract content, crawl websites, or collect data from the internet.
When not to use it
- →When the website prohibits scraping
- →When the user needs real-time interaction
Limitations
- →Respect website terms of service
- →Dynamic content may require browser mode
How it compares
It provides a structured, prompt-based approach to web scraping rather than manual parsing.
Compared to similar skills
web-scraper side by side with the closest alternatives in the catalog.
| Skill | Installs | Updated | Safety | Difficulty |
|---|---|---|---|---|
| web-scraper (this skill) | 0 | 5mo | No flags | Intermediate |
| pi-share | 1 | 8mo | Review | Beginner |
| apify-trend-analysis | 0 | 6mo | Review | Intermediate |
| brightdata-web-mcp | 9 | 9mo | Review | Intermediate |
Try saying
Example prompts that trigger this skill in your AI assistant.
You might also like
pi-share
mitsuhiko
Load and parse session transcripts from shittycodingagent.ai/buildwithpi.ai/buildwithpi.com (pi-share) URLs. Fetches gists, decodes embedded session data, and extracts conversation history.
apify-trend-analysis
Anhvu1107
ALWAYS use this when the request matches Apify Trend Analysis: Discover and track emerging trends across Google Trends, Instagram, Facebook, YouTube, and TikTok to inform content strategy.
brightdata-web-mcp
patchy631
Search the web, scrape websites, extract structured data from URLs, and automate browsers using Bright Data's Web MCP. Use when fetching live web content, bypassing blocks/CAPTCHAs, getting product data from Amazon/eBay, social media posts, or when standard requests fail.
webclaw
0xmassi
Web extraction engine with antibot bypass. Scrape, crawl, extract, summarize, search, map, diff, monitor, research, and analyze any URL — including Cloudflare-protected sites. Use when you need reliable web content, the built-in web_fetch fails, or you need structured data extraction from web pages.
juicebox-core-workflow-b
jeremylongshore
Implement Juicebox candidate enrichment workflow. Use when enriching profile data, gathering additional candidate details, or building comprehensive candidate profiles. Trigger with phrases like "juicebox enrich profile", "juicebox candidate details", "enrich candidate data", "juicebox profile enrichment".
openevidence-core-workflow-b
jeremylongshore
Execute OpenEvidence DeepConsult workflow for comprehensive medical research. Use when implementing deep research synthesis, complex clinical questions, or when physicians need extensive literature review. Trigger with phrases like "openevidence deepconsult", "deep research", "comprehensive evidence", "literature synthesis".