Data Analysis
Website Content Scraper
General-purpose website scraper handling JavaScript rendering and various page structures.
Overview
The Website Scraper Agent is a general-purpose scraper handling various page structures and JavaScript-rendered content. Supports sitemap parsing, screenshot capture, and PDF extraction.
Key Capabilities:
- Page fetching with proper headers
- Content extraction (text, links, metadata)
- JavaScript rendering (Puppeteer)
- Sitemap parsing
- Screenshot capture
- PDF extraction
- robots.txt compliance
Tools & Integrations:
- Puppeteer - JavaScript rendering
- Content Extractor - HTML parsing
- PDF Parser - Document extraction