Crawlio for Developers

Docs and tools to crawl, analyze, and export websites with AI.

Download entire websites for offline use. Detect frameworks, enrich with AI, and export in 7 formats.

34 crawling tools for AI agents. Code mode cuts token usage by 92%.

Automate Chrome with 96 browser commands. Connects to your existing browser sessions.

AI Skills

Teach Claude, Cursor, or any agent how to crawl. 7 skills and 1 agent, installable via npx skills add or Claude Code.

Install the plugin, understand the judgment-vs-mechanics split, fork it for your domain.

Parameters, workflows, and examples for all 7 skills including the canonical 37-tool MCP reference.

Download your first website in under a minute.

Connect Claude, ChatGPT, Cursor, or Windsurf to Crawlio.

3 meta-tools replace 34 individual tools. Saves 92% on tokens.

Detect React, Next.js, Vue, Svelte, and 20+ more frameworks.

22 commands, interactive shell, and autonomous crawl loop.

7 formats: folder mirror, ZIP, single HTML, WARC, PDF, text, and deploy bundle.

Workflows, settings, and recipes

HTTP API, architecture, and file paths

Solve common issues