Connect AI

What is MCP?

Model Context Protocol (MCP) is an open standard that lets AI assistants call external tools. Crawlio's MCP server exposes crawl control, monitoring, export, and enrichment as tools that AI clients can invoke directly.

The init wizard

The fastest way to connect. The wizard auto-detects installed AI clients and writes the correct config for each one.

# If you installed via npm:
npx crawlio-mcp init
 
# If you installed via Homebrew:
crawlio-mcp init
 
# If you have the desktop app:
/Applications/Crawlio.app/Contents/Helpers/CrawlioMCP init

The wizard:

Finds the CrawlioMCP binary (app bundle, Homebrew, npm cache, or auto-downloads it)
Detects which AI clients are installed (15 JSON-format clients + 2 TOML/YAML)
Writes the MCP server entry to each client's config file
Installs 5 skills (crawlio-mcp, crawl-site, audit-site, observe, finding)

Supported clients

The init wizard detects and configures all of these automatically:

Client	Config Path
Claude Code	`~/.claude.json`
Claude Desktop	`~/Library/Application Support/Claude/claude_desktop_config.json`
Cursor	`~/.cursor/mcp.json`
VS Code (Copilot)	`~/Library/Application Support/Code/User/mcp.json`
Windsurf	`~/.codeium/windsurf/mcp_config.json`
Zed	`~/Library/Application Support/Zed/settings.json`
Gemini CLI	`~/.gemini/settings.json`
ChatGPT Desktop	Follow ChatGPT's MCP plugin setup
GitHub Copilot CLI	`~/.copilot/mcp-config.json`
OpenCode	`~/.config/opencode/opencode.json`
Cline	VS Code globalStorage settings
Codex	`~/.codex/config.toml`
Goose	`~/.config/goose/config.yaml`

Any MCP client that supports stdio transport works with Crawlio. Point it at npx crawlio-mcp or the binary directly.

Wizard flags:

Flag	Effect
`--full`	Expose all 49 tools (default: 6 code-mode tools)
`--portal`	Use HTTP transport instead of stdio
`--dry-run`	Preview changes without writing
`--yes` / `-y`	Skip confirmation prompts

Manual configuration

If you prefer to configure manually, add the following to your client's config file.

File: ~/.claude.json or .mcp.json (project-level)

{
  "mcpServers": {
    "crawlio": {
      "command": "npx",
      "args": ["-y", "crawlio-mcp"]
    }
  }
}

File: ~/Library/Application Support/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "crawlio": {
      "command": "npx",
      "args": ["-y", "crawlio-mcp"]
    }
  }
}

Follow ChatGPT's MCP plugin setup, then add the same server entry:

{
  "mcpServers": {
    "crawlio": {
      "command": "npx",
      "args": ["-y", "crawlio-mcp"]
    }
  }
}

File: ~/.cursor/mcp.json

{
  "mcpServers": {
    "crawlio": {
      "command": "npx",
      "args": ["-y", "crawlio-mcp"]
    }
  }
}

File: ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "crawlio": {
      "command": "npx",
      "args": ["-y", "crawlio-mcp"]
    }
  }
}

File: ~/Library/Application Support/Code/User/mcp.json

ℹ️

VS Code uses the servers key, not mcpServers.

{
  "servers": {
    "crawlio": {
      "command": "npx",
      "args": ["-y", "crawlio-mcp"]
    }
  }
}

File: ~/Library/Application Support/Zed/settings.json

{
  "mcpServers": {
    "crawlio": {
      "command": "npx",
      "args": ["-y", "crawlio-mcp"]
    }
  }
}

File: ~/.gemini/settings.json

{
  "mcpServers": {
    "crawlio": {
      "command": "npx",
      "args": ["-y", "crawlio-mcp"]
    }
  }
}

If you installed via Homebrew or have the app bundle, point directly to the binary for faster startup:

{
  "mcpServers": {
    "crawlio": {
      "command": "crawlio-mcp"
    }
  }
}

Or with the full path:

{
  "mcpServers": {
    "crawlio": {
      "command": "/Applications/Crawlio.app/Contents/Helpers/CrawlioMCP"
    }
  }
}

Code mode vs full mode

By default, Crawlio MCP starts in code mode with 6 tools and roughly 800 schema tokens. The AI writes code against typed HTTP endpoints instead of calling individual tools. This means fewer round-trips and smaller context overhead.

To expose all 49 tools, add --full:

{
  "mcpServers": {
    "crawlio": {
      "command": "npx",
      "args": ["-y", "crawlio-mcp", "--full"]
    }
  }
}

Full mode is better for interactive use where you want the AI to see all available operations. Code mode is better for automated pipelines and CLI usage.

See MCP Code Mode for details on writing code against the API.

What AI can do with Crawlio

With the MCP server connected, your AI assistant can:

Start and monitor crawls. "Crawl example.com with depth 3" starts a crawl and reports progress.
Export results. "Export as WARC" or "Extract all text as Markdown" triggers the right export pipeline.
Analyze sites. "Run SEO analysis" invokes 19 built-in analyzers and returns findings with severity levels.
Capture browser state. Framework detection, network waterfalls, console logs, and DOM snapshots via WebKit runtime capture.

Example conversation:

You: Crawl docs.example.com and export as Markdown
 
AI: Starting crawl of docs.example.com with default settings...
    Crawl complete: 47 pages, 312 resources downloaded.
    Exporting as extracted text and Markdown...
    Done. Files saved to ~/Downloads/Crawlio/docs.example.com/

Verify the connection

After configuring, test the connection in your AI client:

You: "What's the crawl status?"
AI: [calls get_crawl_status] "Engine is idle, no active crawl."

If the connection fails, make sure Crawlio.app is running. The MCP server communicates with the app via Unix Domain Socket at ~/Library/Logs/Crawlio/control.sock.

Next steps

MCP Overview: Full MCP server documentation (tools, code mode, resources, prompts)
MCP Setup: Detailed MCP configuration options
AI Skills: 7 skills that teach AI assistants how to crawl and analyze sites
Common Workflows: AI-driven crawl recipes for common tasks

PreviousYour First Crawl NextChangelog