Settings Reference

Download

Control how Crawlio downloads individual files.

Scope

Control which pages and resources Crawlio follows and downloads.

Scope modes

Policy

Control crawl limits, file types, and content handling.

Content type toggles

Proxy

Route crawl traffic through an HTTP, HTTPS, or SOCKS5 proxy.

Proxy credentials (username and password) are accepted at runtime but not persisted to disk.

Crawlio also reads environment variables (http_proxy, https_proxy, no_proxy) when no explicit proxy is configured.

no_proxy matching rules:

• Exact match: example.com matches example.com
• Suffix with dot boundary: example.com matches sub.example.com but not notexample.com
• Leading dot: .example.com matches the domain and all subdomains
• Wildcard: * bypasses all hosts

Example:

crawlio settings set settings.proxyConfiguration '{"type":"http","host":"proxy.corp.com","port":8080,"noProxyHosts":["localhost",".internal.com"]}'

Or via MCP:

update_settings(settings: {
  proxyConfiguration: {
    type: "http",
    host: "proxy.corp.com",
    port: 8080,
    noProxyHosts: ["localhost", ".internal.com"]
  }
})

Security

Certificate pinning, HSTS enforcement, and trust evaluation.

Certificate pinning

Pin specific public keys per host to prevent MITM attacks. Keys are SHA-256 hashes of the DER-encoded SubjectPublicKeyInfo, Base64-encoded.

Example:

update_settings(policy: {
  pinnedPublicKeys: {
    "api.example.com": ["sha256//AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="],
    "cdn.example.com": ["sha256//BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB="]
  }
})

Crawlio validates the server certificate's public key against the pinned values during the TLS handshake. If no pin matches, the connection fails. Host matching is case-insensitive.

HSTS enforcement

Crawlio maintains an HSTS store that tracks Strict-Transport-Security headers observed during crawling. HSTS entries include max-age and includeSubDomains directives per RFC 6797.

When HSTS is active for a host:

• All http:// requests to that host are upgraded to https:// before sending
• Subdomain upgrades apply when includeSubDomains was set
• Entries expire after max-age seconds
• IP addresses are excluded from HSTS (per the RFC)

HSTS state is persisted per project and loaded on crawl resume.

OCR

Optional Vision OCR pipeline for extracting text from downloaded images. Zero overhead when disabled.

OCR runs on raster images only (PNG, JPEG, TIFF, BMP, WebP). SVG and GIF are skipped. Results appear in deploy.json, crawl-manifest.json, and WARC metadata records.

WARC

Control WARC web archive output.

See Export Formats for details on WARC output structure.

Updating settings

Settings can only be changed when the engine is idle (not actively crawling).

update_settings(settings: { maxConcurrent: 20, crawlDelay: 1.0 })
update_settings(policy: { maxDepth: 3, scopeMode: "includeSubdomains" })

crawlio settings set settings.maxConcurrent 20
crawlio settings set policy.maxDepth 3

curl --unix-socket ~/Library/Logs/Crawlio/control.sock \
  -X PATCH http://localhost/settings \
  -H "Content-Type: application/json" \
  -d '{"settings": {"maxConcurrent": 20}, "policy": {"maxDepth": 3}}'

Example: Large SPA crawl

Configure Crawlio for a large single-page application:

# Increase concurrency for fast crawling
crawlio settings set settings.maxConcurrent 20
 
# Enable JS rendering for SPA content
crawlio settings set policy.enableJSRendering true
 
# Allow subdomains (CDN assets)
crawlio settings set policy.scopeMode includeSubdomains
 
# Download cross-domain assets
crawlio settings set policy.downloadCrossDomainAssets true
 
# Extend timeout for slow JS-rendered pages
crawlio settings set settings.timeout 120
 
# Set a depth limit to avoid infinite routes
crawlio settings set policy.maxDepth 10
 
# Start the crawl
crawlio crawl start https://my-spa.com --watch

Or as a single MCP call:

update_settings(
  settings: { maxConcurrent: 20, timeout: 120 },
  policy: {
    enableJSRendering: true,
    scopeMode: "includeSubdomains",
    downloadCrossDomainAssets: true,
    maxDepth: 10
  }
)

Next steps

• MCP Tools Reference: How tools use these settings
• Export Formats: Saving results
• CLI Commands: Configure settings from the terminal