Export Formats

Same structure as folder, compressed into a single .zip archive. Configurable compression level (1-9).

crawlio export run zip --dest ~/archives/example.zip

Inlines all CSS, JavaScript, and images (as base64 data URIs) into a single self-contained HTML file.

crawlio export run singleHTML --dest ~/snapshots/example.html

Supported image formats for inlining: PNG, JPG, JPEG, GIF, SVG, WebP, ICO.

Output is capped at 50 MB. Requires index.html in the downloaded content. External resources not in the download set remain as external references.

Web ARChive (ISO 28500), the standard format used by the Internet Archive's Wayback Machine. Each resource is stored with full HTTP headers, response bodies, and metadata.

crawlio export run warc --dest ~/archives/example.warc

What's included:

• warcinfo header record with Crawlio version and crawl metadata
• Both request and response records with real HTTP headers
• SHA-1 digests (Base32-encoded) for both block and payload per RFC 4648
• Content-Type auto-detected from file extension
• OCR text as metadata records (when OCR is enabled) with WARC-Refers-To linking

Four settings control WARC output:

When splitting is active, files are numbered sequentially (example-00000.warc.gz, example-00001.warc.gz). Each split file gets its own warcinfo record. CDX entries track which split file contains each record.

WARC files can be replayed with ReplayWeb.page or ingested into web archive infrastructure like pywb and warctools.

Full-page PDF rendering via WebKit. Each page is rendered as it would appear in a browser, with full CSS applied.

crawlio export run pdf --dest ~/pdfs/example.pdf

Clean, structured content extracted from each page: normalized HTML, Markdown, metadata, and asset inventories.

output/
  pages/
    index/
      page.html          # Normalized HTML
      content.md         # Markdown conversion
      metadata.json      # Title, description, OG data
      assets.json        # Categorized asset inventory
    about/
      page.html
      content.md
      metadata.json
      assets.json
  manifest.json          # Crawl metadata, site tree

The metadata.json includes:

{
  "url": "https://example.com/",
  "title": "Example Domain",
  "contentType": "text/html",
  "statusCode": 200,
  "headers": { "h1": ["Example Domain"], "h2": ["About", "Contact"] },
  "links": ["https://example.com/about", "https://example.com/contact"],
  "framework": "nextjs",
  "extractedAt": "2026-02-14T10:30:00Z"
}

This format is ideal for AI consumption. Feed content.md files into RAG pipelines, search indexes, or training datasets.

A deploy-ready static site with clean URL routes and machine-readable manifests.

output/
  index.html
  blog/
    index.html              # from blog.html
    series-a/
      index.html            # from blog/series-a.html
  pricing/
    index.html              # from pricing.html
  _assets/
    cdn.example.com/        # cross-domain assets
      fonts/inter.woff2
  _next/static/             # framework assets preserved
  deploy.json               # manifest
  sitemap.xml               # generated

Features:

• Multi-host flattening: main domain at root, cross-domain assets under _assets/{host}/
• HTML pages converted to clean URL routes (blog.html becomes blog/index.html)
• Reference rewriting in HTML/CSS files for the flattened structure
• Skips junk files: API routes, .css_ duplicates, query-variant hash duplicates
• Generates deploy.json manifest and sitemap.xml

Deploy to Vercel, Netlify, S3, or any static hosting.