@purepageio/fetch-engines

Reliable web extraction.

HTTP-first for speed. Browser-backed when needed. Clean Markdown, soft-block handling, and structured extraction for RAG and AI pipelines.

• Why fetch-engines?
• Why trust fetch-engines
• Library vs hosted crawler
• Installation
• Quick start
• Usage patterns
  • Pick an engine
  • Markdown mode
  • Structured extraction
• Configuration
  • Essentials
  • Complete reference
• Error handling
• Tooling and examples
• Contributing
• License

• One API for multiple strategies – Call fetchHTML for rendered pages or fetchContent for raw responses. The library handles HTTP shortcuts and Playwright fallbacks automatically.
• Automatic app-shell & soft-block detection – Shell-like HTTP responses and bot-gate pages (Cloudflare challenges, CAPTCHAs, "verify you're human") are upgraded to Playwright rendering by default, so client-rendered pages and soft blocks work without per-domain rules.
• RAG-ready Markdown – Convert public content pages to clean Markdown with boilerplate, nav, and SVG noise stripped out. Powered by a Rust-native converter.
• HTTP-first, browser-backed when needed – Fast pages stay cheap via plain HTTP, while harder pages automatically benefit from Playwright fallback.
• Structured extraction built in – Define a Zod schema and go from URL to typed data via any OpenAI-compatible API. The page is fetched as Markdown first to minimise tokens.
• Playwright is optional – FetchEngine works without browser dependencies. Playwright is only loaded when you use HybridEngine or PlaywrightEngine.

• 19 live URLs across 7 archetypes (docs, government, knowledge, marketing, commerce, static, access-guarded) validated on every release and nightly via browser-enabled CI
• 85 unit tests + dedicated live browser eval workflow — not just "it compiles," but "it extracts real content from real pages"
• Handles app shells, Cloudflare challenges, CAPTCHAs, and utility-class-heavy doc sites (Tailwind, Vite) without per-domain rules
• Produces clean Markdown with absolute URLs — boilerplate removal typically reduces raw HTML to 10–30% of its original size before it reaches your LLM
• Structured extraction with Zod schemas and any OpenAI-compatible provider, in the same pipeline as page fetching

Choose fetch-engines when you want full control over extraction, data residency, and cost. Choose a hosted crawler when you need managed infrastructure and don't want to run browsers yourself.

pnpm add @purepageio/fetch-engines

Node.js 22 or newer is supported. Repository development uses Node.js 22.23.4 (nvm use) and requires pnpm 11 or newer.

Structured extraction uses AI SDK 6 with the OpenAI and OpenAI-compatible providers. Existing Zod schemas, including optional fields, and the public extraction result and token-usage fields remain supported.

If you plan to use HybridEngine or PlaywrightEngine (which launch a real browser), install the Playwright browsers once:

pnpm exec playwright install

This step is not needed if you only use FetchEngine or StructuredContentEngine.

import { HybridEngine } from "@purepageio/fetch-engines";

const engine = new HybridEngine();

const page = await engine.fetchHTML("https://example.com");
console.log(page.title);

await engine.cleanup();

Pass markdown: true to get clean Markdown instead of raw HTML — ideal for RAG pipelines, LLM context windows, and content indexing. The converter strips navigation, footers, sidebars, SVGs, and high-link-density boilerplate before converting via a Rust-native engine.

import { HybridEngine } from "@purepageio/fetch-engines";

const engine = new HybridEngine({ markdown: true });
const page = await engine.fetchHTML("https://example.com/blog/my-post");

console.log(page.content); // Clean Markdown with headings, tables, code blocks preserved
console.log(page.contentType); // "markdown"

await engine.cleanup();

FetchEngine also supports markdown: true for static pages that don't need JavaScript rendering. HybridEngine now decides whether to render before converting to Markdown, so shell detection still works when callers request Markdown output. Relative links and image URLs in Markdown output are normalized to absolute URLs using the final fetched page URL. The converter strips generic UI chrome (nav/footer/button controls and dense link clusters) using domain-agnostic heuristics, while preserving content on pages without semantic <main>/<article> containers (e.g., Tailwind CSS docs). The extraction path is tuned for publicly accessible content. Paywalled or member-only pages may still return intentionally partial content unless you supply authenticated access yourself.

import { fetchStructuredContent } from "@purepageio/fetch-engines";
import { z } from "zod";

// IMPORTANT: All schema fields must have .describe() calls to guide the AI model
const schema = z.object({
  title: z.string().describe("The title of the article"),
  summary: z.string().describe("A brief summary of the article content"),
});

// model is required - use any model supported by your API provider
const result = await fetchStructuredContent("https://example.com/article", schema, { model: "gpt-4.1-mini" });

console.log(result.data.summary);

Under the hood, structured extraction fetches the page as Markdown first (same boilerplate removal as Markdown mode), then sends the cleaned content to the AI model — keeping token usage low and extraction quality high.

Set OPENAI_API_KEY (or OPENROUTER_API_KEY) before running structured helpers, or use apiConfig to connect to OpenAI-compatible APIs like OpenRouter. The engine automatically adds the Authorization header when you provide an API key:

const result = await fetchStructuredContent("https://example.com/article", schema, {
  model: "anthropic/claude-3.5-sonnet",
  apiConfig: {
    apiKey: process.env.OPENROUTER_API_KEY,
    baseURL: "https://openrouter.ai/api/v1",
    headers: {
      "HTTP-Referer": "https://your-app.com",
      "X-Title": "Your App Name",
    },
  },
});

When you supply a custom baseURL, the engine automatically switches to the Vercel AI SDK's createOpenAICompatible provider (instead of createOpenAI) so OpenAI-compatible services like OpenRouter receive the correct API-key auth flow.

All engines accept familiar fetch options such as custom headers. Additional Hybrid/Playwright options you are likely to tweak:

• markdown – return Markdown instead of HTML.
• Automatic shell detection is enabled by default. spaMode & spaRenderDelayMs still force a more patient render path when you know a page is highly dynamic.
• cacheTTL, maxRetries, and browser pool sizes – control resilience and throughput.

Check the inline TypeScript docs or the /examples directory for end-to-end flows.

Every option from PlaywrightEngineConfig (consumed by HybridEngine) with defaults:

Per-request overrides: fetchHTML accepts fastMode, markdown, spaMode, and headers, while fetchContent supports fastMode and headers.

For Chromium sites whose automatic verification rejects JavaScript-only stealth, opt into Patchright and keep the challenge pass headed and fully resourced:

const juno = new HybridEngine({
  browserDriver: "patchright",
  challengeWaitMs: 15_000,
  defaultFastMode: false,
  markdown: true,
  maxBrowsers: 1,
  playwrightOnlyPatterns: ["juno.co.uk"],
  useHeadedMode: true,
  useHttpFallback: false,
});

const result = await juno.fetchHTML("https://www.juno.co.uk/products/overmono-pure-devotion-vinyl/1156010-01/");
await juno.cleanup();

patchright launches a temporary persistent profile with the browser's native viewport and without custom user-agent or locale injection. It does not solve or submit CAPTCHAs. cdpEndpoint can instead attach Hybrid to an existing Chromium session or a managed browser endpoint; CDP mode preserves the endpoint's default context and cookies and uses a single pooled browser connection.

Failures raise a typed FetchError exposing code, statusCode, and the underlying error. Log these fields to diagnose issues quickly and tune your retry policy.

• Explore the examples directory for scripts you can run end-to-end.
• Ready-to-use TypeScript types ship with the package.
• pnpm test runs the automated suite when you are ready to contribute.
• pnpm test:live:smoke runs the browser-backed live smoke suite, including markdown fetch checks for https://www.bhp.com/ and the Juno Pure Devotion product page. The Juno probe is observe-only in ordinary CI because a hosted runner's network reputation is outside the library's control. Maintainers can explicitly enable a configured, provider-neutral LIVE_CDP_ENDPOINT when manually dispatching the workflow.
• pnpm eval:auto-render runs a live Hybrid-vs-HTTP quality matrix across docs, government, knowledge, marketing, commerce, and access-guarded pages, using a stable gated core plus observe-only sentinels for harder domains.
• pnpm test:live:auto-render runs the same hypothesis as a Vitest live test (LIVE_NETWORK=1) in a Node environment so it exercises the real network/browser stack instead of jsdom.
• GitHub Actions includes a dedicated browser-enabled live eval workflow that runs on main changes, nightly on a schedule, and on manual dispatch. It uploads the JSON report as a build artifact. The default workflow requires no paid browser service.

Issues and pull requests are welcome! Before submitting a change:

1. pnpm test — all tests pass
2. pnpm lint — no errors
3. pnpm format — code is formatted
4. Update CHANGELOG.md under [Unreleased]
5. Bump the version in package.json following semver

See AGENTS.md for detailed guidelines on versioning, testing, and documentation.

Distributed under the MIT license.