Thunderbit Launches High-Fidelity Web Data API, MCP Server, and CLI
Thunderbit, an AI web data platform with over 100,000 users, has launched its developer API, Model Context Protocol (MCP) server, and CLI — giving developers new ways to turn complex, long-tail websites into clean Markdown or structured data for AI agents, RAG pipelines, and automation workflows.
At the center of the launch is Thunderbit Distill, an adaptive HTML-to-Markdown engine designed for high-fidelity conversion across complex web pages. In internal evaluations, Distill scored 0.87 ROUGE-L and produced cleaner, more complete Markdown across product pages, pricing tables, directories, search results, and reviews — without requiring any site-specific rules.
Unlike traditional scraping pipelines that rely on CSS selectors, XPath, or fragile site-specific parsing rules, Thunderbit uses AI models to understand page semantics and adapt to changing structure. Navigation, scripts, ads, and boilerplate are stripped out automatically, so LLMs and databases receive far less noisy input.
"AI agents are only as useful as the web data they can actually reach. We built Thunderbit to turn changing web pages into data that software can use reliably."
— Shuai Guan, Co-founder & CEO, ThunderbitWhy Traditional Scraping Pipelines Break Down
The web is dynamic. Traditional scraping approaches depend on hard-coded CSS selectors or XPath expressions tied to a specific site's layout — meaning a single redesign or A/B test can silently break an entire data pipeline. Maintaining custom scrapers for every site is both expensive and fragile, making consistent data extraction at scale a persistent engineering burden.
Thunderbit takes a fundamentally different approach. Rather than pattern-matching against structure, it uses AI models to comprehend what a page is about, identify its meaningful content, and output clean, well-formatted results regardless of layout changes. The result is a scraping engine that adapts the way a human reader would — understanding context rather than memorizing coordinates.
What Launched: API, MCP Server, and CLI
Thunderbit Open API
A hosted, production-ready API that lets developers scrape any web page without managing infrastructure. Developers obtain an API key and can immediately begin extracting data at scale. It powers both Distill (clean Markdown output) and Extract (structured JSON or CSV via developer-defined JSON Schema), making it suitable for RAG pipelines, knowledge bases, databases, enrichment jobs, and internal tools.
MCP Server (@thunderbit/mcp-server)
An open-source Model Context Protocol server that plugs Thunderbit's extraction capabilities directly into Claude, Cursor, Claude Code, and any MCP-compatible LLM client. The server exposes seven tools covering distill, extract, AI-suggested field detection, and batch processing of up to 100 URLs. Installation is a single configuration block in the MCP client settings, with an API key from thunderbit.com/open-api.
CLI (@thunderbit/thunderbit-cli)
A command-line tool and agent skill installable globally via npm i -g @thunderbit/thunderbit-cli. It can be added as a Claude Code plugin with a single command, giving AI agents real-time web data access during autonomous tasks. From the terminal, developers can distill any page to Markdown, extract structured fields, or batch-process multiple URLs without writing a single line of custom scraping code.
Two Core Extraction Modes: Distill and Extract
Thunderbit surfaces two complementary extraction primitives designed to cover the full spectrum of developer use cases. Distill converts any URL into clean, LLM-ready Markdown — stripping boilerplate, navigation, scripts, and ads to leave only the meaningful page content. It is the foundation for AI agent context, RAG knowledge bases, content ingestion pipelines, and any workflow where an LLM needs to reason over web content.
Extract returns structured JSON or CSV from a URL using a developer-defined schema. Developers specify what fields they need and Thunderbit's AI identifies and pulls those values from the page — without needing to know the site's HTML structure in advance. Together, Distill and Extract address the two primary data formats developers need: unstructured text for AI reasoning, and structured records for databases, spreadsheets, and enrichment workflows.
Compatible Platforms & Environments
Extending Beyond No-Code: Built for Developer Workflows
Thunderbit's no-code Chrome extension and web app are already used by sales, ecommerce, research, and operations teams to extract tens of millions of pages every month. This developer launch extends the same adaptive extraction engine into programmatic environments — enabling engineers and AI application builders to embed it natively into their systems rather than relying on manual tooling.
The open-source monorepo — housing the CLI, MCP server, and Claude Code plugin — is publicly available on GitHub under the thunderbit-open organization. Free credits are available for new users to explore all three products. The AI-suggested fields feature is particularly notable for use cases where developers don't know a page's structure in advance, letting the model propose what data is available before a schema is committed.
Thunderbit has launched a developer API, MCP server, and CLI, bringing its AI-powered web extraction engine to programmatic and agentic workflows for the first time.
Thunderbit Distill, the new HTML-to-Markdown engine, scored 0.87 ROUGE-L in internal evaluations and works across diverse page types without site-specific configuration.
The MCP server integrates directly with Claude, Claude Code, Cursor, and any MCP-compatible client — exposing seven tools including distill, extract, AI field suggestion, and batch processing up to 100 URLs.
Distill and Extract cover two primary developer needs: clean Markdown for AI agents and RAG pipelines, and structured JSON or CSV for databases and enrichment jobs.
All three products are available immediately, backed by a public open-source monorepo on GitHub. Free credits are available for new users getting started.
For developers building AI applications that need reliable, up-to-date web data, Thunderbit's developer launch addresses a persistent and underserved problem: getting clean, structured information out of the open web at scale without maintaining a fleet of brittle scrapers. By combining semantic AI understanding with MCP compatibility, the platform positions itself as a natural fit for the growing ecosystem of agentic AI tools that need to browse, reason over, and act on live web content.
The full API documentation, CLI install guide, and MCP server configuration instructions are available at thunderbit.com/docs, with free credits available for all new users.