mcp-server-webcrawl

mcp-server-webcrawl is a MCP server that runs on your computer. It creates a gateway to web crawler archives so that language models (OpenAI, Claude) can filter and analyze or process the data, autonomously or under your direction. Use mcp-server-webcrawl for technical inference, content management, marketing, SEO, and more. The sky is the limit!

The server supports a host of web crawl options including wget and WARC archival format. InterroBot, Katana, and SiteOne crawlers are also supported.

mcp-server-webcrawl is free and open source.

The API is supposed to stay out of your way, and to a large degree it can be navigated autonomously by your MCP client. Sometimes, however, you may need to nudge the LLM to the correct field or search strategy. The following is the current API interface for your reference.

webcrawl_sites

This tool retrieves a list of sites (project websites or crawl directories).

Parameter	Type	Required	Description
ids	array<int>	No	List of project IDs to retrieve. Leave empty for all projects.
fields	array<string>	No	List of additional fields to include beyond defaults (id, url). Empty list means default fields only. Options include created (ISO 8601), modified (ISO 8601), and norobots (str).

webcrawl_search

This tool searches for resources (webpages, images, CSS, etc.) across projects and retrieves specified fields.

Parameter	Type	Required	Description
sites	array<int>	No	Optional list of project IDs to filter search results to specific sites. In most scenarios, you'd filter to only one site.
query	string	No	Fulltext search query string. Leave empty to return all resources when filtering on other fields for better precision. Supports fulltext and boolean operators (AND, OR, NOT), quoted phrases, and suffix wildcards, but not prefix wildcards. See below for complete boolean and field search capabilities.
fields	array<string>	No	List of additional fields to include beyond defaults (modified, created). Empty list means default fields only. The content field can lead to large results and should be used with LIMIT.
sort	string	No	Sort order for results. Prefixed with + for ascending, - for descending. ? is a special option for random sort, useful in statistical sampling. Options include: +id, -id, +url, -url, +status, -status, ?.
limit	integer	No	Maximum number of results to return. Default is 20, max is 100.
offset	integer	No	Number of results to skip for pagination. Default is 0.
extras	array<string>	No	Optional array of extra features to include in results. Options include markdown, snippets, and thumbnails (see extras table).
extrasRegex	array<string>	No	XPath expressions (e.g. //h1, count(//h1)), enabled when used with regex extra.
extrasXpath	array<string>	No	Regular expressions (e.g. https://[^\s]+), enabled when used with xpath extra.

Crawler Features Support

API support, by parameter, across crawler type.

Parameter	wget	WARC	InterroBot	Katana	SiteOne
Sites/ids	✔	✔	✔	✔	✔
Sites/fields	✔	✔	✔	✔	✔
Search/ids	✔	✔	✔	✔	✔
Search/sites	✔	✔	✔	✔	✔
Search/query	①	✔	✔	✔	①
Search/fields	②	✔	✔	✔	②
Search/sort	✔	✔	✔	✔	✔
Search/limit	✔	✔	✔	✔	✔
Search/offset	✔	✔	✔	✔	✔
Search/extras	✔	✔	✔	✔	✔

Crawler Field Support

API support, by field, across crawler type.

Parameter	wget	WARC	InterroBot	Katana	SiteOne
id	✔	✔	✔	✔	✔
url	✔	✔	✔	✔	✔
type	✔	✔	✔	✔	✔
status	③	✔	✔	✔	✔
size	✔	✔	✔	✔	✔
headers		✔	✔	✔
content	✔	✔	✔	✔	✔

①②③ wget (--mirror) does not index HTTP status beyond 200 OK (HTTP errors not saved to disk). wget and SiteOne crawler implementations do not support field searchable HTTP headers. When used in WARC mode (as opposed to simple mirror), wget is capable of collecting HTTP headers and status.

Crawlers have strengths and weaknesses, judge them on how well they fit your needs. Don't worry too much about field support. You probably don't need HTTP headers, except for specialized web-dev. They all support fulltext boolean search across the crawl data.

Boolean Search Syntax

The query engine supports field-specific (field: value) searches and complex boolean expressions. Fulltext is supported as a combination of the url, content, and headers fields.

While the API interface is designed to be consumed by the LLM directly, it can be helpful to familiarize yourself with the search syntax. Searches generated by the LLM are inspectable, but generally collapsed in the UI. If you need to see the query, expand the MCP collapsable.

Query Example	Description
privacy	fulltext single keyword match
"privacy policy"	fulltext matches exact phrase
boundar*	fulltext wildcard matches results starting with boundar (boundary, boundaries)
id: 12345	id field matches a specific resource by ID
url: example.com/somedir	url field matches results with URLs containing example.com/somedir
type: html	type field matches for HTML pages only
status: 200	status field matches specific HTTP status codes (equal to 200)
status: >=400	status field matches specific HTTP status code (greater than or equal to 400)
content: h1	content field matches content (HTTP response body, often, but not always HTML)
headers: text/xml	headers field matches HTTP response headers
privacy AND policy	fulltext matches both
privacy OR policy	fulltext matches either
policy NOT privacy	fulltext matches policies not containing privacy
(login OR signin) AND form	fulltext matches fulltext login or signin with form
type: html AND status: 200	fulltext matches only HTML pages with HTTP success

Field Search Definitions

Field search provides search precision, allowing you to specify which columns of the search index to filter. Rather than searching the entire content, you can restrict your query to specific attributes like URLs, headers, or content body. This approach improves efficiency when looking for specific attributes or patterns within crawl data.

Field	Description
id	database ID
url	resource URL
type	enumerated list of types (see types table)
size	file size in bytes
status	HTTP response codes
headers	HTTP response headers
content	HTTP body—HTML, CSS, JS, and more

Content Types

Crawls contain a multitude of resource types beyond HTML pages. The type: field search allows filtering by broad content type groups, particularly useful when filtering images without complex extension queries. For example, you might search for type: html NOT content: login to find pages without "login," or type: img to analyze image resources. The table below lists all supported content types in the search system.

Type	Description
html	webpages
iframe	iframes
img	web images
audio	web audio files
video	web video files
font	web font files
style	CSS stylesheets
script	JavaScript files
rss	RSS syndication feeds
text	plain text content
pdf	PDF files
doc	MS Word documents
other	uncategorized

Extras

The extras parameter provides additional processing options, transforming result data (markdown, snippets), or connecting the LLM to external data (thumbnails). These options can be combined as needed to achieve the desired result format.

Extra	Description
thumbnails	Generates base64 encoded images to be viewed and analyzed by AI models. Enables image description, content analysis, and visual understanding while keeping token output minimal. Works with images, which can be filtered using type: img in queries. SVG is not supported.
markdown	Provides the HTML content field as concise markdown, reducing token usage and improving readability for LLMs. Works with HTML, which can be filtered using type: html in queries.
regex	Extracts regular expression matches from crawled files such as HTML, CSS, JavaScript, etc. Not as precise a tool as xpath for HTML, but supports any text file as a data source.
snippets	Matches fulltext queries to contextual keyword usage within the content. When used without requesting the content field (or markdown extra), it can provide an efficient means of refining a search without pulling down the complete page contents. Also great for rendering old school hit-highlighted results as a list, like Google search in 1999. Works with HTML, CSS, JS, or any text-based, crawled file.
xpath	Extracts xpath selector data. Supports count(). Use xpath's text() for text only, element selectors for HTML data. Only supported for HTML, other types will be ignored.

Extras provide a means of producing token-efficient HTTP content responses. Markdown produces roughly 1/3 the bytes of the source HTML, snippets are generally 500 or so bytes per result, and XPath can be as specific or broad as you choose. The more focused your requests, the more results you can fit into your LLM session.

mcp-server-webcrawl provides the toolkit necessary to search web crawl data freestyle, figuring it out as you go, reacting to each query. This is what it was designed for.

It is also capable of running routines (as prompts). You can write these yourself, or use the ones provided. These prompts are copy and paste, and used as raw Markdown. They are enabled by the advanced search provided to the LLM; queries and logic can be embedded in a procedural set of instructions, or even an input loop as is the case with Gopher Service.

Prompt	Download	Category	Description
🔍 SEO Audit	auditseo.md	audit	Technical SEO (search engine optimization) analysis. Covers the basics, with options to dive deeper.
🔗 404 Audit	audit404.md	audit	Broken link detection and pattern analysis. Not only finds issues, but suggests fixes.
⚡ Performance Audit	auditperf.md	audit	Website speed and optimization analysis. Real talk.
📁 File Audit	auditfiles.md	audit	File organization and asset analysis. Discover the composition of your website.
🌐 Gopher Interface	gopher.md	interface	An old-fashioned search interface inspired by the Gopher clients of yesteryear.
⚙️ Search Test	testsearch.md	self-test	A battery of tests to check for Boolean logical inconsistencies in the search query parser and subsequent FTS5 conversion.

If you want to shortcut the site selection (one less query), paste the markdown and in the same request, type "run pasted for [site name or URL]." It will figure it out. When pasted without additional context, you should be prompted to select from a list of crawled sites.