fetchkit

AI-friendly web content fetching tool designed for LLM consumption. Rust library with CLI, MCP server, and Python bindings.

Features

• HTTP fetching - GET and HEAD methods with streaming support
• Pluggable fetchers - URL-aware dispatch to specialized handlers for repos, docs, feeds, videos, papers, and more
• HTML-to-Markdown - Built-in conversion optimized for LLMs, with fetched relative links/images resolved to absolute URLs
• Agent content focus - Optional low-noise extraction mode for AI agents
• Crawl discovery - Optional bounded same-origin page discovery for AI agents
• HTML-to-Text - Plain text extraction with clean formatting
• Content processors - Post-download extraction for text PDFs and Office documents, with an extensible registry
• Binary detection - Returns metadata only for unsupported binary formats
• Timeout handling - 1s first-byte, 30s body with partial content on timeout
• Safety limits - 10 MB default decompressed body cap with truncation
• URL filtering - URL-aware allow/block lists for controlled access
• SSRF protection - Resolve-then-check blocks private IPs by default
• MCP server - Model Context Protocol support for AI tool integration

Built-in Fetchers

Fetchkit routes each request through an ordered fetcher registry. Specialized fetchers match first; the default fetcher handles everything else.

• GitHubCodeFetcher - GitHub source file URLs (/blob/...)
• GitHubIssueFetcher - GitHub issue and pull request URLs
• GitHubRepoFetcher - GitHub repository home pages
• TwitterFetcher - X/Twitter status URLs
• StackOverflowFetcher - Stack Overflow and Stack Exchange question URLs
• PackageRegistryFetcher - PyPI, crates.io, and npm package pages
• WikipediaFetcher - Wikipedia article URLs
• YouTubeFetcher - YouTube watch and youtu.be URLs
• ArXivFetcher - arXiv abstract and PDF URLs
• HackerNewsFetcher - Hacker News item threads
• RSSFeedFetcher - RSS and Atom feed URLs
• DocsSiteFetcher - docs sites with llms.txt/llms-full.txt support
• DefaultFetcher - all remaining HTTP/HTTPS URLs with HTML conversion, streaming, timeout handling, and binary detection

Built-in Content Processors

Content processors run after a fetcher retrieves a bounded response body. They select by final URL, response media type, and requested output, then turn documents into LLM-friendly content without performing their own network requests.

• HtmlProcessor - metadata and focused-content extraction followed by native Markdown or text conversion; accepts a custom HtmlToMarkdownConverter
• DocumentProcessor - PDF and Office documents to Markdown through a configurable chain of document converters (see Document conversion)

Custom processors implement ContentProcessor and are added with ToolBuilder::content_processor (checked before the built-ins) or registered in a ContentProcessorRegistry passed to FetcherRegistry::with_content_processors.

Document Conversion

Documents are always downloaded by fetchkit itself, under the same URL policy, DNS pinning, transport and timeouts as any other fetch. DocumentProcessor then detects the format (Content-Type, then file extension for generic binary types, confirmed by magic bytes) and runs an ordered chain of DocumentConverter backends. The next converter runs when one fails, does not support the input, or returns nothing usable (for example a scanned PDF that needs OCR).

Converter	Cargo feature	Default	Where it runs	Formats
PdfInspectorConverter (pdf-inspector)	docs-pdf	on	local	PDF (text layer; reports OCR-only pages)
MarkkitConverter (markkit)	docs-markkit	on	local	DOCX, XLSX, XLSM, XLS, ODS (via markkit)
OfficeConverter (office), deprecated	docs-office	off	local	DOCX, PPTX, XLSX, XLSM, XLSB, XLS, ODS
CloudflareConverter (cloudflare)	docs-cloudflare	off	remote	PDF, DOCX, XLSX, XLSB, XLS, ODT, ODS
HttpConverter (any name)	docs-http	off	remote (configurable)	any format your service accepts

markkit is the default Office backend. The in-crate office converter is deprecated and kept for one release behind the non-default docs-office feature; it is still the only built-in PPTX and XLSB converter. With both features enabled, markkit runs first and office is the fallback. docs-all enables every backend except the deprecated docs-office. Tune markkit limits (rows and columns per sheet, hidden sheets, archive budgets) by registering MarkkitConverter::with_markkit(Markkit::builder()...build()), which replaces the default markkit converter in place. Remote converters upload document bytes to a third party, so they only run when remote conversion is explicitly allowed, and never under ToolBuilder::hardened().

use fetchkit::documents::CloudflareConverter;
use fetchkit::{DocumentConversion, DocumentFormat, ToolBuilder};

let tool = ToolBuilder::new()
    .document_conversion(
        DocumentConversion::default()                 // pdf-inspector + markkit
            .converter(CloudflareConverter::new(account_id, api_token))
            .allow_remote(true)                       // opt in to uploads
            .chain(DocumentFormat::Pdf, ["pdf-inspector", "cloudflare"])
            .formats([DocumentFormat::Pdf, DocumentFormat::Docx, DocumentFormat::Xlsx])
            .max_document_bytes(25 * 1024 * 1024),    // separate from max_body_size
    )
    .build();

Extension points: implement DocumentConverter for a new backend, FormatDetector to change format detection, ConverterClient to route remote uploads through your own egress, or ContentProcessor to bypass the chain.

CLI and MCP server flags: --convert pdf,docx, --converter markkit,cloudflare, --allow-remote-conversion, --max-document-bytes. The Cloudflare converter reads CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN (build with --features docs-cloudflare). Python: FetchkitTool(document_formats=..., converters=..., allow_remote_conversion=..., max_document_bytes=..., cloudflare_account_id=..., cloudflare_api_token=...).

Installation

From crates.io (recommended)

cargo install fetchkit-cli

From Git

cargo install --git https://github.com/everruns/fetchkit fetchkit-cli

From Source

git clone https://github.com/everruns/fetchkit
cd fetchkit
cargo install --path crates/fetchkit-cli

CLI Usage

# Fetch URL (outputs markdown with frontmatter)
fetchkit fetch https://example.com

# Output as JSON instead
fetchkit fetch https://example.com -o json

# Custom user agent
fetchkit fetch https://example.com --user-agent "MyBot/1.0"

# Hardened outbound policy for cluster/data-plane use
fetchkit fetch https://example.com --hardened

# Discover a small same-origin page map for an agent
fetchkit fetch https://example.com --content-focus agent --crawl --max-pages 5

# Optional JS/DOM rendering for simple SPAs/docs (requires render-rakers feature)
fetchkit fetch https://example.com/app --render-rakers

# Show full documentation
fetchkit --llmtxt

Default output is markdown with YAML frontmatter:

---
url: https://example.com
status_code: 200
source_content_type: text/html; charset=UTF-8
source_size: 1256
quality_score: 1.00
extraction_method: "full"
---
# Example Domain

This domain is for use in illustrative examples in documents...

JSON output (-o json):

{
  "url": "https://example.com",
  "status_code": 200,
  "content_type": "text/html",
  "size": 1256,
  "format": "markdown",
  "content": "# Example Domain\n\nThis domain is for use in illustrative examples..."
}

MCP Server

Run as a Model Context Protocol server:

fetchkit mcp

# Hardened profile for cluster/data-plane use
fetchkit mcp --hardened

Exposes fetchkit tool over JSON-RPC 2.0 stdio transport. Returns markdown with frontmatter (same format as CLI). Compatible with Claude Desktop and other MCP clients.

Library Usage

Add to Cargo.toml:

[dependencies]
fetchkit = "0.2"

Optional rendered fetching:

[dependencies]
fetchkit = { version = "0.2", features = ["render-rakers"] }

render-rakers is not enabled by default. It is lightweight partial rendering: inline JavaScript can update the DOM before markdown/text conversion, but it is not a full browser engine. FetchKit blocks rakers-initiated subresource network access in this mode; the initial page still uses FetchKit's normal URL, DNS, proxy, timeout, and size policies.

Basic Fetch

use fetchkit::{fetch, FetchRequest};

#[tokio::main]
async fn main() {
    let request = FetchRequest::new("https://example.com").as_markdown();

    let response = fetch(request).await.unwrap();
    println!("{}", response.content.unwrap_or_default());
}

With Tool Builder

use fetchkit::{FetchRequest, ToolBuilder};

let tool = ToolBuilder::new()
    .enable_markdown(true)
    .enable_text(false)
    .user_agent("MyBot/1.0")
    .allow_prefix("https://docs.example.com")
    .block_prefix("https://internal.example.com")
    .build();

let request = FetchRequest::new("https://example.com");
let response = tool.execute(request).await.unwrap();

Toolkit Contract Surface

use fetchkit::ToolBuilder;

let builder = ToolBuilder::new().enable_save_to_file(true);
let tool = builder.build();

assert_eq!(tool.name(), "web_fetch");
assert_eq!(tool.display_name(), "Web Fetch");

let definition = builder.build_tool_definition();
let mut service = builder.build_service();

Hardened Tool Profile

use fetchkit::Tool;

let tool = Tool::builder()
    .hardened()
    .allow_prefix("https://docs.example.com")
    .build();

Python Bindings

pip install fetchkit

from fetchkit_py import fetch, FetchRequest, FetchkitTool

# Simple fetch
response = fetch("https://example.com", as_markdown=True)
print(response.content)

# With configuration
tool = FetchkitTool(
    enable_markdown=True,
    user_agent="MyBot/1.0",
    allow_prefixes=["https://docs.example.com"]
)
response = tool.fetch("https://example.com")

Request Fields

Field	Type	Description
url	string	URL to fetch (required, http:// or https://)
method	enum?	GET (default) or HEAD
as_markdown	bool?	Convert HTML to markdown
as_text	bool?	Convert HTML to plain text
save_to_file	string?	Non-blank destination path; validated by FileSaver before fetching
content_focus	string?	"full"/unset returns everything; "main" strips semantic boilerplate; "readable" selects article-like content; "agent" selects the best low-noise strategy for AI agents
crawl	bool?	Fetch the seed URL, then discover and fetch bounded same-origin pages
max_pages	int?	Maximum crawl pages, including the seed; default 5, max 20
if_none_match	string?	ETag for conditional If-None-Match
if_modified_since	string?	Timestamp for conditional If-Modified-Since
render	string?	"rakers" to opt into rendered fetch when enabled

Response Fields

Field	Type	Description
url	string	Fetched URL
status_code	int	HTTP status code
content_type	string?	Content-Type header
size	int?	Content size in bytes
last_modified	string?	Last-Modified header
etag	string?	ETag header (use for next conditional request)
filename	string?	From Content-Disposition
format	string?	"markdown", "text", "raw", or a fetcher-specific format
content	string?	Page content
truncated	bool?	True if content was cut off
method	string?	"HEAD" for HEAD requests
error	string?	Error message if failed
saved_path	string?	Filesystem path when save_to_file succeeded
bytes_written	int?	Bytes saved to file
metadata	object?	Structured PageMetadata (title, description, links, headings, extraction method, …)
quality	object?	Agent-facing PageQuality (score, warnings, link density, suggested next action)
crawl	object?	Bounded crawl discovery result with visited page summaries
word_count	int?	Word count of returned content
redirect_chain	string[]	URLs visited during redirects (empty if none)
is_paywall	bool?	Heuristic paywall signal (soft, not guaranteed)
rendered_by	string?	Rendering backend used before conversion, e.g. "rakers"

Error Handling

Errors are returned in the error field:

• InvalidUrl - Malformed URL
• UrlBlocked - URL blocked by filter
• NetworkError - Connection failed
• Timeout - Request timed out
• HttpError - 4xx/5xx response
• ContentError - Failed to read body
• BinaryContent - Binary content has no registered processor

Security

Fetchkit blocks connections to private/reserved IP ranges by default, preventing SSRF attacks when used in server-side or AI agent contexts.

Blocked by default: loopback, private networks (10.x, 172.16-31.x, 192.168.x), link-local (169.254.x including cloud metadata), IPv6 equivalents, multicast, and other reserved ranges.

// Default: private IPs blocked (safe for production)
let tool = Tool::default();

// Explicit opt-out for local development only
let tool = Tool::builder()
    .block_private_ips(false)
    .build();

DNS pinning prevents DNS rebinding attacks. IPv6-mapped IPv4 addresses are canonicalized before validation. Redirects are followed manually in the default fetcher so each hop is revalidated against scheme and DNS policy. Allow/block prefixes are matched against parsed URLs rather than raw strings, which prevents lookalike host overmatches such as allowed.example.com.evil.test. Proxy environment variables are ignored by default. Use the hardened profile for cluster-facing deployments and opt in with ToolBuilder::respect_proxy_env(true) only when it is part of an intentional egress design.

See the knowledge/security/threat-model.md concept for the full threat model. See docs/hardening.md for deployment guidance.

Configuration

Timeouts And Limits

• First-byte: 1 second (connect + initial response)
• Body: 30 seconds total
• Body size: 10 MB decompressed content by default

Partial content is returned on body timeout or body-size limit with truncated: true.

PDF And Binary Content

When Markdown is requested, text-based PDFs are downloaded within the document size limit (25 MB default) and converted to Markdown. PDF parsing runs locally; no OCR service or additional network request is used. Scanned/image-only PDFs report use_ocr as the suggested next action.

Office documents are converted the same way, locally: Word headings, lists and tables, one section per PowerPoint slide, and one Markdown table per spreadsheet sheet (DOCX, PPTX, XLSX, XLSM, XLSB, XLS, ODS). Legacy DOC and PPT files are not converted by the built-in local converters; route them to a remote or custom converter if needed.

Other binary content returns metadata only:

• Images, audio, video, fonts
• Archives (zip, tar, rar, 7z)
• Legacy Word and PowerPoint files (DOC, PPT)

HTML Conversion

HTML is automatically converted to markdown:

• Headers: h1-h6 → # to ######
• Lists: Proper nesting with 2-space indent
• Code: Language-aware, collision-safe fences and inline backticks
• Links/images: Titles, relative-URL resolution, and highest-resolution srcset
• Tables: Valid Markdown with formatted cells and escaped pipes
• Rich content: Footnotes, LaTeX math, callouts, figures, highlights, strikethrough, details, and task checkboxes
• Strips: document head, scripts, styles, templates, iframes, SVGs
• Adds a bounded Agent resources navigation appendix when discoverable resources are available

License

MIT. See Third-Party Notices for adapted components.