agentic-web-search: Self-Hosted Web Search for Agents (Rust) by Muhammad Husnain Aliagentic-web-search: Self-Hosted Web Search for Agents (Rust) by Muhammad Husnain Ali

agentic-web-search: Self-Hosted Web Search for Agents (Rust)

Portfolio project
Muhammad Husnain Ali
Single Rust microservice that searches the web for AI agents with no third-party search or LLM APIs. Queries 6 engines (Bing, Brave, DuckDuckGo, Yahoo, Mojeek, Google) over HTTP with a pooled headless Chromium fallback for captchas and soft blocks. Concurrent pagination, de-duplication and ranking, per-key GCRA rate limiting, request-coalescing cache, SSRF-guarded content fetch, Prometheus metrics. Lite Docker image ~53 MB, ~22 MB RAM idle.

The problem

Search APIs for agents are expensive and add a third-party dependency.

What I built

Query 6 engines directly, with a pooled headless Chromium fallback for captchas.
Paginate concurrently, de-duplicate and rank.
Harden it: GCRA rate limits, request-coalescing cache, SSRF guard, metrics.

How it works

Query: Agent request, Per-key rate limit
Search: 6 engines over HTTP, Chromium fallback
Merge: Pagination, De-dup + ranking
Return: Results + content, Coalescing cache
Built-in controls: SSRF-guarded fetch, GCRA rate limiting, Prometheus metrics.

Result

One Rust service: ~53 MB image, ~22 MB RAM idle.
My role: Creator

0 likes · 1 view · Nov 13, 2026

Search APIs for agents are expensive and add a third-party dependency. Result: One Rust service: ~53 MB image, ~22 MB RAM idle.

Tags