MCP-Web-Curl — Web Research & Browser Automation Server
Rayhan Saleh
Google Custom Search API
Google Custom Search API is free with usage limits (e.g., 100 queries per day for free, with additional queries requiring payment). For full details on quotas, pricing, and restrictions, see the official documentation.
Web-curl
Developed by Rayss
🚀 Open Source Project 🛠️ Built with Node.js & TypeScript (Node.js v18+ required)
🎬 Demo Video
If your platform supports it, you can also download and play demo/demo_1.mp4 directly.
📚 Table of Contents
📝 Changelog / Update History
See CHANGELOG.md for a complete history of updates and new features.
📝 Overview
Web-curl is a powerful tool for fetching and extracting text content from web pages and APIs. Use it as a standalone CLI or as an MCP (Model Context Protocol) server. Web-curl leverages Puppeteer for robust web scraping and supports advanced features such as resource blocking, custom headers, authentication, and Google Custom Search.
✨ Features
🚀 Deep Research & Automation (v1.4.2)
Advanced Browser Automation: Full control over Chromium via Puppeteer (click, type, scroll, hover, key presses).
Always-On Session Persistence: Browser profiles are now always persistent. Login sessions, cookies, and cache are automatically saved in a local
user_data/ directory.Token-Efficient Snapshots:
Accessibility Tree: Clean, structured snapshots instead of messy HTML.
HTML Slice Mode: Raw HTML with
startIndex/endIndex for safe chunking when needed.Viewport Filtering: Automatically filters out elements not visible on screen, saving up to 90% of context tokens on long pages.
Chrome DevTools Integration (implemented, but hidden from
list_tools):Network Monitoring (
browser_network_requests)Console Logs (
browser_console_messages)Parallel Search:
multi_search: Run multiple Google searches at once (only exposed search tool).Research Pipeline:
research: Decompose a question into sub-queries, run them in parallel, dedupe, and synthesize a markdown report with numbered citations.agent: Collect structured data across pages using a field schema (CSS selectors and/or JSON-LD paths).Structured Extraction:
extract: CSS-selector fields, tables, meta tags, JSON-LD blocks, Readability main content, and schema-type detection.crawl: BFS/DFS traversal, sitemap parsing (incl. index files), or one-page link map, with include/exclude regex, same-domain restriction, and politeness delay.Intelligent Resource Management:
Idle Auto-Close: Browser automatically shuts down after 15 minutes of inactivity to save RAM/CPU.
Tab Rotation: Automatically replaces the oldest tab when the 10-tab limit is reached.
Media & Documents:
Full-Page Screenshots: Capture high-quality screenshots with a 5-day auto-cleanup lifecycle and custom destination support.
Document Parsing: Extract text from PDF and DOCX files directly from URLs.
Storage & Download Details
🗂️ Error log rotation:
logs/error-log.txt is rotated when it exceeds ~1MB (renamed to error-log.txt.bak) to prevent unbounded growth.🧹 Logs & temp cleanup: old temporary files in the
logs/ directory are cleaned up at startup.🛑 Browser lifecycle: Puppeteer browser instances are closed in finally blocks to avoid Chromium temp file leaks.
🔎 Content extraction:
Returns raw text, HTML, and Readability "main article" when available. Readability attempts to extract the primary content of a webpage, removing headers, footers, sidebars, and other non-essential elements, providing a cleaner, more focused text.
Readability output is subject to
startIndex/maxLength/chunkSize slicing when requested.🚫 Resource blocking:
blockResources is now always forced to false, meaning resources are never blocked for faster page loads.⏱️ Timeout control: navigation and API request timeouts are configurable via tool arguments.
💾 Output: results can be printed to stdout or written to a file via CLI options.
⬇️ Download behavior (
download_file):destinationFolder accepts relative paths (resolved against the project root) or absolute paths.The server creates
destinationFolder if it does not exist.Downloads are streamed using Node streams +
pipeline to minimize memory use and ensure robust writes.Filenames are derived from the URL path (e.g.,
https://.../path/file.jpg -> file.jpg). If no filename is present, the fallback name is downloaded_file.Overwrite semantics: by default the implementation will overwrite an existing file with the same name.
🖥️ Usage modes: CLI and MCP server (stdin/stdout transport).
🌐 REST client:
fetch_api returns JSON/text when appropriate and base64 for binary responses.🔍 Google Custom Search: requires
APIKEY_GOOGLE_SEARCH and CX_GOOGLE_SEARCH.🤖 Smart command:
Auto language detection (franc-min) and optional translation (dynamic
translate import).Query enrichment is heuristic-based; results depend on the detected intent.
🏗️ Architecture
This section outlines the high-level architecture of Web-curl.
Loading
CLI & MCP Server:
src/index.ts Implements both the CLI entry point and the MCP server.Web Scraping: Uses Puppeteer for headless browsing and content extraction.
REST Client:
src/rest-client.ts Provides a flexible HTTP client for API requests.⚙️ MCP Server Configuration Example
To integrate web-curl as an MCP server, add the following configuration to your
mcp_settings.json:🔑 How to Obtain Google API Key and CX
Get a Google API Key:
Go to Google Cloud Console.
Create/select a project, then go to APIs & Services > Credentials.
Click Create Credentials > API key and copy it.
Get a Custom Search Engine (CX) ID:
Go to Google Custom Search Engine.
Create/select a search engine, then copy the Search engine ID (CX).
Enable Custom Search API:
In Google Cloud Console, go to APIs & Services > Library.
Search for Custom Search API and enable it.
Replace
YOUR_GOOGLE_API_KEY and YOUR_CX_ID in the config above.🛠️ Installation
Prerequisites: Ensure you have Node.js (v18+) and Git installed on your system.
Puppeteer installation notes
Windows: Just run
npm install.Linux / Ubuntu Server: You must install extra dependencies for Chromium to handle rendering and screenshots in a headless environment. Run:
For more details, see the Puppeteer troubleshooting guide.
🚀 Usage
CLI Usage
The CLI supports fetching and extracting text content from web pages.
Command Line Options
--timeout <ms>: Set navigation timeout (default: 60000)-o <file>: Output result to specified fileMCP Server Usage
Web-curl can be run as an MCP server for integration with Roo Context or other MCP-compatible environments.
Exposed Tools (v1.4.2)
Only the tools below are exposed via
list_tools to reduce tool-chaining in agent clients.browser_flow: One-call browser workflow (optional navigate → optional actions → return ONE result).
browser_configure: Set proxy/user-agent/viewport (session persistence is always on via
user_data/).browser_close: Close browser and tabs (also auto-closes after 15 minutes of inactivity).
multi_search: Run multiple Google searches in parallel (the only exposed search entrypoint).
fetch_api: REST API request with response truncation (
limit).download_file: Download a file from a URL.
parse_document: Extract text from PDF/DOCX URLs.
research: Research pipeline — question decomposition → parallel sub-search → dedup → synthesized markdown report with numbered citations.
extract: Structured page extraction — CSS selectors, tables, meta tags, JSON-LD, Readability main content, schema-type detection.
crawl: Site traversal — BFS/DFS, sitemap (incl. index files), or one-page map, with include/exclude regex and politeness delay.
agent: Structured data collection across pages — field schema with CSS selectors and/or JSON-LD paths, type coercion, required-field validation.
Running as MCP Server
The server will communicate via stdin/stdout and expose the tools as defined in
src/index.ts.🚦 HTML Slicing Example (Recommended for Large Pages)
Use
browser_flow with result: { type: "snapshot", mode: "html" } when you need raw HTML but want to keep the response small.Client request for first slice:
Response (example):
🧩 Configuration
Session Persistence: Always enabled. Logins and cookies are automatically reused across restarts.
Timeout: Set navigation and API request timeouts.
Environment Variables: Used for Google Search API integration (used by
multi_search).💡 Examples {#examples}
Make a REST API Request
Download File
Note:
destinationFolder can be either a relative path (resolved against the project root) or an absolute path. The server will create the destination folder if it does not exist.Configure Browser
Note: Session persistence is always enabled. Cookies and login sessions are automatically stored in the
user_data/ directory.🛠️ Troubleshooting {#troubleshooting}
Timeout Errors: Increase the
timeout parameter if requests are timing out.Google Search Fails: Ensure
APIKEY_GOOGLE_SEARCH and CX_GOOGLE_SEARCH are set in your environment.Error Logs: Check the
logs/error-log.txt file for detailed error messages.🧠 Tips & Best Practices {#tips--best-practices}
Click for advanced tips
For large pages, use
maxLength and startIndex to fetch content in slices.Always validate your tool arguments to avoid errors.
Secure your API keys and sensitive data using environment variables.
Review the MCP tool schemas in
src/index.ts for all available options.🤝 Contributing & Issues {#contributing--issues}
Contributions are welcome! If you want to contribute, fork this repository and submit a pull request.
If you find any issues or have suggestions, please open an issue on the repository page.
📄 Academic Publication
This project is the subject of a peer-reviewed journal article:
Saleh, R. Z., & Lubis, M. (2026). Design and Implementation of MCP-Web-Curl: A Model Context Protocol Server for Web and API Access in Agentic Coding Assistants. JURNAL TEKNIK INFORMATIKA, 19(1), 122–134.
PDF: Download
License: CC BY-SA 4.0
📄 License & Attribution {#license--attribution}
This project was developed by Rayss.
For questions, improvements, or contributions, please contact the author or open an issue in the repository.
Note: Google Search API is free with usage limits. For details, see: Google Custom Search API Overview
Like this project
Posted Sep 22, 2026
Built an open-source Node.js/TypeScript MCP server for web research, browser automation, structured extraction, crawling, APIs, and reliable downloads.
Likes
0
Views
0
