FastCRW

0 reviews

fastCRW is an open-source web scraping, crawling, search, and data extraction API distributed as a single Rust binary. It exposes a /v1 HTTP API compatible with the Firecrawl request and response format, and a search endpoint compatible with the Tavily response format, allowing existing client SDKs to connect by changing only the base URL. The product provides four primary endpoints. Scrape fetches a single URL and returns the page as structured markdown along with metadata, optional screenshots, raw HTML, and link graphs. Crawl traverses an entire site or a subset of paths, supports sitemap discovery, robots.txt rules, canonical URL deduplication, and streams results as pages complete. Search accepts a natural-language query and returns ranked web results with the full page content already converted to markdown in the same response. Extract retrieves structured fields from one or more URLs using either a JSON schema or a natural-language prompt. Map returns the URL graph of a site without fetching individual pages. The binary is written in Rust and distributed as an 8 MB Docker image. Idle resident memory is approximately 6.6 MB. On a public 1,000-URL benchmark, the median request latency is 710 ms and the average is 833 ms. JavaScript rendering is included for pages that require it, and a static fetch path is used automatically when JavaScript execution is not needed. fastCRW is available in two deployment modes. A managed cloud service provides a free tier of 500 credits per month and paid tiers for higher volumes. The full source code is also published under the AGPL-3.0 license on GitHub and can be self-hosted on any infrastructure with no per-request fees. Both modes share the same binary and feature set. Official client SDKs are available for Python and Node.js, along with a command-line interface. Integrations exist for LangChain, LlamaIndex, CrewAI, n8n, and Dify. Distribution packages are available through Homebrew, apt, and Docker. Typical use cases include retrieval-augmented generation pipelines, autonomous LLM agents, competitive intelligence monitoring, lead enrichment workflows, and dataset construction for model training and fine-tuning.