crw
us/crw
Self-hostable web crawler and search API (fastCRW): scraping, crawling, site mapping and SearXNG-backed search.
- Official
- Local integrations
- Cloud integrations
Installation, supported tools and required credentials are maintained by the project.
Project details
- GitHub stars
- 1,171
- Language
- Rust
- License
- AGPL-3.0
- Forks
- 95
- Latest release
- v0.39.0today
- Activity
- updated today
- Platforms listed
- macOS, Windows, Linux
- ai
- ai-agents
- crawler
- data-extraction
- docker
- html-to-markdown
- llm
- markdown
First MCP server?Start with the basics
Web content fetching and processing for AI consumption.
Local web search, fetching, crawling, extraction and research, with multi-engine search, local ML reranking, a SQLite cache and JavaScript rendering.
Real-time web search using the Exa AI Search API.
Search arXiv papers, read original LaTeX sections, get BibTeX from arXiv metadata and set topic watches, with papers stored locally.
Crawl, embed, chunk, search and retrieve information from datasets through Trieve.
Discover, extract and interact with the web via Bright Data, one interface for automated access to the public internet.
Multi-engine web search across Bing, Baidu, DuckDuckGo, Brave, Exa and CSDN.
Web search using DuckDuckGo, with content fetching and parsing.