doc-scraper
Sriram-PR/doc-scraper
Crawl documentation sites into local Markdown or JSONL corpora and serve them with offline BM25 search, page reads, freshness checks and crawl diffs.
- Local integrations
Installation, supported tools and required credentials are maintained by the project.
Project details
- GitHub stars
- 107
- Language
- Go
- License
- Apache-2.0
- Forks
- 17
- Latest release
- v2.21.02d ago
- Activity
- updated 2d ago
- data-preparation
- llm
- web-scraper
- documentation
- golang-cli
- mcp
- mcp-server
- web-crawler
First MCP server?Start with the basics
Web content fetching and processing for AI consumption.
Local web search, fetching, crawling, extraction and research, with multi-engine search, local ML reranking, a SQLite cache and JavaScript rendering.
Real-time web search using the Exa AI Search API.
Search arXiv papers, read original LaTeX sections, get BibTeX from arXiv metadata and set topic watches, with papers stored locally.
Crawl, embed, chunk, search and retrieve information from datasets through Trieve.
Discover, extract and interact with the web via Bright Data, one interface for automated access to the public internet.
Multi-engine web search across Bing, Baidu, DuckDuckGo, Brave, Exa and CSDN.
Web search using DuckDuckGo, with content fetching and parsing.