doc-scraper: Go web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion

by Sriram-PR · MCP Server · ★ 108

Last updated: · Indexed by AgentSkillsHub · Auto-synced every 8h

How to install doc-scraper

go install github.com/Sriram-PR/doc-scraper/v2/cmd/doc-scraper@latest
git clone https://github.com/Sriram-PR/doc-scraper.git

Taken from the project's README on GitHub.

Security audit verdict: SAFE · quality 76/100

None of the 42 rules matched — no credential harvesting, no data exfiltration, no curl-pipe-shell installer. Scanned against 42 published rules built on the SlowMist agent-security taxonomy, refreshed every 8 hours.

The quality score combines nine heuristic checks of the repo and its README: completeness, clarity, specificity, examples, structure, agent readiness, step-by-step content, instruction quality and security notes. It rates how well the project explains itself to an agent, not the code.

data-preparationdocumentationgolang-clillmmcpmcp-serverweb-crawlerweb-scraper

Quick Facts

Stars108
Forks24
LanguageGo
CategoryMCP Server
LicenseApache-2.0
Quality Score76/100
Open Issues65
Last Updated2026-11-07
Created2025-04-27
Platformsbrowser, cli, go, mcp
Est. Tokens~23k

About doc-scraper

LLM Documentation Scraper () A configurable, concurrent, and resumable web crawler written in Go. Specifically designed to scrape technical documentation websites, extract core content, convert it cleanly to Markdown format suitable for ingestion by Large Language Models (LLMs), and save the results locally.

Compatible Skills

These tools work well together with doc-scraper for enhanced workflows:

  • apicat — semantic(0.21)+complementary+same_lang+similar_pop+shared_platform (57%)
  • oxylabs-ai-studio-js — semantic(0.31)+complementary+similar_pop+shared_platform (51%)

doc-scraper alternative? Top 6 similar tools

Looking for a doc-scraper alternative? If you're comparing doc-scraper with other mcp server tools, these 6 projects are the closest alternatives on Agent Skills Hub — ranked by topic overlap, star count, and community traction.

  • web-scout-mcp by pinkpixel-dev · ⭐ 137

    An MCP server providing web search and content extraction capabilities. Integrates DuckDuckGo search functiona

  • mcp-svelte-docs by spences10 · ⭐ 123

    🔍 MCP server that lets you search and access Svelte documentation with built-in caching

  • anansi by mdowis · ⭐ 112

    A self-healing scraper for hostile sites: broken selectors repair themselves, browser rendering kicks in when

  • rhizome by aliengiraffe · ⭐ 66

    Rhizome lets Claude Code, Codex, and OpenCode sessions discover each other, follow selected work, and exchange

  • open-docs by bgauryy · ⭐ 513

    This repository contains documentation created to better understand the open project

  • docfork by docfork · ⭐ 487

    Docfork - Up-to-date docs for AI Coding Agents.

More MCP Server Tools

Explore other popular mcp server tools:

View all MCP Server tools →

Popular Go Agent Tools

How this security grade is produced

Grades come from a rule-based scan built on the SlowMist agent-security taxonomy: 42 published rules, covering credential harvesting, data exfiltration, curl | sh installers and more. It is a first-layer scan, not a manual audit — we say so rather than overstate it.

The scale of the problem is documented independently: Liu et al. (2026), in a study of 31,132 agent skills, report that 26.1% contain security vulnerabilities. Our own full-catalog census is published as a citable open dataset.

Sources & who's responsible:

View on GitHub → Browse MCP Server tools