Server data from the Official MCP Registry
A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.
About
A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.
Security Report
Valid MCP server (2 strong, 3 medium validity signals). No known CVEs in dependencies. ⚠️ Package registry links to a different repository than scanned source. Imported from the Official MCP Registry. Trust signals: 4 highly-trusted packages. 1 finding(s) downgraded by scanner intelligence.
8 files analyzed · 1 issue found
Security scores are indicators to help you make informed decisions, not guarantees. Always review permissions before connecting any MCP server.
Permissions Required
This plugin requests these system permissions. Most are normal for its category.
What You'll Need
Set these up before or after installing:
Environment variable: POLITE_CRAWLER_DIR
Environment variable: POLITE_CRAWLER_CONTACT
Environment variable: POLITE_CRAWLER_ALLOW_PRIVATE
Environment variable: POLITE_CRAWLER_ALLOW_USER_PROVIDED
How to Install
Add this to your MCP configuration file:
{
"mcpServers": {
"io-github-mark1russell7-polite-crawler": {
"env": {
"POLITE_CRAWLER_DIR": "your-polite-crawler-dir-here",
"POLITE_CRAWLER_CONTACT": "your-polite-crawler-contact-here",
"POLITE_CRAWLER_ALLOW_PRIVATE": "your-polite-crawler-allow-private-here",
"POLITE_CRAWLER_ALLOW_USER_PROVIDED": "your-polite-crawler-allow-user-provided-here"
},
"args": [
"-y",
"polite-crawler"
],
"command": "npx"
}
}
}Documentation
View on GitHubFrom the project's GitHub README.
crawler
This repository holds polite-crawler, a polite web crawler for research and for AI agents, and its site.
polite-crawler fetches a source one time, keeps it, and reads it many times. It obeys robots.txt, sends one request at a time to each site, and answers each later question from a local full-text index.
- The package:
packages/polite-crawleris on npm aspolite-crawler. - The site:
packages/siteis the documentation, with interactive demos that use the real code: https://mark1russell7.github.io/crawler/. - The study: Who lets AI agents in? The crawler read the robots.txt files of the top 1,000 domains. The page shows the verdict for 15 agents, with the rule that decided it.
npx polite-crawler fetch https://en.wikipedia.org/wiki/Web_crawler
npx polite-crawler find "robots exclusion"
claude mcp add polite-crawler -- npx -y polite-crawler mcp
What the crawler does for you and for the sites:
- It stops a host after a challenge, a 401, a 402 or a 403, for 1 to 24 hours. It does not try a different identity.
- By default, it fetches only public addresses.
--allow-privatepermits loopback and private addresses. - The MCP server gives page text in an
<untrusted-web-content>envelope, so an assistant can tell data from instructions. A page that refuses AI input stays held back until a person confirms. - The library is
polite-crawler, and the MCP server ispolite-crawler/mcp.
The package README tells the details.
Development
The repository is a pnpm workspace. It needs Node.js 22.19 or later and pnpm 10 or later.
pnpm install
pnpm browser # the headless shell for the renderer test
pnpm check # typecheck, lint, test, lint:ste, build and the smoke test of the package
The prose of the repository (the README files, the doc comments and the site) follows ASD-STE100 Simplified Technical English. The linter ste-lint examines it in CI.
CONTRIBUTING.mdtells how to set up, test and write a change.SECURITY.mdtells how to report a vulnerability privately.CHANGELOG.mdrecords the changes in each version.
License
MIT
Reviews
No reviews yet
Be the first to review this server!
More Developer Tools MCP Servers
Fetch
Freeby Modelcontextprotocol · Developer Tools
Web content fetching and conversion for efficient LLM usage
Git
Freeby Modelcontextprotocol · Developer Tools
Read, search, and manipulate Git repositories programmatically
Worldmonitor
Freeby Koala73 · Developer Tools
Live markets, conflicts, country risk, chokepoints, energy, and China decision signals. 93 tools.
Toleno
Freeby Toleno · Developer Tools
Toleno Network MCP Server — Manage your Toleno mining account with Claude AI using natural language.
MCP Marketplace
Freeby mcp-marketplace · Developer Tools
Search and install MCP servers from inside your AI client.
mcp-creator-python
Freeby mcp-marketplace · Developer Tools
Create, build, and publish Python MCP servers to PyPI — conversationally.