Back to Browse

Ai Bot Access Checker MCP Server

Developer ToolsLow Risk8.2MCP RegistryRemote
Free

Server data from the Official MCP Registry

Checks if GPTBot, ClaudeBot, PerplexityBot, Googlebot and 9 more AI crawlers can reach a page.

About

Checks if GPTBot, ClaudeBot, PerplexityBot, Googlebot and 9 more AI crawlers can reach a page.

Remote endpoints: streamable-http: https://tools.seopaslaugos.io/mcp

Security Report

8.2
Low Risk8.2Low Risk

This is a read-only AI crawler access checker deployed as a remote Streamable HTTP MCP server with no credentials required, which is appropriate for a public, free diagnostic tool. The code shows strong SSRF protections (public-domain-only validation, manual redirect re-validation, private/reserved host blocking), response size and timeout caps, rate limiting, and control-character sanitization of third-party text. Remaining concerns are minor: the server is intentionally unauthenticated, outbound HTTP to arbitrary user-supplied public URLs is inherent to its purpose, and the DNS-rebinding protection relies on hostname checks rather than resolved IP checks. Supply chain analysis found 1 known vulnerability in dependencies (0 critical, 1 high severity).

7 files analyzed · 5 issues found

Security scores are indicators to help you make informed decisions, not guarantees. Always review permissions before connecting any MCP server.

Permissions Found in Source Code

Found by scanning the linked source code. This listing connects to a hosted endpoint, so none of this runs on your machine: it describes what the server software does where it is hosted.

HTTP Network Access

Connects to external APIs or services over the internet.

system_info

Applies to the server that hosts this plugin, not to your machine.

How to Connect

Remote Plugin

No local installation needed. Your AI client connects to the remote endpoint directly.

Add this to your MCP configuration to connect:

{
  "mcpServers": {
    "io-seopaslaugos-ai-bot-access-checker": {
      "url": "https://tools.seopaslaugos.io/mcp"
    }
  }
}

Documentation

View on GitHub

From the project's GitHub README.

AI Bot Access Checker

Can ChatGPT, Claude, Perplexity and Google actually reach your page? This tool checks 13 AI and search crawlers against one URL and tells you, for each, whether robots.txt blocks it or whether your server or firewall refuses it while a normal browser gets the page.

Free, no key, MIT licence.

Why

robots.txt is only half of it. Shared-hosting security rules (ModSecurity), rate limiters that answer 429 to bots on uncached pages, and CDN bot settings (Cloudflare's AI crawler blocking) refuse AI crawlers before robots.txt is ever read, often by default. Site owners then wonder why ChatGPT says it "cannot access" their site. Most checkers only read robots.txt; this one also requests the page as each bot and compares the answer with a browser's.

Bots checked

BotVendorUsed for
GPTBotOpenAITraining
OAI-SearchBotOpenAIChatGPT search index
ChatGPT-UserOpenAILive visits when a user asks (OpenAI: robots.txt may not apply)
ClaudeBotAnthropicTraining
Claude-SearchBotAnthropicClaude search index
Claude-UserAnthropicLive visits when a user asks
PerplexityBotPerplexityPerplexity search index
Perplexity-UserPerplexityLive visits when a user asks (Perplexity: generally ignores robots.txt)
GooglebotGoogleGoogle Search, AI Overviews, AI Mode
Google-ExtendedGoogleGemini training opt-out (robots.txt token only; no effect on Search)
bingbotMicrosoftBing search index
Applebot-ExtendedAppleApple model training opt-out (robots.txt token only; no effect on Apple search)
CCBotCommon CrawlOpen training corpus

Use it

REST

curl "https://tools.seopaslaugos.io/v1/check?url=example.com&format=text"
curl "https://tools.seopaslaugos.io/v1/check?url=https://example.com/pricing&bots=GPTBot,ClaudeBot"
AI bot access for https://www.example.lt/
Browser baseline: HTTP 200
robots.txt: found (HTTP 200)

GPTBot             BLOCKED (server/firewall) - Gets a Cloudflare block page (HTTP 403) instead of the page.
OAI-SearchBot      BLOCKED (server/firewall) - Gets a Cloudflare block page (HTTP 403) instead of the page.
Claude-User        ALLOWED
Googlebot          ALLOWED
...

MCP (Claude Code, Cursor, ChatGPT, any MCP client)

{
  "mcpServers": {
    "ai-bot-access-checker": { "type": "http", "url": "https://tools.seopaslaugos.io/mcp" }
  }
}

Hermes Agent

hermes plugins install ai-bot-access-checker

The package is an Agent Plugins v1 bundle: plugin.json, mcp.json (remote Streamable HTTP server, no keys, no local process) and the skill in skills/. The one tool is read-only; it sends HTTP requests to the URL you ask it to check, from tools.seopaslaugos.io.

Agent skill

Copy skills/ai-bot-access-check into your agent's skills folder, or install it from ClawHub. The skill calls the REST API with curl, so it needs no MCP support.

Results

accessMeaning
allowedrobots.txt allows it and the server answers it like a browser
blocked_by_robotsA Disallow rule applies (rule and user-agent group included). For ChatGPT-User and Perplexity-User the vendors say robots.txt may be ignored; robots.vendorNote says so
blocked_by_serverrobots.txt allows it, the server refuses the User-Agent: 403, 429, dropped connection, challenge or captcha page
inconclusiveThe firewall refuses a fake Googlebot too, so it checks crawler IPs; the real crawler may get in
suspiciousHTTP 200 but under a fifth of the browser's response size
page_errorA normal browser fails as well

How it works and its limits

  • robots.txt is parsed per RFC 9309: most specific user-agent group, merged duplicate groups, longest match wins, Allow wins ties, * and $ wildcards; a 4xx robots.txt means no restrictions, a 5xx means full disallow.
  • The page is requested once with a desktop Chrome User-Agent and once per bot with the bot's published User-Agent, from Cloudflare's network.
  • Requests do not come from the vendors' IP ranges. Blocks by User-Agent (most hosting firewalls) are caught; firewalls that verify crawler IPs are reported as inconclusive instead of guessed.
  • Rate limits: 20 checks a minute per caller IP, and 6 fresh checks a minute per checked website from all callers together, so the checker cannot be used to flood a site. Results are cached 10 minutes per URL.

Security

  • Only public http/https URLs on ports 80/443 with a domain name are checked. IP literals (dotted, decimal, octal, hex, IPv6), localhost and reserved TLDs, wildcard-DNS hosts that point at private addresses (nip.io, sslip.io, ...), URLs with credentials and the checker's own hosts are refused.
  • Redirects are followed by hand (max 5) and every hop is validated the same way, so a public page cannot bounce the checker to an internal address, another port or its own API.
  • At most 64 KiB of each response is read; each request times out after 10 s.
  • robots.txt matching uses a linear wildcard matcher with a work budget and caps on lines, rules and pattern length, so a hostile robots.txt cannot stall the Worker; if the budget runs out the result says inconclusive.
  • Text taken from the checked site (robots.txt rules, URLs, headers) is stripped of control and bidi characters and length-capped before it is returned to people or AI agents.
  • MCP batches are capped at 10 messages and one tools/call; request bodies are capped at 16 KiB.
  • No keys, cookies or user data are stored. Results are cached by URL only.

Found a problem? Open an issue or email seo@seopaslaugos.io.

Run your own

npm install
npm test          # robots.txt parser tests
npx wrangler dev  # local Worker on :8787

Deploy with npx wrangler deploy after changing the route in wrangler.toml.

Author

Built by Mantas Tamošaitis, SEO specialist in Lithuania, who works on Google and AI search visibility for service businesses and e-shops at seopaslaugos.io. Issues and pull requests welcome.

Reviews

No reviews yet

Be the first to review this server!