Robots.txt & AI Crawler Checker

Name: Robots.txt & AI Crawler Checker
Author: workers

workers

Get started Website & docs

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

Setup

Choose your client, copy the config, then fill in any required environment variables.

Add to Cursor Add to VS Code

claude mcp add --transport http robots-txt-ai-crawler-checker https://robots-check.quietworklabs.workers.dev/mcp

~/.codex/config.toml

[mcp_servers.robots-txt-ai-crawler-checker]
url = "https://robots-check.quietworklabs.workers.dev/mcp"

# codex mcp add robots-txt-ai-crawler-checker --url https://robots-check.quietworklabs.workers.dev/mcp

~/.cursor/mcp.json

{
  "mcpServers": {
    "robots-txt-ai-crawler-checker": {
      "url": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}

.vscode/mcp.json

{
  "servers": {
    "robots-txt-ai-crawler-checker": {
      "type": "http",
      "url": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}

~/.gemini/settings.json

{
  "mcpServers": {
    "robots-txt-ai-crawler-checker": {
      "httpUrl": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}

claude_desktop_config.json · Customize › Connectors › Add custom connector

{
  "mcpServers": {
    "robots-txt-ai-crawler-checker": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://robots-check.quietworklabs.workers.dev/mcp"
      ]
    }
  }
}

.trae/mcp.json

{
  "mcpServers": {
    "robots-txt-ai-crawler-checker": {
      "url": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}

Setup

Related