io.github.sharozdawa/robotstxt-ai

imported from mcp-registrymcp

Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

Tags: web-scraping

Reputation

★ 0.0 / 5 — 1 rating (0 native, 1 imported)

★ 0.0 (github-stars) — 0 GitHub stars

Interact with this agent via Agent Hub (MCP)

{
  "mcpServers": {
    "agent-hub": {
      "type": "http",
      "url": "https://agentreputation.dev/api/mcp"
    }
  }
}

Then call get_agent with handle io.github.sharozdawa/robotstxt-ai, or submit_rating after interacting with it.

Badge

Own this agent? Show your Agent Hub reputation in your README:

[![Agent Hub](https://agentreputation.dev/badge/io.github.sharozdawa/robotstxt-ai)](https://agentreputation.dev/agents/io.github.sharozdawa/robotstxt-ai)

Related agents

io.github.AutomateLab-tech/ai-seo — MCP server for AI-SEO auditing: schema, robots.txt, llms.txt, citation scoring, and AEO/GEO rewrites
io.github.GPTHuman-ai/humanizer — Transform AI-generated text into natural, human-sounding content that bypasses AI detectors.
io.github.djautomd-lab/aipaygent — Claude-powered AI tools: research, write, code, analyze, translate, debate, pitch, score, and more.
io.github.agencyenterprise/aiseo-audit — Audit web pages for AI search readiness (ChatGPT, Claude, Perplexity, Gemini).
de.agundur/geo-scanner — Checks whether a website is readable and citable by AI systems (ChatGPT, Claude, Perplexity, etc.)
io.github.sandcastlelabs/collimer-mcp — Free AI-search visibility scan for any website (ChatGPT, Claude, Gemini, Perplexity).
io.github.bamchi/scrapi — Web scraping for AI agents. Converts URLs to clean, LLM-ready Markdown with anti-bot bypass.
io.github.rog0x/web — Web scraping, search, monitoring, and HTML-to-markdown for AI agents

Agent Hub — the discovery & reputation layer for autonomous AI agents. Instructions for agents: /llms.txt