robots-txt

$npx mdskill add kostja94/marketing-skills/robots-txt

Configure, audit, and optimize robots.txt for crawler control.

  • User wants to block or allow specific crawlers on certain paths.
  • Depends on project context files for site URL and indexing goals.
  • Analyzes user request and project context to determine directives.
  • Outputs robots.txt configuration or audit findings as JSON.

SKILL.md

.github/skills/robots-txtView on GitHub ↗
---
name: robots-txt
description: When the user wants to configure, audit, or optimize robots.txt. Also use when the user mentions "robots.txt," "crawler rules," "block crawlers," "AI crawlers," "GPTBot," "allow/disallow," "disallow path," "crawl directives," "user-agent," "block Googlebot," "fix robots.txt," "robots.txt blocking," or "search engine crawling." For indexing, use indexing.
metadata:
  version: 1.2.0
---

# SEO Technical: robots.txt

Guides configuration and auditing of robots.txt for search engine and AI crawler control.

**When invoking**: On **first use**, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On **subsequent use** or when the user asks to skip, go directly to the main output.

## Scope (Technical SEO)

- **Robots.txt**: Configure Disallow/Allow, Sitemap, Clean-param; audit for accidental blocks
- **Crawler access**: Path-level crawl control; AI crawler allow/block strategy
- **Differentiation**: robots.txt = crawl control (who accesses what paths); noindex = index control (what gets indexed). See **indexing** for page-level exclusions.

## Initial Assessment

**Check for project context first:** If `.claude/project-context.md` or `.cursor/project-context.md` exists, read it for site URL and indexing goals.

Identify:
1. **Site URL**: Base domain (e.g., `https://example.com`)
2. **Indexing scope**: Full site, partial, or specific paths to exclude
3. **AI crawler strategy**: Allow search/indexing vs. block training data crawlers

## Best Practices

### Purpose and Limitations

| Point | Note |
|-------|------|
| **Purpose** | Controls crawler access; does NOT prevent indexing (disallowed URLs may still appear in search without snippet) |
| **Advisory** | Rules are advisory; malicious crawlers may ignore |
| **Public** | robots.txt is publicly readable; use noindex or auth for sensitive content. See **indexing** |

### Crawl vs Index vs Link Equity (Quick Reference)

| Tool | Controls | Prevents indexing? |
|------|----------|-------------------|
| **robots.txt** | Crawl (path-level) | No—blocked URLs may still appear in SERP |
| **noindex** (meta / X-Robots-Tag) | Index (page-level) | Yes. See **indexing** |
| **nofollow** | Link equity only | No—does not control indexing |

### When to Use robots.txt vs noindex

| Use | Tool | Example |
|-----|------|---------|
| **Path-level** (whole directory) | robots.txt | `Disallow: /admin/`, `Disallow: /api/`, `Disallow: /staging/` |
| **Page-level** (specific pages) | noindex meta / X-Robots-Tag | Login, signup, thank-you, 404, legal. See **indexing** for full list |
| **Critical** | Do NOT block in robots.txt | Pages that use noindex—crawlers must access the page to read the directive |

**Paths to block in robots.txt**: /admin/, /api/, /staging/, temp files. **Paths to use noindex** (allow crawl): /login/, /signup/, /thank-you/, etc.—see **indexing**.

### Location and Format

| Item | Requirement |
|------|-------------|
| **Path** | Site root: `https://example.com/robots.txt` |
| **Encoding** | UTF-8 plain text |
| **Standard** | RFC 9309 (Robots Exclusion Protocol) |

### Core Directives

| Directive | Purpose | Example |
|-----------|---------|---------|
| `User-agent:` | Target crawler | `User-agent: Googlebot`, `User-agent: *` |
| `Disallow:` | Block path prefix | `Disallow: /admin/` |
| `Allow:` | Allow path (can override Disallow) | `Allow: /public/` |
| `Sitemap:` | Declare sitemap absolute URL | `Sitemap: https://example.com/sitemap.xml` |
| `Clean-param:` | Strip query params (Yandex) | See below |

### Path Format Rules (Critical)

Different directives use different path formats — a common source of errors:

| Directive | Format | Correct | Wrong |
|-----------|--------|---------|-------|
| Disallow / Allow | **Root-relative path only** (starts with `/`) | `Disallow: /admin/` | `Disallow: https://example.com/admin/` |
| Sitemap | **Absolute URL only** | `Sitemap: https://example.com/sitemap.xml` | `Sitemap: /sitemap.xml` |

**Wildcards**: `*` matches any character sequence (`Disallow: /tmp/*`). `$` marks exact URL ending (`Allow: /news/.html$`).

**Priority**: More specific paths take precedence. `Allow: /shop/shoes/` overrides `Disallow: /shop/`. Path matching is case-sensitive: `Disallow: /PDF/` does not match `/pdf/`.

### Critical: Do Not Block

| Do not block | Reason |
|--------------|--------|
| CSS, JS, images | Google needs them to render pages; blocking breaks indexing |
| `/_next/` (Next.js) | Breaks CSS/JS loading; static assets in GSC "Crawled - not indexed" is expected. See **indexing** |
| Pages that use noindex | Crawlers must access the page to read the noindex directive; blocking in robots.txt prevents that |

**Only block**: paths that don't need crawling: /admin/, /api/, /staging/, temp files.

### AI Crawler Strategy

robots.txt is effective for all measured AI crawlers. Set rules per user-agent; check each vendor's docs for current tokens.

| User-agent | Purpose | Typical | Notes |
|------------|---------|---------|-------|
| **OAI-SearchBot** | ChatGPT search | Allow | Respects robots.txt |
| **GPTBot** | OpenAI training | Disallow | Respects robots.txt; shares crawl data with OAI-SearchBot if both allowed |
| **ChatGPT-User** | User-initiated browsing | N/A | **No longer respects robots.txt** (Dec 2025); use server-side controls instead |
| **Claude-SearchBot** | Claude search | Allow | Respects robots.txt |
| **Claude-User** | Anthropic user-initiated browsing | Allow | Respects robots.txt (unlike ChatGPT-User) |
| **ClaudeBot** | Anthropic training | Disallow | Respects robots.txt |
> **Deprecated**: ~~anthropic-ai~~ — retired by Anthropic, replaced by ClaudeBot / Claude-User / Claude-SearchBot. References to `anthropic-ai` in robots.txt have no effect.
| **PerplexityBot** | Perplexity search | Allow | Respects robots.txt |
| **Google-Extended** | Gemini training | Disallow | Respects robots.txt |
| **CCBot** | Common Crawl (LLM training) | Disallow | Respects robots.txt |
| **Bytespider** | ByteDance | Disallow | Respects robots.txt |
| **Meta-ExternalAgent** | Meta | Disallow | Respects robots.txt |
| **AppleBot** | Apple (Siri, Spotlight); renders JS | Allow for indexing | Respects robots.txt |

**Allow vs Disallow**: Allow search/indexing bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot); Disallow training-only bots (GPTBot, ClaudeBot, CCBot) if you don't want content used for model training.

**Important — ChatGPT-User exemption**: As of December 2025, **ChatGPT-User** no longer respects robots.txt directives. OpenAI considers it a proxy for human-initiated browsing. If you need to block it, use server-side controls (WAF rules, IP rate-limiting), not robots.txt. See **site-crawlability** for AI crawler optimization (SSR, URL management).

### Clean-param (Yandex)

```
Clean-param: utm_source&utm_medium&utm_campaign&utm_term&utm_content&ref&fbclid&gclid
```

## Output Format

- **Current state** (if auditing)
- **Recommended robots.txt** (full file)
- **Compliance checklist**
- **References**: [Google robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt)

## Related Skills

- **indexing**: Full noindex page-type list; when to use noindex vs robots.txt; GSC indexing diagnosis
- **page-metadata**: Meta robots (noindex, nofollow) implementation
- **xml-sitemap**: Sitemap URL to reference in robots.txt
- **site-crawlability**: Broader crawl and structure guidance; AI crawler optimization
- **rendering-strategies**: SSR, SSG, CSR; content in initial HTML for crawlers

More from kostja94/marketing-skills

SkillDescription
404-page-generatorWhen the user wants to create, optimize, or audit 404 error pages. Also use when the user mentions "404 page," "404 error," "error page," "page not found," "broken link page," "404 design," "custom 404," "404 redirect," "404 page UX," or "404 recovery." For sitewide page planning, use website-structure.
about-page-generatorWhen the user wants to create, optimize, or audit About page content. Also use when the user mentions "about page," "about us," "company story," "our team," "about section," "company overview," "brand story," "team page," or "who we are." For sitewide page planning, use website-structure.
affiliate-marketingWhen the user wants to plan, implement, or optimize affiliate marketing strategy. Also use when the user mentions "affiliate marketing," "affiliate program strategy," "CPS model," "affiliate recruitment," "commission structure," "affiliate partners," "affiliate network," "affiliate tracking," "affiliate commission," or "partner marketing." For affiliate page, use affiliate-page-generator.
affiliate-page-generatorWhen the user wants to create, optimize, or audit affiliate program page content. Also use when the user mentions "affiliate program," "affiliate page," "partner program," "referral program page," "affiliate landing," "partner landing," "commission page," or "affiliate signup." For affiliate program strategy, use affiliate-marketing.
ai-traffic-trackingWhen the user wants to track AI search traffic in GA4 or GSC. Also use when the user mentions "AI traffic," "ChatGPT referral," "Perplexity traffic," "AI Overviews," "GA4 AI sources," "AI search analytics," "track AI referrals," "AI search traffic," "Claude traffic," or "how to track AI traffic." For AI SEO strategy, use generative-engine-optimization.
alternatives-page-generatorWhen the user wants to create, optimize, or audit alternatives or comparison content (page or blog article). Also use when the user mentions "alternatives page," "alternatives listicle," "X alternatives," "competitor comparison," "vs page," "compare page," "best alternatives to X," "switch from X," "competitor brand traffic," "brand keyword ads," or "intercept competitor search." For competitor research, use competitor-research.
api-page-generatorWhen the user wants to create, optimize, or audit the API introduction/overview page. Also use when the user mentions "API page," "API landing page," "/api page," "API overview," "developer landing," "API marketing," or "API for developers." Note: API documentation (endpoint reference) lives in docs; use docs-page-generator.
app-adsWhen the user wants to run app install ads, user acquisition (UA), or promote mobile apps. Also use when the user mentions "app ads," "app install ads," "UA," "user acquisition," "Google App Campaigns," "Apple Search Ads," "ASA," "UAC," "App Store ads," "Play Store ads," "CPI," or "app promotion." For store listings, use distribution-channels.
article-contentWhen the user wants to write, generate, or create article body content—blog post body, long-form content, how-to guide, listicle. Also use when the user mentions "write article," "article content," "blog post content," "article body," "long-form content creation," "generate article," "article draft," "how-to guide content," "listicle content," "information gain," or "content density." For single post page structure, schema, and SEO metadata, use article-page-generator. For blog index/listing page, use blog-page-generator. For short ad, landing, or email copy, use copywriting.
backlink-analysisWhen the user wants to analyze backlinks, audit link profile, or identify link issues. Also use when the user mentions "backlink analysis," "backlink audit," "referring domains," "toxic links," "link profile," or "disavow file." For acquisition, use link-building.