Validate Robots.txt Syntax & Crawl Rules Instantly
Validate your robots.txt file against the Robots Exclusion Protocol. Checks directive syntax, path formats, Crawl-delay values, and Sitemap URLs. Previews crawl rules per user-agent group. Free and runs entirely in your browser.
How to Use robots.txt Validator
How to Use robots.txt Validator
Step 1: Paste Your robots.txt Content
Copy the full content of your robots.txt file and paste it into the input field. You can also try one of the built-in examples.
Step 2: Click Validate robots.txt
The tool checks:
- Valid directive syntax (directive: value format)
- Recognised directive names (User-agent, Disallow, Allow, Crawl-delay, Sitemap, Host)
- Disallow/Allow paths starting with "/"
- Crawl-delay as a non-negative number
- Sitemap URLs as absolute URLs (http:// or https://)
- Presence of a wildcard User-agent (*) group
- Directives placed outside a User-agent group
Step 3: Review Results
Status banner — valid or invalid with a summary of groups, lines, and sitemaps.
Errors — issues that violate the Robots Exclusion Protocol (must fix).
Warnings — non-blocking recommendations (e.g. missing wildcard agent, empty Allow value).
Sitemap URLs — all Sitemap directives listed for easy review.
Rules per User-Agent — each user-agent group with its Allow and Disallow paths displayed in order.
Tips
- Always include a
User-agent: *group so all unconfigured crawlers have explicit rules - Use
Disallow: /to block all crawlers; useDisallow:(empty value) to allow all - Place your robots.txt at the root of your domain:
https://yourdomain.com/robots.txt - Sitemap URLs in robots.txt must be absolute — use full URLs, not relative paths
- Google respects Allow/Disallow order by path specificity, not by declaration order
Frequently Asked Questions
Most Viewed Tools
TOTP Code Generator
Generate time-based one-time passwords from a TOTP secret key. Enter your base32 secret, choose a period and digit length, and get the current and next codes with a live countdown timer. Useful for testing and debugging 2FA integrations.
Use Tool →JSON to Zod Schema Generator
Generate Zod validation schema code from a JSON sample object. Infers z.string(), z.number(), z.boolean(), z.array(), z.object(), and z.null() types automatically. Handles nested objects, arrays of objects with optional field detection, and outputs copy-ready TypeScript with import and z.infer type alias.
Use Tool →JSONL / NDJSON Formatter
Format, validate, and inspect JSON Lines (JSONL) and NDJSON files. Validates each line individually, reports parse errors by line number, outputs compact JSONL or a pretty-print preview, and lets you download the cleaned file.
Use Tool →TLS Cipher Suite Checker
Check TLS protocol version compatibility and cipher suite strength ratings against current best practices. Supports IANA and OpenSSL cipher names — rates each suite as Strong, Weak, or Deprecated and explains why.
Use Tool →Secret and Credential Scanner
Scan pasted text, code, or config files for accidentally exposed API keys, tokens, passwords, and private keys. Detects 50+ secret types across AWS, GitHub, Stripe, OpenAI, and more — all client-side, nothing leaves your browser.
Use Tool →Password Entropy Calculator
Calculate the information-theoretic bit entropy of any password or API key. Detects character set pools automatically, shows the total number of possible combinations, and estimates crack time across five attack scenarios from rate-limited web logins to GPU cracking clusters.
Use Tool →TOML Config Validator
Validate TOML configuration file syntax and report errors with line numbers. Paste any TOML content — Cargo.toml, pyproject.toml, config.toml — and instantly see a green checkmark with key counts and structure stats, or a precise error message pointing to the exact line. Includes a collapsible JSON structure preview to confirm what was parsed.
Use Tool →Content Security Policy Generator
Build Content Security Policy headers interactively. Toggle directives like script-src, style-src, and img-src, select allowed source tokens, and add custom origins. Instantly outputs your CSP as an HTTP header, meta tag, Nginx directive, or Apache header.
Use Tool →Related DevOps & Infrastructure Tools
Cookie Parser
Parse HTTP cookie strings into readable key-value pairs. Decode URL-encoded values and inspect cookies from browser requests.
Use Tool →API Response Formatter
Format and beautify API responses for better readability. JSON formatter with minify and prettify options.
Use Tool →HTTP Header Analyzer
Parse and analyze HTTP request or response headers. Identifies categories, explains each header, flags missing security headers, and detects duplicates or suspicious values — entirely in your browser.
Use Tool →Query String Parser
Parse URL query strings into readable key-value pairs. Decode parameters and inspect URL search queries with ease.
Use Tool →SSL Certificate Validator
Paste a PEM certificate to instantly validate expiry, signature algorithm, key strength, SAN presence, and trust chain. Get a clear pass/warn/fail report for each check.
Use Tool →User Agent Parser
Parse user agent strings to extract browser, operating system, device, and engine information. Essential for web analytics, device detection, and browser compatibility testing.
Use Tool →Cron Expression Validator
Validate cron expressions, get a plain-English explanation of what they mean, and see the next scheduled run times — all in your browser.
Use Tool →Sitemap Validator
Validate XML sitemaps against the sitemap protocol specification. Checks structure, required fields, URL count, changefreq values, and priority ranges. Supports both URL sitemaps and sitemap index files. Free and runs entirely in your browser.
Use Tool →Share Your Feedback
Help us improve this tool by sharing your experience