robots.txt Validator — Crawl Rule Checker
Validate your robots.txt file against the Robots Exclusion Protocol. Checks directive syntax, path formats, Crawl-delay values, and Sitemap URLs. Previews crawl rules per user-agent group. Free and runs entirely in your browser.
How to Use robots.txt Validator — Crawl Rule Checker
How to Use robots.txt Validator
Step 1: Paste Your robots.txt Content
Copy the full content of your robots.txt file and paste it into the input field. You can also try one of the built-in examples.
Step 2: Click Validate robots.txt
The tool checks:
- Valid directive syntax (directive: value format)
- Recognised directive names (User-agent, Disallow, Allow, Crawl-delay, Sitemap, Host)
- Disallow/Allow paths starting with "/"
- Crawl-delay as a non-negative number
- Sitemap URLs as absolute URLs (http:// or https://)
- Presence of a wildcard User-agent (*) group
- Directives placed outside a User-agent group
Step 3: Review Results
Status banner — valid or invalid with a summary of groups, lines, and sitemaps.
Errors — issues that violate the Robots Exclusion Protocol (must fix).
Warnings — non-blocking recommendations (e.g. missing wildcard agent, empty Allow value).
Sitemap URLs — all Sitemap directives listed for easy review.
Rules per User-Agent — each user-agent group with its Allow and Disallow paths displayed in order.
Tips
- Always include a
User-agent: *group so all unconfigured crawlers have explicit rules - Use
Disallow: /to block all crawlers; useDisallow:(empty value) to allow all - Place your robots.txt at the root of your domain:
https://yourdomain.com/robots.txt - Sitemap URLs in robots.txt must be absolute — use full URLs, not relative paths
- Google respects Allow/Disallow order by path specificity, not by declaration order
Frequently Asked Questions
Most Viewed Tools
Screen Size Converter — Diagonal Dimension Tool
Calculate screen width and height from diagonal size and aspect ratio. Convert between inches and centimeters for displays, TVs, and monitors with instant dimension calculations.
Use Tool →DPI Calculator — Print Resolution Tool
Calculate DPI (dots per inch), image dimensions, and print sizes. Convert between pixels and physical dimensions for printing and displays.
Use Tool →Seconds to Time Converter — HH:MM:SS Formatter
Convert seconds to HH:MM:SS time format instantly with precise calculations and easy-to-read formatting.
Use Tool →TOTP Code Generator — 2FA Testing Tool
Generate time-based one-time passwords from a TOTP secret key. Enter your base32 secret, choose a period and digit length, and get the current and next codes with a live countdown timer. Useful for testing and debugging 2FA integrations.
Use Tool →Password Entropy Calculator — Crack Time Estimator
Calculate the information-theoretic bit entropy of any password or API key. Detects character set pools automatically, shows the total number of possible combinations, and estimates crack time across five attack scenarios from rate-limited web logins to GPU cracking clusters.
Use Tool →JSONL Formatter — Line-by-Line Validator
Format, validate, and inspect JSON Lines (JSONL) and NDJSON files. Validates each line individually, reports parse errors by line number, outputs compact JSONL or a pretty-print preview, and lets you download the cleaned file.
Use Tool →TLS Cipher Suite Checker — Strength Analyzer
Check TLS protocol version compatibility and cipher suite strength ratings against current best practices. Supports IANA and OpenSSL cipher names — rates each suite as Strong, Weak, or Deprecated and explains why.
Use Tool →JSON to Zod — Schema Generator
Generate Zod validation schema code from a JSON sample object. Infers z.string(), z.number(), z.boolean(), z.array(), z.object(), and z.null() types automatically. Handles nested objects, arrays of objects with optional field detection, and outputs copy-ready TypeScript with import and z.infer type alias.
Use Tool →Related DevOps & Infrastructure Tools
SSL Certificate Decoder — Expiry & SAN Inspector
Decode X.509 SSL/TLS certificates and RSA private keys in your browser. View subject, issuer, SANs, validity dates, key type, serial number, and SHA-256/SHA-1 fingerprints. Optionally check if a certificate and private key match.
Use Tool →MIME Type Finder — File Extension Lookup
Find MIME type for file extensions instantly. Look up media types for images, videos, documents, and more.
Use Tool →.htaccess Validator — Apache Rule Checker
Lint Apache .htaccess files for syntax errors, unclosed block tags, and common misconfigurations. Detects RewriteRule issues, invalid status codes, deprecated access directives, and unclosed IfModule blocks. Free and runs entirely in your browser.
Use Tool →GitHub Actions Validator — Workflow Syntax & CI/CD Security Audit
Validate GitHub Actions workflow YAML for syntax errors, missing required fields, deprecated commands, mutable action refs, outdated action versions, and broken job dependencies. Get per-job results with fix hints in real time.
Use Tool →Sitemap Validator — XML Sitemap Checker
Validate XML sitemaps against the sitemap protocol specification. Checks structure, required fields, URL count, changefreq values, and priority ranges. Supports both URL sitemaps and sitemap index files. Free and runs entirely in your browser.
Use Tool →AWS ARN Parser & Validator — Deconstruct ARNs and Generate IAM Snippets
Paste an AWS ARN to extract and display account/region/service/resource components in a structured view
Use Tool →API Response Formatter — JSON Pretty Printer
Format and beautify API responses for better readability. JSON formatter with minify and prettify options.
Use Tool →SSL Certificate Validator — Chain & Expiry Checker
Paste a PEM certificate to instantly validate expiry, signature algorithm, key strength, SAN presence, and trust chain. Get a clear pass/warn/fail report for each check.
Use Tool →Share Your Feedback
Help us improve this tool by sharing your experience