robots.txt Validator — Crawl Rule Checker
Validate your robots.txt file against the Robots Exclusion Protocol. Checks directive syntax, path formats, Crawl-delay values, and Sitemap URLs. Previews crawl rules per user-agent group. Free and runs entirely in your browser.
How to Use robots.txt Validator — Crawl Rule Checker
How to Use robots.txt Validator
Step 1: Paste Your robots.txt Content
Copy the full content of your robots.txt file and paste it into the input field. You can also try one of the built-in examples.
Step 2: Click Validate robots.txt
The tool checks:
- Valid directive syntax (directive: value format)
- Recognised directive names (User-agent, Disallow, Allow, Crawl-delay, Sitemap, Host)
- Disallow/Allow paths starting with "/"
- Crawl-delay as a non-negative number
- Sitemap URLs as absolute URLs (http:// or https://)
- Presence of a wildcard User-agent (*) group
- Directives placed outside a User-agent group
Step 3: Review Results
Status banner — valid or invalid with a summary of groups, lines, and sitemaps.
Errors — issues that violate the Robots Exclusion Protocol (must fix).
Warnings — non-blocking recommendations (e.g. missing wildcard agent, empty Allow value).
Sitemap URLs — all Sitemap directives listed for easy review.
Rules per User-Agent — each user-agent group with its Allow and Disallow paths displayed in order.
Tips
- Always include a
User-agent: *group so all unconfigured crawlers have explicit rules - Use
Disallow: /to block all crawlers; useDisallow:(empty value) to allow all - Place your robots.txt at the root of your domain:
https://yourdomain.com/robots.txt - Sitemap URLs in robots.txt must be absolute — use full URLs, not relative paths
- Google respects Allow/Disallow order by path specificity, not by declaration order
Frequently Asked Questions
Most Viewed Tools
Screen Size Converter — Diagonal Dimension Tool
Calculate screen width and height from diagonal size and aspect ratio. Convert between inches and centimeters for displays, TVs, and monitors with instant dimension calculations.
Use Tool →DPI Calculator — Print Resolution Tool
Calculate DPI (dots per inch), image dimensions, and print sizes. Convert between pixels and physical dimensions for printing and displays.
Use Tool →TOTP Code Generator — 2FA Testing Tool
Generate time-based one-time passwords from a TOTP secret key. Enter your base32 secret, choose a period and digit length, and get the current and next codes with a live countdown timer. Useful for testing and debugging 2FA integrations.
Use Tool →JSONL Formatter — Line-by-Line Validator
Format, validate, and inspect JSON Lines (JSONL) and NDJSON files. Validates each line individually, reports parse errors by line number, outputs compact JSONL or a pretty-print preview, and lets you download the cleaned file.
Use Tool →JSON to Zod — Schema Generator
Generate Zod validation schema code from a JSON sample object. Infers z.string(), z.number(), z.boolean(), z.array(), z.object(), and z.null() types automatically. Handles nested objects, arrays of objects with optional field detection, and outputs copy-ready TypeScript with import and z.infer type alias.
Use Tool →Password Entropy Calculator — Crack Time Estimator
Calculate the information-theoretic bit entropy of any password or API key. Detects character set pools automatically, shows the total number of possible combinations, and estimates crack time across five attack scenarios from rate-limited web logins to GPU cracking clusters.
Use Tool →TLS Cipher Suite Checker — Strength Analyzer
Check TLS protocol version compatibility and cipher suite strength ratings against current best practices. Supports IANA and OpenSSL cipher names — rates each suite as Strong, Weak, or Deprecated and explains why.
Use Tool →Secret Scanner — API Key & Credential Detector
Scan pasted text, code, or config files for accidentally exposed API keys, tokens, passwords, and private keys. Detects 50+ secret types across AWS, GitHub, Stripe, OpenAI, and more — all client-side, nothing leaves your browser.
Use Tool →Related DevOps & Infrastructure Tools
SSL Certificate Decoder — Expiry & SAN Inspector
Decode X.509 SSL/TLS certificates and RSA private keys in your browser. View subject, issuer, SANs, validity dates, key type, serial number, and SHA-256/SHA-1 fingerprints. Optionally check if a certificate and private key match.
Use Tool →Dockerfile Linter — Optimize & Secure Your Container Builds
Lint Dockerfile instructions for best practices, security issues, and layer optimization. Flags unpinned base images, root user, ADD vs COPY, apt-get mistakes, shell-form CMD, and more — with fix guidance for each issue.
Use Tool →User Agent Parser — Browser & Device Decoder
Parse user agent strings to extract browser, operating system, device, and engine information. Essential for web analytics, device detection, and browser compatibility testing.
Use Tool →Cron Expression Validator — Schedule Checker
Validate cron expressions, get a plain-English explanation of what they mean, and see the next scheduled run times — all in your browser.
Use Tool →Cookie Parser — HTTP Cookie Decoder
Parse HTTP cookie strings into readable key-value pairs. Decode URL-encoded values and inspect cookies from browser requests.
Use Tool →API Response Formatter — JSON Pretty Printer
Format and beautify API responses for better readability. JSON formatter with minify and prettify options.
Use Tool →DNS Record Validator — Live Lookup Tool
Look up live DNS records for any domain. Query A, AAAA, MX, TXT, CNAME, NS, SOA, and CAA records instantly via Cloudflare DNS-over-HTTPS — no installation required.
Use Tool →GitHub Actions Validator — Workflow Syntax & CI/CD Security Audit
Validate GitHub Actions workflow YAML for syntax errors, missing required fields, deprecated commands, mutable action refs, outdated action versions, and broken job dependencies. Get per-job results with fix hints in real time.
Use Tool →Share Your Feedback
Help us improve this tool by sharing your experience