robots.txt Validator — Crawl Rule Checker
Validate your robots.txt file against the Robots Exclusion Protocol. Checks directive syntax, path formats, Crawl-delay values, and Sitemap URLs. Previews crawl rules per user-agent group. Free and runs entirely in your browser.
How to Use robots.txt Validator — Crawl Rule Checker
How to Use robots.txt Validator
Step 1: Paste Your robots.txt Content
Copy the full content of your robots.txt file and paste it into the input field. You can also try one of the built-in examples.
Step 2: Click Validate robots.txt
The tool checks:
- Valid directive syntax (directive: value format)
- Recognised directive names (User-agent, Disallow, Allow, Crawl-delay, Sitemap, Host)
- Disallow/Allow paths starting with "/"
- Crawl-delay as a non-negative number
- Sitemap URLs as absolute URLs (http:// or https://)
- Presence of a wildcard User-agent (*) group
- Directives placed outside a User-agent group
Step 3: Review Results
Status banner — valid or invalid with a summary of groups, lines, and sitemaps.
Errors — issues that violate the Robots Exclusion Protocol (must fix).
Warnings — non-blocking recommendations (e.g. missing wildcard agent, empty Allow value).
Sitemap URLs — all Sitemap directives listed for easy review.
Rules per User-Agent — each user-agent group with its Allow and Disallow paths displayed in order.
Tips
- Always include a
User-agent: *group so all unconfigured crawlers have explicit rules - Use
Disallow: /to block all crawlers; useDisallow:(empty value) to allow all - Place your robots.txt at the root of your domain:
https://yourdomain.com/robots.txt - Sitemap URLs in robots.txt must be absolute — use full URLs, not relative paths
- Google respects Allow/Disallow order by path specificity, not by declaration order
Frequently Asked Questions
Most Viewed Tools
Screen Size Converter — Diagonal Dimension Tool
Calculate screen width and height from diagonal size and aspect ratio. Convert between inches and centimeters for displays, TVs, and monitors with instant dimension calculations.
Use Tool →DPI Calculator — Print Resolution Tool
Calculate DPI (dots per inch), image dimensions, and print sizes. Convert between pixels and physical dimensions for printing and displays.
Use Tool →TOTP Code Generator — 2FA Testing Tool
Generate time-based one-time passwords from a TOTP secret key. Enter your base32 secret, choose a period and digit length, and get the current and next codes with a live countdown timer. Useful for testing and debugging 2FA integrations.
Use Tool →Password Entropy Calculator — Crack Time Estimator
Calculate the information-theoretic bit entropy of any password or API key. Detects character set pools automatically, shows the total number of possible combinations, and estimates crack time across five attack scenarios from rate-limited web logins to GPU cracking clusters.
Use Tool →JSONL Formatter — Line-by-Line Validator
Format, validate, and inspect JSON Lines (JSONL) and NDJSON files. Validates each line individually, reports parse errors by line number, outputs compact JSONL or a pretty-print preview, and lets you download the cleaned file.
Use Tool →JSON to Zod — Schema Generator
Generate Zod validation schema code from a JSON sample object. Infers z.string(), z.number(), z.boolean(), z.array(), z.object(), and z.null() types automatically. Handles nested objects, arrays of objects with optional field detection, and outputs copy-ready TypeScript with import and z.infer type alias.
Use Tool →Seconds to Time Converter — HH:MM:SS Formatter
Convert seconds to HH:MM:SS time format instantly with precise calculations and easy-to-read formatting.
Use Tool →TLS Cipher Suite Checker — Strength Analyzer
Check TLS protocol version compatibility and cipher suite strength ratings against current best practices. Supports IANA and OpenSSL cipher names — rates each suite as Strong, Weak, or Deprecated and explains why.
Use Tool →Related DevOps & Infrastructure Tools
Systemd Service Generator — Scaffold, Harden, and Validate Unit Files
Fill in service parameters and generate a ready-to-install systemd unit file for Linux daemons
Use Tool →Port Number Lookup — Common TCP/UDP Service & Protocol Database
Searchable reference for 80+ well-known TCP and UDP ports. Look up any port number or service name to see the official protocol, service description, port range (well-known/registered), and security recommendations for risky ports.
Use Tool →Helm Values Builder — Scaffold, Validate, and Export Helm values.yaml
Configure common Helm chart parameters and generate a values.yaml override file for deployments
Use Tool →GitHub Actions Validator — Workflow Syntax & CI/CD Security Audit
Validate GitHub Actions workflow YAML for syntax errors, missing required fields, deprecated commands, mutable action refs, outdated action versions, and broken job dependencies. Get per-job results with fix hints in real time.
Use Tool →Makefile Target Generator — Scaffold, Validate, and Self-Document Makefiles
Define make target names and commands and dependencies to generate a formatted Makefile with PHONY targets
Use Tool →AWS S3 CORS Generator — Build, Audit, and Export S3 CORS Policies
Configure allowed origins/methods/headers and generate the S3 bucket CORS JSON policy
Use Tool →Dockerfile Linter — Optimize & Secure Your Container Builds
Lint Dockerfile instructions for best practices, security issues, and layer optimization. Flags unpinned base images, root user, ADD vs COPY, apt-get mistakes, shell-form CMD, and more — with fix guidance for each issue.
Use Tool →Sitemap Validator — XML Sitemap Checker
Validate XML sitemaps against the sitemap protocol specification. Checks structure, required fields, URL count, changefreq values, and priority ranges. Supports both URL sitemaps and sitemap index files. Free and runs entirely in your browser.
Use Tool →Share Your Feedback
Help us improve this tool by sharing your experience