robots.txt Validator — Crawl Rule Checker
Validate your robots.txt file against the Robots Exclusion Protocol. Checks directive syntax, path formats, Crawl-delay values, and Sitemap URLs. Previews crawl rules per user-agent group. Free and runs entirely in your browser.
How to Use robots.txt Validator — Crawl Rule Checker
How to Use robots.txt Validator
Step 1: Paste Your robots.txt Content
Copy the full content of your robots.txt file and paste it into the input field. You can also try one of the built-in examples.
Step 2: Click Validate robots.txt
The tool checks:
- Valid directive syntax (directive: value format)
- Recognised directive names (User-agent, Disallow, Allow, Crawl-delay, Sitemap, Host)
- Disallow/Allow paths starting with "/"
- Crawl-delay as a non-negative number
- Sitemap URLs as absolute URLs (http:// or https://)
- Presence of a wildcard User-agent (*) group
- Directives placed outside a User-agent group
Step 3: Review Results
Status banner — valid or invalid with a summary of groups, lines, and sitemaps.
Errors — issues that violate the Robots Exclusion Protocol (must fix).
Warnings — non-blocking recommendations (e.g. missing wildcard agent, empty Allow value).
Sitemap URLs — all Sitemap directives listed for easy review.
Rules per User-Agent — each user-agent group with its Allow and Disallow paths displayed in order.
Tips
- Always include a
User-agent: *group so all unconfigured crawlers have explicit rules - Use
Disallow: /to block all crawlers; useDisallow:(empty value) to allow all - Place your robots.txt at the root of your domain:
https://yourdomain.com/robots.txt - Sitemap URLs in robots.txt must be absolute — use full URLs, not relative paths
- Google respects Allow/Disallow order by path specificity, not by declaration order
Frequently Asked Questions
Most Viewed Tools
TOTP Code Generator — 2FA Testing Tool
Generate time-based one-time passwords from a TOTP secret key. Enter your base32 secret, choose a period and digit length, and get the current and next codes with a live countdown timer. Useful for testing and debugging 2FA integrations.
Use Tool →JSON to Zod — Schema Generator
Generate Zod validation schema code from a JSON sample object. Infers z.string(), z.number(), z.boolean(), z.array(), z.object(), and z.null() types automatically. Handles nested objects, arrays of objects with optional field detection, and outputs copy-ready TypeScript with import and z.infer type alias.
Use Tool →JSONL Formatter — Line-by-Line Validator
Format, validate, and inspect JSON Lines (JSONL) and NDJSON files. Validates each line individually, reports parse errors by line number, outputs compact JSONL or a pretty-print preview, and lets you download the cleaned file.
Use Tool →Screen Size Converter — Diagonal Dimension Tool
Calculate screen width and height from diagonal size and aspect ratio. Convert between inches and centimeters for displays, TVs, and monitors with instant dimension calculations.
Use Tool →Password Entropy Calculator — Crack Time Estimator
Calculate the information-theoretic bit entropy of any password or API key. Detects character set pools automatically, shows the total number of possible combinations, and estimates crack time across five attack scenarios from rate-limited web logins to GPU cracking clusters.
Use Tool →TLS Cipher Suite Checker — Strength Analyzer
Check TLS protocol version compatibility and cipher suite strength ratings against current best practices. Supports IANA and OpenSSL cipher names — rates each suite as Strong, Weak, or Deprecated and explains why.
Use Tool →Secret Scanner — API Key & Credential Detector
Scan pasted text, code, or config files for accidentally exposed API keys, tokens, passwords, and private keys. Detects 50+ secret types across AWS, GitHub, Stripe, OpenAI, and more — all client-side, nothing leaves your browser.
Use Tool →TOML Config Validator — Syntax Error Finder
Validate TOML configuration file syntax and report errors with line numbers. Paste any TOML content — Cargo.toml, pyproject.toml, config.toml — and instantly see a green checkmark with key counts and structure stats, or a precise error message pointing to the exact line. Includes a collapsible JSON structure preview to confirm what was parsed.
Use Tool →Related DevOps & Infrastructure Tools
HTTP Header Analyzer — Security & CORS Audit
Parse and analyze HTTP request or response headers. Identifies categories, explains each header, flags missing security headers, and detects duplicates or suspicious values — entirely in your browser.
Use Tool →Sitemap Validator — XML Sitemap Checker
Validate XML sitemaps against the sitemap protocol specification. Checks structure, required fields, URL count, changefreq values, and priority ranges. Supports both URL sitemaps and sitemap index files. Free and runs entirely in your browser.
Use Tool →MIME Type Finder — File Extension Lookup
Find MIME type for file extensions instantly. Look up media types for images, videos, documents, and more.
Use Tool →Cron Expression Validator — Schedule Checker
Validate cron expressions, get a plain-English explanation of what they mean, and see the next scheduled run times — all in your browser.
Use Tool →Robots.txt Generator — Crawler Control Tool
Generate robots.txt file to control search engine crawlers. Create user-agent rules, allow/disallow paths, set crawl delays, and add sitemap URLs. Perfect for managing bot access to your website.
Use Tool →Port Number Lookup — Common TCP/UDP Service & Protocol Database
Searchable reference for 80+ well-known TCP and UDP ports. Look up any port number or service name to see the official protocol, service description, port range (well-known/registered), and security recommendations for risky ports.
Use Tool →Query String Parser — URL Parameter Decoder
Parse URL query strings into readable key-value pairs. Decode parameters and inspect URL search queries with ease.
Use Tool →Dockerfile Linter — Optimize & Secure Your Container Builds
Lint Dockerfile instructions for best practices, security issues, and layer optimization. Flags unpinned base images, root user, ADD vs COPY, apt-get mistakes, shell-form CMD, and more — with fix guidance for each issue.
Use Tool →Share Your Feedback
Help us improve this tool by sharing your experience