robots.txt Validator
Validate your robots.txt file against the Robots Exclusion Protocol. Checks directive syntax, path formats, Crawl-delay values, and Sitemap URLs. Previews crawl rules per user-agent group. Free and runs entirely in your browser.
How to Use robots.txt Validator
How to Use robots.txt Validator
Step 1: Paste Your robots.txt Content
Copy the full content of your robots.txt file and paste it into the input field. You can also try one of the built-in examples.
Step 2: Click Validate robots.txt
The tool checks:
- Valid directive syntax (directive: value format)
- Recognised directive names (User-agent, Disallow, Allow, Crawl-delay, Sitemap, Host)
- Disallow/Allow paths starting with "/"
- Crawl-delay as a non-negative number
- Sitemap URLs as absolute URLs (http:// or https://)
- Presence of a wildcard User-agent (*) group
- Directives placed outside a User-agent group
Step 3: Review Results
Status banner — valid or invalid with a summary of groups, lines, and sitemaps.
Errors — issues that violate the Robots Exclusion Protocol (must fix).
Warnings — non-blocking recommendations (e.g. missing wildcard agent, empty Allow value).
Sitemap URLs — all Sitemap directives listed for easy review.
Rules per User-Agent — each user-agent group with its Allow and Disallow paths displayed in order.
Tips
- Always include a
User-agent: *group so all unconfigured crawlers have explicit rules - Use
Disallow: /to block all crawlers; useDisallow:(empty value) to allow all - Place your robots.txt at the root of your domain:
https://yourdomain.com/robots.txt - Sitemap URLs in robots.txt must be absolute — use full URLs, not relative paths
- Google respects Allow/Disallow order by path specificity, not by declaration order
Frequently Asked Questions
Most Viewed Tools
Screen Size Converter
Calculate screen width and height from diagonal size and aspect ratio. Convert between inches and centimeters for displays, TVs, and monitors with instant dimension calculations.
Use Tool →DPI Calculator
Calculate DPI (dots per inch), image dimensions, and print sizes. Convert between pixels and physical dimensions for printing and displays.
Use Tool →Paper Size Converter
Convert between international paper sizes (A4, Letter, Legal) with dimensions in mm, cm, and inches. Compare ISO A/B series and North American paper standards.
Use Tool →Fuel Consumption Converter
Convert between MPG (miles per gallon), L/100km (liters per 100 kilometers), and other fuel efficiency units. Compare car fuel economy across different measurement systems.
Use Tool →Product Schema Generator
Generate JSON-LD Product schema markup for SEO. Add product details like name, price, brand, rating, and availability to create structured data for rich search results.
Use Tool →CSV Splitter
Split large CSV files into smaller files by number of rows. Process large datasets in manageable chunks instantly.
Use Tool →Large Text File Viewer
View and search large text files up to 200MB in your browser. Features virtual scrolling, line numbers, search functionality, and file statistics. Perfect for log files, CSV, JSON, and code files.
Use Tool →Seconds to Time Converter
Convert seconds to HH:MM:SS time format instantly with precise calculations and easy-to-read formatting.
Use Tool →Related Marketing & SEO Tools
Open Graph Generator
Generate Facebook Open Graph meta tags for social media sharing. Create og:title, og:description, og:image, og:url, and og:type tags. Perfect for optimizing how your links appear on Facebook, LinkedIn, WhatsApp, and Slack.
Use Tool →Robots.txt Generator
Generate robots.txt file to control search engine crawlers. Create user-agent rules, allow/disallow paths, set crawl delays, and add sitemap URLs. Perfect for managing bot access to your website.
Use Tool →Sitemap Validator
Validate XML sitemaps against the sitemap protocol specification. Checks structure, required fields, URL count, changefreq values, and priority ranges. Supports both URL sitemaps and sitemap index files. Free and runs entirely in your browser.
Use Tool →Twitter Card Generator
Generate Twitter Card meta tags for Twitter/X sharing. Create summary cards, large image cards, app cards, and player cards. Optimize how your links appear on Twitter with custom titles, descriptions, and images.
Use Tool →Breadcrumb Schema Generator
Generate JSON-LD BreadcrumbList schema markup for SEO. Add breadcrumb items with names and URLs to create structured data that helps search engines understand your site hierarchy.
Use Tool →Google SERP Simulator
Preview how your title and meta description appear in Google search results. See real-time character counts, pixel width estimates, and desktop/mobile previews to optimize your SEO.
Use Tool →Meta Tag Generator
Generate HTML meta tags for SEO optimization. Create title, description, keywords, viewport, charset, robots, and author meta tags. Perfect for improving search engine rankings and social sharing.
Use Tool →Product Schema Generator
Generate JSON-LD Product schema markup for SEO. Add product details like name, price, brand, rating, and availability to create structured data for rich search results.
Use Tool →Share Your Feedback
Help us improve this tool by sharing your experience