01
CRAWL RULES
Allow & Disallow
These directives tell matching crawlers which paths they are allowed to request and which paths they are instructed not to crawl.
User-agent: *
Disallow: /admin/
FREE SEO TOOL
Understand what search engines can and can't crawl.
Test your website's robots.txt file, inspect its directives and check whether important URLs are allowed or blocked from crawling.
01
Enter your website URL below to fetch and test its robots.txt rules.
ROBOTS.TXT ANALYZER
Check your crawl rules.
Enter a website URL to fetch its robots.txt file, inspect the directives and test whether a specific URL is allowed or blocked for crawling.
READY
Example: https://example.com
We'll check the standard /robots.txt location on the domain you enter.
ROBOTS.TXT
—
—
HTTP STATUS
—
Server response
RULE GROUPS
—
Detected directives
SITEMAP
—
Detected sitemap reference
01
0 rules
Crawl Rules
Fetch a robots.txt file to inspect its crawl directives.
02
Raw robots.txt
Fetch a robots.txt file to view its contents.
03
Test a URL
Check whether your robots.txt rules allow or block a specific path.
ALLOWED
—
—
04
Robots.txt Signals
Important observations from the file.
Waiting for analysis
Fetch a robots.txt file to see crawl and indexing signals.
Unable to fetch robots.txt.
Please check the URL and try again.
UNDERSTAND THE RESULTS
What your robots.txt is telling crawlers.
Robots.txt controls how compliant crawlers are instructed to access parts of your website. Use the checks above to spot important crawl rules before they become an SEO issue.
02
USER-AGENT
Crawler-specific rules
A robots.txt file can contain different rules for different crawlers. The User-agent directive determines which crawler a rule group targets.
User-agent: Googlebot
Disallow: /private/
03
SITEMAP
Sitemap reference
A robots.txt file can reference your XML sitemap. This gives crawlers a discoverable location for the URLs you want them to find.
Sitemap: https://example.com/sitemap.xml
04
URL TEST
Test important URLs
A broad Disallow rule can unintentionally affect valuable pages, resources or sections of a website. Test important URLs individually.
Disallow: /blog/
→ /blog/article/ is blocked
05
SEO CHECK
Look for accidental blocks
Pay particular attention to rules that block important sections of the website, especially content directories and URLs you expect search engines to discover.
User-agent: *
Disallow: /
06
IMPORTANT
Robots.txt is not a noindex tool
Robots.txt instructions are about crawling. They should not be treated as a general replacement for page-level indexing controls such as noindex.
Crawl control ≠ indexing control
Quick reminder
Robots.txt rules are crawler instructions and support different levels of implementation across search engines. Always review important crawl restrictions in the context of your site's actual SEO architecture.