Can I import an existing robots.txt file?

Yes — click "Import existing robots.txt", paste the file's contents into the box, and click "Parse & replace rules". It reads every User-agent, Allow, Disallow, Crawl-delay, and Sitemap line and rebuilds the rule list from them, so you can tweak an existing file instead of starting from scratch.

Where does robots.txt need to live?

At the root of your domain — e.g. https://example.com/robots.txt. Search engine crawlers only look for it there; a copy nested in a subfolder is ignored.

What does User-agent: * mean?

The asterisk is a wildcard that matches every crawler. Use a specific name like Googlebot or Bingbot if you want a rule to apply only to that crawler, and add a separate block with its own User-agent line for each one.

Does robots.txt actually block pages from appearing in search results?

Not reliably — Disallow tells well-behaved crawlers not to fetch a URL, but a disallowed page can still be indexed (without its content) if other sites link to it. To fully keep a page out of search results, use a noindex meta tag or header instead.

Is my data sent anywhere?

No. The file is built entirely in your browser from the rules you enter — nothing is uploaded until you copy it yourself.

ROBOTS.TXT GENERATOR

Build a robots.txt with Allow/Disallow rules and a sitemap link. Nothing leaves your browser.

Rules

Sitemap URL

Crawl-delay (seconds, optional)

robots.txt

User-agent: *
Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

How to generate a robots.txt file

1
Add one or more rules: a user-agent, Allow or Disallow, and a path — or click "Import existing robots.txt" and paste one in to parse it into rules automatically.
2
Set your sitemap URL and an optional crawl delay.
3
Copy the generated robots.txt and upload it to your site's root as /robots.txt.

Questions

Can I import an existing robots.txt file?: Yes — click "Import existing robots.txt", paste the file's contents into the box, and click "Parse & replace rules". It reads every User-agent, Allow, Disallow, Crawl-delay, and Sitemap line and rebuilds the rule list from them, so you can tweak an existing file instead of starting from scratch.
Where does robots.txt need to live?: At the root of your domain — e.g. https://example.com/robots.txt. Search engine crawlers only look for it there; a copy nested in a subfolder is ignored.
What does User-agent: * mean?: The asterisk is a wildcard that matches every crawler. Use a specific name like Googlebot or Bingbot if you want a rule to apply only to that crawler, and add a separate block with its own User-agent line for each one.
Does robots.txt actually block pages from appearing in search results?: Not reliably — Disallow tells well-behaved crawlers not to fetch a URL, but a disallowed page can still be indexed (without its content) if other sites link to it. To fully keep a page out of search results, use a noindex meta tag or header instead.
Is my data sent anywhere?: No. The file is built entirely in your browser from the rules you enter — nothing is uploaded until you copy it yourself.