Robots.txt Generator

Type your site's address: our AI reads your current file and writes a clean one that tells crawlers what they may visit.

Free, no signup. Only the public page is read.

What is a robots.txt file?

The robots.txt file is a small text file at the root of your site (mysite.com/robots.txt). Crawlers read it before visiting your pages: it tells them which parts of the site they may go through and where to find your sitemap.

This generator writes it for you. Type your site's address: our AI reads the file you have now, explains it in plain words and suggests a clean file. It keeps what your file already blocks, AI crawlers included, removes a block on the whole site if there is one and adds your sitemap.

How to use it

  1. Type your site's address.
  2. Read what we found: what your current file says, in plain words. "What changes" lists the differences with the new one.
  3. Copy or download the file and put it at the root of your site, named exactly robots.txt.

FAQ

Your questions

Contact us
Does blocking a page in robots.txt remove it from Google?

No. robots.txt stops the visit, not the listing: a blocked page can stay in the results if other sites link to it. To remove a page, keep it reachable and give it a noindex tag. And never block your whole site by mistake with "Disallow: /".

Should I block AI crawlers?

It's your choice. Blocking them keeps your texts out of their training, but some of these crawlers also feed assistants' answers (OAI-SearchBot or PerplexityBot, for example): blocking them can make you less visible in those answers. Google-Extended changes nothing about your presence in Google Search. And robots.txt is a request: well-behaved crawlers follow it, but it doesn't technically block access. The file suggested here keeps your current file's choice.

Does my site already have a robots.txt?

Type your address at the top of this page: the tool loads your current file, or tells you that you don't have one yet. You can also type your address followed by /robots.txt in your browser, for example mysite.com/robots.txt. Many CMSs create one automatically: read "What changes" before you replace it.

What happens to the address I type?

semchat reads the public page at that address, and the site's robots.txt, the way any visitor could. Parts of the page go to an AI that writes the result. Nothing is stored beyond a 24-hour cache, which lets a repeat check of the same address answer right away.

What about the rest of your site?

The free check reads your homepage in a few seconds and tells you what to improve first, no signup needed.

All free tools

Ready to get your site off the ground?

Ask your first question - semchat replies with a clear action plan in seconds.

Free · no sign-up · takes 5 seconds