A typo in robots.txt can block your own pages from Google while the site looks fine in a browser. The file is small, so there is no excuse for hand-typing a broken allow or block.
Assemble the file from allow and block lists, sitemap URLs, and crawl-delay options. Copy it to your site root and validate it in Search Console, where a blocked homepage is instantly visible.
Try it now: open the free tool, runs 100{b8b7921c843ac9e5edcbca134af59decad3ed340e0e3b2e6c286158f277affea} in your browser.
Bottom line: write rules search engines actually parse, then test them in GSC.
Your turn: use the tool free here, no signup, nothing leaves your device.
How do you use Robots.txt Generator?
Open the Robots.txt Generator in your browser (Build search-engine crawl rules blocks). Type or paste your input, adjust the options, and copy the result. Everything runs locally on your device with no account and nothing stored, so the three steps below take seconds even for confidential work.
- Paste the data or snippet you want to process.
- Run the tool and watch the output update.
- Copy the result for use in your own code.
Try Robots.txt Generator free. It opens right in your browser.