Use a browser-based robots.txt generator to pick which crawlers may access which paths, then place the generated file at the root of your site as /robots.txt. The file is built locally in your browser, and pairing it with a sitemap line helps search engines find your pages.
Free robots.txt generator. Pick which crawlers to allow or block, add disallow paths, crawl delay and a sitemap URL, then copy a valid robots.txt file. Private.
Open Robots.txt Generator → Free toolFree sitemap XML generator. Paste a list of URLs to build a valid XML sitemap with lastmod, changefreq and priority, ready to submit to search engines.
Open Sitemap XML Generator →A robots.txt is a plain text file of User-agent, Disallow and Allow lines, plus an optional Sitemap line pointing at your sitemap URL. It must live at the site root, /robots.txt, or crawlers will not find it. A generator keeps the syntax right, because a stray character in a Disallow rule can silently block more than you intended, including your whole site with a single slash.
robots.txt is advisory: reputable crawlers like Googlebot follow it, but nothing forces compliance, and the file itself is public. That means it is the wrong tool for hiding sensitive URLs, listing a secret admin path in robots.txt actually advertises it. Use it to manage crawl behavior, and use authentication for anything that must stay private.
At the root of the domain, so it is reachable at /robots.txt. Crawlers do not look for it anywhere else.
No. It is a voluntary standard. Well-behaved crawlers respect it, but it is not access control and the file is publicly readable.
Yes. A Sitemap line pointing at your sitemap.xml helps search engines discover your pages faster.