Generate a properly formatted robots.txt file to control how search engines crawl your website. Block bad bots, protect private pages and improve crawl efficiency.
Templates auto-fill all settings below. Customise before generating.
Add specific rules for individual bots — useful for giving Googlebot extra access while restricting others.
Use the form above to configure your robots.txt settings. Start with a Quick Template that matches your website type — WordPress, eCommerce, or standard website.
Click "Download robots.txt" to save the file. The file must be named exactly robots.txt — lowercase, no spaces.
Upload robots.txt to your website's root folder (where index.php is located). On Hostinger: hPanel → File Manager → public_html → Upload.
Visit https://www.yoursite.com/robots.txt — you should see the text file in your browser.
Open Google Search Console → Settings → robots.txt Tester → paste your URL and test if Google can read it correctly.
Your robots.txt should reference your sitemap.xml URL so Google can find all your pages. We've added this automatically if you provided your sitemap URL.
A robots.txt file is a simple text file that tells search engine crawlers (like Googlebot) which pages or sections of your website they can or cannot visit. It's placed in the root of your website and is the first file crawlers look for.
No! robots.txt only prevents crawlers from visiting pages — it doesn't hide them from search results. For pages you truly want hidden from search results, use the noindex meta tag instead. Disallowed pages can still appear in Google if other sites link to them.
Generally no — Facebook, Twitter, LinkedIn bots read your pages to generate preview cards when someone shares your link. Blocking them means no rich previews when your content is shared on social media.
Your entire website will stop being indexed by Google. Never add "Disallow: /" under "User-agent: Googlebot" unless you intentionally want your site removed from Google Search.
Google recrawls your robots.txt every 24 hours typically. Changes take effect within 1-2 days. If you make urgent changes, you can request a recrawl via Google Search Console.
robots.txt is not required but is highly recommended. Without it, crawlers will try to access every page including admin panels, checkout pages and private areas — wasting your crawl budget on unimportant pages.
