Making sure you can be crawled
robots.txt tells search engines and AI crawlers which parts of your site they may visit. One wrong line, often left over from when the site was built, can hide your whole site from Google.
Your XML sitemap lists every page you want found. Listing it in robots.txt and submitting it to Google Search Console and Bing Webmaster Tools gets new pages indexed faster.
Questions
- Do I need a robots.txt file?
- It isn't required, but it's the standard place to point every crawler (Google, Bing, DuckDuckGo and AI tools) at your sitemap.
- My site isn't on Google at all. Why?
- The most common causes are a noindex tag or 'Disallow: /' left over from development. Run this check and the Website SEO Audit together to find out.