Skip to main content

Pages that can't be crawled

Find out why it's normal for robots.txt and sitemap.xml files to appear as uncrawled in error reports.

Written by Cristiana Ghinea

The following pages are not able to be crawled as they are part of the crawling process:

  • robots.txt

  • sitemap.xml

The sitemap is used by the crawling process to navigate your site and the robots.txt file is used to list any exclusions to the crawling process.
​

📌 Note: If you receive an error report showing these are unable to be crawled, keep in mind that this is expected and normal behaviour.

Did this answer your question?