The following pages are not able to be crawled as they are part of the crawling process:
robots.txt
sitemap.xml
The sitemap is used by the crawling process to navigate your site and the robots.txt file is used to list any exclusions to the crawling process.
📌 Note: If you receive an error report showing these are unable to be crawled, keep in mind that this is expected and normal behaviour.
