Robots.txt File and XML Sitemap
Hibu Websites • June 22, 2020
The robots.txt file tells web robots (search engines) which pages to crawl (i.e. visit and recognize) on a site. Only pages marked "Disallow" will not be crawled.
All Hibu Smart and NextGen Websites include this robots.txt file:
All Hibu Smart and NextGen Websites include an XML Sitemap.
Visit https://ssd.websites.hibu.com/sitemap.xml to see this XML Sitemap associated with this site.
The robots.txt file notifies the search engines bots of the XML Sitemap (sitemap.xml) file — a “blueprint” of every page and blog post on a site.
Sitemap https//www._____.com/sitemap.xml instructs the web crawler to visit the associated /sitemap.xml. So, every time a bot visits one page of a NextGen or Smart Site, it jumps to the sitemap.xml file and re-indexes every page of the site, and any changes on those pages.