Update robots.txt to improve crawl accessibility
#765
Merged
robots.txt to improve crawl accessibility
#765