Verify that the robots.txt is correct and not blocking webpages.