txt file is then parsed and will instruct the robot concerning which internet pages are not to be crawled. Being a search engine crawler may hold a cached copy of the file, it could once in a while crawl web pages a webmaster won't want to crawl. Internet pages typically prevented from remaining crawled include things like login-certain pages such