Technical SEO: Technical Web optimization refers to website and server optimization that assists the crawler in crawling, indexing and ranking functions to rank your web site improved. txt file is then parsed and will instruct the robot concerning which internet pages aren't being crawled. Being a online search engine crawler https://www.youtube.com/watch?v=V7lEomNLUOE&t=8s