Men and women use DA to check out which Web-sites are trusted in their field and to determine exactly where to get superior one-way links. txt file is then parsed and can instruct the robotic as to which pages usually are not for being crawled. As being a search engine https://www.youtube.com/watch?v=0USI3vTZ0aw