SEOClerks

Clear
Sign In Join

Spiders that bypass your robots.txt file

  • Beverly Level X3
    • 2

    Write the reason you're deleting this FAQ

    Spiders that bypass your robots.txt file

    Is it possible for a spider (search engine bot) to bypass your robots.txt file?

    # Website Management
    Share post:

    Comments

    Please login or sign up to leave a comment

    Join
    Ethan

    Ethan

    Some people have speculated that Google bots actually ignore your robots.txt file, though most bots are supposed to be bound to respect this file. There is nothing to say they will though.
    11 years ago

    Are you sure you want to delete this post?

    WynDavies

    WynDavies

    A search engine bot will not ignore the robots.txt file. They were programmed to respect and obey this file for the good of the site owner. If you do not have this file on your site, you can expect that a bot will crawl every page it can find.
    9 years ago

    Are you sure you want to delete this post?