• The robot is used for crawling.
• The robot will obey the robots.txt (otherwise, please drop me an email)

How to control the CentiverseBot

User-agent: * Disallow: /

This will keep away the CentiverseBot and all others that obey.

User-agent: CentiverseBot Disallow: /

If you want to keep only CentiverseBot out of your site use the following syntax;

User-agent: CentiverseBot Disallow: /private_directory_name/

If you want to keep CentiverseBot out of a directory use the following syntax;

User-agent: CentiverseBot Disallow: /private_filename.html

If you want to keep CentiverseBot away from a file use the following syntax;

User-agent: CentiverseBot Disallow: .xhtml

If you want to keep CentiverseBot away form a filetype uses the following syntax;

CentiverseBot-investigator

This Bot is not performing any crawling nor is the downloaded content used. The use of this robot is to verifier a given top link is real, and has some sort of content. Furthermore it will perform some speed measurements towards the server. It will not look for the robots.txt file.