The project now operates with two types of bots to download and verify content.
I’ve chosen to split them into two separate subsystems based on statistic details collected while running stress tests. When mining the crawl statistics different patterns quickly emerge. One of then told me that sites did come and go faster that I could ever had imagined.
I also realized some new and exciting possibilities when splitting the two systems.
The crawling system would be cleaner and it would operate way faster than before.
I had a total encapsulate verification system which could work completely independent of the real crawlers and only launched if considered necessary.
Controll
CentiverseBot