It has been an exciting weekend. The fist cycle ended Sunday afternoon, all my crawlers had preformed well with zero exceptions recorded. The master node launched the CentiverseBots again and instructed the organizer nodes to use the exact same crawling scheme as before.
The only different from the first cycle is the memory map structure a CentiverseBots is now trawling with when re-visiting a site. According to my calculation this memory map should reduce a crawl cycle by 40% or more thanks to the known structures and statistics gained from previous cycles.
In theory these maps will automatically reduce core pressure added through crawler nodes as they currently are able to filter in between noise and silence. So what are these memory maps? It’s actuarial a neural network decision pattern which can analyze the actual content of a given page at runtime. This will give the CentiverseBots the authority to determine when content has changed enough so a re-index is required.