The new server room is now ready – it has been for quite some time now - and the Centiverse is working satisfactorily in the new environment. My only concern is the heavy load on the switches and routers. Therefore I’ am in the process of adding two new IP’s to the system which should rapidly speed things up on the collection interfaces (bots etc.)
Furthermore I've added 3 repository nodes which act like a NLB. Sins all data element know where it should persist it self this add-on really rocks. Until now I have only observed what it did to the crawlers - increased there efficiency by 60%. – I can only imagine what will happen when the system gets 3 IP’s…
The crawling period is almost done and the indexers will soon take over the game. How the system will react regarding the tree repositories when indexing starts will soon be reveal.
[15-oct-2008]
What can I say, this is fast now. The indexing time has been reduced by 2/3. Yesterday, I could see the throughput had increased from around 300 to 2000-2700 request pr. sec on a single language node. I simply needed to see this, so I opened the door into server room and was expecting a lightshow with flashing routers, switches etc. There was no lightshows – well, maybe there was, but my eyes and brain could not register the gabs between the flashes – it was like everything just had the “on” button enabled - this just proves that my design works and it can scale when it’s considered necessary.