Very limited capacity SharePoint 2007 Index Server (indexing own content – i.e. it is also a Web server):
Index corpus = 1841
With unlimited concurrency (Crawler Impact Rules) and full crawl took well over 10 minutes – in fact, it had only indexed around 400 items in this time. Performance increased when Impact Rules were used to throttle crawling.
- 2 Concurrent Requests: Full Crawl in 07:30 (min)
- 4 Concurrent Requests: Full Crawl in 06:25
- 8 Concurrent Requests: Full Crawl in 06:00
- 16 Concurrent Requests: Full Crawl in 05:55
Not particularly scientific, but it looks like the ‘sweet spot’ has been found at 8 Concurrent Requests.