While doing some load-tests recently, I had the need to be able to mimic some crawler traffic on a large site. Something that might hit atypical URLs from time to time and not just the top pages.