While doing some load-tests recently, I had the need to be able to mimic some crawler traffic on a large site. Something that might hit atypical URLs from time to time and not just the top pages.

Surprisingly I found it difficult to find many helpful articles here that didn't involve going through the entire sitemap, so heeeerrrrre we are :)

STEP 1 - Don't waste our time:

Get JMeter installed (D'uh!)

STEP 2 - Basic Configuration:

Start with a fresh template and give your Test Plan a Name. Then right-click and select Add -> Config Element -> User Defined Variables

This can be super helpful to change up your tests easily for different hosts, different load test software, etc. For now we'll just give it a name and use it to define our host:

We then want to add in our HTTP Defaults, this is similar in that it can help us make the test more re-usable in the future. Right-click and select Add -> Config Element -> HTTP Request Defaults

Then put in the following based on your host, these are mine:

• Protocal: https

• Server Name or IP: ${host}

• Port Number: 443

STEP 3 - Thread Group:

Right click your plan and select Add -> Threads -> Thread Group This will allow you to define how many users, loops etc. For now set this as 1's, but once you get the idea you can also use variables defined in Step 2 to make this configurable by variable.

STEP 4 - Extract Your Sitemap:

Right-click the thread group you just created and select Add -> Sampler -> HTTP Request . Give it a name and in the path put your sitemap url relative to the host in step 2: i.e /sitemap.xml

Now right click the HTTP Request, select Add -> Post Processors -> XPath Extractor

Enter The following:

This will place the results of your sitemap into a variable called "urls".

STEP 5 - Random URLs:

Now in my testing, I wanted each user to hit several URLs from the sitemap rather than a run of JMeter getting the entire sitemap just to hit a single URL, so I created a Loop Controller to hit 25 URLs at random per user.

First, create a loop controller under your Thread Group with Add -> Logic Controller -> Loop Controller and give it a loop value (i.e. 25)

Then right-click the new Loop Controller and select Add -> Config Element -> Random Variable

NOTE: I did 10,000 here because my sitemap has that many URLs, adjust as necessary.

Now again right-click the Loop Controller and select Add -> Sampler -> HTTP Request

• Give it the name: ${__V(urls_${random})}

• Give it the path: ${__V(urls_${random})}

Now under the Advanced tab, if you want to simulate an actual page load, check the box "Retrieve All Embedded Resources", otherwise leave as is.

STEP 6 - Test and Record:

I like to either output to a file, or use View Results Tree or Summary Reports to track test results, but I'll leave that to you. Once you've fine-tuned your test, you can start playing with Users, Variables, Loop Counts, etc and you'll have a nice way to test crawler traffic :)