How to Determine the cause of a Server Usage Spike

In this article we’ll review what steps you can take if your server’s load average is spiking, to help determine the root cause of the issues.

For these examples, you would need to be on a VPS (Virtual Private Server), or dedicated server so that you have SSH access to the server to run commands on the command line.

Determining the cause of Server Usage Spike

1. Login to your server via SSH.
2. Check on the load average of your server over a minute with the following command: sar -q 5 12
   02:10:06 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 02:10:11 PM 1 112 1.29 1.36 1.43 02:10:16 PM 3 109 1.27 1.35 1.43 02:10:21 PM 3 108 1.41 1.38 1.44 02:10:26 PM 4 118 1.62 1.42 1.45 02:10:31 PM 0 108 1.73 1.45 1.46 02:10:36 PM 4 119 1.67 1.44 1.46 02:10:41 PM 2 122 1.69 1.45 1.46 02:10:46 PM 0 113 1.64 1.44 1.46 02:10:51 PM 2 112 1.59 1.44 1.46 02:10:56 PM 0 103 1.46 1.41 1.45 02:11:01 PM 1 102 1.42 1.40 1.44 02:11:06 PM 0 97 1.31 1.38 1.44 Average: 2 110 1.51 1.41 1.45

   This will run the sar command with the -q flag that shows load averages.

   The 5 tells it to run a check every 5 seconds, and the 12 tells it to do it 12 times.

   If your ldavg-1 column stays consistently high, or continues to rise during this load check, this is an indication that you could have something on the server spiking its usage.

3. Now we noticed from looking at our load averages that at 2:10:11 PM the load was 1.29, and it continued to spike up till 02:10:31 PM where the load got as high as 1.73.

   It’s very common that websites being accessed, and having to run PHP scripts, or other server side code can cause these spikes in your usage. So you can check in your Apache access logs for what might have been going on around the time of the spike.

   Using the following code we are going to look at our website’s access log to see how many “hits” happened from 2:09PM – 2:10PM (14:09 – 14:10). This way we can see the requests leading up to the spike, as well as after:

   
   egrep "15/Jan/2013:14:09|15/Jan/2013:14:10" /home/userna5/access-logs/example.com | wc -l
502

   So here we can see that there were 502 requests over those 2 minutes. We can take this a bit further even and break down how many requests happened per minute with this code:

   
   egrep "15/Jan/2013:14:09|15/Jan/2013:14:10" example.com | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2":"$3}' | sort -nk1 -nk2 | uniq -c | sed 's/[ ]*//'

164 14:09
338 14:10

   This right here is already a pretty telling sign, our load average started to spike at 2:10 PM (14:10) and during that minute we had almost double the amount of requests to our site as the previous minute. So it would make sense that the server is having to work harder to serve more requests.

4. Now comes the part where we take an even deeper look at what was going on with the requests. Because the server can easily handle 100 or so image or plain HTML page requests with less of a usage spike than having to run PHP scripts for instance, it’s important to know exactly what is getting requested.

   We can use the following command in order to see what duplicate requests have been happening:

   egrep "15/Jan/2013:14:09|15/Jan/2013:14:10" example.com | cut -d" -f2 | awk '{print $1 " " $2}' | cut -d? -f1 | sort | uniq -c | sort -n | sed 's/[ ]*//' 15 GET /wp-content/plugins/s2member/s2member-o.php

22 GET /about-us/
26 GET /wp-content/uploads/2012/06/logo.png

   Here we can see that this happens to be a WordPress site, the highest duplicated request is a logo.png image so that’s probably not going to cause a load spike. However the 22 requests for /about-us/, and 15 for /wp-content/plugins/s2member/s2member-o.php in a 2 minute period might have.

    

   Taking a look at this WordPress site, I noticed that there is currently no form of caching enabled such as using the W3 Total Cache plugin. As such that means that each time the /about-us/ page is getting requested, the server is going to have to re-process the PHP script, connect to the database, and retreive the page. So here we were able to determine within a few minutes of a server load spike that our possible culprit of that spike was a sudden influx in requests for a WordPress page that isn’t cached.

You should now have a basic understanding of how to track down the possible cause of a server load spike. Now you might also be interested in reading our articles about advanced server load monitoring, or about how to create a server load monitoring bash script to alert you via email when your server’s load is spiking.