Linux Performance Analysis

Since I am a shadowman certified; all my testing and workout happening with my x-86 workstation loaded with RedHat Advanced Server-4 (2.6.9-5)

Well… before start off the session; I want to share one thing which I have read on a Performance Tuning book (System Performance Tuning-second edition by Gian Paolo & Mike). The book says five rules that we should keep in mind before going for the analysis.

0. Understand Your Environment
1. TANSTAAFL :- There is Ain’t No Such Thing As A Free Lunch
2. Throughput vs Latencey
3. Do Not Over Utilize A Resource *
4. Design Test Carefully

*Over Utilization is a dangerous thing. As a general rule of thumb, something should be not greater than 70% busy or consumed at any given time : this will provide a margin of safety before serious degradation occures.

So give your own thought to know how well the above rule gonna gel with the Performance Analysis.

Talking about performance on a Linux box mainly deals with its Processor/CPU, Main Memory, Block Device (I/O) and Network etc…

Here I am trying to use some of the basic commands which help to know where the performance gone wrong.

0. Processor/CPU

command $ vmstat 5


(r) - The number of process in runnable and un-interruptable state
(b) - The number of process in blocking or sleep state


swapd - Amount of memory swpped
free - Amount of free memory
buff - used for buffers
cache - used for filesystem cache


Si - Amount of memory swapped in
So - swapped out


bi - Blocks of data received (in) from a block device/sec
bo - sent (out) to a block device/sec


in - Number of interrupts/sec
cs - Number of context switches/sec


us - The percentage of cpu time spent in user space
sy - The percentage of cpu time spent in Kernel space
wa - waiting for I/O
id - Idle time

++ In the case of multiple cpu; the command “vmstat” shows the average and if it is not accurate (when a single threaded heavy process is running)

command $ mpstat -P ALL 5
can be used to show the cpus statistics on an aggregated and on a per cpu basis.

1. Main Memory

Adding more memory to the system may help ya out, if the system shows high availability of Swap Memory but free and cache memory values shows low. To make sure, have a look into swap-in (So) and swap-out (So) numbers.

2. Block Device

If the Block Device is the culprit for the systems performance, you may see many processes are in blocking state (b), the cpu wait (wa) times are high and I/O numbers (bi/bo) are consistently high.

The commands ” iostat -x 5 ” or “lostat -x ” help us to dig more into the I/O details

%util - The percentage of cpu time used to issue the requests to the device.
await - The average time (request) taken to complete the result; including the time spent in the queue.
svctm - Service Access Time, the time which the device takes to service the requests.

The high in numbers for the above parameters tells the device is too dump for the current load.

3. Network

command $ netstat -ci

The output of the command shows the traffic in each interface. The first set of result shows the aggregate total since the last system boot and the subsequent ones are deltas/sec.

The switches
-i : Interface
-c : Continuous
-s : Display summary statistics for each protocol
-r : Kernel routing table

4. Hard Drive Access Tuning

command # hdparm (Hard Disk Parameters)

The following techniques can be applied to (E) IDE drives to utilize 32 bit I/O and DMA more effectively but carefully.

# hdparm /dev/hda for a breif info.
#hdparm -I /dev/hda for a detailed info.
#hdparm -tT /dev/hda for the testing and reading the speed.

[-t] : Timing buffered disk reads; xxMB/sec
[-T] : Timing buffered-cache reads; xxMB/sec (Results normally in 02 seconds)

To increase the speed :-

#hdparm -d1 -c3 -u1 /dev/hda

-d1 : Turn on DMA [ 0 for OFF]
-c3 : IDE chipset support [0 - DISABLE, 1 - ENABLE, 3 - 32 BIT TRANSFER]
-u1 : Get/set interrupt ‘umask’. Drive ignores other interrupts during processing of interrupts [0 - OFF]

file : /etc/sysconfig/harddisk

[-r] : Get/set read-only flag for device.
[-R] : Register an IDE device {careful} (expected hwif-ctrl).
[-U] : Un-register an IDE device {careful}
[-C] : Check the current IDE power mode status, normally “active/idle”
[-S] : Set standby (spindown) timeout for the drive. The value is used by the drive to determine, how long to wait (with no disk activity) before turnig off the spindle motor to save the power.
[-Y] : Force to “sleep mode” (soft/hard reset needed but not for linux IDE HDD).
[-y] : Force to “standby mode” (spind down).

5. Kerenel Level

If you still hanging on Linux kernel-2.2, changing the ‘min_percent’ variable may help you to some extend. This variable represents the system memory available for caching.

#cat /proc/sys/vm/buffermem

2 10 60

To change the default value

# echo “5 10 60″ > /proc/sys/vm/buffermem

5 – min-percent.
10 – max-percent.
60 – borrow-percent.

Its always acceptable, have a try. I tried it and I know there are much more powerful tools to workaround.

Leave a Reply