Monitoring Linux Disk load

Using iostat you can monitor disk load on a specific device.

iostat -dx {device name} 2 10

Examples:
iostat -dx /dev/sdc 2 10

The 2 is the interval and the 10 is the number of iterations. So you will see stats every 2 seconds for 10 iterations.

du command

I was finally able to harness the power of the du command. The -s flag has a cryptic definition in the man page (nothing new for man there). But using the -s flag allows you to summarize the file size for directories.


This command will give you a summary of the directory sizes in root. You can of course target down to other folders like so

Update 8/17/2015

Often I need to sort the output by size. Here you go.

Deleting files older than X on Linux

The following is handy if you need to delete files older than X on a Linux server. I like to run the find command first before actually moving or deleting the files. Just to make sure the files I want to remove are returned from the find command.

find /fullpath -mtime +4 -ls

The -ls flag will list out the details of the file so you can verify that the mtime parameter returns the right files.

find /fullpath -mtime +4 -ls -exec mv {} /destination ;

After you verify that the find command returns the files you want then you are ready to use the -exec flag on the find command to actually execute the move/delete. I also like to use full path here on the find command, just to make sure I move the right files.

Using VMSTAT

vmstat = virtual memory stats

Procs

• r - the number of processes waiting for run time


• b - the number of processes in uninterruptible sleep

Shows you if there are processes blocking your system from running smoothly. The b column shows processes in sleep state. If these numbers are high then you have some waiting processes and may have some problems.

Example r = 5 means that 5 processes are currently waiting to execute. When the CPU is pegged at 100% the severity of the CPU starvation wont be reflected in the percentage of CPU utilization but the run queue will clearly show the impact as processes back up in the queue

b can be used to show CPU power…if the value is constantly greater than zero then you may have a CPU bottleneck. Use ps to list out the most CPU intensive processes.

Memory

• swpd - amount of virtual memory used


• free - freem memory in the system


• buff - the amount of memory used as buffers


• cache - the amount of memory used as cache.


• inact - the amount of inactive memory. (-a option)


• active - the amount of active memory. (-a option)

Swap

• si - memory pages in


• so - memory pages out

If you have a lot of activity in so it may indicate that you need more physical ram. the system is constantly paging memory out to disk. you need to take these stats over a period of time to get a feel for how your system performs.

If the values are constnaly greater than 0 then you have a memory issue. use ps to find out the most memory intensive processes.

IO

• bi - Blocks received from a block device (blocks/s).


• bo - Blocks sent to a block device (blocks/s).

System

• in - The number of interrupts per second, including the clock.


• cs - The number of context switches per second.

CPU

• us - Time spent running non-kernel code. (user time, including nice time)


• sy - Time spent running kernel code. (system time)


• id - Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.


• wa - Time spent waiting for IO. Prior to Linux 2.5.41, included in idle.


• st - Time stolen from a virtual machine. Prior to Linux 2.6.11, unknown.

If us is high there is a possibility that some of the user initiated processes are consuming high CPU

If wa is high then there is an issue with the disk storage subsystem and you can use iostat to find sources of I/O contention.

Adding Disk to Linux

This procedure can be vastly different than the steps below. This is just a quick overivew on how to add a new disk to Linux.

Patitioning
#list devices
fdisk -l

#invoke fdisk for the desired drive
fdisk /dev/[device]

#Create a new partion
Command (m for help): n

#Select primary partition and size to use the entire disk
Command (m for help): p
Partition number (1-4): 1
First cylinder (1-1305): 1
Last cylinder or +size or +sizeM or +sizeK (1-1305): 1305

 #Commit changes
Command (m for help): w

 Formatting
#show existing file systems (for finding the fs type if you are just duplicating existing configuration)

df -hT

#find block size of a file system
tune2fs -l /dev/[partition name] | grep ‘Block size’

#mkfs.ext3 -b [block size] /dev/[partition name]

 Mounting

#First create a mount point
mkdir /u01

#then mount it
mount /dev/[partiton name] /u01

#verify the fs is writeable
touch /u01/test.file ; ls /u01

#set the partition to be mounted on boot by adding a line to /etc/fstab.  This is the example fstab file…

#/dev/VolGroup00/LogVol00  /                   ext3      defaults              1 1
#LABEL=/boot                   /boot              ext3      defaults              1 2
#tmpfs                            /dev/shm         tmpfs     defaults              0 0
#devpts                          /dev/pts          devpts    gid=5,mode=620   0 0
#sysfs                           /sys                 sysfs      defaults              0 0
#proc                             /proc               proc      defaults               0 0
#/dev/VolGroup00/LogVol01 swap             swap     defaults               0 0

vi /etc/fstab
/dev/[partition name] /u01                       ext3       defaults           0 3

makewhatis

So that took longer than it should. There is a command called whatis that gives you a single line description of a command:

whatis grep

grep (1) - print lines matching a pattern
grep (1p) - search a file for a pattern
grep (rpm) - The GNU versions of grep pattern matching utilities

A nice little tool to use. I just installed CentOS 5 in a VM and noticed that all my whatis commands were returning nothing appopriate…I quickly found the makewhatis command on some articles but could not locate it for CentOS 5 at first. Finally found the location:

/usr/sbin/makewhatis

This command builds the database and now my whatis commands are working.

Using PS to find resource intensive processes

#Top (X) Memory Consumers
#just change the head command to see more, this will show you top 10 

ps -auxf | sort -nr -k 4 | head -10

#Top (X) CPU Consumers
#just change the head command to see more, this will show you top 10

ps -auxf | sort -nr -k 3 | head -10

Linux uptime and version info

#uptime:

uptime

output: current time, how long system has been running, how many users are logged in, system load averages for past 1, 5, and 15 minutes

#Version info

uname -a
cat /etc/redhat-release
cat /etc/issue

df output format CSV

# File size on your drives in CSV format
# no header on the DF command, append the date to each row
# good for throwin in Excel and doing some trending.

df -Pk | awk '{print $1,$2,$3,$4,$5,$6,strftime("%Y-%m-%d")}' OFS="," | sed -e 1d