#38 Ensuring Maximally Compressed Files
Chapter 5: System Administration: Managing Users Overview
No sophisticated operating system can run itself without some human intervention, whether it's Windows, Mac OS, or Unix. If you use a multiuser Unix system, someone no doubt is performing the necessary system administration tasks. You might be able to ignore the proverbial "man behind the curtain" who is managing and maintaining everything, or you might well be the All Powerful Oz yourself, the person who pulls the levers and pushes the buttons to keep the system running. Even if you have a single-user system, like a Linux or Mac OS X system, there are system administration tasks that you should be performing, whether you realize it or not.
Fortunately, streamlining life for Unix system administrators is one of the most common uses of shell scripting, and as a result there are quite a few different shell scripts that sysadmins use, from the simple to the complex. In fact, there are usually quite a few commands in Unix that are actually shell scripts, and many of the most basic tasks, like adding users, analyzing disk usage, and managing the filespace of the guest account, can easily be done in relatively short scripts.
What's surprising is that many system administration scripts are no more than 20 to 30 lines long, total. This can be easily calculated on the command line for a given directory:
$ wc -l $(file /usr/bin/* | grep "script" | grep -v perl | cut -d: -f1) | \ sort -n | head -15 3 /usr/bin/bdftops 3 /usr/bin/font2c 3 /usr/bin/gsbj 3 /usr/bin/gsdj 3 /usr/bin/gsdj500 3 /usr/bin/gslj 3 /usr/bin/gslp 3 /usr/bin/gsnd 4 /usr/bin/4odb 4 /usr/bin/4xslt 4 /usr/bin/krdb 5 /usr/bin/4rdf 5 /usr/bin/4xupdate 6 /usr/bin/checkXML 6 /usr/bin/kdb2html
None of the shortest 15 scripts in the /usr/bin/ directory are longer than 6 lines. And at 14 lines, the Red Hat Linux 9.0 script /usr/bin/mute is a fine example of how a little shell script can really improve the user experience:
#! /bin/sh
# $Aumix: aumix/src/mute,v 1.1 2002/03/19 01:09:18 trevor Exp $ # Copyright (c) 2001, Ben Ford and Trevor Johnson
#
# Run this script to mute, then again to un-mute. # Note: it will clobber your saved settings. #
volumes=$(aumix -vq |tr -d ,)
if [ $(echo $volumes | awk '{print $2}') -ne 0 -o \ $(echo $volumes | awk '{print $3}') -ne 0 ]; then aumix -S -v 0
else
fi
Like the mute script, the scripts presented in this chapter are short and useful, offering a range of administrative capabilities, including easy system backups, showing what system services are enabled through both inetd and xinetd, an easy front end to the date command for changing the current date and time, and a helpful tool to validate crontab files.
#39 Analyzing Disk Usage
Even with the advent of very large disks and their continual drop in price, system administrators seem to perpetually be tasked with keeping an eye on disk usage to ensure that the system doesn't fill up.
The most common monitoring technique is to look at the /users or /home directory, using the du command to ascertain the disk usage of all the subdirectories, and then reporting the top five or ten users therein. The problem with this approach, however, is that it doesn't take into account space usage elsewhere on the hard disk(s). If you have some users who have additional archive space on a second drive, or sneaky types who keep MPEGs in a dot directory in /tmp or in an unused and accidentally opened directory in the ftp area, they'll escape detection. Also, if you have home directories spread across multiple devices (e.g., disks), searching each /home isn't necessarily optimal.
Instead, a better solution is to get all the account names directly from the /etc/passwd file and then to search the file systems for files owned by each account, as shown in this script.
The Code
#!/bin/sh# fquota - Disk quota analysis tool for Unix.
# Assumes that all user accounts are >= UID 100.
MAXDISKUSAGE=20
for name in $(cut -d: -f1,3 /etc/passwd | awk -F: '$2 > 99 {print $1}') do
echo -n "User $name exceeds disk quota. Disk usage is: " # You might need to modify the following list of directories to match # the layout of your disk. Most likely change: /Users to /home find / /usr /var /Users -user $name -xdev -type f -ls | \
awk '{ sum += $7 } END { print sum / (1024*1024) " Mbytes" }'
done | awk "\$9 > $MAXDISKUSAGE { print \$0 }"
exit 0
How It Works
By convention, uids 1 through 99 are for system daemons and administrative tasks, while 100 and above are for user accounts. Unix administrators tend to be a fairly organized bunch, and this script takes advantage of that, skipping all accounts that have a uid of less than 100.
The -xdev argument to the find command ensures that find doesn't go through all file systems, preventing it from slogging through system areas, read-only source directories, removable devices, the /proc directory of running processes (on Linux), and similar areas.
It may seem at first glance that this script outputs an exceeds disk quota message for each and every account, but the awk statement after the loop allows reporting of this message only for accounts with usage greater than the predefined MAXDISKUSAGE.
Running the Script
This script has no arguments and should be run as root to ensure access to all directories and file systems. The smart way to do this is by using the helpful sudo command (see man sudo for more details). Why is sudo helpful? Because it allows you to execute one command as root, after which you are back to being a regular user. Each time you want to
run an administrative command, you have to consciously use sudo to do so; using su - root, by contrast, makes you root for all subsequent commands until you exit the subshell, and when you get distracted it's all too easy to forget you are root and then type a command that can lead to disaster.
Note You will likely have to modify the directories listed in the find command to match the corresponding directories in your own disk topography.
The Results
Because it's searching across file systems, it should be no surprise that this script takes rather a while to run. On a large system it could easily take somewhere between a cup of tea and a lunch with your significant other. Here are the results:
$ sudo fquota
User linda exceeds disk quota. Disk usage is: 39.7 Mbytes User taylor exceeds disk quota. Disk usage is: 21799.4 Mbytes
You can see that taylor is way out of control with his disk usage! That's 21GB. Sheesh.
Hacking the Script
A complete script of this nature should have some sort of automated email capability to warn the scofflaws that they're hogging disk space. This enhancement is demonstrated in the very next script.
#40 Reporting Disk Hogs
Most system administrators seek the easiest way to solve a problem, and the easiest way to manage disk quotas is to extend the fquota script, Script #39, to include the ability to email warnings directly to users who are consuming too much space.
The Code
#!/bin/sh# diskhogs - Disk quota analysis tool for Unix; assumes all user # accounts are >= UID 100. Emails message to each violating user # and reports a summary to the screen.
MAXDISKUSAGE=20 violators="/tmp/diskhogs0.$$"
trap "/bin/rm -f $violators" 0
for name in $(cut -d: -f1,3 /etc/passwd | awk -F: '$2 > 99 { print $1 }') do
echo -n "$name "
# You might need to modify the following list of directories to match # the layout of your disk. Most likely change: /Users to /home find / /usr /var /Users -user $name -xdev -type f -ls | \ awk '{ sum += $7 } END { print sum / (1024*1024) }'
done | awk "\$2 > $MAXDISKUSAGE { print \$0 }" > $violators
if [ ! -s $violators ] ; then
echo "No users exceed the disk quota of ${MAXDISKUSAGE}MB" cat $violators
exit 0 fi
while read account usage ; do
cat << EOF | fmt | mail -s "Warning: $account Exceeds Quota" $account Your disk usage is ${usage}MB, but you have been allocated only
${MAXDISKUSAGE}MB. This means that you need to either delete some of your files, compress your files (see 'gzip' or 'bzip2' for powerful and
easy-to-use compression programs), or talk with us about increasing your disk allocation.
Thanks for your cooperation in this matter.
Dave Taylor @ x554 EOF
echo "Account $account has $usage MB of disk space. User notified."
done < $violators
exit 0
Note the addition of the fmt command in the email pipeline: cat << EOF | fmt | mail -s "Warning: $account Exceeds Quota" $account
It's a handy trick to improve the appearance of automatically generated email when fields of unknown length, like $account, are embedded in the text. The logic of the for loop in this script is slightly different from the logic of the for loop in Script #39, fquota. Because the output of the loop in this script is intended purely for the second part of the script, during each cycle it simply reports the account name and disk usage rather than a disk quota exceeded error message.
Running the Script
Like Script #39, this script has no starting arguments and should be run as root for accurate results. This can most safely be accomplished by using the sudo command.