Short Tutorial on Linux shell-scripting

Short Tutorial on Linux shell-scripting

Content

Content ... 1

What is a Shell Script?... 2

Defining the Shell Type... 2

Types of Shells ... 2

Four Types of Lines... 2

Shell Variables... 4

Calculations ... 5

Preset Shell Variables ... 5

Special variables ... 6

Array variables... 6

Branching (if statement)... 7

Comparisons ... 8

Search with "grep" ... 8

Loops... 9

while do done... 9

until do done ... 9

for var in list do done... 9

continue...break ... 9

Command Line Arguments ("positional parameters") ... 10

Variable Manipulations ... 11

Removing something from a variable... 11

Replacing a variable if it does not yet exits ... 11

Exiting and stating something if variable is not set... 11

Just check for the variable ... 11

KSH Regular Expressions ... 11

Functions ... 12

Return ... 12

Data Redirection... 12

Output from the command line into a file... 12

Standard error redirection ... 13

Input from a file into the command line ... 13

Combine input and output redirection ... 13

Commands into program... 13

What is a Shell Script?

Shell script is a text file that contains commands which could have been typed directly into the shell (There is no difference in syntax between interactive command line use and placing the commands in a file). A shell script can be as simple as a sequence of commands that you type regularly. By putting them into a script, you reduce them to a single command.

Defining the Shell Type

Any shell can be used for writing a shell script. To allow for this, the first line of every script is:

#!/path/to/shell (e.g. #!/bin/ksh or #!/bin/bash). To make a ksh script crate a new file with a starting line like: #!/usr/bin/ksh

It is important that the path to the ksh is proper and that the line does not have more than 32 characters. The shell from which you are starting the script will find this line and hand the whole script over to ksh. Without this line the script would be

interpreted by the same type of shell as the one, from which it was started. NOTE: since the syntax is different for all shells, it is necessary to define the shell with that line.

Types of Shells

sh - "Bourne" shell, written by Steve Bourne for Unix. Simple, very few internal commands, calls external programs for even the simplest of tasks. It is always available in any Unix operational system.

csh - "C" shell, written by Bill Joy. Like sh + many enhancements to improve interactive use. The internal commands are different from sh and similar to the "C" language syntax.

tcsh - "TC" or “TURBO csh” shell. Based on csh + many additional features to make interactive use more convenient. Freely available.

ksh - "Korn" shell, written by David Korn. Is an upgrade to sh + additional internal commands for the most frequently used functions + commands of tcsh for interactive use (for example the command line history).

bash - "Bourne again" shell. Default shell for Linux and Mac OS-X. A functional clone of sh + with additional features to enhance interactive use + and (partial)

compatibility with ksh.

zsh - freeware functional clone of sh, with parts of ksh, bash and full POSIX

compliance, and many new interactive command-line editing features. It was installed as the default shell on early MacOSX systems.

Four Types of Lines

A script has four types of lines: The shell defining line at the top, empty lines, commentary lines starting with a # and command lines. See the following top of a script as an example for these types of lines:

#!/usr/bin/ksh # Commentary... file=/path/file if [[ $file = $1 ]];then command fi

# - as the first non-whitespace character on a line flags the line as a comment, and the rest of the line is completely ignored. Use comments liberally in your scripts, as in all other forms of programming.

\ - as the last character on a line causes the following line to be logically joined before interpretation. This allows single very long commands to be entered in the script in a more readable fashion. You can continue the line as many times as needed.

; - separator between words on a line is interpreted as a newline. It allows you to put multiple commands on a single line:

print -n "Name: "; read name; print ""

One can continue commands over more than one line with a "\" immediately followed by a newline sign which is made be the return key:

grep filename | sort -u | awk '{print $4}' | \ uniq -c >> /longpath/file

The script must not have a name which is identical to a Unix command: So the script must NOT be called "test"!

Prior to implementing the script the script-file filename should be made executable using chmod:

chmod 700filename or

chmod u+x filename

Example 1.

Create file myscript.sh echo> myscript.sh Make the file executable chmod u+x myscript.sh

Edit the file, write into myscript.sh the following commands: #!/bin/sh

print Current date is: date

print Current directory is: pwd

print Total space used is: du –ds

./myscript.sh

Example 2. #!/bin/ksh

# For the purposes of display, parts of the script have # been rendered in glorious technicolor.

## Some comments are bold to flag special sections # Line numbers on the left are not part of the script. # They are just added to the HTML for reference.

# Built-in commands and keywords (e.g. print) are in blue # Command substitutions are purple. Variables are black print "Disk usage summary for $USER on `date`"

# Everything else is red - mostly that is external # commands, and the arguments to all of the commands. print These are my files: # end of line comment for print

# List the files in columns ls -C

# Summarize the disk usage print

print Disk space usage: df -h .

exit 0

Shell Variables

By using variables, a script can be made generic and applicable to different

situations. Variable names consist of letters, numbers and underscores ([a – z, A - Z, 0 – 9, _], cannot start with a number, and are case sensitive. Several special

variables (always uppercase names) are used by the system, thus, resetting these may cause unexpected behavior. Some special variables may be read-only. Using lowercase names for your own variables is safest.

When filling into a variable then one uses just its name: state="US" and no blanks. There is no difference between strings and numbers: price=50.

state="US" # string variable price=50 # numeric variable

When using a variable one needs to put a $ sign in front of it: print $state $price. Example 3.

#!/bin/ksh

echo "Example 3: operations with names and values of variables"

x=5 y=10

echo "x = $x" echo "y = $y"

z=xy

echo "z = xy = $z"

# concatenation of variables names and symbol "+" z=x+y

echo "z = x+y = $z"

# concatenation of variables values and symbol "+" z=$x+$y

echo "z = $x+$y = $z"

# calculating sum of numerical values of variables let "z=x+y"

echo "let "z=x+y" -> z = $z"

Calculations

Simple calculations are done with either a "let" in front of it or within (( ... )). One can increment a variable within the (( )) without a "$":

(( a+=1 )) or

let a+=1

For more complex calculations use "bc":

$result=$(print "n=1;for(i=1;i<8;i++)n=i*n;n"|bc)

Preset Shell Variables

Several special variables are used by the system - you can use these, but can not change them. Some variables are set by the login process and inherited by the shell (e.g. $USER), while others are used only by the shell.

Try running set or env and you will see a long list of all preset variables fro your current session.

Login environment:

$USER, $LOGNAME - the currently logged-in username.

$PATH - the list of directories that will be searched for external commands. You can change this in a script to make sure you get the programs you intend, and don't accidentally get other versions which might have been installed.

$EDITOR - if set, this contains the name of the program which the user prefers to use for text file editing. A program which needs to have the user manually edit a file might choose to start up this program instead of some built-in default (e.g. "crontab -e". This also determines the default command-line-editing behaviour in interactive shells. Shell internal settings:

$PWD - the current working directory (readonly)

$OLDPWD - the previous directory (before the most recent cd command). ksh/bash additional features:

$SECONDS (readonly) - integer number of seconds since this shell was started. $RANDOM - returns a random integer in the range 0-32k; if set to "seed" the random number generator.

Example 4. #!/bin/ksh

echo "Example 4: set default editor programm" echo "current default editor programm is $EDITOR " EDITOR=/e/rhome/routkli/bin/vi

echo "current default editor programm is $EDITOR "

Special variables

$# Number of arguments on command line. $? Exit status of last command.

$$ Process id of current program.

$! Process id of last background job or background function.

$0 Program name including the path if started from another directory. $1..n Command line arguments, each at a time.

$* All command line arguments in one string.

Array variables

Another type of variable supported by the ksh is the array. An array contains a collection of values (elements) that may be accessed individually or as a group. When assigning or accessing array elements, a subscript is used to indicate each element's position within the array. The subscript is enclosed by brackets after the array name:

arrayname[subscript]

The first element in an array uses a subscript of 0, and the last element position (subscript value) is dependent on what version of the ksh you are using. Review your system's ksh man page to identify this value.

In the following example the colors red, green, and blue are assigned to the first three positions of an array named colors.

$ colors[0]=RED $ colors[1]=GREEN $ colors[2]=BLUE

Alternatively, the same assignments can be done using a single command: $ set -A colors RED GREEN BLUE

Adding a dollar sign and an opening brace to the front of the general syntax and a closing brace on the end allows you to access individual array elements:

Using the array we defined above, each array element can be accessed one by one: print ${colors[0]}

print ${colors[1]} print ${colors[2]}

If you access an array without specifying a subscript, 0 will be used: $ print ${colors[]}

Will result in: RED

The while construct can be used to loop through each position in the array: i=0 while [ $i -lt 3 ] do print ${colors[$i]} (( i=i+1 )) done

Branching (if statement) if then fi

if [[ $value -eq 7 ]];then print "$value is 7" fi or: if [[ $value -eq 7 ]] then print "$value is 7" fi or:

if [[ $value -eq 7 ]];then print "$value is 7";fi if then else fi

if [[ $name = "John" ]];then

print "Your welcome, ${name}." else

print "Good bye, ${name}!" fi

if then elif then else fi

if [[ $name = "John" ]];then

print "Your welcome, ${name}." elif [[ $name = "Hanna" ]];then

print "Hello, ${name}, who are you?" else

print "Good bye, ${name}!" fi

case esac case $var in

john|fred) print $invitation;; martin) print $declination;;

*) print "Wrong name...";; esac

Comparisons

To compare strings one uses "=" for equal and "!=" for not equal.

To compare numbers one uses "-eq" for equal "-ne" for not equal as well as "-gt" for greater than and "-lt" for less than.

if [[ $name = "John" ]];then # commands....

fi

if [[ $size -eq 1000 ]];then # commands....

fi

With "&&" for "AND" and "||" for "OR" one can combine statements: if [[ $price -lt 1000 || $name = "Hanna" ]];then # commands....

fi

if [[ $name = "Fred" && $city = "Denver" ]];then # commands....

fi

Example 5. #!/bin/ksh

echo "Example 5: construct if then else" x=33

y=20

echo "x = $x" echo "y = $y"

if [ $x -ge $y ]; then echo "x = $x"

else

echo "y = $y" fi

Search with "grep"

Search for the occurrence of a pattern in a file: grep 'pattern' file. If one just wants to know how often something occurs in a file:

grep -c 'pattern file This can be used in a script like:

if [[ $(grep -c 'pattern' file) != 0 ]];then ...;fi. The condition is fulfilled if the pattern was found.

Loops

while do done

while [[ $count -gt 0 ]];do print "\$count is $count" (( count -= 1 ))

done

until do done

until [[ $answer = "yes" ]];do

print -n "Please enter \"yes\": " read answer

print "" done

for var in list do done for foo in $(ls);do

if [[ -d $foo ]];then

print "$foo is a directory" else

print "$foo is not a directory" fi

done

continue...break

One can skip the rest of a loop and directly go to the next iteration with: "continue". while read line

do if [[ $line = *.gz ]];then continue else print $line fi done

One can also prematurely leave a loop with: "break". while read line;do

if [[ $line = *!(.c) ]];then break else print $line fi done Example 6. #!/bin/ksh

echo "Example 6: while loop" a=0

amax=5

while [ ${a} -lt ${amax} ] do

let "a = a + 1" echo "a = $a" done

Command Line Arguments ("positional parameters")

The number of command line arguments is stored in $# so one can check for arguments with:

if [[ $# -eq 0 ]];then print "No Arguments" exit

fi

The single Arguments are stored in $1, ....$n and all are in $* as one string. The arguments cannot directly be modified but one can reset the whole command line for another part of the program.

If we need a first argument $first for the rest of the program we do: if [[ $1 != $first ]];then

set $first $* fi

One can iterate over the command line arguments with the help of the shift command. Shift indirectly removes the first argument.

until [[ $# -qe 0 ]];do # commands ....

shift done

One can also iterate with the for loop, the default with for is $*: for arg;do

print $arg done

The program name is stored in $0 but it contains the path also!

Example 7. #!/bin/ksh

echo "Example 7: find minimum and maximum in array a" set -A a 20 10 2 33 103 333 30 74 2 0

n=${#a[*]}

echo "Array a = ${a[*]} "

echo "There are $n elements in this array" amin=${a[0]}

amax=${a[0]}

echo "Initialization minimum = $amin maximum = $amax"

for (( i=0 ; i<n ; i++)) do

amin=${a[$i]} fi

if [ ${a[$i]} -gt $amax ]; then amax=${a[$i]}

fi done

echo "Real minimum = $amin maximum = $amax"

Variable Manipulations

Removing something from a variable

Variables that contain a path can very easily be stripped of it: ${name##*/} gives you just the filename.

Or if one wants the path: ${name%/*}. % takes it away from the left and # from the right.

%% and ## take the longest possibility while % and # just take the shortest one. Replacing a variable if it does not yet exits

If we wanted $foo or if not set 4 then: ${foo:-4} but it still remains unset. To change that we use:

${foo:=4}

Exiting and stating something if variable is not set

This is very important if our program relays on a certain variable: ${foo:?"foo not set!"}

Just check for the variable

${foo:+1} gives one if $foo is set, otherwise nothing.

KSH Regular Expressions

ksh has its own regular expressions.

Use an * for any string. So to get all the files ending with .c use *.c.

A single character is represented with a ?. So all the files starting with any sign followed bye 44.f can be fetched by: ?44.f.

Especially in ksh there are quantifiers for whole patterns: ?(pattern) matches zero or one times the pattern.

*(pattern) matches any time the pattern.

+(pattern) matches one or more time the pattern. @(pattern) matches one time the pattern.

So one can question a string in a variable like: if [[ $var = fo@(?4*67).c ]];then ...

Functions

A function (= procedure) must be defined before it is called, because ksh is

interpreted at run time. It “knows” all the variables from the calling shell except the command line arguments. But a function has its own command line arguments so that one can call it with different values from different places in

the script. It has an exit status but cannot return a value like a C-function does. One can make one in either of the following two ways:

function myfunction { # commands... } myfunction(){ # commands... }

To call it just put its name in the script: my_function

To give it arguments:

my_function arg1 arg2 ...

The arguments are there in the form of $1...$n and $* for all at once like in the main code.

And the main $1 is not influenced bye the $1 of a particular function. Return

The return statement exits the function immediately with the specified return value as an exit status.

Data Redirection

Data redirection is done with the following signs: ">", ">>", "<", "<<". Every program has at least a standard-input, standard-output and standard-error-output. All of these can be redirected.

Output from the command line into a file

For writing into a new file or for overwriting a file: command > file For example:

echo “Hello!” > newfile.txt

The command echo will create a new file with filename ‘newfile.txt’ that contains the text “Hello!”

For appending to a file: command >> file For example:

echo “New line” >> newfile.txt

The command echo will add the text “New line” into the file ‘newfile.txt’; now the file ‘newfile.txt’ contains:

Hello! New line

Standard error redirection

To redirect the error output of a command: command 2> file To discard the error all together: command 2>/dev/null

To put the error to the same location as the normal output: command 2>&1 Input from a file into the command line

If a program needs a file for input over standard input: command < file Combine input and output redirection

command < infile > outfile

command < infile > outfile 2>/dev/null Commands into program

Every Unix command can take its commands from a text like listing with: command <<EOF

input1 input2 input3 EOF

From eof to eof all is feed into the above mentioned command.

Pipes

For a serial processing of data from one command to the next: command1 | command2 | command3 ...