Pipes and How They Are Used

Look at Figure 4-6 carefully. It shows another powerful feature of the UNIX shell, using the output of one command as input of another command. We call this process piping due to its similarity to the real-world use of a pipe. At the command line, both of the processes (commands) are connected using the vertical bar symbol "|". This symbol is often called a pipe symbol. When two commands are connected through a pipe, the first command sends its output to the pipe instead of sending it to the terminal screen. The second command reads its input from the pipe rather than from the keyboard. Both of the commands still send error messages to the terminal screen, as shown in the figure. The first command takes input from the keyboard (stdin), and the second command sends its output to the terminal screen (stdout). If the second command needs input data but nothing is available in the pipe, it just waits for the first command to send something into the pipe.

Figure 4-6. Use of pipes.

Pipes are often used to filter, modify, or manipulate data output of one command. Multiple levels of pipes can be used in one command line. Similarly, pipes can also be used in combination with I/O redirection symbols.

Use of Pipes as Filters

Many times we don't need all of the output produced by a command. In such a case, we can filter the desired information from the output produced by a command. Filtering means extracting useful data and throwing away the rest. We have already studied the who command, which is used to see the names of logged-in users. In large systems, where hundreds of users are logged in simultaneously, it is difficult to find out whether a particular user is currently logged in. In this situation, we use the filter to get the desired information. We can use the who command with the grep command, where grep acts as a filter. Consider the next example, where we want to find if a user "mike" is logged in. First we use only the who command and then we combine the who and grep commands.

$ who

operator pts/ta Aug 30 16:05 boota pts/tb Aug 30 15:59 mike pts/tc Aug 30 15:44 linda pts/td Aug 30 14:34 $

Now we use a pipe to filter out our required information. $ who | grep mike

mike pts/tc Aug 30 15:44 $

As you can see, only the line containing the word "mike" is now displayed. We have used the grep command previously to find a string from one or multiple files. The grep commands, at that time, used file names as input. In this example, it did the same thing but took its input from the pipe.

How did grep know that no more data were coming from the pipe and that it should stop processing? Well, this is quite simple. The who command sends an end of file (EOF) character when it has completed sending output to the pipe. The grep command checks the EOF character and stops execution when it finds the character. In case there are no data in the pipe and the grep command has not received the EOF character, it will just wait until it gets more data or the EOF character.

As another example, we can get only login names from the who command by using another filter known as cut. We will discuss the cut command in more detail in the last chapter, but for the time being just see how we use it to extract the first word of each line and throw away the rest.

$ who | cut -f 1 -d " " operator boota mike linda $

The cut command takes its input as fields separated by space characters and picks the first field from each input line. Since the first field of all output lines is the login name, we got the login names only in the output.

You can also use multiple levels of pipes as shown below. $ who | cut -f 1 -d " "| grep mike

mike $

Try to explain what is happening here. We have filtered the output of one command and then again filtered the output of the second command. You can continue this process as far as you want.

Use of Pipes for Data Manipulation

As we have used pipes for filtering data, we can also use them for reorganizing and manipulating data. What if you need to get output of a command in sorted form? Yes, it is quite simple if you pass it through the sort command using a pipe. Consider the above example of using the who command. See how the output changes without and with a sort pipe.

$ who

operator pts/ta Aug 30 16:05 boota pts/tb Aug 30 15:59 mike pts/tc Aug 30 15:44 linda pts/td Aug 30 14:34 $

Now we use a pipe with the sort command. $ who | sort

boota pts/tb Aug 30 15:59 linda pts/td Aug 30 14:34

mike pts/tc Aug 30 15:44 operator pts/ta Aug 30 16:05 $

The sort command has arranged the output of the who command in alphabetical order. If there are many users logged in, the output of the who command just scrolls up and you see only the last page. In that case, you can use the more command as a filter to stop the scrolling at the end of each page.

$ who | more

Filters can do many things for you in a very simple way. If you were using some other operating system, you might need to write separate programs!

.6 The T-Junction

This is a special type of pipe similar to a T pipe junction in real life. This is used to redirect incoming data in a pipe to more than one place. Please see Figure 4-7 to get an idea how the T-junction works.

Figure 4-7. The T-junction.

The tee command is used to form a T-junction. It takes its input from stdin and writes the same thing to stdout as well as to another file at the same time. Consider the same example of the who command. If you want to display the output of the who command at the terminal as well as save it in whofile for future use, the command line and result will be as follows.

$ who | tee whofile

operator pts/ta Aug 30 16:05 boota pts/tb Aug 30 15:59 mike pts/tc Aug 30 15:44 linda pts/td Aug 30 14:34 $

Now if we see the contents of the whofile, it will contain the same data. $ cat whofile

operator pts/ta Aug 30 16:05 boota pts/tb Aug 30 15:59 mike pts/tc Aug 30 15:44 linda pts/td Aug 30 14:34 $

Like ordinary pipes and redirection symbols, multiple levels of t-junction pipe can be used to send data to many places. Can you use the sort or head commands with the tee command now? How about using the spell command to check spellings of a command output?

Table 4-2 is a summary of the redirection and pipe symbols used in HP-UX.

Table 4-2. Standard I/O Redirection

Symbol Function Syntax

> Redirect stdout and overwrite or create a file prog > file

< Redirect stdin prog < file

>> Redirect stdout and append to, or create a file prog >> file

2> Redirect stderr prog2> file

| Pipe stdout of prog1 to stdin of prog2 prog1 | prog2 |& Pipe stdout and stderr of prog1 to stdin of

prog2 prog1 |& prog2 Note:prog, prog1, and prog2 represent a command or executable program, while file is any file

Chapter Summary

Whenever a UNIX command is executed, it opens three standard files for standard input and output. UNIX uses file descriptor 0 for standard input (stdin), 1 for standard output (stdout), and 2 for standard error (stderr) messages. In this chapter, you learned the meaning of standard input, output, and error and how to redirect them. You used the output redirection symbol >, input redirection symbol <, and error

redirection symbol 2>. In addition, you learned how to append output of a command to a file with the >> symbol. You also used 2>&1 to redirect standard output and standard error to one place.

Use of pipes was the other important matter discussed in the chapter. You used the pipe symbol "|" and also used the tee command to pipe output of a command to more than one place.

Chapter Review Questions 1

: UNIX is a "file-based" system. What does that mean? 2

: What if we redirect output of a command to a file that already exists? 3

: What if we pipe output of a command to another command that expects nothing as input? 4

: Can we redirect both input and output to same file?

Test Your Knowledge

1: What is the file descriptor used for stderr?

1 0 2 3

2: The symbol used to append to a file when redirecting stdout to that file is:

> >> < 2>

3: When you redirect both stdout and stderr to the same location, you use:

2&> 2&>1 2>&1 1>&2 4: A pipe is used to:

send output of one command to the input of another command filter certain data from the output of a command

reorder output of a command all of the above

5: Which is not true?

Pipes can be used with redirection symbols.

Pipes cannot be used when stdin redirection is used in a command. It is possible to redirect all stdin, stdout, and stderr at the same time.

Chapter Syllabus

5.1 Modes Used in vi