Streams, redirection, and filters

Synopsis

Turn three field observations into a count for each species. Run the commands in /bin/sh under an ordinary account. The exercise needs a writable practice directory and three lines of text. Both the input and the expected result appear below.

Prepare the observations

Use the practice directory under the home directory. These commands also create it for a first visit to this chapter.

Shell input
mkdir -p "$HOME/unix-practice"
cd "$HOME/unix-practice"
pwd

Check that the path ends in /unix-practice. Save observations.txt there. At a text console, the following command creates the same file. It replaces observations.txt if that name already exists, so use it only for the practice copy.

Shell input
printf '%s\n' sparrow robin sparrow > observations.txt

Display the file before processing it.

Shell input
cat observations.txt

File contents · observations.txt

sparrow
robin
sparrow

Each line records one sighting. The two sparrow sightings occur on separate lines with a robin between them.

Inspect each filter

Run the counting utility directly on the original file.

Shell input
uniq -c observations.txt

Each count is one. The uniq utility compares adjacent lines. To bring the two sparrow lines together, sort the input.

Shell input
LC_ALL=C sort observations.txt

Expected output

robin
sparrow
sparrow

The variable assignment selects the C locale for sort, giving the example a fixed sorting order. A filter reads input and writes a transformed result. Connect the filters in a pipeline with the vertical bar.

Shell input
LC_ALL=C sort observations.txt | uniq -c

Example output

      1 robin
      2 sparrow

Look for a count of one beside robin and two beside sparrow. The width of the count field can vary. The shell connects the standard output of sort to the standard input of uniq. The final command writes its standard output to the terminal. OpenBSD uniq(1) · Debian sort(1)

Save the report

Choose report.txt for the disposable report. The > operator creates or empties its destination before the last command writes. Check that the destination differs from the input file and that any older report can be replaced.

Shell input
LC_ALL=C sort observations.txt | uniq -c > report.txt

On success, the prompt returns and the terminal displays no counts. Read the destination.

Shell input
cat report.txt

The file contains the same two rows. Repeating the pipeline with > replaces the report. The operator >> appends instead and would duplicate the rows on a second run.

Diagnose a missing input

Use the deliberately misspelled input observatons.txt, with the second i missing. Keep the existing report by leaving off its redirection.

Shell input
LC_ALL=C sort observatons.txt | uniq -c

Expect a message from sort identifying the missing file. A separate stream called standard error still leads to the terminal. The pipe carries standard output, so uniq receives no observations and prints no counts. Inspect the pipeline’s exit status immediately.

Shell input
printf '%s\n' "$?"

In the default practice shell with pipeline failure reporting disabled, expect zero. The last command, uniq, succeeded on empty input. A zero status here gives no assurance that sort opened the file. Check the real name with ls observations.txt and repeat the successful pipeline.

Continue with Processes, jobs, and signals, or follow the connections between the two filters in How the shell runs a pipeline.

UNIX is a registered trademark of The Open Group.