Writing Shell Scripts

A Few Notes about Writing Shell Scripts
=======================================

Most of this is taken from a lecture I gave as part of the "Linux
System Administration for Researchers" series. You can see those
slides here:

http://galileo.phys.virginia.edu/compfac/courses/sysadmin1/05-scripts/presentation-notes.pdf

Slides from the whole series are here:

http://galileo.phys.virginia.edu/compfac/courses/sysadmin1/

===============================================================================

Some general notes about shell scripts:
---------------------------------------

* What's a shell script? A simple shell script is a file that
contains some commands that you would normally type at the
command line.

* Why is it useful to learn about shell scripts?

- Shell scripts are widely used, so sooner or later you'll
encounter a script you'll need to understand.

- You'll inevitably want to easily repeat a series of commands
in a consistent way. (Avoiding questions like "What switches
did I use the last time I compiled that program?" and "Do I
really have to type almost the same command a thousand times
to process all of these files?")

- The same scripting techniques we'll talk about today are
used when you write shell commands in the Makefiles we
talked about last week.

* When are shell scripts appropriate?

I think of shell scripts as one part of a hierarchy of
techniques:

1. One-off commands, where you just type a command at the
command prompt. This is appropriate if you only need to do
something once, or if what you're doing is trivial.

2. Shell scripts: These are appropriate when you need to
repeat the same (or a similar) set of commands many times,
and/or when you need to be careful to do exactly the same
command at some time in the future.

3. Interpreted languages (e.g., perl or python): These are
appropriate when you need to deal with complex data
structures, or need a little more speed than shell scripts
can provide. My rule of thumb is that when you find
yourself needing an array, you should think about moving
from shell scripts and learning an interpreted language.

4. Compiled languages (e.g., C or Fortran): When you need more
speed, this is where you should go.

* Which shell should I use when writing my scripts?

It's important to understand that there are several different
shells. A shell is just a program that accepts commands from
you and executes them. When you log in to a computer, your
"login shell" starts up, and you can start typing commands.
Which shell is used depends on your account's settings. By
default, we give our users a shell called "tcsh".

tcsh is a fine login shell, but it's not often used for
writing shell scripts. That's OK though. You don't have to
use the same shell for your login and for your shell scripts.

Most shell scripts are written in what's called the "Bourne
Shell". There are lots of different versions of the Bourne
Shell, but the one found most often on Linux systems is called
"bash" (from "the Bourne Again Shell"). bash provides some
extensions that aren't available in other implementations of
the Bourne Shell, but in what I'm going to show you I won't
use any of those extensions.

All of the examples I'm going to show will use the Bourne
Shell as their scripting language.

Basic script-writing style:
---------------------------

Let's start out by looking at an example script:

#!/bin/sh
echo 'Hello World!'
echo "Here is where I am"
pwd
echo "Here is who I am:"
whoami
echo "Here is my home directory:"
echo $HOME
# And now for something completely different:
echo "Here is a really, really long command:"
ls -l /tmp | \
grep bkw1a | \
awk '{print $5/1000,$NF}' | \
sed -e 's/\..*$//' | \
/bin/mail -s Information elvis@graceland.heaven.gov

* Creating and running a script:

The text above could be typed into any text editor (say, emacs, vi or
nano) and saved into a file (say, "myprogram").

The commands in the file are all things that could be typed, one by
one, on the command line.

You can insert comments into the file by starting them with "#".

Once you've written the file, there are a couple of ways you could run
it. First, you could type:

sh myprogram

If you'd rather just type the file's name, you'll first need to make
the file "executable". You can do that with the "chmod" command, like
this:

chmod +x myprogram

After that, you can run the program by just typing:

myprogram

(This assumes that the directory is in your search path, and -- if
tcsh is your login shell -- that you've either typed "rehash" or
logged out and back in.)

You should always begin your script with a "shebang" (#!) followed by
the name of the shell your script should be run by (/bin/sh in this
case, wich is the Bourne Shell). This is how you can write scripts in
whatever shell you want, regardless of your login shell. When you
type the name of an executable script, the operating system looks at
the shebang line and figures out what shell to use for this script.

* What about very long lines?:

As you can see in the example script above, you can extend commands
across more than one line by using a "\" at the end of the line. This
means "continued on next line". Note, however, that you aren't
allowed to put anything after the \ on the same line, not even a
space.

* What commands can I use in a script?:

Anything you can type at the command line. We'll look at a catalog of
particularly useful commands for shell scripts later.

* Debugging a script:

I'll note here that, if your script doesn't do what you want it to do,
you can get some useful debugging information by using the "-x" switch
and running the script like this:

sh -x myprogram

Shell Script Plumbing:
----------------------

In addition to letting you run commands, the shell provides some
plumbing that allows you to connect commands together. The output of
one command can be sent to a file, or even to the input of another
command.

Each command has three input/output "channels" associated with it:

0 "Standard Input" (stdin)
1 "Standard Output" (stdout)
2 "Standard Error" (stderr)

The command reads input from stdin, writes normal output to stdout,
and writes any errors to stderr. When you run a command from the
command line, all three of these channels just point to your screen
and keyboard. (The command accepts input from your keyboard, and
displays any output or errors on your screen.)

The shell allows you to redirect these channels to other locations,
though.

For example, you can redirect stdout into a file:

myprogram > file.dat

If you already have a file, and you want a command to append onto the
end of it, you can use ">>":

myprogram >> file.dat

Note that these will only redirect "stdout", not "stderr". In the
examples above, any errors would still be printed out on your screen.
If you want errors to go into a file, you can redirect them too, by
using "2>":

myprogram > file.dat 2> file.err

In the example above, normal output goes into file.dat and any errors
would go into file.err. Nothing would be printed on your screen. You
can also use 2>> to append. Finally, 1> and 1>> are just the same as
> and >>.

If you want to just have stdout and stderr go into the same file, you
can type this:

myprogram > file.dat 2>&1

Here, "2>&1" means "send stderr to the same place as stdout".

You can also redirect stdin. Normally, commands read input from your
keyboard, but you can tell them to read it from someplace else. Look
at the following example:

myprogram < inputfile.dat 1> outputfile.dat 2> errorfile.dat

In the command above, we tell the program to read input from
inputfile.dat, then write normal output into outputfile.dat and errors
into errorfile.dat. The input might be answers to any questions the
program would ask us.

What if we connect the output of one program to the input of another?
That's easy to do, and it's one of the most powerful features of
shells in Unix-like operating systems.

Consider the following command:

myprogram | grep bryan

The "grep" command looks for strings that match a given pattern. Run
this way, it reads data from stdin, searches for the pattern, and
prints matching lines out on stdout (where they would appear on your
screen unless you redirected them into a file).

The "|" symbol is called a "pipe", and in the command above it
connects the stdout of myprogram to the the stdin of grep. We'll see
a lot of examples of this as we go along.

You'll use grep a lot in shell scripts, so lets take a look at how to
use it.

Using Variables
---------------

Before we go any farther, let's take a look at how to define
variables. Variables in shell scripts are used in ways that you're
probably already familiar with:

DELAY=24
NAME=Bryan
FULLNAME="Bryan Wright"
ADDRESS="Right Here"
NAMEANDADDRESS="$NAME at $ADDRESS"

Note that there can be no spaces on either side of the equals sign.

Look at the last example. It shows that you can get the values of
variables by using a dollar sign in front of the variable name. The
variable NAMEANDADDRESS now has the value "Bryan Wright at Right
Here".

What would have happened if we'd used single quotes instead?

NAMEANDADDRESS='$NAME at $ADDRESS'

In that case, NAMEANDADDRESS would contain the string "$NAME at
$ADDRESS". Variables aren't expanded when they appear inside single
quotes.

Several symbols have special meaning to the shell when they're not
enclosed within single quotes. These are *, ?, `, $, ! and \. If you
want to use one of these characters inside double quotes (or without
quotes at all), you can prepend a backslash, like "\$". For example:

echo "\$NAME"

would print the string "$NAME" (the echo command just displays the text
you give it), but

echo "$NAME"

would print "Bryan Wright". You can get a literal backslash by using
"\\".

Here's an interesting special case: What if you want to use a literal
single-quote inside a single-quoted string? You can't type:

echo 'this is a single quote: \''

because the backslash character itself is literal inside single
quotes. Here's one way to do it (can you figure out why it works?):

echo 'this is a single quote: '"'"

* The read command:

As we saw above, we can assign a value to a variable
by just typing the value after an equals sign. We can also
have our script ask the user to supply a variable's value
when he or she runs the script. Consider the following
example:

echo "Please enter your name: "
read NAME
echo "You said your name was $NAME"

The "read" command causes the script to pause and wait for the
user to type in something. Whatever is typed is put into the
variable "NAME", and can be used later.

* Using Backticks:

Backticks are another way of putting values into variables.
Here's an example:

CFILES=`ls *.c`

In the command above, the shell executes what's between the backticks,
then sticks the output of the command into the variable called
CFILES. In this example, the variable would contain a list of files
with names like *.c in the current directory.

* Doing arithmetic with variables:

What does the following do?

VARIABLE=0
VARIABLE=$VARIABLE+1
echo $VARIABLE

You may be surprised that it prints out "0+1". That's because
shell variables are usually treated like strings. No arithmetic
is done on the second line of the example above.

How about this?:

VARIABLE=0
((VARIABLE=$VARIABLE+1))
echo $VARIABLE

Now the script will print out "1", as expected. By enclosing
the second line in double-parentheses, we cause it to treat
the variables as numbers, and do arithmetic with them.

Some useful commands for writing scripts:
-----------------------------------------

Now let's look at the toolkit of commands we have available
for use in shell scripts:

* The grep command

The grep command searches for strings that match a given
pattern. The command can be run in either of two different ways:

myprogram | grep bryan
or
grep bryan file.dat

The first example is the same as the one we saw above, where grep
reads stdin. In the second example, grep reads a file and prints out
any lines it finds that match the given pattern.

Patterns for grep can be as simple as a string, like the ones above,
or quite complex. Grep uses a pattern-specifying system called
"Regular Expressions" (or "regexp").

There are actually two different pattern-matching systems that are
commonly used for shell commands. The first is the kind of pattern
matching you've probably been doing for a long time, whenever you type
a command like:

ls *.dat

The string "*.dat" specifies a pattern that will match any file name
ending in ".dat". This way of specifying patterns is called
"glob"-style pattern matching. When you type a pattern like this on
the command line, the shell actually expands it into a list of
matching file names before invoking the command ("ls", in this case).
From the perspective of the command, it looks just like you'd typed a
command like this:

ls file1.dat file2.dat a.dat b.dat

with all of the matching filenames written out. Glob-style patterns
can include "wildcards" like "*", which matches any string of
characters and "?", which matches any single character.

Things are different with commands like grep, though. They use the
more complex and flexible regexp pattern matching system. This can
cause confusion in at least a couple of ways.

First of all, both glob and regexp patterns may use the * and ?
characters, but they use them in different ways.

Second, as I mentioned above, the shell automatically interprets
glob-style patterns before passing the matched filenames on to any
command you type. This means that we need to be careful to tell the
shell not to interpret our regexp patterns, but instead to pass them
along, unchanged, to commands like grep.

Let's first take a look at regular expression syntax. Here's a table
showing what some special symbols mean when used within a regular
expression:

Symbol Meaning -E?
------ ------- ---
. Match any single character.
* Match zero or more of the preceding item.
+ Match one or more of the preceding item. Y
? Match zero or one of the preceding item. Y
{n,m} Match at least n, but not more than m, of the Y
preceding item.
^ Match the beginning of the line.
$ Match the end of the line.
[abc123] Match any of the enclosed list of characters.
[^abc123] Match any character not in this list.
[a-zA-Z0-9] Match any of the enclosed ranges of characters.
this|that Match “this” or “that”. Y
\., \*, etc. Match a literal “.”, “*”, etc.

Note that some of these symbols will only be interpreted this way if
you tell grep to use "extended" regular expressions. To enable
extended regexps, type "grep -E" instead of just "grep". The last
column of the table tells you whether you need to give grep the "-E"
qualifier in order to use a particular symbol.

Here are a few examples showing grep in action:

grep elvis
grep ^elvis$
grep '[eE]lvis'
grep 'B[oO][bB]' (matches Bob, BOB, BoB, or BOb)
grep '^$' (matches a blank line)
grep '[0-9][0-9]'
grep '[a-zA-Z]'
grep '[^a-zA-Z0-9]'
grep -E '[0-9]{3}-[0-9]{4}' (matches a phone number)
grep '^.$' (matches a line with only one character)
grep '^\.'
grep '^\.[a-z][a-z]'

Notice that we have to take care to enclose the patterns in
single-quotes in cases where the shell would otherwise try to
interpret one of the characters as part of a glob- style expression.
For example, if we typed the command:

grep '1234.0*' file.txt

grep would look for strings like '1234.', '1234.0', '1234.00', etc.,
in the file. However, if we just typed:

grep 1234.0* file.txt

the shell would try to expand 1234.0* by looking for files with names
beginning like "1234.0" in the current directory. That's not what we
intended at all.

Here are some examples showing grep in combination with other
commands, connected by pipes:

cat file.txt | grep important
This line is important!

ls | grep myprogram
myprogram
oldmyprogram
myprogram.c
myprogram.h

ls | grep ^myprogram
myprogram
myprogram.c
myprogram.h

ls | grep -v old
myprogram
myprogram.c
myprogram.h

ls | grep '\.c$'
myprogram.c

Here's another potential stumbling-block to look out for: What if you
want to search for the string '-this-'? Grepnormally interprets
anything beginning with a dash as being a switch intended to control
how grep operates. Given the string above, grep would probably
complain that "-this-" was an unrecognized option. We can get around
this by telling grep to treat dashes as literal, like this:

grep -- -this- file.dat

Anything after the "--" is treated as a pattern, even if it begins
with a dash. We can put real options before the -- if we want to,
like this:

grep -E -- -this- file.dat

One useful option you may want to use with grep is "-q". This tells
grep not to print out what it finds, but rather just to return an
"exit status" saying whether it succeeded in finding a match, or
failed.

In Unix-like operating systems, each command returns an exit status
which is interpreted by the shell as either "success" or "failure".
The shell provides us with several ways to make use of these status
flags. One way is through the "&&" and "||" expressions.

Consider the following command:

grep -q Elvis file.dat && echo Found elvis!

The && means "execute the following command only if the preceding
command succeeds". In this case, the "echo" command would only be
executed if grep found the string "Elvis" in the file.

We can test for failure using the || expression:

grep -q Elvis file.dat || echo Not found.

and we can combine both && and || to deal with both possibilities:

grep -q Elvis file.dat && echo Found || echo Not found

Finally: Why is the command called "grep"? Grep was invented by Ken
Thompson, the inventor of Unix, and he named it after a commonly-used
command in an early editor called "ed": g/re/p. This told ed to do a
"global search" for a regular expression (which you'd insert in place
of "re") and "print" the result. Just what the grep command does now.

* The sed command

sed is a "stream editor". It's used to transform a stream of
data by substituting one string for another, or deleting strings. sed
uses regular expressions for pattern matching, just as grep does.

Like grep, sed can either read from its "standard input" or from a
file.

Here's one way you might use sed:

cat file.txt | sed -e 's/HERE/Charlottesville/g' > newfile.txt

The command above tells sed to substitute the string "Charlottesville"
wherever it finds the string "HERE". sed speaks a complex language
all its own, but simple commands like this are the most common way to
use sed.

The "-e" qualifier tells sed to "execute" the following command (which
will be given in sed's own terse language). The command itself is
enclosed in single-quotes. It begins with "s", which means "make a
substitution. Then, there are two strings like this:

/string1/string2/

String1 is a regexp pattern to look for, and string2 is some text to
drop in place of any matches. The final "g" tells sed to do this
"globally", meaning that it should do the replacement for all matches.
Otherwise, it would just replace the first match.

Here's another sed command:

echo 555-1212 | sed -e 's/-//'

In this example, we delete the dash from the middle of a phone number.
The result would be "5551212".

You'll notice that the first sed example, above, processes data from
one file, and writes the modified data into a new file. sed is also
capable of editing a file "in-place", modifying the original file. To
do this, use the "-i" qualifier:

sed -i -e 's/HERE/Charlottesville/g' file.txt

Remember to use "-i" with caution. It's very easy to mess up your
original file. Unless you're very sure that sed will do what you
expect, you should probably write out a new file, then examine it
before replacing your old file with it.

* The awk command

grep is useful for selecting lines of input, but
what about selecting columns? The awk command provides this functionality.
awk is actually a whole programming language that can do quite sophisticated
manipulations of data, but few people use it for complex tasks these days.
For these, awk has generally been superseded by perl, an interpreted
language that incorporates awk's abilities and much more.

awk is still often used as a convenient way to pick out columns of
input, though. Consider the following command:

ls -l

The command's output might look like this:

lrwxrwxrwx 1 bkw1a bkw1a 11 Mar 26 2010 clus.pdf -> cluster.pdf
-rw-r----- 1 bkw1a bkw1a 20601 Jan 18 2009 cluster.pdf
-rw-r--r-- 1 bkw1a bkw1a 29 Jan 18 2009 data-for-everybody.1.dat
-rw------- 1 bkw1a bkw1a 41 Jan 18 2009 ForYourEyesOnly.dat
drwxr-x--- 3 bkw1a bkw1a 4096 Jan 18 2009 phase1
drwxr-x--- 2 bkw1a bkw1a 4096 Jul 16 2009 phase2
drwxrwxr-x 2 bkw1a bkw1a 4096 Jan 26 2012 phase3
-rw-r----- 1 bkw1a bkw1a 9552 Jan 18 2009 ReadMe.txt
drwxr-xr-x 2 bkw1a bkw1a 4096 Apr 14 2009 tmp

This tells us a lot of information about the files in the current
directory. The fifth column, for example, is the file size, in bytes.
What if we just wanted to get the file sizes? We could use awk to
do it:

ls -l | awk '{print $5}'

The command above would produce output like this:

11
20601
29
41
4096
4096
4096
9552
4096

The text enclosed in single quotes in the command above is written
in awk's own language. It tells awk to print out column 5 of each
line. By default, awk defines by looking for white space (any
number of spaces or tabs), but you can tell it to use any other "field
separator" by using awk's "-F" qualifier. For example, say you have
a file that contains columns of numbers, separated by commas. You
could pick out the third column by typing:

cat file.dat | awk -F, '{print $3}'

What if you want to pick out the last column, but don't want
to count the columns (or maybe there's a variable number of columns)?
awk provides a way to do that:

cat file.dat | awk -F, '{print $NF}'

$NF stands for "number of fields". The command above will print out
the last column in each line. If you want the next-to last column,
you can say something like this:

cat file.dat | awk -F, '{print $(NF-1)}'

(Note the unexpected placement of the parentheses.) Similarly,
$(NF-2) would get you the column two spaces from the end, and so
forth.

Looking back at the "ls -l" example above: What if we wanted to
print out both the file size and file name? Here's how to do that:

ls -l | awk '{print $5,$NF}'

This would print:

11 cluster.pdf
20601 cluster.pdf
29 data-for-everybody.1.dat
41 ForYourEyesOnly.dat
4096 phase1
4096 phase2
4096 phase3
9552 ReadMe.txt
4096 tmp

You can also rearrange columns, and add text:

ls -l | awk '{print "the size of",$NF,"is",$5,"bytes"}'

would produce:

the size of cluster.pdf is 11 bytes
the size of cluster.pdf is 20601 bytes
the size of data-for-everybody.1.dat is 29 bytes
the size of ForYourEyesOnly.dat is 41 bytes
the size of phase1 is 4096 bytes
the size of phase2 is 4096 bytes
the size of phase3 is 4096 bytes
the size of ReadMe.txt is 9552 bytes
the size of tmp is 4096 bytes

Finally, awk lets you do arithmetic manipulations on columns:

cat file.dat | awk -F, '{print $1/$2, $3-$4, 2*$1+3*$6}'

The command above would print out the ratio of the values in
columns 1 and 2, the difference of the values in columns 3 and 4,
and a linear combination of the values in columns 1 and 6.

* The sort command

Consider, again, the output of one of the examples
above:

ls -l | awk '{print $5,$NF}'

This might print something like:

11 file1.dat
206 file2.dat
100 file3.dat
2 file4.dat
210 file5.dat

What if we wanted to sort this list, in order of increasing
file size? There's a command called "sort" that might seem
to do the trick, but its output probably isn't what we expect:

ls -l | awk '{print $5,$NF}' | sort

100 file3.dat
11 file1.dat
206 file2.dat
210 file5.dat
2 file4.dat

(Notice here that we can connect any number of commands together
with pipes. We can build up long chains of commands to do
complex jobs.)

Why aren't the files sorted the way we expect? Because "sort"
sorts things, by default, in dictionary order. In a dictionary,
we'd expect to see all of the things that begin with "1" first,
and then all of the things that begin with "2", and so forth.

We can get "sort" to sort numerically by giving it the "-n"
qualifier:

ls -l | awk '{print $5,$NF}' | sort -n

This command would give:

2 file4.dat
11 file1.dat
100 file3.dat
206 file2.dat
210 file5.dat

* The uniq command

What if we were looking at a shared folder full of files, and we
wanted to find out which user owns the most files in the folder? Looking
at the folder with "ls -l" we could get each file's owner by asking awk
to show us column number 3, like this:

ls -l /tmp | awk '{print $3}'

The output might look like this:

bkw1a
abk5s
bj6h
cb8nw
bj6h
cb8nw
bkw1a
...etc.

If I wanted to find the number of files for each user, I could
just count the number of times the user's name occurs, but that
would be tedious. I can make things a little better by sorting
the names (this time in dictionary order):

ls -l /tmp | awk '{print $3}' | sort

This might give:

abk5s
abk5s
bkw1a
bkw1a
bj6h
bj6h
bj6h
cb8nw
cd6j
...etc.

So now, all of the files for each user are grouped together, making
it slightly easier to count them, but not much.

If we just wanted to get a list of user names, eliminating
duplicates, we coult use the "uniq" command:

ls -l /tmp | awk '{print $3}' | sort | uniq

which might give:

abk5s
bkw1a
bj6h
cb8nw
cd6j
...etc.

That doesn't help us with counting, but it turns out that we can
give uniq the "-c" flag to tell it to count the number of times
each name occurs:

ls -l /tmp | awk '{print $3}' | sort | uniq -c

3 abk5s
5 bkw1a
10 bj6h
25 cb8nw
13 cd6j
...etc.

OK, so we now have a count of how many files each user owns!
We could even sort it in order of number of files, by adding
one more "sort" command, this time sorting numerically:

ls -l /tmp | awk '{print $3}' | sort | uniq -c | sort -n

3 abk5s
5 bkw1a
10 bj6h
13 cd6j
25 cb8nw
...etc.

* The wc command

What about situations where you just want to count the
number of lines in a file? Maybe the file has one line per
data point, and you want to count your data points. The wc
command will do that for you:

wc file.dat

This would return three numbers, like this:

3 12 72

The first number is the number of lines, then the number of
"words" (defined as strings separated by white space), and the
third number is the number of characters.

If you just want the number of lines, you can type "wc -l".
There are also "wc -c" and "wc -w" qualifiers to get just
character count or word count, respectively.

* The tail and head commands

What if you just want to get the last line of a file?
For that, there's the "tail" command:

tail -1 file.dat

This would print out the last line of the file. To get the
last two lines, type "tail -2", and so forth. With no number,
the last ten lines are printed.

There's a similar "head" command to get the first n lines of
a file.

Like many other commands, these can read stdin as well as
a file. For example, if we go back to our list showing
how many files belong to each user:

ls -l /tmp | awk '{print $3}' | sort | uniq -c | sort -n | tail -1

The command above would just print out the last line, showing
the number of files owned by the biggest hog.

* The find command

The "find" command is a versatile tool for finding files and
manipulating them. The command "find ." (note the dot) will begin
in the current directory, and recursively print out the names of all files and
directories underneath. You can also find files whose name matches a
given pattern (glob-style). For example:

find . -name '*.dat'

would print out the names of any files matching *.dat. Note the need for
single-quotes around the wildcard filename, to prevent the shell from
trying to expand this into a list of files in the current directory before
invoking "find".

You can tell "find" to ignore the difference between upper- and lower-case
letters by using "-iname" instead of "-name" (for "ignore case").

What if you want to find all of the files created during the last day?
You could use the -mtime qualifier (for "modification time"):

find . -mtime -1

The -mtime qualifier takes a numerical argument that's an integer, optionally
preceded by + or -. In this case, -1 means "less than one day ago". If
we'd used +1, we'd mean "more than one day ago". If we just typed 1, we'd
mean "exactly one day ago". There's also a -mmin qualifier that works the
same, except the times are given in minutes instead of days.

The general form of a "find" command is:

find (starting somewhere) (select files that match some criteria) (do something)

The "starting somewhere" will be a directory name, possibly "." for the
current working directory. The "criteria" will be given by qualifiers like
-name or -mtime. The "something" to do can be left off, in which case find
will just print the file name. Other things to do are:

-print This is the default
-ls Print out detailed information about the file, like "ls"
-exec Execute a given command for each file found.

The last of these is very powerful, and can be used to easily do the
same operation on lots of files. For example:

find . -type f -exec gzip {} \;

The command above would find all files under the current directory
and compress each of them with gzip. Note that the syntax of the
exec qualifier is:

-exec (some command) \;

The space between the command and the backslash is mandatory.
Inside the command, you can put {}, which is a placeholder for
the name of each file found.

Here's another -exec example:

find . -name '*.dat' -exec mv {} /tmp/ \;

This would find all files underneath the current directory with
names like "*.dat" and move them into the directory /tmp.

Here's one last -exec example:

find . -name '*.ps' -exec ps2pdf {} \;

This would create a pdf version of every postscript file.

Finally, note that several search criteria can be combined together
with "find"'s equivalent of "and" and "or" statements. For example:

find . -name '*.dat' -a -mtime -1

The command above would find files with names matching "*.dat" AND
modification times of less than one day ago. The qualifier for
"or" is -o.

* The test command

The final command we'll talk about in detail is "test".
This is a tool that just checks a condition to see if it's true
or false, and returns an exit status. It's so commonly used that
it has a one-character alternative name: "[".

General usage for "test" would be something like this:

[ condition to test for ] && echo True

where "condition to test for" is an expression that test understands.
Note that the spaces after "[" and before "]" are mandatory.

Also note that, because this command is called "test" and is so
widely used, things will probably break badly if you name one of
your own executable files "test". (Don't do that.)

Here's some of the syntax for test conditions:

STRING1 = STRING2 the strings are equal
STRING1 != STRING2 the strings are not equal

INTEGER1 -eq INTEGER2 INTEGER1 is equal to INTEGER2
INTEGER1 -ge INTEGER2 INTEGER1 is greater than or equal to INTEGER2
INTEGER1 -gt INTEGER2 INTEGER1 is greater than INTEGER2
INTEGER1 -le INTEGER2 INTEGER1 is less than or equal to INTEGER2
INTEGER1 -lt INTEGER2 INTEGER1 is less than INTEGER2
INTEGER1 -ne INTEGER2 INTEGER1 is not equal to INTEGER2

-e FILE FILE exists
-f FILE FILE exists and is a regular file
-d FILE FILE exists and is a directory
-L FILE FILE exists and is a symbolic link
-s FILE FILE exists and has a size greater than zero
-x FILE FILE exists and is executable

EXPRESSION1 -a EXPRESSION2 both EXPRESSION1 and EXPRESSION2 are true
EXPRESSION1 -o EXPRESSION2 either EXPRESSION1 or EXPRESSION2 is true
! EXPRESSION EXPRESSION is false

And here are some examples of "test" usage:

[ -f /etc/motd ] && cat /etc/motd
[ -x /local/bin/update ] && /local/bin/update
[ -d /tmp/mytemp ] || mkdir /tmp/mytemp
[ “$ANSWER” = “yes” ] && echo Okay. || echo Nope.
[ “$ANSWER” \!= “no” ] && echo Didn't SAY no...
[ $SIZE -gt 10000 ] && echo Found a big file.
[ $SIZE -le 1000 -o -f /etc/preserve ] && echo OK.

Note that some shells will require that you put a backslash in front
of any “!” characters, to prevent the shell from interpreting them
before they're passed to “test”.

Positional Variables
--------------------

As we saw above, we can make a script executable by using the
chmod command, and then we can run the script by just typing its name.
Let's say we've created a script called myprogram. What happens if we
type other things (maybe the names of some files we want the script to
work on) after the name of the program?

myprogram file1.txt file2.txt file3.txt

Inside the program, several variables are automatically defined for us
when the program is run. Among these are the "positional variables".
In the example above, these would be:

$0, which has the value myprogram
$1, which has the value file1.txt
$2, which has the value file2.txt
$3, which has the value file3.txt

We can use these inside our program just like any other
variable:

echo "The first file is $1"
echo "The second file is $2"

There are also variables that contain all of the file names, as a
list. These are $* and $@. These two variables behave the same way,
except when they're enclosed in quotes. Here are some examples
showing commands, followed by the output of each command:

echo $*
file1 file2 file3

echo "$*"
"file1 file2 file3"

echo $@
file1 file2 file3

echo "$@"
"file1" "file2" "file3"

Notice the difference between the second and fourth examples.
In the second example, the file names are all put together into
one string. In the fourth example, they remain separately-quoted
individual strings.

Flow control
------------

OK, so what if you want to do something to each of the files
you've named on the command line? Say, for example, that we wanted
to compress each file with gzip. You could just write out commands
like:

gzip $1
gzip $2
gzip $3

and so forth. But what if you have different numbers of files at
different times? And what if you want to execute a whole series
of complicated commands for each file? We can solve those problems
by using the shell's loop-control mechanisms.

* making loops:

Consider the following program:

#!/bin/sh

for FILE in "$@"
do
echo "Processing file $FILE..."
gzip $FILE
done

The "for" command creates a loop. Each time through the loop, the
variable FILE is given one of the values in the list "$@" (which,
as we saw above, is just the list of arguments we've given the program
on the command line). We can specify any number of files on the command
line, and the script will loop through all of them. If our script
is called "myprogram", we might use it like this:

myprogram *.ps

This would pass the names of all files in the current directory that
match *.ps to the script as positional variables $1, $2, $3, ...etc.
The script would then gzip each file, one at a time.

Here are some more examples. Can you figure out what they do?

#!/bin/sh

for FILE in *.ps
do
ps2pdf $FILE
done

#!/bin/sh

for PID in `ps auxwww| grep elvis | awk '{print $1}'`
do
renice $PID
done

#!/bin/sh

LIST=`find . -name '*.txt'`
for F in $LIST
do
dos2unix $F
done

* Conditional statements:

We've already seen that we can use the exit status of a command
along with the && and || operators to cause our script to make different
decisions in different situations.

The Bourne Shell also provides the kind of "if" statements that you may
be used to in other languages. Consider the following:

#!/bin/sh

USER=$1

if grep -q ^$USER: /etc/passwd
then
echo "User already exists"
else
echo "Creating user..."
useradd $USER
fi

The example above shows a simple script for adding a new user account.
It first checks to see if the user already has an account, by looking
for the user's name in the /etc/passwd file. If the name is found,
the script tells us so, and stops. If the name isn't found, it executes
some appropriate commands to create the new account.

The "if" statement executes the following command, and does different
things based on the exit status (success or failure) of that command.

Notice that the first thing I've done is to store the value of $1
(the user name, as supplied to my script on the command line) in
a variable with the more informative name "USER". It's good practice
to store positional variables in other variables with readable names
right at the top of your scripts. That way, you don't have to remember
which argument is $1, $2, etc. later in the script, and you're free
to rearrange the command line arguments in the future, without having
to rewrite anything except the top part of your script.

Very often, we'll use the "test" command (aka "[") in "if" statements.
Here are some examples:

if [ -f /tmp/junk.dat ]
then
rm /tmp/junk.dat
fi

if [ "$ANSWER" = "yes" ]
then
echo "Do good stuff."
elif [ "$ANSWER" = "maybe" ]
then
echo "Maybe do some stuff".
else
echo "Must be no"
fi

Note that we can omit the "else" part of the statement, or we can add
as many "else if" ("elif" here) statements as we want. When we have
"elif" statements, only the first matched "elif" is acted upon.
Any other matching "elif"s are ignored, as you'd expect.

* The case statement

But what if we wanted to have a whole lot of "elif"s? That
could get very messy. The Bourne Shell provides us with an abbreviated
way of dealing with situations like that. Look at the following:

echo "Enter your answer: "
read ANSWER

case $ANSWER in
yes|YES|y|Y)
echo "Answered yes."
;;
no|NO|n|N)
echo "Answered no."
;;
*)
echo "I do not understand."
;;
esac

The case statement above looks at the value of ANSWER and compares it
with several lists. If ANSWER matches any of the lists, the commands
associated with that list are executed. In the example above, if
ANSWER is any of the strings "yes", "YES", "y" or "Y", the script will
say "Answered yes.".

Note that each set of commands is terminated with ";;".

Also note that the lists can contain glob-style patterns. In the last
case, the "*" will match anything.

* The while statement (another loop):

The Bourne Shell also provides a way to do a counting loop.
Consider the following example:

echo 'How many should I do?'
read LIMIT

VALUE=1
while [ $VALUE -le $LIMIT ]
do
echo "Value is now $VALUE"
((VALUE=$VALUE+1))
done

Look at the expression enclosed in double-parentheses. Without the
parentheses, VALUE would be set to "1+1", then "1+1+1", then "1+1+1+1",
and so forth. In order to make the shell do arithmetic on the values,
we enclose them in double-parentheses.

"while" loops continue until some given condition is no longer true.
The while loop in the example above will continue until VALUE is
greater than LIMIT.

Special Characters and Line Endings:
------------------------------------

Finally, a note about a few special characters that you
can use within double-quotes:

\n gives a "newline" (linefeed)
\r gives a "carriage return"
\t gives a tab
\\ gives a literal backslash

The first two characters are important things to know about if you're
planning to copy scripts from one operating system to another (say,
Windows to Linux, or Linux to OS X). This is because each of these
three operating systems signifies the end of a line in a different way:

Operating System Line Ending
---------------- -----------
Linux \n
Windows \r\n
OS X \r

This means that you might find that your script doesn't work when
you copy it to a different operating system, because the operating
system isn't able to find the ends of the lines in the script. It
might thing that your script is just one long run-on line, or it might
see extra characters that are confusing.

There are several tools available to convert line endings from one
format to another. These are "dos2unix", "unix2dos" and "mac2unix",
with self-explanatory names.