# 1. The Shell

Without a shell, to do simple operations with files on an operating system, one is usually required to write a program in a low-level language such as c. For example, the following c program renames a file from /tmp/a.txt to /tmp/b.txt:

#include <stdio.h>
main () {
int status;
status = rename("/tmp/a.txt", "/tmp/b.txt");
}


A shell is a program that provides an interface to the operating system and allows simpler syntax for basic operations. To do the above with a shell, one would need only to enter

mv /tmp/a.txt /tmp/b.txt


on the shell's command line.

The word shell means that it is layer that surrounds the operating system.

There are many different shells available for Unix/Linux systems. The difference between them is usually in the number of features and the syntax used.

For a more extensive introduction to the shell, see http://linuxcommand.org/lc3_learning_the_shell.php.

# 2. Bash

Most Unix/Linux systems use Bash as the default shell. (Bash stands for "Bourne again shell" [2] ) Unless you are a very, very advanced user, Bash has all of the capabilities that you need. Bash is also available on OS-X (natively) and Windows (via install of Cygwin). The default shell for Windows is called the DOS Shell and has a syntax that differs significantly from Unix/Linux shells.

Most of the commands shown below should work in other Unix-based shells (e.g., sh, tcsh, zsh).

The shell command line is accessed from a terminal program. Many terminal programs exist (e.g., gnome-terminal, XTerm, UXTerm, etc.) and their primary difference is the features available (ability to change fonts and colors using a menu, etc.).

References:

## 2.1. Basic commands

pwd         # Print working directory
ls          # List contents of current directory
ls -l       # List contents of current directory in long form
ls -l -a    # List contents of current directory in long form and list all files. ls -al does same thing.
ls -l FNAME # List information about file FNAME
cd /tmp     # Change directory to /tmp
cd          # Change directory to your home directory (usually /home/username)
history     # List commands that you have entered previously
!3          # Repeat the 3rd command in the history list (pronounced "bang three")


## 2.2. Compound commands

Multiple commands can be executed without hitting the RETURN key by separating the commands by a semicolon.

cd /tmp; ls


is equivalent to entering (press RETURN after entering cd /tmp)

cd /tmp
ls


## 2.3. Creating directories and empty files

touch /tmp/x.txt    # Create an empty file named x.txt in /tmp/x.txt
mkdir /tmp/zz       # Create a directory named zz
touch /tmp/zz/x.txt # Create an empty file named x.txt in /tmp/zz/x.txt


The command touch is used for creating an empty file or for modifying the timestamp of the last edit of the file (which is displayed by ls -l).

Note that the -p argument may be used to create many directories at once:

mkdir -p /tmp/a/b/c/d


is the same as

mkdir /tmp/a
mkdir /tmp/a/b
mkdir /tmp/a/b/c
mkdir /tmp/a/b/c/d


## 2.4. Renaming and copying files

Copy file /tmp/x.txt to y.txt

cp /tmp/x.txt /tmp/y.txt


or

cd /tmp; cp x.txt y.txt


In the first case, the absolute path of the file was specified because the file was not in the current working directory (directory displayed when pwd is entered). In the second case, a relative path was entered.

Rename file /tmp/x.txt to /tmp/y.txt

mv /tmp/y.txt /tmp/z.txt


or

cd /tmp; mv y.txt z.txt


mv stands for "move". A better name for this command would have been ren for "rename".

## 2.5. Renaming and copying directories

Create directory /tmp/zzcopy that is a copy of /tmp/zz

cp -r /tmp/zz /tmp/zzcopy


or

cd /tmp; cp -r zz zzcopy


Rename directory /tmp/zz to /tmp/zznewname

mv /tmp/zz /tmp/zznewname


or

cd /tmp; mv zz zznewname


## 2.6. Removing files and directories

rm /tmp/x.txt  # Remove file named x.txt in /tmp directory
rm x.txt       # Remove file named x.txt in working directory
rm -rf /tmp/zz # Remove directory zz in /tmp directory
rm -rf zz      # Remove directory zz in working directory

Be very careful when using rm -r. You could easily delete your operating system by entering
(don't do this) rm -rf /


CTRL-a    # Move to start of line
CTRL-e    # Move to end of line
L/R Arrow # Move left or right on a line
U/D Arrow # Show commands in history list
Backspace # Delete a character
TAB       # Autocomplete


## 2.8. Viewing Files

To inspect any file using the command line, use cat:

cat SOMEFILE


This program is usually used to concatenate two files together and then display the result. When used with only one file, concatenates that file to nothing. To display the contents of SOMEFILE twice, enter

cat SOMEFILE SOMEFILE


Create a text file named catdemo.txt with contents "Concatenation demonstration" and then enter

cat catdemo.txt


and

cat catdemo.txt catdemo.txt


on the command line.

To inspect only parts of a file, use head and tail:

head FILENMAME


will print the first 10 lines of FILENAME. And

tail FILENAME


will print the last 10 lines. To display only the first 3 lines, use

head -3 FILENAME


To display only the last 3 lines, use

tail -3 FILENAME


## 2.9. Pipes (|) and I/O (>, >> and <)

In Unix,

• A pipe is used to send the output of one program to the input of another program. The symbol is | and the syntax looks like PROGRAM1 | PROGRAM2, which means "evaluate PROGRAM1 and send its output not to the screen, but to PROGRAM2 for further processing".
• Indirection is used to send the output of a program to a file (using >) or to send the contents of a file to a program (using <). The syntax looks like PROGRAM > FILENAME or PROGRAM < FILENAME.

Consider the program ls. Normally, the output of this command is displayed on the screen (stdout, for "standard output"). To send (pipe) this output to another command, we use the symbol |.

ls | ANOTHERPROGRAM


where ANOTHERPROGRAM is the name of a program that can process text. A standard program on most Unix systems is wc, which simply counts the number of words in a file. To test this program, create a file named test.txt with content Hello, save, and enter on the command line

wc test.txt


(To understand the output of wc, enter man wc.)

Now, suppose that instead of copying the content output by ls to a text file, say, lsoutput.txt an then entering wc lsoutput.txt, we wanted to skip the file creation part. We could pipe the output of ls to wc using

ls | wc


Compare the output of the above with typing ls, copying the content into a file named lsoutput.txt and then entering

wc lsoutput.txt


The result should be identical.

As an example of indirection, consider sending the contents of lsoutput.txt to wc:

wc < lsoutput.txt


Or, if we want to save the output of wc, we could use

ls | wc > wcoutput.txt


or

wc lsoutput.txt > wcoutput.txt


COMMAND > filename.txt means "Write output of COMMAND to file filename.txt; overwrite filename.txt if it exists.

COMMAND >> filename.txt means "Append output of COMMAND to file filename.txt and create file if it does not exist.

What do you think

ls | wc > wcoutput.txt
ls | wc > wcoutput.txt
ls | wc >> wcoutput.txt


will do? Will the result be different from

rm -f wcoutput.txt
ls | wc >> wcoutput.txt
ls | wc >> wcoutput.txt


What do you think

wc lsoutput.txt | wc


will do?

Examples:

The program curl is used to download a web page. Try

curl http://www.google.com/


and compare with what you see by selecting View Page Source when this page is open in a web browser. (It should be the same.)

We can download and save the contents of a file using indirection:

curl http://www.google.com/ > google.txt


Download, count words, and save result (all should give same file googlewc.txt) Send output of curl to wc, and send output of wc to a file named googlewc.txt:

curl http://www.google.com/ | wc > googlewc.txt


or, Send output of curl to a file named google.txt and count the words in this file:

curl http://www.google.com/ > google.txt ; wc google.txt


or Send output of curl to a file and execute a new command that states "send contents of google.txt to wc" and output the results to a file named googlewc.txt:

curl http://www.google.com/ > google.txt ; wc < google.txt > googlewc.txt


References:

## 2.10. Echo

The command echo simply displays text to stdout (the screen). Entering

echo "Some text"


results in Some text being displayed. To display multiple lines of text, use the \n symbol to create a new line and the argument -e to tell echo to not literally display \n:

echo -e "Line1\nLine2"


To see what happens if the argument -e is omitted, enter

echo -e "Line1\nLine2"


We can analyze this string by piping the output to wc

echo "Some text" | wc


Or, we can use echo to write text into a file:

echo "Some text" > demoecho.txt


If, after executing the previous command, you enter

echo "Some text" | cat demoecho.txt -


the result should be Some text displayed twice.

## 2.11. Help

For a full manual page, enter

man COMMAND


e.g.,

man ls


For a short syntax list, enter

COMMAND -h


e.g.,

ls -h


## 2.12. Searching a File: grep

The command grep is used to globally search a regular expression and print. A regular expression is a pattern used for search. grep simply prints out the lines that match a regular expression.

Basic usage:

Create a file and search the file for lines with the letter H:

echo "Hello" > hello.txt; grep "H" hello.txt


Same as above, but skip the file creation:

echo "Hello" | grep "H"


To show how grep works, first try the echo commands and then pipe the output through grep:

echo -e "Hello\nGoodbye" | grep "H"


Prints only the first line

echo -e "Hello\nGoodbye" | grep "ood"


Prints only the second line

The most basic regular expression is ^, which means "start of line". To search for all lines that start with "H", the expression would be

echo -e "Hello1\n Hello2" | grep "^H"


This command only prints Hello1. Because there is a space between the newline (\n) and H, the first character on the second line is a space. To match the second line only, we could enter

echo -e "Hello1\n Hello2" | grep "^ H"


There are entire books written on regular expressions and to get started, I recommend doing a search on "grep regular expression tutorials". In practice, it takes quite a bit of trial-and-error to figure out the regular expression that matches the way you want.

## 2.13. Manipulating a File: sed

For a detailed tutorial on sed, see [3].

The program sed stands for stream editor. Its most basic use is to replace strings in a file and display the result:

sed 's/OLD/NEW/' file.txt


Where OLD is a string to replace the first instance of NEW on all lines of the file file.txt.

A common use of this command is with the -i argument, which tells sed to modify the file in place. You should only use this option if you are confident that the command will work as expected (or have made a copy of file.txt).

sed -i 's/OLD/NEW/' file.txt


To replace all instances of OLD with NEW, use the modifier g (for global):

sed 's/OLD/NEW/g' file.txt


Example

Create a file

echo -e "I wish I was hear.\nPlease come hear. hear hear" > file.txt


Fix the grammar

sed 's/hear/here/' file.txt


Note that only the first instance of hear was replaced on each line. To change all instances, use the g modifier:

sed 's/hear/here/g' file.txt


Like grep, complex regular expressions can be used with sed. For example, to replace the first character in each line with a x if the line starts with I,

sed 's/^I/x/' file.txt


Like grep there are entire books and manuals written on using sed to replace strings that match a given pattern. In general, the regular expression syntax used by sed is similar to that of grep, but exceptions often exist.

## 2.14. Manipulating a File: perl

Perl is a programming language that can be used for stream editing on the command line. One advantage of Perl is that the regular expression syntax is used in many programming languages, whereas that used in sed is not (However, regular expressions written for sed will often work with perl. For more information on using Perl as a stream editor, see [4] and [5].

## 2.15. Manipulating a File: cut

The program cut allows one to extract a column of information from a file. For example, suppose that you had a file

Value 19.1 good
Value 20.2 good
Value 31.1 good


which could be created using

echo -e "Value 19.1 good\nValue 20.2 good\nValue 31.1 bad\nValue 31.1 good" > file.txt


and wanted only the numbers. You could use

cut -b 7-10 file.txt


to display only the numbers and

cut -b 7-10 file.txt > file2.txt


to save the numbers to a file.

In the above command, the -b 7-10 tells cut to extract columns 7 through 10 of each row of the file. (The documentation for cut says -b means "select only these bytes". Each character is a byte, so a column and byte mean the same thing in this context.)

The above file was nicely structured, but what if the file looked like

Value 19.001 good
Value 20 good
Value 31.1 good


which could be created using

echo -e "Value 19.001 good\nValue 20 good\nValue 31.1     bad\nValue 31.1 good" > file.txt


In this case,

cut -b 7-10 file.txt


results in 19.0 20 g 31.1 31.1 which has cut off two decimals for the first number and include a letter in the second row. For this file, we need to tell code how each column is separated in a row (the delimiter) and what column we want to select.

cut -f 2 -d' ' file.txt


where the -f 2 means "select the second item (field) in each row assuming that the items are separated by the delimiter of a space -d' '.

A common delimiter used in ASCII files is the comma. If the file contained

Value,19.001,good
Value,20,good
Value,31.1,good


which could be created using

echo -e "Value,19.001,good\nValue,20,good\nValue,31.1,bad\nValue,31.1,good" > file.txt


then

cut -f 2 -d',' file.txt


will extract the second item (field) in each row.

Example:

Use cut and grep to extract only rows containing the word "good" from a file with contents

Value:19.001:good
Value:20:good


Answer: Create the file using

echo -e "Value:19.001:good\nValue:20:good\nValue:31.1:bad\nValue:99:bad" > file.txt


Show only lines containing the word "good"

grep good file.txt


Send these lines to a file

grep good file.txt > filegood.txt


Select the number from the file with only "good" measurements

cut -f 2 -d':' filegood.txt


Note that the creation of the intermediate file filegood.txt is not necessary:

grep good file.txt | cut -f 2 -d':'


Here we are piping the output of the grep command to cut. Instead of processing a file, cut processes the content that was sent through a pipe to it.

## 2.16. Manipulating Many Files

find

# 3. Problems

## 3.1. Basic Commands

Use ls to determine the number of files and directories that are in the directory /etc/.

Open a text editor and create a file named one.txt. Enter the letter A, save the file in /tmp, and close. How large do you expect the file to be?

Use ls -l to determine the size of the file in bytes and the time that the file was created.

What is the difference between the commands ls /tmp/one.txt and ls -l /tmp/one.txt?

List previous commands that you have entered and repeat the first command using the "bang" character (!).

## 3.2. Compound commands

Compare the output when the command

cd /tmp; ls -l


is entered versus

ls -l /tmp


versus

cd /tmp; ls -l .


## 3.3. Creating directories and empty files

Create an empty file named a.txt in /tmp/. Create a subdirectory of /tmp/ named tmp2. Create an empty file named b.txt in /tmp/tmp2/.

List the files in the directory /tmp/tmp2 using ls. Compare this with what you see in a file browser program.

List details about the files in the directory /tmp/tmp2 using ls and an argument. Compare this with what you see in a file browser program.

## 3.4. Renaming and copying files

Copy the file /tmp/one.txt to /tmp/tmp2/two.txt. Use ls to verify that your command worked.

Rename the file /tmp/one.txt to /tmp/tmp2/three.txt. Use ls to verify that your command worked.

## 3.5. Renaming and copying directories

Rename the directory /tmp/tmp2 to /tmp/tmp3

Copy the contents of /tmp/tmp3 to /tmp/tmp3copy

## 3.6. Removing files and directories

Create a directory with multiple subdirectories in the directory /tmp.

Write a command that removes only one of the subdirectories.

Write a command that removes the directory and all of its subdirectories.

Write commands that create a directory named /tmp/mynumbers/files/ and moves numbers.txt to that directory.

## 3.8. Directories

Write a series of Bash commands that creates an empty file named a.txt, places it into a directory named /tmp/A/B/C/D, creates a copy of a.txt in /tmp/A/B and renames it to /tmp/A/B/b.txt.

If successful, after entering the commands

ls -l /tmp/A/B/C/D/


and

ls -l /tmp/A/B


I should see something like

total 0
-rw-rw-r-- 1 user user 0 Jan 30 14:21 a.txt


and

total 0
-rw-rw-r-- 1 user user 0 Jan 30 14:21 b.txt


touch a.txt
mkdir /tmp/A
mkdir /tmp/A/B
mkdir /tmp/A/B/C
mkdir /tmp/A/B/C/D
mv a.txt /tmp/A/B/C/D
cp /tmp/A/B/C/D/a.txt /tmp/A/B


or

touch a.txt
mkdir -p /tmp/A/B/C/D
mv a.txt /tmp/A/B/C/D
cd /tmp/A/B
cp C/D/a.txt .


Create a directory with a long name

mkdir /tmp/averylongdirectoryname


cd to that directory using

cd /tmp/aver


and after typing the "r", press the TAB key. You should see the rest of the long name written out.

## 3.10. Viewing Files

### 3.10.1. cat

Create two text files and write a command with only one use of cat that displays the contents of the file on after the other. Modify this command to display the files in reverse order.

Create three text files and write a command with only one use of cat that displays the contents of the file on after the other.

### 3.10.2. head and tail

Create a text file containing

1
2
3
4
5
6
7
8
9
10
11


Use head to display the first 5 lines. Use tail to display the last 5 lines of the file.

### 3.10.3. head and tail

Use the Bash commands head and tail to create a file that contains lines 100 through 110 of astro.csv. Save your commands in a file named HW3d.sh.

## 3.11. Pipes (|) and I/O (> and <)

Write a command to download the content of a web page and pipe it to wc.

Write a command that uses echo and > to create a file with content of 1 2 3 4.

Write a set of commands that download the content of two web pages and create a single a file that contains both pages.

The following two commands do the same thing. Explain the difference.

wc /etc/passwd
cat /etc/passwd | wc


The following two commands do the same thing. Explain the difference.

wc /etc/passwd > /tmp/passwd_wc.txt
cat /etc/passwd | wc > /tmp/passwd_wc.txt


Write a series of commands to place the output of ls -1 /tmp/ and ls -l /home in a file named /tmp/ls2x.txt.

Write a series of commands that displays the output of wc for http://www.google.com/, http://www.yahoo.com/, and http://cnn.com/ in a file. (The file should have three columns and three rows.)

Create a text file named file.txt containing

1
2
3
4
5
6
7
8
9
10
11


Use head to display the first 5 lines. Use tail to display the last 3 lines of the output from the head command. When executed you should see

3
4
5


## 3.12. Echo

Write an echo command so that after it is executed, cat nine.dat shows

1 2 3
4 5 6
7 8 9


## 3.13. Echo

Write a command using echo that creates a file named numbers.txt with with the following contents (typing head numbers.txt should result in the numbers being displayed as below with three rows):

9 9 9
1 1 1
3 3 3


## 3.14. Echo

Write a command using echo that creates a file named info.txt with the content

1,2,3,4
5,6,7,8
9,10,11,12
13:14:15:16


## 3.15. Help

Enter

ls -lh


or

ls -l -h


and compare with the output of

ls -l


Use

man ls


to determine the difference.

## 3.16. Searching a File

Write a grep command prints all lines with the letter H in the file /tmp/sixlines.txt created by

echo -e "Hello1\nHello2\nHello3\nGoodbye1\nGoodbye2\nGoodbye3" > /tmp/sixlines.txt


Pipe the output of the grep command above through wc (see man wc) and verify that wc produced the expected output.

Write a grep command prints all lines with the letter H but keep only the last two lines (using tail).

## 3.17. Manipulating a File

### 3.17.1. I

The Unix command ps aux provides a list of programs that are running. First inspect the output of

ps aux


Suppose that you wanted to create a file containing only the %CPU column. Write a program that uses cut to do this.

### 3.17.2. II

The Unix command dmesg lists operating system messages. For example, if you plug in a USB drive, you may see

[11312034.362806] usb-storage: device found at 25


Suppose that you wanted to create a file that contained only the number (with no brackets) and the message for all lines of the output of dmesg that contained the string usb-storage. Write a program that uses sed and grep to do this.

(The number in brackets is the time in seconds relative to when the machine was last booted. The message above means that at that time, the operating system detected a new device plugged into a USB port.)

### 3.17.3. III

Challenge

Extract the RSS column. You will need to use sed to replace occurrences of one or more continuous spaces with a comma or a single space - cut can only handle simple delimiters.

## 3.18. Manipulating a File

Create a text file containing the following content:

1 2 3 4
5 6 7 8
4 3 2 1
8 7 6 5


Using cut, head, and |, write a command that, when executed, displays

3
7
2
6


Using cut, head, tail, and |, write a command that, when executed, displays

6 7
3 2


Using cut, head, and |, write a command that, when executed, displays

3
7


## 3.19. Manipulating a File

Write a command using sed that replaces all occurrences of 3 with 4 in numbers.txt and places the result in /tmp/mynumbers/files/numbers-modified.txt.

Use the cat command to create a file named /tmp/mynumbers/files/numbers-all.txt, which contains the content of numbers.txt and numbers-modified.txt.

Use the program cut and tail to extract the third column of numbers, except the first row, of numbers-modified.txt and place it in a file named numbers-all-c3.txt.

## 3.20. Manipulating a File

We want to extract the numbers 3,7,11,15 using cut, but the last line has a delimiter of : instead of ,. Use sed to replace all occurrences of : with , and pipe the result through cut so that what is displayed after executing the commands is

3
7
11
15


Pipe the output of the command ls -l /etc/ to cut so that only the date column is displayed.

Pipe the output of the command ls -l through cut and then grep so that only files containing the letter r are displayed.

## 3.21. Manipulating a File

Use curl, cut, and grep to display the rows for Iceland and Ireland in the file [7].

## 3.22. Manipulating a File

Download the file http://mag.gmu.edu/git-data/cds302/bou20130402vmin.min using curl. Pipe the output of curl through a combination of one or more of head, tail, cut and grep so that only the numbers in the BOUD column are displayed.

Download the file http://mag.gmu.edu/git-data/cds302/bou20130402vmin.min using curl. Pipe the output of curl through a combination of head, tail, and grep so that only lines associated with data in hour 12 are displayed.

Download the file http://mag.gmu.edu/git-data/cds302/bou20130402vmin.min using curl. Pipe the output of curl through a combination of one or more of head, tail, and grep so that only lines before the line starting in DATE are displayed.

Download the file http://mag.gmu.edu/git-data/cds302/bou.htm using curl and use cut and grep command so that only the text below is displayed (a total of 687 lines).

bou20130402vmin.min
bou20130403vmin.min
bou20130404vmin.min
...
bou20150220vmin.min


Write a command that displays the number of lines that were output from the previous command.