Shell-Scripting-1.pdf

Shell-Scripting-1.pdf

1. Shell Scripting
Alexander B. Pacheco
LTS Research Computing

2. Outline
1
Introduction
Types of Shell
Variables
File Permissions
Input and Output
2
Shell Scripting
Getting Started with Writing Simple Scripts
Arithmetic Operations
Flow Control
Arrays
Command Line Arguments
Functions
3
Unix Utilities
grep
sed
4
awk programming
5
Wrap Up
2 / 76

3. Introduction

4. Introduction
What is a SHELL
The command line interface is the primary interface to Linux/Unix operating systems.
Shells are how command-line interfaces are implemented in Linux/Unix.
Each shell has varying capabilities and features and the user should choose the shell
that best suits their needs.
The shell is simply an application running on top of the kernel and provides a powerful
interface to the system.
OtherSoftwareShellKernelHardware
4 / 76

5. Types of Shell
sh: Bourne Shell
Developed by Stephen Bourne at AT&T Bell Labs
csh: C Shell
Developed by Bill Joy at University of California, Berkeley
ksh: Korn Shell
Developed by David Korn at AT&T Bell Labs
backward-compatible with the Bourne shell and includes many features of the C
shell
bash: Bourne Again Shell
Developed by Brian Fox for the GNU Project as a free software replacement for
the Bourne shell (sh).
Default Shell on Linux and Mac OSX
The name is also descriptive of what it did, bashing together the features of sh,
csh and ksh
tcsh: TENEX C Shell
Developed by Ken Greer at Carnegie Mellon University
It is essentially the C shell with programmable command line completion,
command-line editing, and a few other features.
5 / 76

6. Shell Comparison
shcshkshbashtcsh
Programming Language 3 3 3 3 3
Shell Variables 3 3 3 3 3
Command alias 7 3 3 3 3
Command history 7 3 3 3 3
Filename completion 7 M M 3 3
Command line editing 7 7 M 3 3
Job control 7 3 3 3 3
3: Yes
7: No
M: Yes, not set by default
http://www.cis.rit.edu/class/simg211/unixintro/Shell.html
6 / 76

7. Variables
A variable is a named object that contains data used by one or more applications.
There are two types of variables, Environment and User Dened and can contain a
number, character or a string of characters.
Environment Variables provides a simple way to share conguration settings between
multiple applications and processes in Linux.
As in programming languages like C, C++ and Fortran, dening your own variables
makes the program or script extensible by you or a third party
Rules for Variable Names
1
Variable names must start with a letter or underscore
2
Number can be used anywhere else
3
DO NOT USE special characters such as@, #, %, $
4
Case sensitive
5
Examples
Allowed: VARIABLE, VAR1234able, varname,VAR
Not Allowed: 1VARIABLE, %NAME, $myvar, VAR@NAME
To reference a variable, environment or user dened, you need to prepend the variable
name with "$" as in $VARIABLE, $PATH, etc.
7 / 76

8. Variables
Its a good practice to protect your variable name withinf. . .gsuch as $fPATHgwhen
referencing it. (We'll see an example in a few slides)
Assigning value to a variable
Type sh,ksh,bash csh,tcsh
Shell name=value set name = value
Environment export name=value setenv name value
sh,ksh,bashTHERE IS NO SPACE ON EITHER SIDE OF =
csh,tcshspace on either side of = is allowed for thesetcommand
csh,tcshThere is no = in thesetenvcommand
8 / 76

9. File Permissions
In *NIX OS's, you have three types of le permissions
1
read (r)
2
write (w)
3
execute (x)
for three types of users
1
user
2
group
3
world i.e. everyone else who has access to the system
drwxr-xr-x. 2 user user 4096 Jan 28 08:27 Public
-rw-rw-r- -. 1 user user 3047 Jan 28 09:34 README
The rst character signies the type of the le
dfor directory
lfor symbolic link
-for normal le
The next three characters of rst triad signies what the owner can do
The second triad signies what group member can do
9 / 76

10. File Permissions
The third triad signies what everyone else can do
d rwx
|{z}
u
g
z}|{
rx rx
|{z}
o
Read carries a weight of 4
Write carries a weight of 2
Execute carries a weight of 1
The weights are added to give a value of 7 (rwx), 6(rw), 5(rx) or 3(wx) permissions.
chmodis a *NIX command to change permissions on a le
To give user rwx, group rx and world x permission, the command is
chmod 751 filename
Instead of using numerical permissions you can also use symbolic mode
u/g/o or a
+/-
r/w/x
10 / 76

11. File Permissions
Give everyone execute permission:
chmod a+x hello.sh
chmod ugo+x hello.sh
Remove group and world read & write permission:
chmod go-rw hello.sh
Use the-Rag to change permissions recursively, all les and directories and their
contents.
chmod -R 755 $fHOMEg/*
What is the permission on $fHOMEg?
HPC Users
If you want to share your les with your colleagues
1
Make your home directory read accessible to the world
chmod 755 $fHOMEg
do not use the recursive -R flag
2
Change to your home directory and give read access to the directory that you want to
share using the -R ag
11 / 76

12. Input/Output
For reading input from screen/keyboard/prompt
bashread
tcsh$<
Thereadstatement takes all characters typed until the
Enter key is pressed and
stores them into a variable.
Syntaxread <variable name>
Exampleread name
Enter
Alex Pacheco
$<can accept only one argument. If you have multiple arguments, enclose the$<
within quotes e.g."$<"
Syntax:set <variable> = $<
Example:set name = "$<"
Enter
Alex Pacheco
In the above examples, the name that you enter in stored in the variablename.
12 / 76

13. Input/Output
The commandechois used for displaying output to screen
Use theechocommand to print the variablenameto the screen
echo $name
Enter
Theechostatement can print multiple arguments.
By default,echoeliminates redundant whitespace (multiple spaces and tabs) and
replaces it with a single whitespace between arguments.
To include redundant whitespace, enclose the arguments within double quotes
echo Welcome to HPC Training
(more than one space between HPC and
Training)
echo "Welcome to HPC Training"
read name
orset name = "$<"
Alex Pachecoecho $nameecho "$name"
13 / 76

14. Input/Output
You can also use theprintfcommand to display output
Syntax:printf <format> <arguments>
Example:printf "$name"
printf "%snn" "$name" Format Descriptors
%s
%d
%f
nn
you can add a width for the argument between the % andfs,d,fgelds
%4s, %5d, %7.4f
Theprintfcommand is used inawkto print formatted data (more on this later)
14 / 76

15. I/O Redirection
There are three le descriptors for I/O streams
1
STDIN: Standard Input
2
STDOUT: Standard Output
3
STDERR: Standard Error
1 represents STDOUT and 2 represents STDERR
I/O redirection allows users to connect applications
<: connects a le to STDIN of an application
>: connects STDOUT of an application to a le
>>: connects STDOUT of an application by appending to a le
j: connects the STDOUT of an application to STDIN of another application.
Examples:
1
write STDOUT to le:ls -l > ls-l.out
2
write STDERR to le:ls -l 2> ls-l.err
3
write STDOUT to STDERR: ls -l 1>&2
4
write STDERR to STDOUT: ls -l 2>&1
5
send STDOUT as STDIN: ls -l | wc -l
15 / 76

16. Shell Scripting

17. What is a scripting language?
Ascripting languageorscript languageis aprogramming languagethat supports
the writing ofscripts.
Scripting Languagesprovide a higher level of abstraction than standard
programming languages.
Compared to programming languages, scripting languages do not distinguish between
data types: integers, real values, strings, etc.
Scripting Languages tend to be good for automating the execution of other programs.
analyzing data
running daily backups
They are also good for writing a program that is going to be used only once and then
discarded.
Ascriptis a program written for a software environment that automate the execution
of tasks which could alternatively be executed one-by-one by a human operator.
The majority of script programs are \quick and dirty", where the main goal is to get
the program written quickly. 17 / 76

18. Writing your rst script
Three things to do to write and execute a script
1
Write a script
A shell script is a le that contains ASCII text.
Create a le,hello.shwith the following lines
#
#
e c h o
2
Set permissions
~
OR
~
3
Execute the script
~
H e l l o W o r l d !
4
If you do not set execute permission for the script, then
~
H e l l o W o r l d !
18 / 76

19. Description of the script
My First Script
#
#
e c h o
The rst line is called the "ShaBang" line. It tells the OS which interpreter to use. In
the current example, bash
Other options are:
sh : #!/bin/sh
ksh : #!/bin/ksh
csh : #!/bin/csh
tcsh: #!/bin/tcsh
The second line is a comment. All comments begin with "#".
The third line tells the OS to print "Hello World!" to the screen.
19 / 76

20. Special Characters
#:
$:
n:
f g:
;
on the same line.
;;
.
$?
$$
[ ]
[[ ]]
$[ ], (( ))
jj, &&, !
20 / 76

21. Quotation
Double Quotation" "
Enclosed string is expanded ("$", "/" and "`")
Example:echo "$myvar"prints the value ofmyvar
Single Quotation' '
Enclosed string is read literally
Example:echo '$myvar'prints$myvar
Back Quotation` `
Used for command substitution
Enclosed string is executed as a command
Example:echo `pwd`prints the output of thepwdcommand i.e. print working
directory
Inbash, you can also use$( )instead of` `
e.g.$(pwd)and`pwd`are the same
21 / 76

22. Example
#
H I = H e l l o
e c h o
e c h o
e c h o
e c h o
e c h o
e c h o
e c h o
e c h o
e c h o
~
H I
H e l l o
$ H I
H e l l o
$ H I
H e l l o A l e x
/
/
~
22 / 76

23. Arithmetic Operations
You can carry out numeric operations on integer variables
Operation Operator
Addition +
Subtraction -
Multiplication *
Division /
Exponentiation ** (bashonly)
Modulo %
Arithmetic operations inbashcan be done within the$(( ))or$[ ]commands
FAdd two numbers:$((1+2))
FMultiply two numbers:$[$a*$b]
FYou can also use theletcommand:let c=$a-$b
For use theexprcommand:c=`expr $a - $b`
23 / 76

24. Arithmetic Operations
Intcsh,
FAdd two numbers:@ x = 1 + 2
FDivide two numbers:@ x = $a / $b
FYou can also use theexprcommand:set c = `expr $a % $b`
Note the use of space
bashspace required around operator in theexprcommand
tcshspace required between @ and variable, around = and numeric operators.
You can also use C-style increment operators
bashlet c+=1orlet c--
tcsh@ x -= 1or@ x++
/=,*=and%=are also allowed.
bash
The above examples only work for integers.
What about oating point number?
24 / 76

25. Arithmetic Operations
Using oating point inbashortcshscripts requires an external calculator like GNU
bc.
FAdd two numbers:
echo "3.8 + 4.2" | bc
FDivide two numbers and print result with a precision of 5 digits:
echo "scale=5; 2/5" | bc
FCallbcdirectly:
bc<<<"scale=5; 2/5"
FUsebc -lto see result in oating point at max scale:
bc -l<<<"2/5"
You can also useawkfor oating point arithmetic.
25 / 76

26. Flow Control
Shell Scripting Languages execute commands in sequence similar to programming
languages such as C, Fortran, etc.
Control constructs can change the sequential order of commands.
Control constructs available inbashandtcshare
1
Conditionals:if
2
Loops:for, while, until
3
Switches:case, switch
26 / 76

27. ifstatement
Anif/thenconstruct tests whether the exit status of a list of commands is 0, and if
so, executes one or more commands.
bash
i f
s o m e c o m m a n d s
e l i f
s o m e c o m m a n d s
e l s e
s o m e c o m m a n d s
f i
tcsh
i f
s o m e c o m m a n d s
e l s e
s o m e c o m m a n d s
e l s e
s o m e c o m m a n d s
e n d i f
Note the space betweenconditionand "[" "]"
bashis very strict about spaces.
tcshcommands are not so strict about spaces.
tcshuses theif-then-else if-else-endifsimilar to Fortran.
27 / 76

28. Comparison Operators
Integer Comparison
Operation bash tcsh
equal to if [ 1 -eq 2 ] if (1 == 2)
not equal to if [ $a -ne $b ] if ($a != $b)
greater than if [ $a -gt $b ] if ($a > $b)
greater than or equal toif [ 1 -ge $b ] if (1 >= $b)
less than if [ $a -lt 2 ] if ($a < 2)
less than or equal toif [ $a -le $b ] if ($a <= $b)
String Comparison
operation bash tcsh
equal to if [ $a == $b ] if ($a == $b)
not equal to if [ $a != $b ] if ($a != $b)
zero length or nullif [ -z $a ] if ($%a == 0)
non zero length if [ -n $a ] if ($%a > 0)
28 / 76

29. File Test & Logical Operators
File Test Operators
Operation bash tcsh
le exists if [ -e .bashrc ] if ( -e .tcshrc )
le is a regular le if [ -f .bashrc ]
le is a directory if [ -d /home ] if ( -d /home )
le is not zero size if [ -s .bashrc ] if ( ! -z .tcshrc)
le has read permission if [ -r .bashrc ] if ( -r .tcshrc)
le has write permissionif [ -w .bashrc ] if ( -w .tcshrc)
le has execute permissionif [ -x .bashrc ] if ( -x .tcshrc)
Logical Operators
Operation bash tcsh
Operation bash tcsh
NOT if [ ! -e .bashrc ] if ( ! -z .tcshrc)
AND if [ $a -eq 2 ] && [ $x -gt $y ] if ($a == 2 && $x<=$y )
OR if [[ $a -eq 2jj$x -gt $y ]] if ($a == 2jj$x<=$y )
29 / 76

30. Examples
Condition tests using theif/thenmay be nested
r e a d
i f
i f
e c h o
a n d
f i
f i
s e t
i f
i f
e c h o
0
e n d i f
e n d i f
This is same as
r e a d
i f
e c h o
5
f i
O R
i f
e c h o
5
f i
s e t
i f
e c h o
a n d
e n d i f
30 / 76

31. Loop Constructs
Aloopis a block of code that iterates a list of commands as long as theloop control
conditionis true.
Loop constructs available in
bash:for, whileanduntil
tcsh:foreachandwhile
31 / 76

32. bash: for loops
Theforloop is the basic looping construct inbash
f o r
d o
s o m e c o m m a n d s
d o n e
theforanddolines can be written on the same line:forarginlist;do
forloops can also use C style syntax
f o r
s o m e c o m m a n d s
d o n e
f o r
d o
t o u c h f i l e $ { i } . d a t
d o n e
f o r
t o u c h f i l e $ { i } . d a t
d o n e
f o r
d o
t o u c h f i l e $ { i } . d a t
d o n e
32 / 76

33. tcsh: foreach loop
Theforeachloop is the basic looping construct intcsh
f o r e a c h
s o m e c o m m a n d s
e n d
f o r e a c h
t o u c h f i l e $ i . d a t
e n d
33 / 76

34. while Construct
Thewhileconstruct tests for a condition at the top of a loop, and keeps looping as
long as that condition is true (returns a 0 exit status).
In contrast to aforloop, awhileloop nds use in situations where the number of loop
repetitions is not known beforehand.
bash
w h i l e
d o
s o m e c o m m a n d s
d o n e
tcsh
w h i l e
s o m e c o m m a n d s
e n d
factorial.sh
#
e c h o
r e a d
f a c t o r i a l = 1
w h i l e
d o
f a c t o r i a l = $ ( ( $ f a c t o r i a l * $ c o u n t e r ) )
c o u n t e r = $ ( ( $ c o u n t e r - 1 ) )
d o n e
e c h o
factorial.csh
#
e c h o
s e t
s e t
w h i l e
@
@
e n d
e c h o
34 / 76

35. until Contruct (bash only)
Theuntilconstruct tests for a condition at the top of a loop, and keeps looping as
long as that condition is false (opposite ofwhileloop).
u n t i l
d o
s o m e c o m m a n d s
d o n e
factorial2.sh
#
e c h o
r e a d
f a c t o r i a l = 1
u n t i l
f a c t o r i a l = $ [ $ f a c t o r i a l * $ c o u n t e r ]
i f
b r e a k
e l s e
l e t
f i
d o n e
e c h o
35 / 76

36. Nested Loops
for, while & untilloops can nested. To exit from the loop use thebreakcommand
nestedloops.sh
#
#
e c h o
f o r
e c h o
f o r
c = $ ( ( $ a * $ b ) )
i f
e c h o
e l s e
e c h o
b r e a k
f i
d o n e
d o n e
e c h o
e c h o
e c h o
f o r
e c h o
b = 1
w h i l e
c = $ ( ( $ a * $ b ) )
i f
e c h o
e l s e
e c h o
b r e a k
f i
l e t
d o n e
d o n e
e c h o
nestedloops.csh
#
#
e c h o
f o r e a c h
e c h o
f o r e a c h
@
i f
e c h o
e l s e
e c h o
b r e a k
e n d i f
e n d
e n d
e c h o
e c h o
e c h o
f o r e a c h
e c h o
s e t
w h i l e
@
i f
e c h o
e l s e
e c h o
b r e a k
e n d i f
@
e n d
e n d
e c h o
~ / T u t o r i a l s / B A S H / s c r i p t s / d a y 1 / e x a m p l e s > . /
n e s t e d l o o p s . s h
N e s t e d f o r l o o p s
V a l u e o f a i n o u t e r l o o p : 1
a * b = 1 * 1 = 1
a * b = 1 * 3 = 3
a * b = 1 * 5 = 5
V a l u e o f a i n o u t e r l o o p : 2
a * b = 2 * 1 = 2
a * b = 2 * 3 = 6
2 * 5 > 1 0
V a l u e o f a i n o u t e r l o o p : 3
a * b = 3 * 1 = 3
a * b = 3 * 3 = 9
3 * 5 > 1 0
V a l u e o f a i n o u t e r l o o p : 4
a * b = 4 * 1 = 4
4 * 3 > 1 0
V a l u e o f a i n o u t e r l o o p : 5
a * b = 5 * 1 = 5
5 * 3 > 1 0
= = = = = = = = = = = = = = = = = = = = = = = =
N e s t e d f o r a n d
V a l u e o f a i n o u t e r l o o p : 1
a * b = 1 * 1 = 1
a * b = 1 * 3 = 3
1 * 5 > 5
V a l u e o f a i n o u t e r l o o p : 2
a * b = 2 * 1 = 2
2 * 3 > 5
V a l u e o f a i n o u t e r l o o p : 3
a * b = 3 * 1 = 3
3 * 3 > 5
V a l u e o f a i n o u t e r l o o p : 4
a * b = 4 * 1 = 4
4 * 3 > 5
V a l u e o f a i n o u t e r l o o p : 5
5 * 1 > 5
= = = = = = = = = = = = = = = = = = = = = = = =
~ / T u t o r i a l s / B A S H / s c r i p t s > . / d a y 1 / e x a m p l e s /
n e s t e d l o o p s . c s h
N e s t e d f o r l o o p s
V a l u e o f a i n o u t e r l o o p : 1
a * b = 1 * 1 = 1
a * b = 1 * 3 = 3
a * b = 1 * 5 = 5
V a l u e o f a i n o u t e r l o o p : 2
a * b = 2 * 1 = 2
a * b = 2 * 3 = 6
2 * 5 > 1 0
V a l u e o f a i n o u t e r l o o p : 3
a * b = 3 * 1 = 3
a * b = 3 * 3 = 9
3 * 5 > 1 0
V a l u e o f a i n o u t e r l o o p : 4
a * b = 4 * 1 = 4
4 * 3 > 1 0
V a l u e o f a i n o u t e r l o o p : 5
a * b = 5 * 1 = 5
5 * 3 > 1 0
= = = = = = = = = = = = = = = = = = = = = = = =
N e s t e d f o r a n d
V a l u e o f a i n o u t e r l o o p : 1
a * b = 1 * 1 = 1
a * b = 1 * 3 = 3
1 * 5 > 5
V a l u e o f a i n o u t e r l o o p : 2
a * b = 2 * 1 = 2
2 * 3 > 5
V a l u e o f a i n o u t e r l o o p : 3
a * b = 3 * 1 = 3
3 * 3 > 5
V a l u e o f a i n o u t e r l o o p : 4
a * b = 4 * 1 = 4
4 * 3 > 5
V a l u e o f a i n o u t e r l o o p : 5
5 * 1 > 5
= = = = = = = = = = = = = = = = = = = = = = = =
36 / 76

37. Switching or Branching Constructs
Thecaseandselectconstructs are technically not loops, since they do not iterate the
execution of a code block.
Like loops, however, they direct program ow according to conditions at the top or bottom
of the block.
caseconstruct
c a s e
"
s o m e
; ;
"
s o m e o t h e r
; ;
e s a c
selectconstruct
s e l e c t v a r i a b l e [ l i s t ]
d o
c o m m a n d
b r e a k
d o n e
37 / 76

38. Switching or Branching Constructs
tcshhas theswitchconstruct
switchconstruct
s w i t c h
c a s e
s o m e c o m m a n d
b r e a k s w
e n d s w
38 / 76

39. dooper.sh
#
e c h o
r e a d
e c h o
o p e r a t i o n s = ' a d d s u b t r a c t m u l t i p l y d i v i d e e x p o n e n t i a t e
m o d u l o a l l q u i t '
s e l e c t
c a s e
"
e c h o
; ;
"
e c h o
; ;
"
e c h o
; ;
"
e c h o
; ;
"
e c h o
; ;
"
e c h o
; ;
"
e c h o
e c h o
e c h o
e c h o
e c h o
e c h o
; ;
* )
e x i t
; ;
e s a c
d o n e
dooper.csh
#
e c h o
s e t
s e t
e c h o
e c h o
s e t
s w i t c h
c a s e
@
e c h o
b r e a k s w
c a s e
@
e c h o
@
e c h o
@
e c h o
@
e c h o
@
e c h o
b r e a k s w
c a s e
@
e c h o
b r e a k s w
e n d s w
39 / 76

40. ~
P r i n t t w o n u m b e r s
1 4
W h a t o p e r a t i o n d o y o u w a n t t o d o ?
1 ) a d d 3 ) m u l t i p l y 5 ) e x p o n e n t i a t e 7 ) a l l
2 ) s u b t r a c t 4 ) d i v i d e 6 ) m o d u l o 8 ) q u i t
#
1 + 4 = 5
1 - 4 = -3
1 * 4 = 4
1 * * 4 = 1
1
1 % 4 = 1
#
~
P r i n t t w o n u m b e r s o n e a t a
1
5
W h a t o p e r a t i o n d o y o u w a n t t o d o ?
E n t e r + , - , x ,
a l l
1 + 5 = 6
1 - 5 = -4
1 * 5 = 5
1
1 % 5 = 1
40 / 76

41. dooper1.sh
#
e c h o
r e a d
e c h o
e c h o
e x p o n e n t i a t e
r e a d
c a s e
"
e c h o
; ;
"
e c h o
; ;
"
e c h o
; ;
"
e c h o
; ;
"
e c h o
; ;
"
e c h o
; ;
"
e c h o
e c h o
e c h o
e c h o
e c h o
e c h o
; ;
* )
e x i t
; ;
e s a c
~
P r i n t t w o n u m b e r s
2 5
W h a t o p e r a t i o n d o y o u w a n t t o d o ?
O p t i o n s a r e a d d , s u b t r a c t , m u l t i p l y , e x p o n e n t i a t e ,
d i v i d e , m o d u l o a n d a l l
a l l
2 + 5 = 7
2 - 5 = -3
2 * 5 = 1 0
2 * * 5 = 3 2
2
2 % 5 = 2
41 / 76

42. Arrays
bashandtcshsupports one-dimensional arrays.
Array elements may be initialized with thevariable[xx]notation
variable[xx]=1
Initialize an array during declaration
bashname=(firstname 'last name')
tcshset name = (firstname 'last name')
reference an elementiof an arrayname
$fname[i]g
print the whole array
bash$fname[@]g
tcsh$fnameg
print length of array
bash$f#name[@]g
tcsh$f#nameg
42 / 76

43. Arrays
print length of elementiof arrayname
$f#name[i]g
Note: Inbash$f#namegprints the length of the rst element of the array
Add an element to an existing array
bashname=(title $fname[@]g)
tcshset name = ( title "$fnameg")
Intcsheverything within "..." is one variable.
In the abovetcshexample,titleis rst element of new array while the second
element is the old arrayname
copy an arraynameto an arrayuser
bashuser=($fname[@]g)
tcshset user = ( $fnameg)
43 / 76

44. Arrays
concatenate two arrays
bashnameuser=($fname[@]g$fuser[@]g)
tcshset nameuser=( $fnameg$fuserg)
delete an entire array
unset name
remove an elementifrom an array
bashunset name[i]
tcsh@ j = $i - 1
@ k =$i + 1
set name = ( $fname[1-$j]g$fname[$k-]g)
bashthe rst array index is zero (0)
tcshthe rst array index is one (1)
44 / 76

45. Arrays
name.sh
#
e c h o
r e a d
n a m e = ( $ f i r s t n a m e $ l a s t n a m e )
e c h o
e c h o
r e a d
e c h o
r e a d
n a m e = ( $ t i t l e
e c h o
u n s e t
e c h o
name.csh
#
e c h o
s e t
e c h o
s e t
s e t
e c h o
e c h o
s e t
e c h o
s e t
s e t
e c h o
@
s e t
e c h o
~
P r i n t y o u r f i r s t a n d l a s t n a m e
A l e x P a c h e c o
H e l l o A l e x P a c h e c o
E n t e r y o u r s a l u t a t i o n
D r .
E n t e r y o u r s u f f i x
t h e f i r s t
H e l l o D r . A l e x P a c h e c o t h e f i r s t
H e l l o D r . A l e x t h e f i r s t
~
P r i n t y o u r f i r s t n a m e
A l e x
P r i n t y o u r l a s t n a m e
P a c h e c o
H e l l o A l e x P a c h e c o
E n t e r y o u r s a l u t a t i o n
D r .
E n t e r y o u r s u f f i x
t h e f i r s t
H e l l o D r . A l e x P a c h e c o t h e f i r s t
H e l l o D r . A l e x t h e f i r s t
45 / 76

46. Command Line Arguments
Similar to programming languages,bash(and other shell scripting languages) can also
take command line arguments
./scriptname arg1 arg2 arg3 arg4 ...
$0,$1,$2,$3, etc: positional parameters corresponding to
./scriptname,arg1,arg2,arg3,arg4,...respectively
$#: number of command line arguments
$*: all of the positional parameters, seen as a single word
$@: same as$*but each parameter is a quoted string.
shift N: shift positional parameters fromN+1to$#are renamed to variable
names from$1to$# - N + 1
Incsh,tcsh
an arrayargvcontains the list of arguments withargv[0]set to name of script.
#argvis the number of arguments i.e. length ofargvarray.
46 / 76

47. shift.sh
#
U S A G E =
i f
e c h o
e x i t
f i
e c h o
e c h o
e c h o
e c h o
w h i l e
e c h o
e c h o
s h i f t
d o n e
shift.csh
#
s e t
i f
e c h o
e x i t
e n d i f
e c h o
e c h o
e c h o
e c h o
w h i l e
e c h o
e c h o
s h i f t
e n d
d y n 1 0 0 0 8 5 :
N u m b e r o f A r g u m e n t s : 5
L i s t o f A r g u m e n t s : 1 2 3 4 5
N a m e o f s c r i p t t h a t y o u a r e r u n n i n g :
C o m m a n d Y o u E n t e r e d :
A r g u m e n t L i s t i s : 1 2 3 4 5
N u m b e r o f A r g u m e n t s : 5
A r g u m e n t L i s t i s : 2 3 4 5
N u m b e r o f A r g u m e n t s : 4
A r g u m e n t L i s t i s : 3 4 5
N u m b e r o f A r g u m e n t s : 3
A r g u m e n t L i s t i s : 4 5
N u m b e r o f A r g u m e n t s : 2
A r g u m e n t L i s t i s : 5
N u m b e r o f A r g u m e n t s : 1
d y n 1 0 0 0 8 5 :
N u m b e r o f A r g u m e n t s : 5
L i s t o f A r g u m e n t s : 1 2 3 4 5
N a m e o f s c r i p t t h a t y o u a r e r u n n i n g :
C o m m a n d Y o u E n t e r e d :
A r g u m e n t L i s t i s : 1 2 3 4 5
N u m b e r o f A r g u m e n t s : 5
A r g u m e n t L i s t i s : 2 3 4 5
N u m b e r o f A r g u m e n t s : 4
A r g u m e n t L i s t i s : 3 4 5
N u m b e r o f A r g u m e n t s : 3
A r g u m e n t L i s t i s : 4 5
N u m b e r o f A r g u m e n t s : 2
A r g u m e n t L i s t i s : 5
N u m b e r o f A r g u m e n t s : 1
47 / 76

48. Declare command
Use thedeclarecommand to set variable and functions attributes.
Create a constant variable i.e. read only variable
Syntax:
declare -r var
declare -r varName=value
Create an integer variable
Syntax:
declare -i var
declare -i varName=value
You can carry out arithmetic operations on variables declared as integers
~
1 0
~
2
48 / 76

49. Functions
Like "real" programming languages,bashhas functions.
A function is a subroutine, a code block that implements a set of operations, a "black
box" that performs a specied task.
Wherever there is repetitive code, when a task repeats with only slight variations in
procedure, then consider using a function.
f u n c t i o n
c o m m a n d
}
O R
f u n c t i o n _ n a m e ( ) {
c o m m a n d
}
49 / 76

50. Functions
shift10.sh
#
u s a g e ( ) {
e c h o
e x i t
}
[ [
e c h o
e c h o
e c h o
e c h o
e c h o
e c h o
$ { 1 1 }
e c h o
e c h o
s h i f t
e c h o
e c h o
d y n 1 0 0 0 8 5 :
U S A G E :
d y n 1 0 0 0 8 5 :
U S A G E :
d y n 1 0 0 0 8 5 :
N u m b e r o f A r g u m e n t s : 1 1
L i s t o f A r g u m e n t s : 1 3 5 7 9 1 1 1 3 1 5 1 7 1 9 2 1
N a m e o f s c r i p t t h a t y o u a r e r u n n i n g :
C o m m a n d Y o u E n t e r e d :
2 1
F i r s t A r g u m e n t 1
T e n t h a n d E l e v e n t h a r g u m e n t 1 0 1 1 1 9 2 1
A r g u m e n t L i s t i s : 1 3 5 7 9 1 1 1 3 1 5 1 7 1 9 2 1
N u m b e r o f A r g u m e n t s : 1 1
A r g u m e n t L i s t i s : 1 9 2 1
N u m b e r o f A r g u m e n t s : 2
d y n 1 0 0 0 8 5 :
N u m b e r o f A r g u m e n t s : 1 2
L i s t o f A r g u m e n t s : 2 1 2 3 2 5 2 7 2 9 3 1 3 3 3 5 3 7 3 9 4 1 4 3
N a m e o f s c r i p t t h a t y o u a r e r u n n i n g :
C o m m a n d Y o u E n t e r e d :
3 7 3 9 4 1 4 3
F i r s t A r g u m e n t 2 1
T e n t h a n d E l e v e n t h a r g u m e n t 2 1 0 2 1 1 3 9 4 1
A r g u m e n t L i s t i s : 2 1 2 3 2 5 2 7 2 9 3 1 3 3 3 5 3 7 3 9 4 1 4 3
N u m b e r o f A r g u m e n t s : 1 2
A r g u m e n t L i s t i s : 3 9 4 1 4 3
N u m b e r o f A r g u m e n t s : 3
50 / 76

51. Functions
You can also pass arguments to a function.
All function parameters or arguments can be accessed via $1, $2, $3,..., $N.
$0 always point to the shell script name.
$* or $@ holds all parameters or arguments passed to the function.
$# holds the number of positional parameters passed to the function.
Array variable calledFUNCNAMEcontains the names of all shell functions currently in
the execution call stack.
By default all variables are global.
Modifying a variable in a function changes it in the whole script.
You can create a local variables using thelocalcommand
Syntax:
local var=value
local varName
51 / 76

52. Functions
A function may recursively call itself even without use of local variables.
factorial3.sh
#
u s a g e ( ) {
e c h o
e x i t
}
f a c t o r i a l ( ) {
l o c a l
l o c a l
d e c l a r e
d e c l a r e
i f
e c h o
e l i f
e c h o
e l s e
f = $ ( ( $ i - 1 ) )
f = $ ( f a c t o r i a l $ f )
f = $ ( ( $ f * $ i ) )
e c h o
f i
}
i f
u s a g e
e l s e
f o r
x = $ ( f a c t o r i a l $ i )
e c h o
d o n e
f i
d y n 1 0 0 0 8 5 :
F a c t o r i a l o f 1 i s 1
F a c t o r i a l o f 3 i s 6
F a c t o r i a l o f 5 i s 1 2 0
F a c t o r i a l o f 7 i s 5 0 4 0
F a c t o r i a l o f 9 i s 3 6 2 8 8 0
F a c t o r i a l o f 1 1 i s 3 9 9 1 6 8 0 0
52 / 76

53. Scripting for Job Submission
Problem Description
I have to run more than one serial job.
Solution: Create a script that will submit and run multiple serial jobs.
I don't want to submit multiple jobs using the serial queue since
Cluster Admins give lower priority to jobs that are not parallelized
The number of jobs that I want to run exceed the maximum number of jobs that
I can run simultaneously
How do I submitonejob which can run multiple serial jobs?
One Solution of many
Write a script which will log into all unique nodes and run your serial jobs in
background.
Easy said than done
What do you need to know?
1
Shell Scripting
2
How to run a job in background
3
Know what thewaitcommand does
53 / 76

54. [ a l p 5 1 4 . s o l ] ( 1 0 1 2 ) :
#
#
#
#
#
#
#
#
#
#
e x p o r t
s r u n - s h o s t n a m e > h o s t f i l e
e x p o r t
N O D E S = ( `
U N O D E S = ( ` s o r t h o s t f i l e | u n i q ` )
e c h o
e c h o
e c h o
i = 0
f o r
s s h - n $ n o d e s '
l e t
d o n e
w a i t
e c h o
i = 0
N P R O C S = ` s o r t h o s t f i l e | u n i q | w c - l | g a w k ' / / { p r i n t $ 1 } ' `
l e t
w h i l e
s s h - n $ { U N O D E S [ $ i ] } '
l e t
d o n e
[ a l p 5 1 4 . s o l ] ( 1 0 1 3 ) : s b a t c h - p i m l a b c h e c k n o d e s . s l r
S u b m i t t e d b a t c h j o b 6 2 0 0 4 5
54 / 76

55. [ a l p 5 1 4 . s o l ] ( 1 0 1 4 ) : c a t n o d e t e s t . o u t
N o d e s A v a i l a b l e : s o l - b 4 1 1 s o l - b 4 1 1 s o l - b 4 1 1 s o l - b 4 1 1 s o l - b 4 1 3 s o l - b 4 1 2 s o l - b 5 0 1 s o l - b 4 1 3 s o l - b 4 1 3 s o l - b 4 1 3
s o l - b 4 1 2 s o l - b 4 1 2 s o l - b 4 1 2 s o l - b 5 0 1 s o l - b 5 0 1 s o l - b 5 0 1
U n i q u e N o d e s A v a i l a b l e : s o l - b 4 1 1 s o l - b 4 1 2 s o l - b 4 1 3 s o l - b 5 0 1
G e t H o s t n a m e s f o r a l l p r o c e s s e s
s o l - b 5 0 1 1 4
s o l - b 5 0 1 6
s o l - b 5 0 1 1 5
s o l - b 5 0 1 1 3
s o l - b 4 1 3 4
s o l - b 4 1 3 9
s o l - b 4 1 2 1 1
s o l - b 4 1 2 1 0
s o l - b 4 1 3 7
s o l - b 4 1 3 8
s o l - b 4 1 2 5
s o l - b 4 1 1 1
s o l - b 4 1 2 1 2
s o l - b 4 1 1 3
s o l - b 4 1 1 2
s o l - b 4 1 1 0
G e t H o s t n a m e s f o r a l l u n i q u e n o d e s
s o l - b 4 1 1 0
s o l - b 4 1 2 1
s o l - b 4 1 3 2
s o l - b 5 0 1 3
55 / 76

56. Unix Utilities

57. grep
grepis a Unix utility that searches through either information piped to it or les in the
current directory.
egrepis extended grep, same asgrep -E
Usezgrepfor compressed les.
Usage:grep <options> <search pattern> <files>
Commonly used options
Option Operation
-i ignore case during search
-r search recursively
-v invert match i.e. match everything except pattern
-l list les that match pattern
-L list les that do not match pattern
-n prex each line of output with the line number within its input le.
-A num print num lines of trailing context after matching lines.
-B num print num lines of leading context before matching lines.
57 / 76

58. sed
sed (\stream editor") is Unix utility for parsing and transforming text les.
sed is line-oriented, it operates one line at a time and allows regular expression
matching and substitution.
sed has several commands, the most commonly used command and sometime the only
one learned is the substituion command,s
~
#
#
e c h o
List of sed pattern ags and commands line options
Pattern Operation
s substitution
g global replacement
p print
I ignore case
d delete
G add newline
w write to le
x exchange pattern with hold buer
h copy pattern to hold buer
Command Operation
-e combine multiple commands
-f read commands from le
-h print help info
-n disable print
-V print version info
-i in le subsitution
sed one-liners:http://sed.sourceforge.net/sed1line.txt
sed is a handy utility very useful for writing scripts for le manipulation.
58 / 76

59. awk programming

60. The Awk text-processing language is useful for such tasks as:
FTallying information from text les and creating reports from the results.
FAdding additional functions to text editors like \vi".
FTranslating les from one format to another.
FCreating small databases.
FPerforming mathematical operations on les of numeric data.
Awk has two faces:
Fit is a utility for performing simple text-processing tasks, and
Fit is a programming language for performing complex text-processing tasks.
awk comes in three variations
awk
nawk
gawk
symbolic link to gawk.
Simplest form of using awk
awkpatternfactiong
Most common action:print
Print le dosum.sh:awk 'fprint $0g' dosum.sh
Print line matching bash in all les in current directory:
awk '/bash/fprint $0g' *.sh
60 / 76

61. awk patterns may be one of the following
BEGIN
Mostly used for preprocessing, setting constants, etc. before input is read.
END
Mostly used for postprocessing after input has been read.
/regular expression/
is read
relational expression
&&
Execute action if pattern1 and pattern2 are true
jj: logical OR operator used as pattern1 || pattern2.
Execute action if either pattern1 or pattern2 is true
!
Execute action if pattern is not matched
?:
If pattern1 is true use pattern2 for testing else use pattern3
pattern1, pattern2
pattern1 continuing until a record has been reached that matches pattern2
print expressionis the most common action in the awk statement. If formatted output
is required, use theprintf format, expressionaction.
Format speciers are similar to the C-programming language
%d,%i
%e,%E ]dd. The %E format
uses E instead of e.
%f
61 / 76

62. %g,%G
format uses %E instead of %e
%s
Format speciers have additional parameter which may lie between the % and the
control letter
0
with zeroes instead of spaces.
width
with spaces. If the 0 ag has been used, it is padded with zeroes.
.prec
string constants supported by awk
nn: Literal backslash
nn
nr
nt
nv
~
n ' ' , $1 , $2 , $3 , $ 3 } '
h e l l o 0 . 2 4 8 5 0 0
5
0 0 0 0 5
The print command puts an explicit newline character at the end while the printf
command does not.
62 / 76

63. awk has in-built support for arithmetic operations
Operation Operator
Addition +
Subtraction -
Multiplication *
Division /
Exponentiation **
Modulo %
Assignment Operation Operator
Autoincrement ++
Autodecrement {
Add result to varibale +=
Subtract result from variable -=
Multiple variable by result *=
Divide variable by result /=
~
1
~
2
awk also supports trignometric functions such as sin(expr) and cos(expr) where expr is
in radians and atan2(y/x) where y/x is in radians
~
3 . 1 4 1 5 9 1 . 2 2 4 6 5 e - 1 6 -1
63 / 76

64. Other Arithmetic operations supported are
exp(expr)
int(expr)
log(expr)
sqrt(expr)
rand() Nbetween 0 and 1 such that 0N <1
srand(expr)
provided, time of day is used.
awksupports the if and while conditional and for loops
if and while conditionals work similar to that in C-programming
i f
c o m m a n d 1 ;
c o m m a n d 2
}
w h i l e
c o m m a n d 1 ;
c o m m a n d 2
}
64 / 76

65. awk supports if ... else if .. else conditionals.
i f
c o m m a n d 1 ;
c o m m a n d 2
}
c o m m a n d 3
}
c o m m a n d 4
}
Relational operators supported by if and while
==
!=
>: Is greater than
>=
<: Is less than
<=
: String Matches to
!: Doesn't Match
~
1 :
2 :
3 :
4 :
5 :
65 / 76

66. The for command can be used for processing the various columns of each line
~
} }
6 1
Like all progamming languages, awk supports the use of variables. Like Shell, variable
types do not have to be dened.
awk variables can be user dened or could be one of the columns of the le being
processed.
~
#
#
e c h o
~
#
#
e c h o
Unlike Shell, awk variables are referenced as is i.e. no $ prepended to variable name.
awk one-liners:http://www.pement.org/awk/awk1line.txt
66 / 76

67. awk can also be used as a programming language.
The rst line in awk scripts is the shebang line (#!) which indicates the location of the awk
binary. Usewhich awkto nd the exact location
On my Linux desktop, the location is /usr/bin/awk.
If unsure, just use /usr/bin/env awk
hello.awk
#
B E G I N {
p r i n t
}
~
H e l l o W o r l d !
To support scripting, awk has several built-in variables, which can also be used in one line
commands
ARGC
ARGV
FILENAME
FS
OFS
ORS
67 / 76

68. awk permits the use of arrays
arrays are subscripted with an expression between square brackets ([ ])
hello1.awk
#
B E G I N {
x [ 1 ] =
x [ 2 ] =
x [ 3 ] =
f o r
p r i n t f
}
~
H e l l o , W o r l d !
Use the delete command to delete an array element
awk has in-built functions to aid writing of scripts
length
toupper
tolower
split
separator
gsub
pattern",string)
68 / 76

69. getline
Similar to bash, GNU awk also supports user dened function
#
{
i f
e r r o r ( ` ` E x p e c t e d 4 f i e l d s ' ' ) ;
}
p r i n t ;
}
}
f u n c t i o n
i f
p r i n t f
}
p r i n t f
` `/
}
69 / 76

70. getcpmdvels.sh
#
n a r g = ( $
i f
e c h o
e x i t
f i
n a t o m = $ 1
v e l s = $ 2
c a t
a w k
p r i n t f
}
p r i n t f
} \
} ' > $ v e l s
getengcons.sh
#
b a s h b a s h
G M S O U T = $ 1
g r e p ' T I M E M O D E ' $ G M S O U T | h e a d -1 > e n e r g y . d a t
a w k
70 / 76

71. #
n a r g = ( $
i f
e c h o
V e l o c i t y
e x i t
f i
g m s o u t = $ 1
n a t o m s = $ 2
d e l t a t = $ 3
c o o r d s = $ 4
v e l s = $ 5
f t v e l s = $ 6
a u 2 a n g = 0 . 5 2 9 1 7 7 1
s e c 2 f s = 1 e 1 5
m a s s = m a s s . d a t
r m - r f $ v e l s $ c o o r d s $ f t v e l s
#
c a t
e g r e p - i = | \
s e d
x a r g s |
#
#
#
#
#
#
#
#
#
a w k
i c o u n t = 3 ; \
p r i n t f
w h i l e
p r i n t $ 0 ; \
+ + i c o u n t \
} \
} ' $ g m s o u t |
#
71 / 76

72. #
#
c a t
a w k
p r i n t f
}
p r i n t $ 0 \
} \
} ' > $ c o o r d s
c a t
a w k
p r i n t f
} \
e l s e
p r i n t f
} \
} ' > $ v e l s
r m - r f t m p . $ $
o c t a v e - q < < E O F
v e l s = l o a d (
a t m a s s = l o a d (
a t m a s s = d i a g ( a t m a s s ) ;
m w v e l s = v e l s * a t m a s s ;
f t m w v e l s = a b s ( f f t ( m w v e l s ) ) ;
N = r o w s ( f t m w v e l s ) ;
M = c o l u m n s ( f t m w v e l s ) ;
d e l t a w = 1 / N / $ d e l t a t ;
f i d = f o p e n (
f o r
s u m f t = 0 ;
f o r
s u m f t = s u m f t + f t m w v e l s ( I , J ) ^ 2 ;
e n d f o r
f p r i n t f ( f i d ,
e n d f o r
f c l o s e ( f i d ) ;
E O F
72 / 76

73. getmwvels.awk
#
B E G I N
i f
p r i n t f
e x i t
}
g a u l o g = A R G V [ 1 ] ;
n a t o m = A R G V [ 2 ] ;
v e l s = A R G V [ 3 ] ;
d e l e t e
d e l e t e
}
/ ^ * M W C a r t e s i a n v e l o c i t y : / {
i c o u n t = 1 ;
w h i l e
i f
g s u b
p r i n t f
}
+ + i c o u n t ;
}
p r i n t f
}
73 / 76

74. gettrajxyz.awk
#
B E G I N
i f
p r i n t f
e x i t
}
g a u l o g = A R G V [ 1 ] ;
n a t o m = A R G V [ 2 ] ;
c o o r d s = A R G V [ 3 ] ;
d e l e t e
d e l e t e
}
/ ^ * I n p u t o r i e n t a t i o n : / {
i c o u n t = 1 ;
p r i n t f
w h i l e
i f
p r i n t f
}
+ + i c o u n t ;
}
}
74 / 76

75. Wrap Up

76. References & Further Reading
BASH Programming http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html
Advanced Bash-Scripting Guidehttp://tldp.org/LDP/abs/html/
Regular Expressionshttp://www.grymoire.com/Unix/Regular.html
AWK Programming http://www.grymoire.com/Unix/Awk.html
awk one-liners:http://www.pement.org/awk/awk1line.txt
sedhttp://www.grymoire.com/Unix/Sed.html
sed one-liners:http://sed.sourceforge.net/sed1line.txt
CSH Programming http://www.grymoire.com/Unix/Csh.html
csh Programming Considered Harmful
http://www.faqs.org/faqs/unix-faq/shell/csh-whynot/
Wiki Bookshttp://en.wikibooks.org/wiki/Subject:Computing
76 / 76