Unix Shell Script - 2 Days Session.pptx

Bootcamp – Unix & Shell Script
By Rajesh Kumar
Day 1

What is Unix/Linux?
Unix is a modular OS made up of a number of essential components, including the kernel, shell, file
system and a core set of utilities or programs. At the heart of the Unix OS is the kernel, a master
control program that provides services to start and end programs.
 Unix was originally developed in 1969 by a group of AT&T employees at Bell Labs, including Ken Thompson, Dennis Ritchie,
Douglas McIlroy, and Joe Ossanna.
 There are various Unix variants available in the market. Solaris Unix, AIX, HP Unix and BSD are few examples. Linux is also a
flavor of Unix which is freely available
 Several people can use a UNIX computer at the same time; hence UNIX is called a multiuser system.
 A user can also run multiple programs at the same time; hence UNIX is called multitasking.
 UNIX OS is commercial and more select audience (i.e.. servers). Where Linux is a free, open-source operating system (OS).
Linux is designed for a general-purpose audience (i.e.. Desktops/Laptops/Mini server’s).

Unix Architecture:
• The main concept that unites all versions of UNIX is the following four
basics:
• Kernel: The kernel is the heart of the operating system. It interacts with
hardware and most of the tasks like memory management, task
scheduling and file management.
• Shell: The shell is the utility that processes your requests. When you
type in a command at your terminal, the shell interprets the command
and calls the program that you want. The shell uses standard syntax for
all commands. C Shell, Bourne Shell and Korn Shell are most famous
shells which are available with most of the Unix variants.
• Commands and Utilities: There are various command and utilities
which you would use in your day-to-day activities. cp, mv, cat and grep
etc. are few examples of commands and utilities. There are over 250
standard commands plus numerous others provided through 3rd party
software. All the commands come along with various optional options.
• Files and Directories: All data in UNIX is organized into files. All files are
organized into directories. These directories are organized into a tree-
like structure called the filesystem.

UNIX Filesystem
• Unix uses a hierarchical file system structure, much like an upside-down
tree, with root (/) at the base of the file system and all other directories
spreading from there.
/ → This is the root directory which should contain only the directories needed at the top level of the file structure.
/bin → essential command binaries
/boot → static files of the boot loader
/dev → device files (special files to access your devices)
/etc → host-specific system configuration files
/home → user home directories (e.g., /home/peter, /home/sarah, etc)
/lib → essential shared libraries and kernel modules
/media → mount point for removable media (e.g., CD-ROMs & flash disks)
/mnt → “old-style” mount point for any media
/tmp → system-wide temporary folder, writable by anyone
/sbin → Contains binary (executable) files, usually for system administration. For example, fdisk and ifconfig utlities.

Files and Directories Permissions
File & directory has the following attributes:
• Owner permissions: The owner's permissions determine what actions the owner of the file can perform on the file.
• Group permissions: The group's permissions determine what actions a user, who is a member of the group that a file belongs to, can perform
on the file.
• Other (world) permissions: The permissions for others indicate what action all other users can perform on the file.

Shell (.bashrc file) and Environment Variables
• .bashrc configuration file - .bashrc file is a script file that's executed when a user logs
in. The file itself contains a series of configurations for the terminal session. This
includes setting up or enabling coloring, completion, shell history, command aliases,
and more. It is a hidden file and simple ls command won't show the file.
• #Print all and Check the default environment variables
printenv or env
echo $SHELL
echo $HOME
• #EXPORT command will set the new variable and add the path into existing variable
export V_TEST="Hello World“
echo $V_TEST
export PATH=$PATH:/place/with/the/file
echo $PATH

User Login and Password
• #ssh or telnet – command to login into server with ip address or hostname
$ssh username@ip_address_or_hostname
• # su command lets you switch the current user to any other user.
$su – <username>
• # If you prefix “sudo” with any command, it will run that command with
elevated privileges or in other words allow a user with proper permissions to
execute a command as another user. This is the equivalent of “run as
administrator”
$ sudo su – <username>
• #passwd command to change your user password
$passwd <username>
• #The whoami command allows Linux users to see the currently logged-in user
$whoami
•
Tools to connect servers via (SSH and telnet client):
• Putty, WinSCP etc.

Vi Editor
• The default editor that comes with the UNIX operating system is called vi
(visual editor). Using vi editor, we can edit an existing file or create a new file
from scratch. we can also use this editor to just read a text file.
• vi filename : Creates a new file if it already not exist, otherwise opens existing
file.
• vi -R filename : Opens an existing file in read only mode.
• view filename : Opens an existing file in read only mode.
• #Create a new file
$vi filename.txt
• Press i to insert mode
• :wq <save>
:q <not-save>
•
#Search Text in file
/<searchtext> Enter
• Press n for next shift+n for previous

Unix Commands
• ls – is a command to list computer files and directories
• cd – is a command-line shell command used to change the current working directory
• pwd – the pwd command writes the full pathname of the current working directory to the standard output.
• chown – chown command is used to change the file Owner or group
• $chown -v username file1.txt
• chmod – the chmod command is used to change the access mode of a file
• $ chmod u=rw two.py
• $ chmod o=rw two.py
• $ chmod g=rw two.py
• $ ls -l two.py
• -rw-rw-rw- 1 EAD+rkumar446 EAD+Domain Users 337 May 3 16:25 two.py
• touch – The touch command's primary function is to modify a timestamp. Commonly, the utility is used for file creation.
• $ touch -d "2012-10-19 12:12:12.000000000 +0530" two.py
• $ ls --full-time two.py
• -rw-rw-rw- 1 EAD+rkumar446 EAD+Domain Users 337 2012-10-19 12:12:12.000000000 +0530 two.py

• mkdir – The mkdir command in Linux/Unix allows users to create or make new directories.
• rmdir – rmdir command is used remove empty directories from the filesystem
• rm – The rm command removes the entries for a specified file, group of files, or certain select files from a list within a directory.
• echo – echo command in linux is used to display line of text/string that are passed as an argument .
• cat – The cat command is a utility command in Linux. One of its most common usages is to print the content of a file onto the standard
output stream.
• grep – The grep filter searches a file for a particular pattern of characters and displays all lines that contain that pattern.
• $ grep "run" two.py
• $ grep -n "run" two.py
• $ grep -i "ruN" two.py
• $ grep -n -C1 "run" two.py
• head – head command prints the first lines of one or more files (or piped data) to standard output.
• tail –The tail command, as the name implies, print the last N number of data of the given input.
• more –command is used to view the text files in the command prompt, displaying one screen at a time in case the file is large
• uname – To display system information, use the uname command. Displays the operating system name as well as other system related
information like node, release, OS version, hardware name, and processor type etc. Exp: uname -a

• find – The “find” command is used to find, filter, or search files and folders in your system according to user-specified conditions and
perform several operations.
• $ find ./ -name "two.py"
• $ find ./ -type f -name "two.py"
• $ find ./ -type f -name "*.py"
• $ find ./ -type d -name "tt"
• $ find ./ -type f -name "*.py" -exec grep 'run' {} ;
• $ find ./ -type f -name "*.py" -print -exec grep -n 'run' {} ;
• ping – PING (Packet Internet Groper) command is used to check the network connectivity between host and server/host.
• top – top command is used to show the Linux processes. It provides a dynamic real-time view of the running system.
• file – file command is used to determine the type of a file. .file type may be of human-readable(e.g. 'ASCII text') or MIME type (e.g.
'text/plain; charset=us-ascii').
• cp – cp stands for copy. This command is used to copy files or group of files or directory.
• mv – The mv command is one of the basic Linux commands that is used to move files and directories from one location to another.
• wc – is a command line utility for printing newline, word and byte counts for files. Exp: wc -l/w/c
• cut – cut is a command for cutting out the sections from each line of files and writing the result to standard output.
• $ echo 'An|exception|occurred|at|runtime'|cut -d '|' -f1
• $ echo 'An|exception|occurred|at|runtime'|cut -d '|' -f2
• $ echo 'An|exception|occurred|at|runtime'|cut -c 1-5

• which – The Linux which command is used to locate the executable files or location of a
program from the file system.
• tr – The tr command is a Linux command-line utility that translates or deletes characters from
standard input ( stdin ) and writes the result to standard output ( stdout ).
• $ echo 'An|exception|occurred|at|runtime'|tr e E
• $ echo 'An|exception|occurred|at|runtime'|tr A-Z a-z
• $ echo 'An|exception|occurred|at|runtime'|tr -d 'e'
• ps – ps command is used to list the currently running processes and their PIDs along with
some other information depends on different options. It reads the process information from
the virtual files in /proc file-system.
• $ ps
• PID PPID PGID WINPID TTY UID STIME COMMAND
971 1 971 24872 ? 1074028310 15:12:41 /usr/bin/minty
972 971 972 24324 pty0 1074028310 15:12:44 /usr/bin/bash
1351 972 1351 20808 pty0 1074028310 11:38:18 /usr/bin/ps
• kill – kill command in Linux (located in /bin/kill), is a built-in command which is used to
terminate processes manually.
• $ kill -l
• $ kill pid
• $ kill -9 pid
• history – history command is used to view the previously executed command.

• du – to check how much space a file or a directory takes, the du (Disk Usage) command
• df – to get a report on the system’s disk space usage, shown in percentage and KBs.
• tar – The Linux ‘tar’ stands for tape archive, is used to create Archive and extract the Archive
files.
• $ tar cvf file.tar *.c
• $ tar xvf file.tar
• expr – The expr command in Unix evaluates a given expression and displays its corresponding
output.
• $ expr --help
• $ expr 12 + 12
• $ expr 12 * 2
• Help yourself
• man – man command in Linux is used to display the user manual of any command that we can run on
the terminal. It provides a detailed view of the command which includes NAME, SYNOPSIS,
DESCRIPTION, OPTIONS, EXIT STATUS, RETURN VALUES, ERRORS, FILES, VERSIONS, EXAMPLES, AUTHORS
and SEE ALSO.
• [username@host ~]$ man ls
...

Bootcamp – Unix & Shell Script
By Rajesh Kumar
Day 2

What is Shell Script?
• A shell script is a computer program designed to be run by the Unix shell, a command-line interpreter. The various
dialects of shell scripts are considered to be scripting languages. Typical operations performed by shell scripts
include file manipulation, program execution, and printing text. A script which sets up the environment, runs the
program, and does any necessary cleanup or logging, is called a wrapper (another way to extend primary
object/commands behavior).
• The basic concept of a shell script is a list of commands, which are listed in the order of execution. A good shell
script will have comments, preceded by a pound sign, #, describing the steps.
• There are conditional tests, such as value A is greater than value B, loops allowing us to go through massive
amounts of data, files to read and store data, and variables to read and store data, and the script may include
functions.
• Shell scripts and functions are both interpreted. This means they are not compiled.
The whole code of shell is in the text file ASCII Character

Why Shell Script ?
 Shell scripts can be used to prepare input files, job monitoring, and output processing etc.
 To avoid repetitive work and automation
 Useful to create own commands with help of alias (to list all execute $alias –p, alias l=‘ls –la’)
 Save lots of time on file processing.
 To automate some task of day-to-day activity.
 System Administration part can be also automated.
 Use shell scripting for routine backups, text Parsing, cleanup etc. activity.

Components of a script
• A series of OS commands for execution Stored in a text file.
• A shell script have syntax just like any other programming language. A shell script comprises
following elements:
• -Shell Keywords – if, else, break etc.
-Shell commands – cd, ls, echo, pwd, touch etc.
-Functions
-Control flow – if..then..else, case and shell loops etc.
#!/bin/sh
rm -f /tmp/listing.tmp > /dev/null 2>&1
touch /tmp/listing.tmp
ls -l [a-z]*.doc | sort > /tmp/listing.tmp
lpr -Ppostscript_1 /tmp/listing.tmp
rm -f /tmp/listing.tmp

Sample .sh Code
#!/bin/sh
rm -f /tmp/listing.tmp > /dev/null 2>&1
touch /tmp/listing.tmp
# This is a comment
ls -l [a-z]*.doc | sort > /tmp/listing.tmp
lpr -Ppostscript_1 /tmp/listing.tmp
rm -f /tmp/listing.tmp
Shell Interpreter (sh, csh, ksh)
Comment
Command

How to invoke a script ?
• Make it executable
• $ chmod +x path/to/our/file/my_script.sh
• Correct way
$ /bin/ksh my_script arg_1 arg_2
• $ /bin/csh my_script arg_1 arg_2
• $ /bin/sh my_script arg_1 arg_2
• Simple way
$ my_script arg_1 arg_2
Or
• $ ./my_script arg_1 arg_2

Execute a script in background
• At Command Prompt
$ nohup sh my_script arg_1 arg_2 > /dir/file.log &
Or
• $ nohup sh my_script arg_1 arg_2 > /dir/file.log 2>&1
• Means:
• 2 refers to the second file descriptor of the process, i.e.
stderr.
• > means redirection.
• &1 target of the redirection should be the same location as the
first file descriptor, i.e. stdout.
•
So this command first redirects stdout to /dev/null and then
redirects stderr there as well.

Useful Metacharacters
• $ Variable Substitution or expand the value of Variable.
• > used for Output Redirection.
• >> used for Output Redirection to append.
• < Input redirection.
• << used for input redirection and is also known as here document.
• * Match any number of characters, Substitution wildcard for zero or more characters
• ? Match one character, Substitution wildcard for 1 character
• [ ] Match range of characters, Substitution wildcard for any character between brackets
• `cmd` Replace cmd with the command to execute and will execute that, Substitution wildcard for command execution
• $(cmd) Replace cmd with the command to execute and will execute that, Substitution wildcard for command execution
• | Pipe is a Redirection the output of one command/program/process to another command for further processing.
• ; Command separator is used to execute 2 or more commands with one statement.
• || OR conditional execution of the commands.
• && AND conditional execution of the commands.
• ( ) Groups the command in to one output stream.

Example (metacharacters)
List files having prefix new
$ ls new*
Cat files having prefix ch and one more letter
$ cat ch?
List files starting by letters from D to E
$ ls [D-R]*
• Use of Array, Used in special cases for variables with the $.
• $ numbers=( 1 2 3 4 5 )
• $ echo ${numbers[1]}

Quoting
“ “ Everything taken literally, except
 $ (variable substitution)
 ` (command substitution)
 “ (ending mark)
‘ ‘ Everything taken literally
Escape next Character
# to comment something.

Example (quoting)
$ echo 'my class is "unix and tools"'
My class is "unix and tools“
$ echo "Well, isn't that "good" ?"
Well, isn't that "good" ?
$ echo "You have `ls | wc –l` files in `pwd`"
You have 34 files in /home/user
$ echo "The value of $x is $x"
The value of $x is 100

Command forms
cmd1 ; cmd2 Multiple commands on the same line
{ cmd1 ; cmd2 } Commands as a group in current shell
(cmd1 ; cmd2) Commands as a group in a subshell
cmd1 | cmd2 Pipe
cmd1 && cmd2 AND; cmd1, then cmd2 if (cmd succeeds)
cmd1 || cmd2 OR

Example (command forms)
$ cd; ls
Execute sequentially
$ (date; who; pwd) > logfile
All output is redirected
$ sort file | pr -3
Sort file, page output, then print
$ grep XX file && cat file
Print file if it contains the pattern;
$ grep XX file || echo "XX not found"
otherwise, echo not found message

Arguments
#!/bin/sh
##############################
echo "Script name is [$0]"
echo "First argument is [$1]"
echo "Second argument is [$2]"
echo "This process ID is [$$]"
echo "This argument count is [$#]"
echo "All arguments [$@]“
• The variable $*, is similar, but does not preserve any whitespace, and
quoting,
so "File with spaces" becomes "File" "with" "spaces".

Example (Arguments) Output:
hoai@moon:~> my_script.sh DBNAME 1 BACKUP
Script name is [my_script.sh]
First argument is [DBNAME]
Second argument is [1]
This process ID is [5401]
This argument count is [3]
• All arguments [DBNAME 1 BACKUP]

Simple commands
sort Sort lines
grep Search for regular expressions
find search for files in a directory hierarchy
sed stream editor for filtering and transforming text
cut Chop up a text by strings or characters
awk Pattern scanning and processing language
tr 'a' 'b' Transform characters
expr Simple arithmetic processor
date Create date strings
head/tail Access lines in files

Example (Simple commands -1)
• #Sort
sort data.txt >sort_f.txt OR sort -o output.txt data.txt
sort -k 2 data.txt
• #grep
• grep –nv ^$ file.txt
• grep –w “string” filename
• #Sed
• sed –n ‘25p’ file.txt
• sed ‘s/n/xyz/’ file.txt OR sed ‘s/n/xyz/g’ file.txt
• sed –e ‘s/f/rr/g’ –e ‘s/a/1234/g’ –e ‘s/n/UNIX/g’ file.txt
• #Find
• find /path1/path2/ -type f -name "Alca*tel.txt"
• find ./ -type f|xargs grep “string”

Example (Simple commands -2)
• #Cut
• cut –c10-20 file.txt
• cut -d "," -f4 file.txt
• #awk
• cat /passwd|awk -F" " '{print NF}' OR cat /passwd|awk -F" " '{print NR}'
• cat /passwd|awk –F":" '{print $1}'cat /passwd|awk -F":" '{if($3>30)print $0}'
• #tr
• echo "have a good day" | tr "[:lower:]" "[:upper:]"
• echo "linuxserver" | tr -d "linux"
• echo "have a good day:"| tr ':' '.'
• #expr
• sum=èxpr 5 + 3` OR multi=èxpr 7 * 9` OR
• divid=èxpr 6 / 4` OR modulus=èxpr 6 % 4`

Crontab Job Scheduling
• Command
Crontab –l , -e, -r (l-list, e-edit, r-remove). Crontab
have six fields separated either of space or tab. The beginning
five fields represent time to run tasks and last field is for
command.
• -Minute (hold values between 0-59)
• -Hour (hold values between 0-23)
• -Day of Month (hold values between 1-31)
• -Month of the year (hold values between 1-12)
• -Day of week (hold values between 0-6 or Sun-Sat)
• -Command
• Example:
• 00 11,13 * * * sh /home/user/backup_job.sh DB >/home/user/file_bkp.log

Example Script-1 (for loop)
#!/bin/sh
alphabet="a b c d e" # Initialise a string
count=0 # Initialise a counter
for letter in $alphabet # Set up a loop control
do # Begin the loop
count=`expr $count + 1` # Increment the counter
# Display the result
echo "Letter $count is [$letter]"
done

Example Script-2 (if/else condition)
• #!/bin/sh
• a=10
• b=20
•
if [ $a == $b ]
• then
• echo "a is equal to b"
• elif [ $a -gt $b ]
• then
• echo "a is greater than b"
• elif [ $a -lt $b ]
• then
• echo "a is less than b"
• else
• echo "None of the condition met"
• fi

Example Script-3 (functions)
• #!/bin/sh
• PrintHello () {
• echo "Hello World from function."
• return 100
• }
• Argumentfun () {
• echo "Your Arguments are: $1 $2"
• }
• # Call function
• PrintHello
• echo “Returned Exit status: $?"
• Argumentfun Tea Coffee
• echo “Exit status: $?"

Example Script-4 (case statements)
• #!/bin/sh
• echo "Please talk to me ..."
• while :
• do
• read INPUT_STRING
• case $INPUT_STRING in
• hello)
• echo "Hello yourself!" ;;
• bye)
• echo "See you again!"
• break ;;
• *)
• echo "Sorry, I don't understand";;
• esac
• done
• echo
• echo "That's all folks!"

Example: Database Connection
• echo "Starting......."
• sqlplus -s /nolog<<-EOF
• conn user/password@DBNAME
• set pages 200 lines 200 heading off feedback off;
@insert.sql;
• EOF
• echo "Finishing ......“
•
#running job in background
nohup sh scriptname.sh Arg1 Arg2 Arg3 > scriptname.log &

Some Day-to-Day Useful Commands
• How to kill multiple process at a time.
• $ ps –aef|grep oracle|awk {print $2}|xargs kill -9
• Find 30 days older file in directories.
• $ find /path/to/files* -mtime +30 -exec rm {} ;
• List 5 huge size file in current path
• $ find ./ -type f -printf '%s %pn'|sort -rn|head -5
• Multiple pattern replacement using sed
• $ sed –e ‘s/12/45/g’ –e ‘s/a/A/g’ –e ‘s/To/For/g’
file.txt

Unix Shell Script - 2 Days Session.pptx

More Related Content

Similar to Unix Shell Script - 2 Days Session.pptx (20)

Recently uploaded (20)

Unix Shell Script - 2 Days Session.pptx