GNU and UNIX commands
Monday, 23. July 2007, 09:26:46
- the bash shell is one of several shells available for linux
- shell is a program that executes commands (many commands on linux system are scripts)
- shells use standard I/O streams (stdin,strout,strerr)
- if a line contains a # character, then all remaining characters on the line are ignored
[ian@echidna ian]$ echo -n No new line
No new line[ian@echidna ian]$ echo -e "No new line\c"
No new line[ian@echidna ian]$ echo "A line with a typed
> return"
A line with a typed
return
[ian@echidna ian]$ echo -e "A line with an escaped\nreturn"
A line with an escaped
return
[ian@echidna ian]$ echo "A line with an escaped\nreturn but no -e option"
A line with an escaped\nreturn but no -e option
[ian@echidna ian]$ echo -e Doubly escaped\\n\\tmetacharacters
Doubly escaped
metacharacters
[ian@echidna ian]$ echo Backslash \
> followed by newline \
> serves as line continuation.
Backslash followed by newline serves as line continuation.
- bash displays a special prompt (>) when you type a line with unmatched quotes
- bash has several metacharacters: besides a blank, these are '|', '&', ';', '(', ')', '<', and '>'
- if you want to include it in part of your text it must be quoted or escaped using a baskslash
[ian@echidna ian]$ echo line 1;echo line 2; echo line 3 line 1 line 2 line 3 [ian@echidna ian]$ echo line 1&&echo line 2&&echo line 3 line 1 line 2 line 3 [ian@echidna ian]$ echo line 1||echo line 2; echo line 3 line 1 line 3
- && se izvede samo, ce je prvi ukaz uspesno zakljucil
- || se izvede samo, ce je prvi ukaz neuspesno zakljucil
- you can terminate a shell using "exit" command (if running a window, the window will close)
ENVIRONMENT VARIABLES
- when you are running in a bash shell, many things constitute your environment (form or your prompt...)
- your environment includes many variables that may have been set by bash or by you
- bash shell also allows you to have shell variables, which you may export to your environment
- some of the common variables are:
USER The name of the logged-in user UID The numeric user id of the logged-in user HOME The user's home directory PWD The current working directory SHELL The name of the shell $ The process id (or PIDof the running bash shell (or other) process PPID The process id of the process that started this process (that is, the id of the parent process) ? The exit code of the last command
- variables are case sensitive
- when you create a shell variable, you want to export it to the environment so it will be available to other processes that you start from this shell (variables that you export are not available to a prent shell)
- there is a difference between single and double quotes
- The shell expands shell variables that are between double quotes ($quot;), but expansion is not done when single quotes (') are used.
[ian@echidna ian]$ echo "$SHELL" '$SHELL' "$$" '$$' /bin/bash $SHELL 19244 $$
ENV
- the env command displays the current environment variables
- the -i option clears the current environment before running the command, while the -u option usets variables that you do not want to pass
- you can use unset command to unset a variable and remove it from the shell variable list
EXEC
- exec command runs another program that replaces the current shell
[ian@echidna ian]$ echo $$ 22985 [ian@echidna ian]$ bash [ian@echidna ian]$ echo $$ 25063 [ian@echidna ian]$ exec ksh $ echo $$ 25063 $ exit [ian@echidna ian]$ echo $$ 22985
- starts a child bash shell and then uses exec to replace it with a Korn shell. Upon exit from the Korn shell, you are back at the original bash shell (PID 22985, in this example).
COMMAND HISTORY
- If you are typing in commands as you read, you may notice that you often use a command many times, either exactly the same, or with slight changes.
- The good news is that the bash shell can maintain a history of your commands. By default, history is on
- You can turn it off using the command set +o history and turn it back on using set -o history
- an envoronment variable HISTSIZE tells bash how many history lines to keep
- some of the commands that you can use with the history facility are:
history Displays the entire history history N Displays the last N lines of your history history -d N Deletes line N form your history; you might do this if the line contains a password, for example !! Your most recent command !N The Nth history command !-N The command that is N commands back in the history (!-1 is equivalent to !!) !# The current command you are typing !string The most recent command that starts with string !?string? The most recent command that containsstring
- By default, the keys and key combinations used to move through the history or edit lines are similar to those used in the GNU Emacs editor. Emacs keystroke combinations are usually expressed as C-x or M-x, where x is a regular key and C and M are the Control and Meta keys, respectively. On a typical PC system, the Ctrl key serves as the Emacs Control key, and the Alt key serves as the Meta key.
- history editing:
C-f Right arrow Move one space to the right C-b Left arrow Move one space to the left M-f Alt-f Move to beginning of next word; GUI environments usually take this key combination to open the File menu of the window M-b Alt-b Move to beginning of previous word C-a Home Move to beginning of line C-e End Move to end of line Backspace Backspace Delete the character preceding the cursor C-d Del Delete the character under the cursor (Del and Backspace functions may be configured with opposite meanings) C-k Ctrl-k Delete (kill) to end of line and save removed text for later use M-d Alt-d Delete (kill) to end of word and save removed text for later use C-y Ctrl-y Yank back text removed with a kill command
- If you prefer to manipulate the history using a vi-like editing mode, use the command set -o vi to switch to vi mode. Switch back to emacs mode using set -o emacs.
PATHS
- where does the shell find commands?
- if you type a command name, then bash looks for that command on your PATH variable, which is a colon separated list
- If you want to know what command will be executed if you type a particular string, use the which or type command
- Try running dircolors --print-database to see how the color codings are controlled and which colors are used for what kind of file.
- If you use cd with no parameters, the change will be to your home directory. A single hyphen (-) as a parameter means to change to the previous working directory.
- Your home directory is stored in the HOME environment variable, and the previous directory is stored in the OLDPWD variable, so cd alone is equivalent to cd $HOME and cd - is equivalent to cd $OLDPWD
Command substitution
- The bash shell has an extremely powerful capability that allows the results of one command to be used as input to another; this is called command substitution. This can be done by enclosing the command whose results you wish to use in backticks (`). This is still common, but a way that is easier to use with multiply nested commands is to enclose the command between $( and ).
- this shows an example of how to get the absolute path of a directory from a relative path, how to find which RPM provides the /bin/echo command, and how (as root) to list the labels of three partitions on a hard drive. The last one uses the seq command to generate a sequence of integers.
[ian@echidna ian]$ echo '../../usr/bin' dir is $(cd ../../usr/bin;pwd) ../../usr/bin dir is /usr/bin [ian@echidna ian]$ which echo /bin/echo [ian@echidna ian]$ rpm -qf `which echo` sh-utils-2.0.12-3 [ian@echidna ian]$ su - Password: [root@echidna root]# for n in $(seq 7 9); do echo p$n `e2label /dev/hda$n`;done p7 RH73 p8 SUSE81 p9 IMAGES
MAN
- The primary (and traditional) source of documentation is the manual pages, which you can access using the man command.
- Two important commands related to man are whatis and apropos. The whatis command searches man pages for the name you give and displays the name information from the appropriate manual pages. The apropos command does a keyword search of manual pages and lists ones containing your keyword.
[ian@lyrebird ian]$ whatis man man (1) - format and display the on-line manual pages man (7) - macros to format man pages man [manpath] (1) - format and display the on-line manual pages man.conf [man] (5) - configuration data for man [ian@lyrebird ian]$ whatis mkdir mkdir (1) - make directories mkdir (2) - create a directory [ian@lyrebird ian]$ apropos mkdir mkdir (1) - make directories mkdir (2) - create a directory mkdirhier (1x) - makes a directory hierarchy
Text streams and filters
- Text filtering is the process of taking an input stream of text and performing some conversion on the text before sending it to an output stream.
- Although either the input or the output can come from a file, in the Linux and UNIX environments, filtering is most often done by constructing a pipeline of commands where the output from one command is piped or redirected to be used as input to the next
Piping with |
- shell use 3 standard I/O streams: stdin,stdout,stderr
piping output from command1 to input of command2
command1 | command2
- either command may have options and arguments
Output redirection with >
- While it is nice to be able to create a pipeline of several commands and see the output on your terminal, there are times when you want to save the output in a file. You do this with the output redirection operator (>).
Cat, tac, od, and split
- displaying contents with cat
[ian@echidna lpi103]$ cat text1 1 apple 2 pear 3 banana
- creating a text file with cat
[ian@echidna lpi103]$ cat>text2 9 plum 3 banana 10 apple
- in this case cat keep reading from stdin
Occasionally, you might want to display a file in reverse order. Naturally, there is a text filter for that too, called tac (reverse of cat).
- Now, suppose you display the two text files using cat or tac and notice alignment differences. To learn what causes this, you need to look at the control characters that are in the file. Since these are acted upon in text display output rather than having some representation of the control character itself displayed, we need to dump the file in a format that allows you to find and interpret these special characters. The GNU text utilities include an od (or Octal Dump) command for this purpose.
- There are several options to od, such as the -A option to control the radix of the file offsets and -t to control the form of the displayed file contents. The radix may be specified as o, (octal - the default), d (decimal), x (hexadecimal), or n (no offsets displayed). You can display output as octal, hex, decimal, floating point, ASCII with backslash escapes or named characters (nl for newline, ht for horizontal tab, etc.).
bash-3.1# od test 0000000 060553 067553 071440 005151 060553 005152 060544 062556 0000020 005163 0000022 bash-3.1# od -A d -t c test 0000000 k a k o s i \n k a j \n d a n e 0000016 s \n 0000018
- The -A option of cat provides an alternate way of seeing where your tabs and line endings are.
- Our sample files are very small, but sometimes you will have large files that you need to split into smaller pieces. For example, you might want to break a large file into CD-sized chunks so you can write it to CD for sending through the mail to someone who could create a DVD for you. The split command will do this in such a way that the cat command can be used to recreate the file easily. By default, the files resulting from the split command have a prefix in their name of 'x' followed by a suffix of 'aa', 'ab', 'ac', ..., 'ba', 'bb', etc. Options permit you to change these defaults. You can also control the size of the output files and whether the resulting files contain whole lines or just byte counts
bash-3.1# split -l 2 test bash-3.1# ls test xaa xab xorg.conf.new bash-3.1# cat xaa kako si kaj bash-3.1# cat xab danes bash-3.1# cat x* kako si kaj danes
Wc, head, and tail
- wc (word count) command - to see how big the file is (it displays the number of lines, words and bytes in a file)
- you can also find the number of bytes using ls -l
- Two commands allow you to display either the first part (head) or last part (tail). These commands are the head and tail commands. They can be used as filters,or they can take a file name as an augument. By default they display the first (or last) 10 lines of the file or stream
- you ca use:
----> dmesg | tail
----> dmesg | head
----> dmesg | wc
Expand, unexpand, and tr
- sometimes in your files you want to swap tabs for spaces or vice versa. The expand and unexpand commands do this.
- expand: Convert tabs in each FILE to spaces, writing to standard output
- unexpand: convert spaces in FILE to tabs (requires at least 2 spaces to convert to tab)
- tr: Translate, squeeze, and/or delete characters from standard input
bash-3.1# cat test
kako si kaj
danes
bash-3.1# expand -t 1 test
kako si kaj
danes
bash-3.1# expand -t 2 test
bash-3.1# unexpand test
kako si kaj
danes
bash-3.1# unexpand -t 1 test
kako si kaj
danes
bash-3.1#
Pr, nl, and fmt
- pr: format file for printing (the default header includes the file name and the dile creation date and time along with page number and two lines of blank footer)
-nl: numbers lines (cat -n would do the same)
- fmt: formats text so it fits within margins (you can join several short lines as well as split long ones)
Sort and uniq
- sort: sorts the input using the collating sequence for the locale (LC_COLLATE). It can also merge already sorted files and check wheter a file is sorted or not
- uniq: removes the identical lines from any file
Cut, paste, and join
- cut: extracts fields from text files (the default field delimiter is the tab character)
- paste: pastes lines from two or more files side-by-side
- join: joins file based on a matching field
SED
- is a stream editor (can work as a filter, can take its input from a file...)
- sed has many commands
bash-3.1# cat test kako si kaaj daneaaas bash-3.1# sed 's/a/A/' test kAko si kAaj dAneaaas bash-3.1# sed 's/a/A/g' test kAko si kAAj dAneAAAs bash-3.1# sed '2d;$s/a/A/g' test kako si dAneAAAs
- in the first example we use s(substitute) command to substitute an upper case for lowe case on each line. This replaces only the first a, so in the second example we add the g(global) falg to cause sed to chande all occurances
- In the third script, we introduce the d (delete) command to delete a line. In our example, we use an address of 2 to indicate that only line 2 should be deleted. We separate commands using a semi-colon (
- in addition to operating on individual lines, sed can operate on a range of lines.
- The beginning and end of the range are separated by a comma (,) and may be specified as a line number, a caret (^) for beginning of file, or a dollar sign ($) for end of file. Given an address or a range of addresses, you can group several commands between curly braces ({ and }) to have these commands operate only on lines selected by the range.
- -e option to add commands to the pattern space. When using braces, we must separate commands in this way.
[ian@echidna lpi103]$ sed -e '2,${' -e 's/a/A/g' -e '}' text1
1 apple
2 peAr
3 bAnAnA
[ian@echidna lpi103]$ sed -e '/pear/,/bana/{' -e 's/a/A/g' -e '}' text1
1 apple
2 peAr
3 bAnAnA
Listing details
- On a storage device, a file or directory is contained in a collection of blocks. Information about a file is contained in an inode which records information such as the owner, when the file was last accessed, how large it is, whether it is a directory or not, and who can read or write it. The inode number is also known as the file serial number and is unique within a particular filesystem.
-rw-rw-r-- 1 root root 22 Jul 24 19:26 test
- the second value (1) is a number which tells us the number of hard links to the file. We said that the inode contains information about the file. The file's directory entry contains a hard link (or pointer) to the inode for the file, so every entry listed should have at least one hard link. Directory entries have an additional one for the . entry and one for each subdirectory's .. entry
- The -i option of the ls command will display the inode numbers for you.
Wildcards and globbing
- a string containing any of the characters '?', '*' or '[', is a wildcard pattern. Globbing is the process by which the shell (or possibly another program) expands these patterns into a list of pathnames matching the pattern. The matching is done as follows.
? matches any single character. * matches any string, including an empty string. [ introduces a character class. A character class is a non-empty string, terminated by a ']'. A match means matching any single character enclosed by the brackets. There are a few special considerations.
- The '-' character between two others represents a range which includes the two other characters and all between in the collating sequence. For example, [0-9a-fA-F] represents any upper or lower case hexadecimal digit. You can match a '-' by putting it either first or last within a range.
- The '!' character specified as the first character of a range complements the range so that it matches any character except the remaining characters. For example [!0-9] means any character except the digits 0 through 9
- Globbing is applied separately to each component of a path name. You cannot match a '/', nor include one in a range.
sh-3.1$ touch text1 text2 text3 text5 sh-3.1$ ls text[2-4] text2 text3
TOUCHING FILES
- touch: can update file access and modification times or create empty files
- The touch command with no options takes one or more filenames as parameters and updates the modification time of the files.
sh-3.1$ echo "pojdi spat">file; ls -l file ;sleep 60; touch file ; ls -l file -rw-rw-r-- 1 eleanor eleanor 11 Jul 24 20:58 file -rw-rw-r-- 1 eleanor eleanor 11 Jul 24 20:59 file
- if you specify a filename that does not exist, then touch will normally create an empty file unless you use -c option (then it will not create that file)
- you can also set a files mtime to a specific date and time using -d or -t option. The -d is very flexible in the date and time formats that it will accept, while the -t option needs at least an MMDDhhmm time with optional year and seconds values (date command resolves the same kind of date formats than touch can)
Finding files
- it will search for files using all or part of the name, or by other search criteria: size, type, owner, creation date...
find . -name "*[1k]*"
- search in current directory where for all files that have either a '1' or a 'k' in their name
- If you want to find a file or directory whose name begins with a dot, such as .bashrc or the current directory (.), then you must specify a leading dot as part of the pattern.
- Use the -type parameter along with one-letter type to restrict the search. Use 'f' for regular files, 'd' for directories, and 'l' for symbolic links.
find . -type d
- We can also search by file size, either for a specific size (n) or for files that are either larger (+n) or smaller than a given value (-n). By using both upper and lower size bounds, we can find files whose size is within a given range.
find . -size -26c -size +23c -print
- -print option which is an example of an action that may be taken on the results returned by the search. In the bash shell, this is the default action if no action is specified. On some systems and some shells, an action is required, otherwise there is no output.
- Other actions include -ls which prints file information equivalent to that from the ls -lids command, or -exec which executes a command for each file. The -exec must be terminated by a semi-colon which must be escaped to avoid the shell interpreting it first. Also specify {} wherever you want the returned file used in the command
find . -size -26c -size +23c -ls
find . -size -26c -size +23c -exec ls -l '{}' \;
//this removes all the files in a directory tree
find . -empty -exec rm '{}' \;
//this renames all .htm files to .html files
find . -name "*.htm" -exec mv '{}' '{}l' \;
- When used with -mtime -2, the find command finds all files modified within the last two days. A day in this case is a 24-hour period relative to the current date and time. Note that you would use -atime if you wanted to find files based on access time rather than modification time.
Streams, pipes, and redirects
-









