High performance computing for non-programmers

Glib Ivashkevych
junior researcher, NSC KIPT

Lecture #7: bash & a bit of Linux

Linux shell text processing

cut - get sections of line
useful options: -f -d -s

paste - merge lines

useful options: -d

sort - sort lines

useful options: -V -k -f (see man sort for more)

Linux shell text processing

uniq - get or skip repeated lines
useful options: -c -i -f

wc - count words, lines and characters

useful options: -l -L -w

join - join lines by common field

useful options: -a -j -1 -2 -e (see man join for more)

Linux shell text processing

tr - translate/delete characters
useful options: -c

tee - read from input, write to output and files

useful options: -a

date, cal - date and calendar

Linux shell text processing

grep - print matching lines
useful options: -c -i -l -m -v (and many more, see man grep)

sed - stream editor

Swiss army knife later

awk - scanning and processing

Swiss army knife later

Linux shell search for files

find - search for files
useful options: -name -type -exec (and many more, see man find)

ack - search for files with some pattern

useful options: -c -A -B -C

which - show full path of command

useful options: -a

Linux shell redirections

Results of program execution are sent to standard output Errors standard error Input standard input In Unix everything is a file. So, we can change I/O directions.

Linux shell redirections

Standard output redirection: > or >> (1>, actually) Standard error redirection: 2> or 2>> Both: &> Standard input redirection: <, <<<

Linux shell redirections

To throw smth. away: /dev/null Trick: named pipes mkfifo /some/path/pipe Trick: process substitution
command1 <(command2) command1 1> >(command2) 2> >(command3)

Linux shell pipes

Pass STDOUT of one command to STDIN of another: command1 | command2

Linux shell expansions

Brace expansion: {a,b,c,...} {start..end[..incr]} a b c ... Tilde expansion: ~/foo - directory foo of home directory ~name/foo - directory foo of names home directory

Linux shell expansions

Parameters expansion: $VARNAME ${VARNAME} Command substitution: $(command) $(command) Arithmetic expansion: $((expr))

Linux shell expansions

Process substitution: <(command) >(command) Word splitting: everything not inside is splitted Filename expansion: * ? [ (pattern matching here)

Linux shell quoting

... - strong quoting no special meaning of characters ... - weak quoting no special meaning, except $, \, ` \ - escape prevent special meaning of next character

Linux shell environment variables

Name/value pairs to influence behavior of software Some are: $PATH $HOME $LD_LIBRARY_PATH $USER $PS1 Process local, inherited from parent process To create one: export $ENVVARNAME=value To get: env or echo

Linux shell config

set command ~/.bash_profile - is executed at each login ~/.bashrc - is executed at each subshell start ~/.bash_logout - is executed at logout

Linux going further

Linux is multiuser multitasking POSIX-compliant open-source secure flexible OS (kernel)

Linux open source

Linux is most amazing success in OSS movement GPL license Huge ecosystem

Linux open source

GPL: Linux, GNU coreutils BSD: BSD, Darwin Apache: Android, Apache Python, Ruby, git nginx, MySQL PostgreSQL, MongoDB

Linux usage
HPC: 96% of Top-500 Web: not just Linux, but OSS Embedded Mobile: Android

Questions?