Indispensable UNIX utilities

A list of utilities which might have saved me some work if I had known of them sooner. Also but not primarily contains the standards "everybody" knows. I added some brief explanations, and the place to download them from if your distro does not contain them. Sometimes there are several which do the same thing — usually developed independently. Those marked "shameless plug" are my own.

Files, pipes and processes

File basics

cat

Output file(s)

split

Split file into smaller files

packcd

Group files into sets of given maximum size

file

Determine file type / format

find

Search the file system

Multiple file operations

mmv

Multiple file copy, move, append or link

rename

Multiple file rename by Perl expression

multi

Run command multiple times on different file (or non-file) arguments

Directories

pushd, popd, dirs

bash commands for changing and saving directory locations on a stack

mkcd

Create a directory and descend (shameless plug)

rmcd

Ascend and remove empty directory (shameless plug)

pwdx

Working directory of a process

du, ncdu

Disk space used by files and directories

Pipes and redirection

tee

Copy input to output and to file

pee

Copy input to output and to another command's input

reservoir, sponge

Buffer input, then write to file; allows in-place automated editing of files

vipe

Edit data piped between commands

mispipe

Make pipe return exit status of first command

ifne

Run command on non-empty input

Processes

ps

List running processes

kill

Send signal to process, by process ID

pgrep

Obtain process IDs from command name

pkill

Send signal to process, by command name

skill

Send signal to process, by various criteria

watch

Run a process periodically and redisplay output

nice

Run a command with a given priority

renice, snice

Change the priority of a process

nohup

Run a command, ignoring signals

sudo

Run a command with the rights of a different user

taskset

Set the CPU affinity of a process

Commands

script

Record shell session

yes

Output "y" or other string repeatedly; useful for piping into interactive commands

zrun

Run command on decompressed copy of file

lckdo

Prevent concurrent execution

xargs

Run command on every argument from stdin, possibly in parallel

multi

Run command multiple times on argument and transformed argument

para

Run multiple instances of a command in parallel on different files/arguments (shameless plug)

Plain text processing

Line-based

tac

Reverse line order

head

Output initial lines only

tail

Output trailing lines only, or lines newly written

since

Output lines newly written (and keep track of status between calls)

csplit

Split text file on lines matching pattern

dog

Print file or URL; allows some editing such as line numbering, case conversion etc.

sort

Sort lines

comm

Print lines contained or not contained in two sorted files

uniq

Print or remove sequentially repeated lines

combine

Print lines which are in both/either/one of two files, which need not be sorted

nl

Number lines

ts

Prepend lines with time and date

sed

Stream editor; programmable automated line editing

perl -n -e 'expression'

Filter text line by line with Perl code

iconv

Convert between character encodings (part of GNU libc project)

Column-based

column

Fill words into columns the way ls does

colrm

Remove columns

cut

Remove columns, by word, character of byte

paste

Concatenate corresponding lines from two files

join

Concatenate lines of sorted files when starting with same word

awk

Line-by-line (by default) automated text processing

Paragraph formatting

fold

Line breaking

fmt

Line breaking and refilling

par

More powerful alternative to fmt; supports customised line prefixes, line suffixes and character types.

Searching

grep

Line filtering based on regular expressions

numgrep

Line filtering based on ranges of numbers or divisibility

bool

Word search with boolean operators and proximity

Comparison

diff

List line-by-line differences between files

patch

Apply differences found by diff to other file

wdiff

Find word-by-word differences between files

diff3

List line-by-line differences between three files

gendiff

Recursively diff files of a directory tree against originals

Program code

expand, unexpand

Expand and unexpand tab characters

unifdef

Expand C preprocessor #ifdef's

Binary files

Output

strings

Extract ASCII strings from binary data

hexdump, od

Output hexadecimal listing of binary files

dog

Among others, allows to hexdump a URL

xxd

Convert binary files to hexdump or vice versa; allows to do editing with sed or other text filters in between (comes with vim)

Editing

hexedit

Hexadecimal editor running in text terminal; supports bit mask search with this patch

bbe

Block-based binary editor

bvi

vi-like binary editor

xxd

Convert binary files to hexdump or vice versa; allows to do editing with sed or other text filters in between (comes with vim)

bined

Shameless plug: a Perl-based automated hexadecimal editor

Comparison

cmp

Find byte-by-byte difference between binary files

xdelta

Find, print or patch differences between binary files

diffn, duff

Compare a set of files to find duplicates

Binary executables

ldd

Print dynamic library dependencies

nm

List symbols

strip

Discard symbols

size

List section sizes

readelf, objdump

Print various information about object files

Arithmetic

seq, range

Print an arithmetic progression (an equidistant sequence of numbers)

factor

Print the prime factorisation of a number

bc

Interactive or batch calculator

dc

Reverse polish calculator

$RANDOM

Magic bash variable returning random numbers

/dev/random

Source of binary random numbers

units

Command-line unit conversion

Networking and WWW

Networking

ifdata

Machine-readable output of network interface properties

ping

Check if host is reachable by sending echo requests

nc, netcat

"Network cat", connect your terminal to network socket

cryptcat

netcat with twofish encryption

ncat

netcat clone with additional features, such as proxy support, session log, authentication etc.

socat

"Socket cat", more powerful than netcat, relay between two endpoints, which may be sockets, your terminal, files, other programs etc.

nmap

Port scanner and vulnerability checker

WWW

httping

Check connection to web server

curl

Download and output a set of URLs, supports glob-like syntax, and also allows HEAD (header) requests

wget

Download web pages recursively

wput

FTP uploader

lynx

Web browser which runs in text terminal; can also output formatted page

w3m

Web browser which runs in text terminal; can also output formatted page and supports tables

links

Browser both for text terminal and X (where it supports images); allows colour calibration

dog

Download URL and optionally number lines, hexdump, extract links, print line range etc.

xmllint --shell

Structure browser for XML and HTML files

TOS / Impressum