Useful commands and scripts

Some here are scripts, those begin with a shebang

1
#!

• Main
  • Control characters
  • skeleton template path
  • Special variables and parameters of the shell
  • Shell operators, add to variable
  • print all environment variables
  • man pages and paths
  • replace in place with sed
  • Find, grep, sed find files with pattern and replace
  • -u to turn off buffering for sed
  • pz (pythonize shell)
  • Vim
  • Bash history expansion
  • Bash job control
  • grep
  • find delete empty directories
  • mail
  • moreutils
  • data structure or configuration processing tools
  • linkchecker
  • ncdu equivalent
• Rust tools
  • Bacon
  • Faster alternatives written in Rust
  • Other alternatives written in Rust
• Extra
  • pandoc to convert between text formats
  • pass
  • find and play videos flexibly
  • Rude wget robot to recursively download a site
  • ImageMagick
  • FFMpeg
  • nginx
  • Systemd get specific service properties
  • Systemd test if service is failed, get exit code

Main

Control characters

• EOT - End Of Transmission, ^D (Ctrl+d)
• ETX - End Of Text, ^C
• LF - Line Feed, ^J
• CR - Carriage Return, ^R

skeleton template path

1
/etc/skel

Special variables and parameters of the shell

man bash does not mention special parameters in full form, they are listed under the Special Parameters section. This is due to all variables being prefixed with $ on use. For example, $$ is $ in the manual. Same with dash.

Other shells may have different conventions.

• fish - man fish-doc provides translations to bash, search for Special variables
• zsh - man zshparams, search for PARAMETERS SET BY THE SHELL
• nu

  1
  2
  wget --content-disposition \
  https://github.com/nushell/nushell.github.io/blob/main/book/special_variables.md?raw=true
  

Shell operators, add to variable

For help on operators, search for test expression.

1
2
man $(basename $SHELL) # bash, dash
help test # fish

Helper function for environmental variables.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
push_path_to_var() {
  if [ $# -ne 2 ] || [ ! -x "$2" ] || [ ! -d "$2" ]; then
    printf "Usage: push_path_to_var VAR executable_directory\n"
    printf 'Current $1: %s $2: %s\n' "$1" "$2"
    return 1
  fi
  # Appends the content of VAR to the argument array.
  # 'set --' to change the positional parameters without changing any options
  eval set -- "$1 $2 \$$1"

  # If the variable is empty, set it
  if [ -z "$3" ]; then
    eval "export $1=$2"
  else
    case ":$3:" in
      # If path already exists in VAR do nothing
      *":$2:"*) ;;
      # prepend path to VAR
      *) eval "$1=$2:$3" ;;
    esac
  fi
}

push_path_to_var PATH "$HOME/bin"

print all environment variables

1
2
3
4
printenv
jq -n env
jq -nr 'env|.HOME' # -r = raw
set # list values, including functions

man pages and paths

manpath is installed as part of man-db.

1
manpath --debug

User man pages should be installable to $XDG_DATA_HOME/man if XDG spec is followed and XDG_DATA_HOME is the default $HOME/.local/share. If it is a custom path, the relationship with $HOME/.local/bin can break.

1
2
3
4
5
6
manpath --debug

... snip ...
path directory /home/user/.local/bin is not in the config file
  adding /home/user/.local/share/man to manpath
... snip ...

manp.c from man-db repository contains

1
2
3
4
/* The directory we're working on isn't in the config file.
  See if it has ../man, man, ../share/man, or share/man
  subdirectories.  If so, and they haven't been added to
  the list, do. */

replace in place with sed

1
2
sed -i -e 's/^/#/' filename
sed -i '1s/^/added line before 1st line of file\n/' filename

Empty files don't have that first line and are not affected.

Find, grep, sed find files with pattern and replace

1
2
3
4
5
find group_vars/ -type f -name '*.yml' ! -exec grep -q -- '---' '{}' \; -print

find group_vars/ -type f -name '*.yml' \
! -exec grep -q -- '---' '{}' ';' \
-exec sed -i '1s/^/---\n/' '{}' '+'

-u to turn off buffering for sed

1
dmesg -T --follow | sed -u -e "s/^/$HOSTNAME/"

pz (pythonize shell)

operate on s (the line variable).

1
echo -e "example\nwikipedia" | pz 's += ".com"'

Vim

• Delete empty lines.

  1
  2
   :g/^$/d
   :help :g
  

• Show shell executable.

  1
   :set shell?
  

• Execute a child shell process. Exiting returns to vi.

  1
   :sh
  

• Run given shell commands and exit after pressing enter.

  1
   :!printf ":help :!"
  

• Execute current line ('.') in sh and replace it in the vi buffer with the output.

  1
   :.!sh
  

• Insert the output of the shell command in the buffer.

  1
   :r!date
  

• Get the output of a shell command with system ( :help system ).

  1
   :call system('touch /tmp/$(date +%Y%m%d)')
  

  1
   :echo system('ls -lt /tmp/$(date +%Y)* | tail')
  

• Start a job that doesn't wait to finish ( :help job_status ), execute with no shell.

  1
   :call job_start(['/bin/bash', '-c', '{ sleep 60 && printf "DONE"; }'])
  

• Registers (+ * ~ / : % . -)
  • c - characterwise text
  • l - linewise text
  • b - blockwise text

    1
    2
    :registers
    "+p # paste in visual mode
    

    1. unnamed register "" fills when using delete or yank
    2. numbered registers "0-"9 (deletion history stack)
    3. small delete register "- (one line or less deleted)
    4. named registers "a-"z"A-"Z (write to F gets appended, f replaced)
    5. read only registers:
       • ": most recently executed command;
       • "% current file name;
       • ". last inserted text.
    6. alternate file name register "# (in case of multiple open files)
    7. expression register "=
    8. last search result register "/
    9. GUI primary "+ and secondary "* , middle click clipboard
    10. GUI drop register "~ (drag and drop)

Bash history expansion

Run last command. man bash Event Designators

1
sudo !!

Bash job control

Ctrl+Z to suspend a program in bash. Sends SIGTSTP (Keyboard stop).

Start a job in the background.

1
sleep 4500 &

Bash shell builtins for jobs. Applies to current shell process, not all user's shells. ####

1
jobs --help

1
2
3
fg # Bring to foreground. No job spec, targets shells notion of the current job
bg %1 # Sends the first job to execute in the background
wait 20110 # Process ID or job spec

Remove the second job from current shell's job list, keeping it in the process table. Not reliable for running a program in the background (in case the parent shell is destroyed).

1
disown %2

Run a program in a new session. Allows bypassing potential signals from initial parents.

1
setsid --fork sleep 3600 # always create a new process

Format text in a way that is safe to use as shell input.

1
printf '%q\n' "It's magic!"

grep

Search directory recursively for lines starting 0 or more whitespace and $.

1
grep -ER '^(\w*)\$'

find delete empty directories

Starts from the deepest, deletes recursively up.

1
find . -depth -type d -empty -delete

mail

mailutils.txt

3.5.2.1 Syntax of mail internal commands

Open the default inbox (mbox) instead of the spool file.

1
mail -f

With mail open, match string from the body.

1
h :/\\[SPAM\\]

Display help.

1
?

List messages (print message headers).

1
2
h
f *

Match string from the header From.

1
h From:/root

Header is Subject if omitted. Escaped to prevent an attempt to POSIX regex match.

1
h /\\[SPAM\\]

Delete matching messages.

1
d /\\[SPAM\\]

Touch all messages, they'll be acted on as if they were read.

1
tou *

Quit without removing system mailbox.

1
x

moreutils

• chronic: runs a command quietly unless it fails
• combine: combine the lines in two files using boolean operations
• errno: look up errno names and descriptions
• ifdata: get network interface info without parsing ifconfig output
• isutf8: check if a file or standard input is utf-8
• ifne: run a command if the standard input is not empty
• lckdo: execute a program with a lock held (deprecated)
• mispipe: pipe two commands, returning the exit status of the first
• parallel: run multiple jobs at once
• pee: tee standard input to pipes
• sponge: soak up standard input and write to a file
• ts: timestamp standard input
• vidir: edit a directory in your text editor
• vipe: insert a text editor into a pipe
• zrun: automatically uncompress arguments to command

data structure or configuration processing tools

• jq
• yq
• tomlq
• rq (see: open policy agent, Rego, kube-mgmt)

linkchecker

Check HTML documents and websites for broken links. Return value is 1 when

• invalid links were found or
• link warnings were found and warnings are enabled Supports clamav integration.

ncdu equivalent

For when ncdu is not installed.

1
du -d1 -h | sort -h

Rust tools

Bacon

bacon - background code checker designed for minimal interaction. It has analyzers for Rust, python, javascript, typescript, C++, Go.

Kondo recursively cleans project directories. Interface to rm -rf build files.

1
kondo --dry-run

Faster alternatives written in Rust

fd - alternative to find. debian package name fd-find, executable name on debian is

1
fdfind

ripgrep - alternative to grep. Debian package name ripgrep.

1
rg --pcre2 '(?!\\)_'

runiq filters duplicate lines. Alternative to uniq.

Uv claims to replace virtualenv, pip, pip-tools, pipx, poetry, pyenv, twine.

1
uv

pre-commit, husky alternative

1
prek

watchexec watches a path and runs a command whenever it detects modifications.

Other alternatives written in Rust

eza - ls

bat - cat

xh - httpie. Friendlier interface than curl, can print out an equivalent curl command.

zellij - tmux, screen

du-dust - du like

dua-cli - ncdu

hyperfine - benchmarking, default repeated runs

helix, evil-helix - vim

just, mask - make

Extra

pandoc to convert between text formats

General markup coverter, supports lots of formats including epub, html, odt, csv, json.

Markdown to plain text.

1
pandoc -t plain README.md

Render markdown as html and view.

1
pandoc -t html README.md | w3m -T text/html

pass

Unix style password management script, encrypts with gpg.

find and play videos flexibly

Using mpv, my preferred video player.

To use as is, create a shell script with below contents, arguments:

1st argument - how long ago was the video file last changed.

The rest of the arguments: path(s) to look for videos (glob, like

1
/media/videos/201*

can be used as argument).

1
2
#!/bin/sh
find -L "${@:2}" \( -iname '*.mp4' -o -iname '*.webm' -o -iname '*.avi' \) -a -ctime -"$1" -print0 | xargs -0 mpv 

Rude wget robot to recursively download a site

1
2
3
4
5
6
7
8
9
10
11
#!/bin/sh
# shows the correct script path even if called from another script
if [ -z "$1" -o -z "$2" ]; then
  echo "Usage: $(basename $(readlink -nf $0)) site_to_download_recursively destination_directory" 
  exit 1
else
  wget --continue --recursive --execute robots=off --wait 1 --directory-prefix="$2" "$1"
  # Below line for offline mirroring, test on each site.
  # wget  --mirror --convert-links --adjust-extension --page-requisites --no-parent --continue --recursive --execute robots=off --wait 1 --directory-prefix="$2" "$1"
  exit 0
fi

ImageMagick

Requires ImageMagick

Resize .jpg pictures in current directory level to roughly 1500x2000 pixels in size.

1
mogrify -resize 1500x2000 *.{jpg,JPG}

Create a black background in required size (for example, to use in a video).

1
convert -size 1920x1080 xc:black bg.png

FFMpeg

Dump audio using parallel. Parallel uses 1 thread for each core by default.

Make sure to change audio format according to source.

1
parallel ffmpeg -i '{}' -map 0:1 -c:a copy '{.}.m4a' ::: /media/video/source_video_file.mkv

nginx

Dump config (along with testing it)

1
nginx -T

Systemd get specific service properties

1
systemctl show --property ActiveState nginx.service

Systemd test if service is failed, get exit code

1
ecode=$(systemctl is-failed --quiet nginx.service)