Useful commands and scripts

Some here are scripts, those begin with a shebang

1
#!

• replace in place with sed
• -u to turn off buffering for sed
• pz (pythonize shell)
• Vi(m)
• Bash job control
• grep
• mail
• moreutils
• data structure or configuration processing tools
• linkchecker
• ncdu equivalent
• Faster alternatives written in Rust
• pass
• find and play videos flexibly
• Rude wget robot to recursively download a site
• ImageMagick
• FFMpeg

replace in place with sed

1
sed -i -e 's/^/#/' filename

-u to turn off buffering for sed

1
dmesg -T --follow | sed -u -e "s/^/$HOSTNAME/"

pz (pythonize shell)

operate on s (the line variable).

1
echo -e "example\nwikipedia" | pz 's += ".com"'

Vi(m)

Show shell executable.

1
:set shell?

• Execute a child shell process. Exiting returns to vi.

  1
  :sh
  

• Run given shell commands and exit after pressing enter.

  1
  :!printf ":help :!"
  

• Execute current line ('.') in sh and replace it in the vi buffer with the output.

  1
  :.!sh
  

• Insert the output of the shell command in the buffer.

  1
  :r!date
  

  Get the output of a shell command with system ( :help system ).

  1
  :call system('touch /tmp/$(date +%Y%m%d)')
  

  1
  :echo system('ls -lt /tmp/$(date +%Y)* | tail')
  

  Start a job that doesn't wait to finish ( :help job_status ), execute with no shell.

  1
  :call job_start(['/bin/bash', '-c', '{ sleep 60 && printf "DONE"; }'])
  

Bash job control

Ctrl+Z to suspend a program in bash. Sends SIGTSTP (Keyboard stop).

Start a job in the background.

1
sleep 4500 &

Bash shell builtins for jobs. Applies to current shell process, not all user's shells. ####

1
jobs --help

1
2
3
fg # Bring to foreground. No job spec, targets shells notion of the current job
bg %1 # Sends the first job to execute in the background
wait 20110 # Process ID or job spec

Remove the second job from current shell's job list, keeping it in the process table. Not reliable for running a program in the background (in case the parent shell is destroyed).

1
disown %2

Run a program in a new session. Allows bypassing potential signals from initial parents.

1
setsid --fork sleep 3600 # always create a new process

Format text in a way that is safe to use as shell input.

1
printf '%q\n' "It's magic!"

grep

Search directory recursively for lines starting 0 or more whitespace and $.

1
grep -ER '^(\w*)\$'

mail

mailutils.txt

3.5.2.1 Syntax of mail internal commands

Open the default inbox (mbox) instead of the spool file.

1
mail -f

With mail open, match string from the body.

1
h :/\\[SPAM\\]

Match string from the header From.

1
h From:/root

Header is Subject if omitted. Escaped to prevent an attempt to POSIX regex match.

1
h /\\[SPAM\\]

Delete matching messages.

1
d /\\[SPAM\\]

moreutils

• chronic: runs a command quietly unless it fails
• combine: combine the lines in two files using boolean operations
• errno: look up errno names and descriptions
• ifdata: get network interface info without parsing ifconfig output
• isutf8: check if a file or standard input is utf-8
• ifne: run a command if the standard input is not empty
• lckdo: execute a program with a lock held (deprecated)
• mispipe: pipe two commands, returning the exit status of the first
• parallel: run multiple jobs at once
• pee: tee standard input to pipes
• sponge: soak up standard input and write to a file
• ts: timestamp standard input
• vidir: edit a directory in your text editor
• vipe: insert a text editor into a pipe
• zrun: automatically uncompress arguments to command

data structure or configuration processing tools

• jq
• yq
• tomlq
• rq (see: open policy agent, Rego, kube-mgmt)

linkchecker

Check HTML documents and websites for broken links. Return value is 1 when

• invalid links were found or
• link warnings were found and warnings are enabled Supports clamav integration.

ncdu equivalent

For when ncdu is not installed.

1
du -d1 -h | sort -h

Faster alternatives written in Rust

fd - alternative to find. debian package name fd-find, executable name on debian is

1
fdfind

ripgrep - alternative to grep. debian package name ripgrep.

1
rg

pass

Unix style password management script, encrypts with gpg.

find and play videos flexibly

Using mpv, my preferred video player.

To use as is, create a shell script with below contents, arguments:

1st argument - how long ago was the video file last changed.

The rest of the arguments: path(s) to look for videos (glob, like

1
/media/videos/201*

can be used as argument).

1
2
#!/bin/sh
find -L "${@:2}" \( -iname '*.mp4' -o -iname '*.webm' -o -iname '*.avi' \) -a -ctime -"$1" -print0 | xargs -0 mpv 

Rude wget robot to recursively download a site

1
2
3
4
5
6
7
8
9
10
11
#!/bin/sh
# shows the correct script path even if called from another script
if [ -z "$1" -o -z "$2" ]; then
  echo "Usage: $(basename $(readlink -nf $0)) site_to_download_recursively destination_directory" 
  exit 1
else
  wget --continue --recursive --execute robots=off --wait 1 --directory-prefix="$2" "$1"
  # Below line for offline mirroring, test on each site.
  # wget  --mirror --convert-links --adjust-extension --page-requisites --no-parent --continue --recursive --execute robots=off --wait 1 --directory-prefix="$2" "$1"
  exit 0
fi

ImageMagick

Requires ImageMagick

Resize .jpg pictures in current directory level to roughly 1500x2000 pixels in size.

1
mogrify -resize 1500x2000 *.{jpg,JPG}

Create a black background in required size (for example, to use in a video).

1
convert -size 1920x1080 xc:black bg.png

FFMpeg

Dump audio using parallel. Parallel uses 1 thread for each core by default.

Make sure to change audio format according to source.

1
parallel ffmpeg -i '{}' -map 0:1 -c:a copy '{.}.m4a' ::: /media/video/source_video_file.mkv