Useful commands and scripts

Some here are scripts, those begin with a shebang

1
#!

• Main
  • print all environment variables
  • replace in place with sed
  • Find, grep, sed find files with pattern and replace
  • -u to turn off buffering for sed
  • pz (pythonize shell)
  • Vim
  • Bash job control
  • grep
  • find delete empty directories
  • mail
  • moreutils
  • data structure or configuration processing tools
  • linkchecker
  • ncdu equivalent
• Rust tools
  • Bacon
  • Faster alternatives written in Rust
  • Other alternatives written in Rust
• Extra
  • pass
  • find and play videos flexibly
  • Rude wget robot to recursively download a site
  • ImageMagick
  • FFMpeg
  • nginx
  • Systemd get specific service properties
  • Systemd test if service is failed, get exit code

Main

print all environment variables

1
2
3
printenv
jq -n env
jq -nr 'env|.HOME' # -r = raw

replace in place with sed

1
2
sed -i -e 's/^/#/' filename
sed -i '1s/^/added line before 1st line of file\n/' filename

Empty files don't have that first line and are not affected.

Find, grep, sed find files with pattern and replace

1
2
3
4
5
find group_vars/ -type f -name '*.yml' ! -exec grep -q -- '---' '{}' \; -print

find group_vars/ -type f -name '*.yml' \
! -exec grep -q -- '---' '{}' ';' \
-exec sed -i '1s/^/---\n/' '{}' '+'

-u to turn off buffering for sed

1
dmesg -T --follow | sed -u -e "s/^/$HOSTNAME/"

pz (pythonize shell)

operate on s (the line variable).

1
echo -e "example\nwikipedia" | pz 's += ".com"'

Vim

• Delete empty lines.

  1
  2
   :g/^$/d
   :help :g
  

• Show shell executable.

  1
   :set shell?
  

• Execute a child shell process. Exiting returns to vi.

  1
   :sh
  

• Run given shell commands and exit after pressing enter.

  1
   :!printf ":help :!"
  

• Execute current line ('.') in sh and replace it in the vi buffer with the output.

  1
   :.!sh
  

• Insert the output of the shell command in the buffer.

  1
   :r!date
  

• Get the output of a shell command with system ( :help system ).

  1
   :call system('touch /tmp/$(date +%Y%m%d)')
  

  1
   :echo system('ls -lt /tmp/$(date +%Y)* | tail')
  

• Start a job that doesn't wait to finish ( :help job_status ), execute with no shell.

  1
   :call job_start(['/bin/bash', '-c', '{ sleep 60 && printf "DONE"; }'])
  

• Registers (+ * ~ / : % . -)
  • c - characterwise text
  • l - linewise text
  • b - blockwise text

    1
    2
    :registers
    "+p # paste in visual mode
    

    1. unnamed register "" fills when using delete or yank
    2. numbered registers "0-"9 (deletion history stack)
    3. small delete register "- (one line or less deleted)
    4. named registers "a-"z"A-"Z (write to F gets appended, f replaced)
    5. read only registers:
       • ": most recently executed command;
       • "% current file name;
       • ". last inserted text.
    6. alternate file name register "# (in case of multiple open files)
    7. expression register "=
    8. last search result register "/
    9. GUI primary "+ and secondary "* , middle click clipboard
    10. GUI drop register "~ (drag and drop)

Bash job control

Ctrl+Z to suspend a program in bash. Sends SIGTSTP (Keyboard stop).

Start a job in the background.

1
sleep 4500 &

Bash shell builtins for jobs. Applies to current shell process, not all user's shells. ####

1
jobs --help

1
2
3
fg # Bring to foreground. No job spec, targets shells notion of the current job
bg %1 # Sends the first job to execute in the background
wait 20110 # Process ID or job spec

Remove the second job from current shell's job list, keeping it in the process table. Not reliable for running a program in the background (in case the parent shell is destroyed).

1
disown %2

Run a program in a new session. Allows bypassing potential signals from initial parents.

1
setsid --fork sleep 3600 # always create a new process

Format text in a way that is safe to use as shell input.

1
printf '%q\n' "It's magic!"

grep

Search directory recursively for lines starting 0 or more whitespace and $.

1
grep -ER '^(\w*)\$'

find delete empty directories

Starts from the deepest, deletes recursively up.

1
find . -depth -type d -empty -delete

mail

mailutils.txt

3.5.2.1 Syntax of mail internal commands

Open the default inbox (mbox) instead of the spool file.

1
mail -f

With mail open, match string from the body.

1
h :/\\[SPAM\\]

Match string from the header From.

1
h From:/root

Header is Subject if omitted. Escaped to prevent an attempt to POSIX regex match.

1
h /\\[SPAM\\]

Delete matching messages.

1
d /\\[SPAM\\]

moreutils

• chronic: runs a command quietly unless it fails
• combine: combine the lines in two files using boolean operations
• errno: look up errno names and descriptions
• ifdata: get network interface info without parsing ifconfig output
• isutf8: check if a file or standard input is utf-8
• ifne: run a command if the standard input is not empty
• lckdo: execute a program with a lock held (deprecated)
• mispipe: pipe two commands, returning the exit status of the first
• parallel: run multiple jobs at once
• pee: tee standard input to pipes
• sponge: soak up standard input and write to a file
• ts: timestamp standard input
• vidir: edit a directory in your text editor
• vipe: insert a text editor into a pipe
• zrun: automatically uncompress arguments to command

data structure or configuration processing tools

• jq
• yq
• tomlq
• rq (see: open policy agent, Rego, kube-mgmt)

linkchecker

Check HTML documents and websites for broken links. Return value is 1 when

• invalid links were found or
• link warnings were found and warnings are enabled Supports clamav integration.

ncdu equivalent

For when ncdu is not installed.

1
du -d1 -h | sort -h

Rust tools

Bacon

bacon - background code checker designed for minimal interaction. It has analyzers for Rust, python, javascript, typescript, C++, Go.

Kondo recursively cleans project directories. Interface to rm -rf build files.

1
kondo --dry-run

Faster alternatives written in Rust

fd - alternative to find. debian package name fd-find, executable name on debian is

1
fdfind

ripgrep - alternative to grep. debian package name ripgrep.

1
rg

runiq filters duplicate lines. Alternative to uniq.

Uv claims to replace virtualenv, pip, pip-tools, pipx, poetry, pyenv, twine.

1
uv

pre-commit, husky alternative

1
prek

watchexec watches a path and runs a command whenever it detects modifications.

Other alternatives written in Rust

eza - ls

bat - cat

xh - httpie. Friendlier interface than curl, can print out an equivalent curl command.

zellij - tmux, screen

du-dust - du like

dua-cli - ncdu

hyperfine - benchmarking, default repeated runs

helix, evil-helix - vim

just, mask - make

Extra

pass

Unix style password management script, encrypts with gpg.

find and play videos flexibly

Using mpv, my preferred video player.

To use as is, create a shell script with below contents, arguments:

1st argument - how long ago was the video file last changed.

The rest of the arguments: path(s) to look for videos (glob, like

1
/media/videos/201*

can be used as argument).

1
2
#!/bin/sh
find -L "${@:2}" \( -iname '*.mp4' -o -iname '*.webm' -o -iname '*.avi' \) -a -ctime -"$1" -print0 | xargs -0 mpv 

Rude wget robot to recursively download a site

1
2
3
4
5
6
7
8
9
10
11
#!/bin/sh
# shows the correct script path even if called from another script
if [ -z "$1" -o -z "$2" ]; then
  echo "Usage: $(basename $(readlink -nf $0)) site_to_download_recursively destination_directory" 
  exit 1
else
  wget --continue --recursive --execute robots=off --wait 1 --directory-prefix="$2" "$1"
  # Below line for offline mirroring, test on each site.
  # wget  --mirror --convert-links --adjust-extension --page-requisites --no-parent --continue --recursive --execute robots=off --wait 1 --directory-prefix="$2" "$1"
  exit 0
fi

ImageMagick

Requires ImageMagick

Resize .jpg pictures in current directory level to roughly 1500x2000 pixels in size.

1
mogrify -resize 1500x2000 *.{jpg,JPG}

Create a black background in required size (for example, to use in a video).

1
convert -size 1920x1080 xc:black bg.png

FFMpeg

Dump audio using parallel. Parallel uses 1 thread for each core by default.

Make sure to change audio format according to source.

1
parallel ffmpeg -i '{}' -map 0:1 -c:a copy '{.}.m4a' ::: /media/video/source_video_file.mkv

nginx

Dump config (along with testing it)

1
nginx -T

Systemd get specific service properties

1
systemctl show --property ActiveState nginx.service

Systemd test if service is failed, get exit code

1
ecode=$(systemctl is-failed --quiet nginx.service)