The Arcology Garden

I should be collecting these in the grand tradition of "shell history isn't good enough". Neither, probably, is Atuin ... So I take inspiration from Maya Kate and make my own grymoire of shell spells.

I am always updating this page, check it in your feed reader: https://arcology.garden/grymoires/shell.xml

shell source: :tangle ~/bin/gopro-convert.sh :shebang #! /usr/bin/env nix-shell :comments none
#!nix-shell -p ffmpeg exiftool -i bash
USAGE="$(cat <<EOF
$(basename "$0") -i <input_file> [-o <out_dir>]
    -i: input file path
    -o: output directory, create if not exist. default to current directory.

EOF
)"
while getopts 'hi:o:' opt; do case "$opt" in
    i)    input_file="$OPTARG" ;;
    o)    out_dir="$OPTARG" ;;
    h|*)  echo "$USAGE" >&2; exit 1 ;;
esac done
shift $((OPTIND-1))

: "${div:=65}"
: "${input_file:?}"
: "${out_dir:=.}"

base_name=$(basename "$input_file")
mkdir -p "$out_dir"
echo "Created output directory: $out_dir"
outfile=$out_dir/${base_name}.mp4;

#ffmpeg -i "$input_file" -frames:v 1500 -y  -filter_complex "
ffmpeg -i "$input_file" -y -filter_complex "
[0:0]crop=128:1344:x=624:y=0,format=yuvj420p,
geq=
lum='if(between(X, 0, 64), (p((X+64),Y)*(((X+1))/"$div"))+(p(X,Y)*(("$div"-((X+1)))/"$div")), p(X,Y))':
cb='if(between(X, 0, 64), (p((X+64),Y)*(((X+1))/"$div"))+(p(X,Y)*(("$div"-((X+1)))/"$div")), p(X,Y))':
cr='if(between(X, 0, 64), (p((X+64),Y)*(((X+1))/"$div"))+(p(X,Y)*(("$div"-((X+1)))/"$div")), p(X,Y))':
a='if(between(X, 0, 64), (p((X+64),Y)*(((X+1))/"$div"))+(p(X,Y)*(("$div"-((X+1)))/"$div")), p(X,Y))':
interpolation=b,crop=64:1344:x=0:y=0,format=yuvj420p,scale=96:1344[crop],
[0:0]crop=624:1344:x=0:y=0,format=yuvj420p[left], 
[0:0]crop=624:1344:x=752:y=0,format=yuvj420p[right], 
[left][crop]hstack[leftAll], 
[leftAll][right]hstack[leftDone],

[0:0]crop=1344:1344:1376:0[middle],

[0:0]crop=128:1344:x=3344:y=0,format=yuvj420p,
geq=
lum='if(between(X, 0, 64), (p((X+64),Y)*(((X+1))/"$div"))+(p(X,Y)*(("$div"-((X+1)))/"$div")), p(X,Y))':
cb='if(between(X, 0, 64), (p((X+64),Y)*(((X+1))/"$div"))+(p(X,Y)*(("$div"-((X+1)))/"$div")), p(X,Y))':
cr='if(between(X, 0, 64), (p((X+64),Y)*(((X+1))/"$div"))+(p(X,Y)*(("$div"-((X+1)))/"$div")), p(X,Y))':
a='if(between(X, 0, 64), (p((X+64),Y)*(((X+1))/"$div"))+(p(X,Y)*(("$div"-((X+1)))/"$div")), p(X,Y))':
interpolation=b,crop=64:1344:x=0:y=0,format=yuvj420p,scale=96:1344[cropRightBottom],
[0:0]crop=624:1344:x=2720:y=0,format=yuvj420p[leftRightBottom], 
[0:0]crop=624:1344:x=3472:y=0,format=yuvj420p[rightRightBottom], 
[leftRightBottom][cropRightBottom]hstack[rightAll], 
[rightAll][rightRightBottom]hstack[rightBottomDone],
[leftDone][middle]hstack[leftMiddle],
[leftMiddle][rightBottomDone]hstack[bottomComplete],



[0:5]crop=128:1344:x=624:y=0,format=yuvj420p,
geq=
lum='if(between(X, 0, 64), (p((X+64),Y)*(((X+1))/"$div"))+(p(X,Y)*(("$div"-((X+1)))/"$div")), p(X,Y))':
cb='if(between(X, 0, 64), (p((X+64),Y)*(((X+1))/"$div"))+(p(X,Y)*(("$div"-((X+1)))/"$div")), p(X,Y))':
cr='if(between(X, 0, 64), (p((X+64),Y)*(((X+1))/"$div"))+(p(X,Y)*(("$div"-((X+1)))/"$div")), p(X,Y))':
a='if(between(X, 0, 64), (p((X+64),Y)*(((X+1))/"$div"))+(p(X,Y)*(("$div"-((X+1)))/"$div")), p(X,Y))':
interpolation=n,crop=64:1344:x=0:y=0,format=yuvj420p,scale=96:1344[leftTopCrop],
[0:5]crop=624:1344:x=0:y=0,format=yuvj420p[firstLeftTop], 
[0:5]crop=624:1344:x=752:y=0,format=yuvj420p[firstRightTop], 
[firstLeftTop][leftTopCrop]hstack[topLeftHalf], 
[topLeftHalf][firstRightTop]hstack[topLeftDone],

[0:5]crop=1344:1344:1376:0[TopMiddle],

[0:5]crop=128:1344:x=3344:y=0,format=yuvj420p,
geq=
lum='if(between(X, 0, 64), (p((X+64),Y)*(((X+1))/"$div"))+(p(X,Y)*(("$div"-((X+1)))/"$div")), p(X,Y))':
cb='if(between(X, 0, 64), (p((X+64),Y)*(((X+1))/"$div"))+(p(X,Y)*(("$div"-((X+1)))/"$div")), p(X,Y))':
cr='if(between(X, 0, 64), (p((X+64),Y)*(((X+1))/"$div"))+(p(X,Y)*(("$div"-((X+1)))/"$div")), p(X,Y))':
a='if(between(X, 0, 64), (p((X+64),Y)*(((X+1))/"$div"))+(p(X,Y)*(("$div"-((X+1)))/"$div")), p(X,Y))':
interpolation=n,crop=64:1344:x=0:y=0,format=yuvj420p,scale=96:1344[TopcropRightBottom],
[0:5]crop=624:1344:x=2720:y=0,format=yuvj420p[TopleftRightBottom], 
[0:5]crop=624:1344:x=3472:y=0,format=yuvj420p[ToprightRightBottom], 
[TopleftRightBottom][TopcropRightBottom]hstack[ToprightAll], 
[ToprightAll][ToprightRightBottom]hstack[ToprightBottomDone],
[topLeftDone][TopMiddle]hstack[TopleftMiddle],
[TopleftMiddle][ToprightBottomDone]hstack[topComplete],

[bottomComplete][topComplete]vstack[complete], [complete]v360=eac:e:interp=cubic,crop=4032:2388:x=0:y=0[v]" -map "[v]" -map "0:a:0"  -c:v h264 -c:a aac -f mp4 "$outfile"

exiftool -api LargeFileSupport=1  -overwrite_original -XMP-GSpherical:Spherical="true" -XMP-GSpherical:Stitched="true" -XMP-GSpherical:StitchingSoftware=dummy -XMP-GSpherical:ProjectionType=equirectangular "$outfile"

echo "Location of File: $outfile"

freebie Emacs Grymoire entry while i'm at it insert-uri:

elisp source: 
(defun insert-uri (uri)
  (interactive "sURI: ")
  (require 'url)
  (condition-case err
      (progn
        (let ((buf (url-retrieve-synchronously uri)))
          (with-current-buffer 
              (insert-buffer-substring buf)))
        'ok)
    (error (message "Error fetching URI %s:
%s" uri (error-message-string err)) nil)))

I've recently been watching a build for a package on my server fail inconsistently ~50% of the times I build updates, and I was confused as to why it was missing the binary cache in the first. Oh fun, how reproduceable. I spent an hour or two yesterday reading the code and the failing tests and realized it was because the test was stress-testing inotify system calls:

 source: 
  Traceback (most recent call last):
    File "/nix/store/zv1kaq7f1q20x62kbjv6pfjygw5jmwl6-python3-3.12.7/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
      self.run()
    File "/build/source/src/documents/tests/test_management_consumer.py", line 30, in run
      self.cmd.handle(directory=settings.CONSUMPTION_DIR, oneshot=False, testing=True)
    File "/build/source/src/documents/management/commands/document_consumer.py", line 251, in handle
      self.handle_inotify(directory, recursive, options["testing"])
    File "/build/source/src/documents/management/commands/document_consumer.py", line 294, in handle_inotify
      inotify = INotify()
                ^^^^^^^^^
    File "/nix/store/3ziqbc4xcs58hhh5srx7pfl2n9mwj22g-python3.12-inotifyrecursive-0.3.5/lib/python3.12/site-packages/inotifyrecursive/inotifyrecursive.py", line 31, in __init__
      inotify_simple.INotify.__init__(self)
    File "/nix/store/1qv923rjqijj7nbhhm9k1bz53jh9pb3a-python3.12-inotify-simple-1.3.5/lib/python3.12/site-packages/inotify_simple.py", line 91, in __init__
      FileIO.__init__(self, _libc_call(_libc.inotify_init1, flags), mode='rb')
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/nix/store/1qv923rjqijj7nbhhm9k1bz53jh9pb3a-python3.12-inotify-simple-1.3.5/lib/python3.12/site-packages/inotify_simple.py", line 39, in _libc_call
      raise OSError(errno, os.strerror(errno))
  OSError: [Errno 24] Too many open files

inotify is an API that allows your process to be notified when files are accessed, changed, deleted, etc. Because inotify queues can use a fair bit of memory, Linux implements specific interfaces to limit these calls:

The following interfaces can be used to limit the amount of kernel memory consumed by inotify:

/proc/sys/fs/inotify/max_queued_events The value in this file is used when an application calls inotify_init(2) to set an upper limit on the number of events that can be queued to the corresponding inotify instance. Events in excess of this limit are dropped, but an IN_Q_OVERFLOW event is always generated.

/proc/sys/fs/inotify/max_user_instances This specifies an upper limit on the number of inotify instances that can be created per real user ID.

/proc/sys/fs/inotify/max_user_watches This specifies an upper limit on the number of watches that can be created per real user ID.

I found a Nixpkgs GitHub Issue about the build failure where folks saw the "too many open files" and assumed it was ulimit configuration issues on the build hosts that stumped even the NixOS Super Posters, but it's this other more obscure limit that nevertheless raises the same Errno when it's hit. Classic Linux! These values are set to arbitrarily low values by default, and I remembered that the inotify watcher in the The Arcology Project 's FastAPI prototype bumped up against these limits when I deployed it way back when. Classic Linux!

There are two machines in /etc/nix/machines, my Framework 13 Laptop and My Homelab Build . Presumably the build works on one but not the other, but I no longer explicitly set these values, so something in NixOS itself must be, this is easy enough to check:

shell source: :results drawer
pushd ~/arroyo-nix
grep -ri fs.inotify.max_user_watches
pushd ~/Code/nixpkgs
grep -ri fs.inotify.max_user_watches

~/arroyo-nix ~/org
~/Code/nixpkgs ~/arroyo-nix ~/org
nixos/modules/services/misc/graphical-desktop.nix:      "fs.inotify.max_user_watches" = lib.mkDefault 524288;
nixos/modules/virtualisation/lxd.nix:      "fs.inotify.max_user_watches" = 1048576;
nixos/modules/virtualisation/incus.nix:      "fs.inotify.max_user_watches" = lib.mkOverride 1050 1048576; # override in case conflict nixos/modules/services/x11/xserver.nix

So it's set to a higher value by enabling LXD (which I believe Waydroid does) but also by enabling any graphical desktop. So the package would build on my laptop but not my server or any "stock" NixOS server... a 50-50 shot.

 source: 
sudo sysctl -w fs.inotify.max_user_watches=524288
sudo sysctl -w fs.inotify.max_user_instances=524288

I set this temporarily on my server and the build ran reliably 5 times though the tests are sooooooo freakin' slow... I hope the Hydra instance that publishes the binary cache gets around to something like this, rather than disabling the tests that validate that document consumption in this document scanning/processing service works properly....

It can be set "for good" with this NixOS configuration:

nix source: 
boot.kernel.sysctl = {
  "fs.inotify.max_user_instances" = 524288;
  "fs.inotify.max_user_watches" = 524288;
};

shell source: 
sudo tail -n 10000 /var/log/nginx/access.log | awk '$10 == "502" {print $10 " " $1 " " $8}' | sort | uniq -c | sort -n

shell source: :session *piss*
ssh fontkeming.fail sudo cat /etc/ssl/certs/fontkeming.fail_cert.pem | ssh last-bank tee /home/rrix/fontkeming.fail_cert.pem
ssh -t last-bank sudo cp /home/rrix/fontkeming.fail_cert.pem /var/lib/nginx/certs/fontkeming.fail_cert.pem
ssh -t last-bank sudo systemctl reload nginx